Phase 0: Centralize interwiki language links for Wiktionary
Open, NormalPublic

Description

https://www.wikidata.org/wiki/Wikidata:Wiktionary/Development/Proposals/2014-10#Phase_0:_Interwiki_links

Phase 0: Interwiki links

The easiest and most immediate gain that can be had by transforming Wiktionary to a Wikidata format comes from leveraging the existing data model that Wiktionary uses at its most basic level.

Each localized Wiktionary has pages whose names are based on representation. Thus, Phase 0 is merely importing the combined set of (main namespace) pages from all Wiktionaries.

These can then be used to fill in any blanks in the interwiki link web, giving a single centralized location for interwiki links.

Data model
  • representation (R)
    • script

Hackathon coordination

Volunteers

Volunteers confirmed:

Volunteers interested (if you are not yet sure, or want to learn more):

  • (@YourPhabricatorUsername, #YourIRCUsername, interest & skills volunteered to this project, at the event or remote online?)

Progress report

  • TBD: (Write down achievements, blockers requiring attention, and other important updates. This information will be priceless at the hackathon showcase and after the event!)
GPHemsley changed Security from none to None.
GPHemsley edited the task description. (Show Details)
GPHemsley edited the task description. (Show Details)
GPHemsley edited the task description. (Show Details)
GPHemsley added a subscriber: GPHemsley.
Gilles triaged this task as "Normal" priority.Nov 24 2014, 1:44 PM
Gilles added a subscriber: Gilles.
Lydia_Pintscher changed the title from "Implement Phase 0: Interwiki links" to "language links for Wiktionary".Nov 27 2014, 10:39 AM
Lydia_Pintscher lowered the priority of this task from "Normal" to "Low".
GPHemsley edited the task description. (Show Details)Dec 1 2014, 3:18 AM
Lydia_Pintscher removed a subscriber: Unknown Object (MLST).
GPHemsley changed the title from "language links for Wiktionary" to "Phase 0: Centralize interwiki language links for Wiktionary".Mar 8 2015, 3:19 PM
GPHemsley raised the priority of this task from "Low" to "Normal".
GPHemsley moved this task from Backlog to Up Next on the Wiktionary board.Mar 8 2015, 3:26 PM

Is there a reason why only "main namespace" pages are mentioned in the above proposal?
Also, does this proposal takes into account the different ways the different communities represent words? E.g. different apostrophes used for aujourd'hui in French and in English (with corresponding redirects in each project).

mxn added a subscriber: mxn.Mar 17 2015, 6:21 AM
Gilles removed a subscriber: Gilles.Apr 23 2015, 6:11 AM

When will this task get completed ?

I unfortunately can't tell you yet. I am currently looking for someone to work on it.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2015, 5:54 AM
JAnD added a subscriber: JAnD.Oct 9 2015, 9:36 AM
dg711 added a subscriber: dg711.Dec 30 2015, 1:20 AM
Qgil added a subscriber: Qgil.Mar 31 2016, 9:34 AM

The Wikimedia-Hackathon-2016 starts tomorrow and this task is featured at T119703. We want to use T130776: Wikimedia Hackathon 2016 Opening Session to promote these projects and help recruiting volunteers to work for them.

If this task is ripe for hackathon work, please follow these instructions. If it is not ready, remove it from T119703 in order to avoid volunteers' frustration. Thank you!

gabriel-wmde edited the task description. (Show Details)Apr 2 2016, 9:16 AM
gabriel-wmde added a comment.EditedApr 2 2016, 9:53 AM

Some preliminary thoughts regarding the implementation:

The task will be implemented as an extension that does two things:

  • On page edit, update a central data store that the page exists in the language of the currently used wiktionary.
  • On page render, query the central data store for all existing translations and insert the information in the page structure.

The "central data store" must be a database that is accessible from all wiktionary projects. Proposed table structure:

langtitlelast_updated

lang and title must be indexed.

On initialization the table must be filled with the existing data. This could be done by querying the API and crawling each wiktionary with a bot or by direct database access. This still has to be determined.

Regarding the comment by @Darkdadaah: Wiktionary is in the unique situation that each word should be the same in each language version. Where they differ, there must be a redirect page, as in the case pointed out by Darkdadaah.

JAnD added a comment.Apr 2 2016, 6:47 PM

Regarding the comment by @Darkdadaah: Wiktionary is in the unique situation that each word should be the same in each language version. Where they differ, there must be a redirect page, as in the case pointed out by Darkdadaah.

There are some exception cases:

  • proverbs are in some anguage as sentence (First capital and dot at the end.), in some with firs lowcase and without dot
  • words containing '
Qgil removed a subscriber: Qgil.Apr 3 2016, 7:18 AM

@gabriel-wmde worked on this at Wikimedia-Hackathon-2016 - "not ready but getting there"

@gabriel-wmde It'd be great if you can post a status update here. Then I can help you figure out who can review it and so on. If you want I can also see if you can continue this on work-time.

gabriel-wmde added a comment.EditedApr 6 2016, 9:14 AM

Current status:

Next steps:

  • Add code for the remaining TODO comments.
  • Address the cases where the title is different in the original language (as raised by @JAnD and @Darkdadaah, I'll add a separate comment with a proposal on how to address this).
  • Create a maintenance script that fills the database.
  • ???
  • Profit Deploy

After the comments from @JAnD and @Darkdadaah I have lots of questions on how disparate the page titles for each language are.

As far as I understand, some Wiktionary projects have their own, localized spelling that deviates from the "standard" spelling of Wiktionary, for adhering to the spelling standards of the language they are in.

Some questions:

  • Do wiktionary words exist in more than 2 variations of a word across all translations? If they do, how big is the variation?
  • Is the wiktionary project doing the localized spelling only for the language they are for (e.g. the French Wiktionnaire has French apostrophes in french words and phrases, but leaves the apostrophes of other languages words and phrases as-is).
  • Is the localized spelling consistent inside one wiktionary?
  • For which percentage (roughly) of localized pages does a redirect from the standard spelling to the localized spelling exist?
  • Would it be possible to "Normalize" page titles algorithmically (search and replace with regular expressions)?
  • Words in different scripts (Cyrillic, Hebrew, Arabic, etc.) don't have these issues? Or Do they have even graver issues?

Is there someone who can reliably answer these questions for all wiktionaries? Otherwise the next step in this issue would be to write a program that analyzes the deviations.

hoo added a subscriber: hoo.Thu, Apr 14, 9:27 AM

@gabriel-wmde I did some work like this a long time ago, see https://www.wikidata.org/wiki/Wikidata_talk:Wiktionary#First_and_second_phases. I'll see if I can do an update...

Add Comment