9 ms·

I was very actively involved in MediaWiki development & Wikimedia ops (less so though) in 2004-2006 back when IIRC there were just 1-4 paid Wikimedia employees.

by avar 10y ago

I was very actively involved in MediaWiki development & Wikimedia ops (less so though) in 2004-2006 back when IIRC there were just 1-4 paid Wikimedia employees. It was a very different time, and the whole thing was run much more like a typical open source project. I think the whole project has gone in completely the wrong direction since then. Wikipedia itself is still awesome, but what's not awesome is that the typical reader / contributor experience is pretty much the same as it was in 2005. Moreover, because of the growing number of employees & need for revenue the foundation's main goal is still to host a centrally maintained site that must get your pageviews & donations directly. The goal of Wikipedia should be to spread the content as far & wide as possible, the way OpenStreetMap operates is a much better model. Success should be measured as a function of how likely any given person is to see factually accurate content sourced from Wikipedia, and it shouldn't matter if they're viewing it on some third party site. Instead it's still run as one massive monolithic website, and it's still hard to get any sort of machine readable data out of it. This IMO should have been the main focus of Wikimedia's development efforts.

jacquesm 10y ago
> and it's still hard to get any sort of machine readable data out of it. Huh? https://dumps.wikimedia.org/ https://dumps.wikimedia.org/ Doesn't that qualify?
- rspeer 10y ago
  That gives you Wikitext encapsulated in XML. How do you get at the content of the Wikitext? I work on a Wikitext parser [1]. So do many other people, in different ways. Wikitext syntax is horrible and it mixes content and presentation indiscriminately (for example, it contains most of HTML as a sub-syntax). The problem is basically unsolvable, as the result of parsing a Wiki page is defined only by a complete implementation of MediaWiki (with all its recursively-evaluated template pages, PHP code, and Lua code), but if you run that whole stack what you get in the end is HTML -- just the presentation, not the content you presumably wanted. So people solve various pieces of the problem instead, creating approximate parsers that oversimplify various situations to meet their needs. One of these solutions is DBPedia [2], but if you use DBPedia you have to watch out for the parts that are false or misleading due to parse errors. [1] https://github.com/LuminosoInsight/wikiparsec https://github.com/LuminosoInsight/wikiparsec [2] http://wiki.dbpedia.org/ http://wiki.dbpedia.org/
  - shshhdhs 10y ago
    Thanks for working on that! Didn't know it was so bad. The following is a possibly stupid idea, but I'd like to hear your thoughts: What if you just render the content into HTML and then "screen scraped" the text, and then convert into a more useful format (MarkDown, JSON, etc). Is that plausible?
    - GhostVII 10y ago
      That would allow a basic UI change on Wikipedia to break your code. Sometimes it is necessary, but not usually the best option in my experience, and it's pretty annoying to do.
    - 3131s 10y ago
      Wikipedia used to do HTML dumps but stopped a long time ago, unfortunately.
      - int_19h 10y ago
        You can get what amounts to an HTML dump (which is then indexed and compressed in a single huge archive) from Kiwix. Although they do them basically twice a year or so.
  - lacksconfidence 10y ago
    You could download the search indexes, also on the dumps site, that has the text content among other things.
  - deleted 10y ago
    [deleted]
  - 10165 10y ago
    "That gives you Wikitext encapsulated in XML." avar: "The goal of Wikipedia should be to spread the content as far & wide as possible, the way OpenStreetMap operates is a better model." I am confused. Doesn't OSM data come encapsulated in XML or some binary format? As for dispersion of content, I could have sworn I have seen Wikipedia content on non-Wikipedia websites. Is there some restriction that prohibits this? I have seen Wikipedia data offered in DNS TXT records as well.
- nullc 10y ago
  Go click through those links. Most of them are hardly maintained. E.g. last static HTML dump was in 2008. Current enwiki raw data dump is in progress and reads: "2017-05-07 23:24:34: enwiki (ID 13918) 103 pages (0.0|1.2/sec all|curr), 921000 revs (14.4|11.4/sec all|curr), 100.0%|100.0% prefetched (all|curr), ETA 2019-01-23 15:17:29 [max 779130995]" There are real logistical challenges in making these dumps and making them _useful_. For all Wikimedia's spending, they have not invested sufficiently in this area.
- 3131s 10y ago
  No, not at all. Years back Wikipedia released HTML dumps of the entire site, which was closer to providing the actual content of Wikipedia as structured data, but that was discontinued.
mboto 10y ago
How would you quantify the success of spreading data as far and as wide as possible?
smsm42 10y ago
> the foundation's main goal is still to host a centrally maintained site Wikimedia universe is way bigger than one site. There's Wikidata, Commons, Wikisource, Wiktionary, Wikivoyage, Wikibooks and so on. And there's a lot of language versions too - English is not the only way to store knowledge, you know. > The goal of Wikipedia should be to spread the content as far & wide as possible The requires a) creating the content and b) presenting the content in the form consumable by the users. Creating tools for this is far from trivial, especially if you want it to be consumable and not just unpalatable infodump accessible only to the most determined. > Instead it's still run as one massive monolithic website This is not accurate. A lot has changed since 2004. It's not one monolithic website, it's a constellation of dozens, if not hundreds, of communities. They are using common technical infrastructure (datacenters, operations, etc.) and common software (Mediawiki, plus myriad of extensions for specific projects), but they are separate sites and separate communities, united by common goal of making knowledge available to everyone. > it's still hard to get any sort of machine readable data out of it Please check out Wikidata. This is literally the goal of the project. You can also be interested in "structured Commons" and "structured Wiktionary" projects, both in active development as we speak. > This IMO should have been the main focus of Wikimedia's development efforts. It is. One of focuses - for a project of this size, there's always several directions. BTW, right now Wikimedia is in the process of formulating movement strategy, with active community participation. You are welcome to contribute: https://meta.wikimedia.org/wiki/Strategy/Wikimedia_movement/2017 https://meta.wikimedia.org/wiki/Strategy/Wikimedia_movement/... Disclosure: working for WMF, not speaking for anybody but myself.
- derefr 10y ago
  "One massive monolithic website" is, I think, meant to be read as referring to the WMF sites being a thing that have a shared telecommunications Single Point of Failure—a "choke point" where a given piece of information can only get from a given WMF site, to a user, by travelling through WMF-managed Internet infrastructure. Remember Napster, back in the day? It was able to be shut down because it had an SPOF: Napster-the-corporation owned and maintained all the "supernodes" that formed the backbone of the network. Or consider the Great Firewall of China. If the Great Firewall can block your site/content entirely with a single rule, you have an SPOF. The answer to such problems isn't simple sharding-by-content-type into "communities" like you're talking about; this is still centralized, in the sense of "centralized allocation." Instead, to answer such problems, you need true distribution. This can take the form of protocols allowing Wiki articles to be accessed and edited in a peer-to-peer fashion with no focal point that can be blocked; this can take the form of Wikipedia "apps" that are offline-first, such that you can "bring Wikipedia with you" to places where state actors would rather you don't have it; this can take the form of preloaded "Wikipedia mirror in a box" appliances (plus a syncing logistics solution, ala AWS Snowball) which can be used by local libraries in countries with little internet access to allow people there access to Wikipedia.
EGreg 10y ago
Don't they have MediaWiki APIs and dumps? Look at other wiki sites. Also there is Kiwix and various offline apps. Have you seen them? Thoughts? Also what's the difference between WikiData and DBPedia?
- Tpt 10y ago
  > Also what's the difference between WikiData and DBPedia? Wikidata is a Wikimedia project with aim to create a structured knowledge based. It is mostly filled and curated by humans: https://www.wikidata.org https://www.wikidata.org DBPedia is a knowledge base which content is extracted from Wikipedia (mostly from the infoboxes). It is a project run by researchers: http://dbpedia.org http://dbpedia.org
sudoscript 10y ago
Random thought. Why can't something like Wikipedia be run distributed through a blockchain? Edits are just transactions that are broadcast over the network. I imagine the total cost of that to individual contributors of nodes would be less than the millions they're paying right now. EDIT: Turns out someone is already working on it: https://lunyr.com/ https://lunyr.com/
- pseud0r 10y ago
  Saw this the other day. Someone working on making Wikipedia on MaidSafe https://safenetforum.org/t/safe-drive-wikipedia-on-safe-technology-preview/13094 https://safenetforum.org/t/safe-drive-wikipedia-on-safe-tech... https://github.com/loureirorg/wikirofs https://github.com/loureirorg/wikirofs Not sure how far along it is, but it looks interesting.
- ChristianKle 9y ago
  Most of Wikipedia's money goes to pay for things besides hosting. Centralized hosting also happens to be more efficient than a decentralized version.
frik 10y ago
Completely with you. The goal of Wikipedia should be to spread the content and allow more new content. Sadly it seems the opposite is true, whole parts of Wikipedia are infested by cancer (aka corrupt/out-od-mind admins who are acting in their own world/turf and interest), have a closer look at certain languages like de.Wikipedia.org where more new and old articles get deleted than content can grow (source: various German news media incl. Heise.de reported about it). And why is Wikia a thing? And why is it from the Wikipedia founder, has he a double interest!? And now he is starting a news offspring as well! Something like the Wikipedia frontpage and WikiNews, just under his own company. And on the otherside Wikipedia banned trivia sections to make the Wikia offspring even possible (happened 10 years ago, but you probably remember it; yet Wikipedia deleted/buried the trivia section history). Why even delete non-corporate-spam artices? Why are fictional manga creatures all o er on Wikipedia but info about movie characters all deleted? Many Wikipedia admin seem to be deletionists that care only about their turf, the care about "their own" articles, they revert changes to them just for their own sake. Look at the WikiData project. Why is it implemented by a German Wikipedia org that has little to do with intl Wikimedia foundation, it's not a sister project, they do their own fundraiser and media news reported not so nice things over the years. Look at OpenStreetMap project, it works a lot better. Maybe the Wikipedia project should be transfered over or forked by OpenStreetMaps project. And delete all admin rights, and start over with the in some way toxic community that scares away most normal people who don't want to engage in childish turf wars and see their contributions deleted and cut down for no reason but admin power play.

by avar 10y ago

jacquesm 10y ago

> and it's still hard to get any sort of machine readable data out of it. Huh? https://dumps.wikimedia.org/ https://dumps.wikimedia.org/ Doesn't that qualify?

rspeer 10y ago
That gives you Wikitext encapsulated in XML. How do you get at the content of the Wikitext? I work on a Wikitext parser [1]. So do many other people, in different ways. Wikitext syntax is horrible and it mixes content and presentation indiscriminately (for example, it contains most of HTML as a sub-syntax). The problem is basically unsolvable, as the result of parsing a Wiki page is defined only by a complete implementation of MediaWiki (with all its recursively-evaluated template pages, PHP code, and Lua code), but if you run that whole stack what you get in the end is HTML -- just the presentation, not the content you presumably wanted. So people solve various pieces of the problem instead, creating approximate parsers that oversimplify various situations to meet their needs. One of these solutions is DBPedia [2], but if you use DBPedia you have to watch out for the parts that are false or misleading due to parse errors. [1] https://github.com/LuminosoInsight/wikiparsec https://github.com/LuminosoInsight/wikiparsec [2] http://wiki.dbpedia.org/ http://wiki.dbpedia.org/
- shshhdhs 10y ago
  Thanks for working on that! Didn't know it was so bad. The following is a possibly stupid idea, but I'd like to hear your thoughts: What if you just render the content into HTML and then "screen scraped" the text, and then convert into a more useful format (MarkDown, JSON, etc). Is that plausible?
  - GhostVII 10y ago
    That would allow a basic UI change on Wikipedia to break your code. Sometimes it is necessary, but not usually the best option in my experience, and it's pretty annoying to do.
  - 3131s 10y ago
    Wikipedia used to do HTML dumps but stopped a long time ago, unfortunately.
    - int_19h 10y ago
      You can get what amounts to an HTML dump (which is then indexed and compressed in a single huge archive) from Kiwix. Although they do them basically twice a year or so.
- lacksconfidence 10y ago
  You could download the search indexes, also on the dumps site, that has the text content among other things.
- deleted 10y ago
  [deleted]
- 10165 10y ago
  "That gives you Wikitext encapsulated in XML." avar: "The goal of Wikipedia should be to spread the content as far & wide as possible, the way OpenStreetMap operates is a better model." I am confused. Doesn't OSM data come encapsulated in XML or some binary format? As for dispersion of content, I could have sworn I have seen Wikipedia content on non-Wikipedia websites. Is there some restriction that prohibits this? I have seen Wikipedia data offered in DNS TXT records as well.
nullc 10y ago
Go click through those links. Most of them are hardly maintained. E.g. last static HTML dump was in 2008. Current enwiki raw data dump is in progress and reads: "2017-05-07 23:24:34: enwiki (ID 13918) 103 pages (0.0|1.2/sec all|curr), 921000 revs (14.4|11.4/sec all|curr), 100.0%|100.0% prefetched (all|curr), ETA 2019-01-23 15:17:29 [max 779130995]" There are real logistical challenges in making these dumps and making them _useful_. For all Wikimedia's spending, they have not invested sufficiently in this area.
3131s 10y ago
No, not at all. Years back Wikipedia released HTML dumps of the entire site, which was closer to providing the actual content of Wikipedia as structured data, but that was discontinued.

mboto 10y ago

How would you quantify the success of spreading data as far and as wide as possible?

smsm42 10y ago

> the foundation's main goal is still to host a centrally maintained site Wikimedia universe is way bigger than one site. There's Wikidata, Commons, Wikisource, Wiktionary, Wikivoyage, Wikibooks and so on. And there's a lot of language versions too - English is not the only way to store knowledge, you know. > The goal of Wikipedia should be to spread the content as far & wide as possible The requires a) creating the content and b) presenting the content in the form consumable by the users. Creating tools for this is far from trivial, especially if you want it to be consumable and not just unpalatable infodump accessible only to the most determined. > Instead it's still run as one massive monolithic website This is not accurate. A lot has changed since 2004. It's not one monolithic website, it's a constellation of dozens, if not hundreds, of communities. They are using common technical infrastructure (datacenters, operations, etc.) and common software (Mediawiki, plus myriad of extensions for specific projects), but they are separate sites and separate communities, united by common goal of making knowledge available to everyone. > it's still hard to get any sort of machine readable data out of it Please check out Wikidata. This is literally the goal of the project. You can also be interested in "structured Commons" and "structured Wiktionary" projects, both in active development as we speak. > This IMO should have been the main focus of Wikimedia's development efforts. It is. One of focuses - for a project of this size, there's always several directions. BTW, right now Wikimedia is in the process of formulating movement strategy, with active community participation. You are welcome to contribute: https://meta.wikimedia.org/wiki/Strategy/Wikimedia_movement/2017 https://meta.wikimedia.org/wiki/Strategy/Wikimedia_movement/... Disclosure: working for WMF, not speaking for anybody but myself.

derefr 10y ago
"One massive monolithic website" is, I think, meant to be read as referring to the WMF sites being a thing that have a shared telecommunications Single Point of Failure—a "choke point" where a given piece of information can only get from a given WMF site, to a user, by travelling through WMF-managed Internet infrastructure. Remember Napster, back in the day? It was able to be shut down because it had an SPOF: Napster-the-corporation owned and maintained all the "supernodes" that formed the backbone of the network. Or consider the Great Firewall of China. If the Great Firewall can block your site/content entirely with a single rule, you have an SPOF. The answer to such problems isn't simple sharding-by-content-type into "communities" like you're talking about; this is still centralized, in the sense of "centralized allocation." Instead, to answer such problems, you need true distribution. This can take the form of protocols allowing Wiki articles to be accessed and edited in a peer-to-peer fashion with no focal point that can be blocked; this can take the form of Wikipedia "apps" that are offline-first, such that you can "bring Wikipedia with you" to places where state actors would rather you don't have it; this can take the form of preloaded "Wikipedia mirror in a box" appliances (plus a syncing logistics solution, ala AWS Snowball) which can be used by local libraries in countries with little internet access to allow people there access to Wikipedia.

EGreg 10y ago

Don't they have MediaWiki APIs and dumps? Look at other wiki sites. Also there is Kiwix and various offline apps. Have you seen them? Thoughts? Also what's the difference between WikiData and DBPedia?

Tpt 10y ago
> Also what's the difference between WikiData and DBPedia? Wikidata is a Wikimedia project with aim to create a structured knowledge based. It is mostly filled and curated by humans: https://www.wikidata.org https://www.wikidata.org DBPedia is a knowledge base which content is extracted from Wikipedia (mostly from the infoboxes). It is a project run by researchers: http://dbpedia.org http://dbpedia.org

sudoscript 10y ago

Random thought. Why can't something like Wikipedia be run distributed through a blockchain? Edits are just transactions that are broadcast over the network. I imagine the total cost of that to individual contributors of nodes would be less than the millions they're paying right now. EDIT: Turns out someone is already working on it: https://lunyr.com/ https://lunyr.com/

pseud0r 10y ago
Saw this the other day. Someone working on making Wikipedia on MaidSafe https://safenetforum.org/t/safe-drive-wikipedia-on-safe-technology-preview/13094 https://safenetforum.org/t/safe-drive-wikipedia-on-safe-tech... https://github.com/loureirorg/wikirofs https://github.com/loureirorg/wikirofs Not sure how far along it is, but it looks interesting.
ChristianKle 9y ago
Most of Wikipedia's money goes to pay for things besides hosting. Centralized hosting also happens to be more efficient than a decentralized version.

frik 10y ago

Completely with you. The goal of Wikipedia should be to spread the content and allow more new content. Sadly it seems the opposite is true, whole parts of Wikipedia are infested by cancer (aka corrupt/out-od-mind admins who are acting in their own world/turf and interest), have a closer look at certain languages like de.Wikipedia.org where more new and old articles get deleted than content can grow (source: various German news media incl. Heise.de reported about it). And why is Wikia a thing? And why is it from the Wikipedia founder, has he a double interest!? And now he is starting a news offspring as well! Something like the Wikipedia frontpage and WikiNews, just under his own company. And on the otherside Wikipedia banned trivia sections to make the Wikia offspring even possible (happened 10 years ago, but you probably remember it; yet Wikipedia deleted/buried the trivia section history). Why even delete non-corporate-spam artices? Why are fictional manga creatures all o er on Wikipedia but info about movie characters all deleted? Many Wikipedia admin seem to be deletionists that care only about their turf, the care about "their own" articles, they revert changes to them just for their own sake. Look at the WikiData project. Why is it implemented by a German Wikipedia org that has little to do with intl Wikimedia foundation, it's not a sister project, they do their own fundraiser and media news reported not so nice things over the years. Look at OpenStreetMap project, it works a lot better. Maybe the Wikipedia project should be transfered over or forked by OpenStreetMaps project. And delete all admin rights, and start over with the in some way toxic community that scares away most normal people who don't want to engage in childish turf wars and see their contributions deleted and cut down for no reason but admin power play.