9 ms·

Wikipedia's in Trouble (2019)

103 points by sanqui 6y ago

pierewoehl 6y ago
I would be highly interested in Moving Wikipedia to Markdown in sphinx with own repo in git. But this poses issue to people who want to use a WYIWYG Editor and not git it away.
- klodolph 6y ago
  It also poses a major problem for advanced usage (templates, math, citations, etc.) Markdown is a decent language for basic stuff like Reddit, Stack Overflow, GitHub READMEs, etc. Stuff where the most important feature is making it easy for people to make content. Places like Wikipedia or blogs are where Markdown doesn’t work so well. Sure, you want it to be relatively easy for people to edit Wikipedia pages, but it’s worth sacrificing ease of editing because you want to improve the reading experience.
- mhh__ 6y ago
  They'd need to think a lot bigger and better than markdown. Honestly, wikipedia would be a great proving ground for a hypothetical evolution of LaTeX
  - warmwaffles 6y ago
    The whole right to left and left to right orientation is definitely a difficult thing to solve. On top of that, you'd need to figure out a way to have translations live along side the article itself.
  - BlueTemplar 6y ago
    LaTeX itself is a big mess...
kayodelycaon 6y ago
I think this article shoots itself in the foot by suggesting markdown. Wikipedia predates markdown by several years. The only truly “universal” markup language I know of at the time was bbcode. Even if markdown has been designed, it’s lacking support for templates and tables that wikis make heavy use of. Every wiki software I’ve used has had a custom language for one reason or another. Usually good reasons.
- burnte 6y ago
  And also that it says that WikiPedia articles are in an impenetrable, UNPARSABLE format that you can't run scripts against. This is absurd. If it's unparsable, how does Mediawiki manage? because it IS parsable! oh my, in order to work with the content you need to either use the MW parser or write your own. Well, literally everything is like that. just because there aren't 5 different MW parser engines out there doesn't mean it's unparsable. I stopped reading the article early on.
  - detaro 6y ago
    Mediawiki manages by having defined "whatever our code does is how it is", and when they needed a second parser for their new WYSIWG editor (because the one in the main Mediawiki code base isn't really usable for other purposes) it added a lot of extra effort and pain. So no, it's not literally "impossible", but a large hurdle to working with it. When I had to deal with mediawiki markup I built all kinds of really ugly tricks that fed markup snippets into a mediawiki instance, parsed the HTML, adjusted the snippets based on that, repeat...
    - dredmorbius 6y ago
      POSIWID meet POMLIWIR https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
      - jwilk 6y ago
        What's POMLIWIR?
        dredmorbius 6y ago
        The purpose of a markup language is what it renders.
  - deleted 6y ago
    [deleted]
bzb6 6y ago
This blog is unreadable in mobile Safari.
- cglong 6y ago
  I had to zoom out on Chrome for Android to read it.
- Hnrobert42 6y ago
  Same for Firefox Focus on iPhone.
awinter-py 6y ago
> Work is being ravenously deleted all the time this reminds me of the library in the david brin uplift books, where the myth is that it contains perfect knowledge but in fact there are sinister memory holes (not saying it's sinister in this case)
- ncmncm 6y ago
  Yet, it is.
The_Colonel 6y ago
Most of those mentioned issues are quite minor - some of them (like talk pages containing copy pasted content) seem actually like non-issues. Or the category thing - yes, it has circular references which breaks the nice theory behind it, but what practical issues to the readers or editors does this present? The syntax is messy, yes and it's a problem. But converting it to a markdown/HTML is downright absurd. I think the current approach of improving it step by step is the most reasonable. Wikipedia is a project which can afford to sustain its own custom syntax (because of history, compatibility concerns and wikipedia's special needs). I think the biggest issue isn't technical in nature - it's the actual task of building the encyclopedia - to decide together on its scope, decide what is notable and what isn't. To fight against brigading from interest groups, fight against attempts to game their rules. All of these are intrinsically difficult tasks and of course wikipedia has made many mistakes. But even then it still fares pretty well in my book.
- dalbasal 6y ago
  Well said. Difficulties are what they are. Part of why things are difficult is because perfect, mechanical solutions aren't feasible. If they were, the problem would be solveable, commoditized and something else would be difficult instead. This article does a good job presenting difficulties, and they are all important areas where doing a good job is important. Eg "Or the category thing - yes, it has circular references which breaks the nice theory behind it, but what practical issues to the readers or editors does this present?" There probably isn't a good, legible solution that will "fix" this. It's an epistemological problem about knowledge itself. Wikipedia's approach is an approach. It isn't perfect because there are no perfect solutions to knowledge problems. It is however, a workable approach given the problem space. None of these need 100% solutions. They just need workable solutions.
- fireattack 6y ago
  > decide what is notable and what isn't. I feel like Wikipedia (at least English Wikipedia) has long past this, and pretty much keep everything that has even the slightest of notability. And I have no issue with it. I see this as one of the (big) advantage of Wikipedia over traditional encyclopedia. (Don't get me wrong, I knew https://en.wikipedia.org/wiki/Wikipedia:Notability https://en.wikipedia.org/wiki/Wikipedia:Notability still exists.)
artagnon 6y ago
Yes, this is the unfortunate reality. It happens to the best of software as it ages. The article doesn't address the elephant in the room: MediaWiki is really the _only_ solution to a difficult problem that we have today. Which other markup language is so comprehensive? Yes, it's unnecessarily complicated, but let's understand that the constraints are very tight: the data can't be separated out of the markup, and it's impossible to make backward-incompatible changes. Many of the really great pages on Wikipedia render beautifully. It's untrue that they haven't added features over the years: they have, but they do it at a glacial pace. Consider one of the recent ones: when you hover over a link, a box pops up and shows a preview. They used to render math on the client-side, but now, they just convert to SVG: as a result, it loads instantly, and renders consistently. The WYSIWIG editor rollout was slow, because people felt that it would attract low-quality low-commitment edits. They first released it as an optional feature, and then turned it on by default, when they were confident that it worked as intended. Oh, and my favorite? Allowing an article to start with a lower-cased word (say iOS); I remember that there were a bunch of redirects just to correct for this deficiency. Yes, it is a giant pain to edit some pages in that arcane syntax, but nothing else even comes close in terms of features. Yes, there are an enormous number of templates, but in practice, an infrequent contributor just finds a page that uses a similar template, and copies it out. Yes, there are lots of bots, and they try very hard to guard against spam, without making you sign up or even solve a CAPTCHA to edit. Plenty of bot edits are "good" edits: they revert rage-rewrites, rage-deletions, and all kinds of malicious user behavior. What you don't understand about redirects is that, the good ones can't be automated. It's not a string-matching problem. Yes, they could automate /some/ of the redirects, and they try. I've personally never run into a typo-redirect in recent years. Yes, it can get political at times, and it's _very_ difficult to have objective guidelines about which pages are worthy of existing. Politicians' pages often get locked, when there's an upcoming election, and this means that you need an account to edit. Again, MW has lots of great features. Wikipedia is aging, and nobody can deny that, but who would want to do the thankless work of parsing the markup and porting it to another system, AND correct the breakages? What commercial value does it have, and who's going to fund it?
fortran77 6y ago
It bothers me that he doesn't know "its" from "it's." It probably bothers others when he edits articles and spells it wrong. I think the biggest problem he mentioned is the "exclusionists." There's no harm in including a page on every local elected official, every Nobel prize winner, even every band that has a reference or two somewhere. Most likely it will remain a little article, but so what? The history of small local bands, for example, would be very interesting 100, 200 years from now. (Wouldn't it be fun to read about some town's local musical groups in 18th century America?)
- rm445 6y ago
  Oddly, this article made me understand the case that exists for deletionism (though it's still awful). It always seemed to be one of those weird things that keeps happening despite nobody being willing to stand up for it. I think I see now, that to the people who consider themselves the self-appointed guardians of Wikipedia's quality, each extra page spreads the work of guarding a little thinner. The counter to which is, of course, that Wikipedia doesn't have to be run by a relatively-tiny number of highly-dedicated editors. Over the hill somewhere just out of sight is a set of technologies, policies and social norms that would lead to still more people participating.
PaulHoule 6y ago
If you are extracting information from Wikipedia and trying to parse the markup you're doing it wrong. It's the difference between "screwed around with handwritten parsers for years and it still doesn't work 100% right" and "look this CSS selector extracts what you want." Freebase, DBpedia, and many others (me) have tried, but the reality is that the markup language is poorly defined and the only path that is really tested is the one that ends up rendering HTML. If you feed HTML from Wikipedia into a web parser that supports the DOM (say Beautiful Soup) you can generally parse out what you want pretty effectively. Once I switched from the "markup rabbithole" to "parsing standard HTML" I was able to turn my MediaWiki extractor into a Flickr extractor in about 15 minutes.
- indymike 6y ago
  This is exactly what I was thinking. Wikipedia articles carry semantic data (ld-json), so it should be pretty easy to just scrape away.
  - PaulHoule 6y ago
    The LD-JSON data is quite limited. I looked at this page https://en.wikipedia.org/wiki/Thomas_Sowell https://en.wikipedia.org/wiki/Thomas_Sowell I went and used this https://addons.mozilla.org/en-US/firefox/addon/openlink-structured-data-sniff/ https://addons.mozilla.org/en-US/firefox/addon/openlink-stru... and the JSON-LD data comprised of "wrapper metadata" for the document, a photograph of that person, and a link to a concept in Wikidata where you would find some of the facts that you'd be looking for if you mistook the wikipedia page for the person. The page at Wikidata for that person https://www.wikidata.org/wiki/Q553449 https://www.wikidata.org/wiki/Q553449 has the birth date, gender and other facts such as he taught at both Cornell and Stanford and also links to identifiers for that person in many bibliographic databases.
  - bawolff 6y ago
    There's also an alternate html output that has more semantic tags (and the html is consistent enough there are parsers to take this html and turn it back into wikitext): https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einstein https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einste...
thrower123 6y ago
None of the technical issues listed amount to even a molehill next to the mountain of cultural and political issues that plague Wikipedia.
- indymike 6y ago
  Sounds a lot like human knowledge. Its filled with cultural and political issues.
Simulacra 6y ago
I cannot recall the last time I used Wikipedia. The hyperaccurate wikis make even the most benign research task a daunting experience. It's like there are perfectionists out there that won't stop tweaking the wiki's. I think Wikipedia has served its usefulness and should be allowed to be absorbed or recreated into something better. For now it's just a trust fund for Jimmy Wales.
- lacksconfidence 6y ago
  How is it a trust fund for Jimmy Wales?
  - 20after4 6y ago
    Simple: it isn't.
- t0astbread 6y ago
  Isn't that sort of accuracy natural for an encyclopedia? Of course, depending on your familiarity with a subject, articles might be hard to read but I think in general Wikipedia does a good job at distilling its sources down to a useful summary.
- deleted 6y ago
  [deleted]
- belorn 6y ago
  May I ask, do you ever look up trivial information online? I do believe there exist people who never look up information online. Such people must exist somewhere and thus I should not exclude them from existence. For the rest of people I strongly believe they either use a search engine, a virtual assistant, or go directly to Wikipedia. The search engine scrape Wikipedia, the virtual assistant use the search engine to find answers and Wikipedia is Wikipedia. In either case it is highly likely that one is using Wikipedia directly or indirectly.
  - approxim8ion 6y ago
    Looking up trivial information is my #1 use case for wikipedia. For instance, I use it to look up movies/series, their cast and public reception all the time. It's presented much better on Wiki than Google/IMDB and just works for me.
8bitsrule 6y ago
Wikipedia is a complicated mess? So's the world, and for many of the same reasons. Look at the average town ... yours, for example. Where -wouldn't- you take visitors? What matters to Wikipedia visitors is content quality (and managed depth) and reliability. All the rest is froth. Layout? Are you kidding? Could it be better? Hell yes. It'd be great to see many articles, written by committee, visited by a professional editor with years of proven experience -in that category-. The Foundation's got the money to pay one per category. Each has to visit 5 articles per day. Each they finish is locked and marke 'pro-edited in (the year)'. Make that fact a search filter. ('Administrators' choosing? Brrrr.) Then there's Wikibestia.org. Every year, copy a limit of 1 million articles there. Chosen how? Poll the visitors, it's theirs! 'Wikibestia ... the people's choice!' 50 net downvotes? Guillotine! Big contest, media advertising, whatever. Articles that can only be read by subject-matter experts completely miss the point. They're just there for vanity or whatever. Flag them, give them one year to move to Wikiexpertia, while versions for non-experts are prepared (or not), then delete.
- dredmorbius 6y ago
  Wikipedia is a complicated mess? So's the world Map resembles terrain!
  - 8bitsrule 6y ago
    Be careful, or the ghost of Korzybski might come looking for you! (I just learned today that the Institute of General Semantics he started in 1938 still exists!)
    - dredmorbius 6y ago
      Fun rabbit hole! https://en.wikipedia.org/wiki/Alfred_Korzybski https://en.wikipedia.org/wiki/Alfred_Korzybski I knew the expression (and was obviously alluding to it), not its origin. Thanks!
liminal 6y ago
I didn't know that Freebase offered their db contents for free and Wikimedia declined. What a stupid, short-sighted loss to the community.
- zozbot234 6y ago
  They did not 'decline' - the work is in progress, but there are lots of known issues wrt. the quality of existing data in Freebase. See e.g. https://www.wikidata.org/wiki/Wikidata:Freebase https://www.wikidata.org/wiki/Wikidata:Freebase for the details.
SmokeyHamster 6y ago
Yeah, I generally agree. Technologically, it's a dumpster fire. And even ideologically, it's just become a lefty political circle-jerk where any facts that are not endorsed, in some cases literally, by MSNBC are immediately purged by self-appointed gate keepers and admins. But that's fine. It'll create more innovation as people ditch it and create better tools. The one part of the article I disagree with is where they say it's "unparseable", which obviously it's not. It's parsed millions of times a day. Even by third party tools that harvest it for semantic data. That means it can be transformed and used to seed other databases using non-terrible description languages like Markdown.
- approxim8ion 6y ago
  Ah MSNBC, the world-renowned bastion of leftism.
ultimateocelot 6y ago
If you want to extract info from Wikipedia, you could use MediaWiki to parse the content and then scrape the (relatively) consistently formatted pages. Might be an easier angle of attack.
- bawolff 6y ago
  Especially if you use the alternate semantic html output: https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einstein https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einste...
WorldPeas 6y ago
Could one not scrape the html-rendered pages, then transfer it to markdown? I see the bulk of the complication with the Einstein article is in citations, could one not simply write a script to track all citations, and create numeric links to them at the end of the document?
biryani_chicken 6y ago
Is there some kind of universal intermediate format that could be targeted by parsers and renders into html? maybe something from pandoc?
hateful 6y ago
The solution to all this seems simple to me. Have the site just accept both formats, the old wiki format and the new [not yet decided] format. Then each page will just have to have a format option. Then, slowly, pages can be converted, even if manually - perhaps just auto-converting and then fixing the manual stuff. And I'm sure there's a number of articles that are just simple links and they can be converted right away. You can put bounties or some kind of incentive for people to take on the conversion, but I think if the new format is better, a number of people will feel strongly enough about it to want to convert each article when it's updated. And even if 12 years from now some articles aren't converted, so what?
- bawolff 6y ago
  We have support for multiple formats already in mediawiki (Special:changeContentModel). For example you can mark a page ss only containing plaintext. There however is a lack of political interest in changing markup formats. Imo changing to markdown at this stage would be a bad idea.
4cao 6y ago
This is a good summary of some of the technical issues Wikipedia is facing. Those can be overcome and eventually will. The really serious problems are organizational in nature: the overgrowth of bureaucracy, the jungle of policies and "sort-of" policies that can be weaponized, the emergence of an inner clique skilled in navigating the quagmire, and the resulting impossibly high barrier to entry for new contributors. The experience of contributing to Wikipedia these days is inherently adversarial. People come and write something genuinely useful or at least do so in good faith, and then realize they unleashed on themselves a whole lengthy process when they have to defend what they did on numerous talk pages over an extended period of time, or succumb to deletionists. Even if they prevail, the experience is hardly pleasant, so they eventually leave and never look back. An artefact of the above is also that the editorial base suffers from an overrepresentation of people with vested interests, as they will always be the most highly-motivated to stay. There are of course many great editors too but the way things are going, the project is not sustainable, and its quality is being compromised due to an organizational failure. What Wikipedia needs is not Markdown (not that I have anything against it) but courage, fresh air, and less of the siege mentality.
- samat 6y ago
  Thank you for writing this. Some years ago I served as a CTO for a major news media org. Google approached us and asked to implement some 'structured web' stuff on our pages, so I dug in and realised there is a mandatory field for a link to a book or movie reviewers personal page in wikipedia. So I've found out that one of the most notable Russian book reviewers does not have an wikipedia page. No big deal, I've created one with some very basic info and some links to prove she's 'notable enough'. What ensued shortly after is kafkaesque story of some entitled shitheads who have power because they know some local lingo and have unlimited time resources to waste. Luckily, someone poured too much time and effort and made that page 'standard wikipedia article about a person' with all those 'templates and boxes' and used the right lingo to defend it's notability. I swear I tried to do it myself and failed. I am not stupid, the wikipedia editing is just too complicated. Thanks for reading, a few years passed and apparently I still have not got it fully out of my system. That feeling of complete helplessness, when those bureaucrats are clearly toying with you and enjoying it. It sucks big time.
bawolff 6y ago
I disagree with most of this (for context im a mediawiki developer. I used to work for WMF but dont anymore. My opinions are my own): -wikitext unparsable - wikitext is a bit insane but there exists a parser called parsoid. If we couldn't parse it it would be impossible to make a visual editor - most pages are redirects: not sure what the problem is. -boilerplate text - there is a template system to have repeated text in only one place. Not sure what the issue is. - he goes on a rant about no search without giving much context. There is a search feature based on elastic search (older versions were based on lucene directly). In my opinion its a pretty decent search engine (especially compared to most sites that make their own search). I'm not sure what the actual complaint is. -complaints about wikidata - this is more political than technical, however "If wikidata was a company, it would not exist anymore, and you wouldn't have heard of it." seems patently false. Wikidata is pretty popular even outside of academia, and is used quite extensively. - category tree being a graph not a tree - that's kind of unfortunate but what exactly is the problem here. Its a problem on commons, but i've never really seen how its an issue in practise on wikipedia. Complex categorization is being taken over by wikidata anyways. Template ecosystem is complex - it could certainly be better, but the complexity here is a trade-off allowing more flexibility and allowing the system to evolve. Inclusionist vs deletionist: no comment UI design: perhaps a fair point here, although i do kind of like the stability of the current design. Most of the modern web sucks imho. Moral failure: i dont want wikipedia to fix the world. That's not its role. Its job is to document, not to partake. Viz editor not being enabled by default: i agree, although i think it was pushed too hard in early days when there was still kinks, but its long past time now. To be clear, i definitely dont think wikipedia is perfect, i just disagree with some of these specific criticisms.
- entropicdrifter 6y ago
  - most pages are redirects: not sure what the problem is. With respect to this point, I think the author is saying that it's a dated system that introduces a ton of potentially unnecessary link rot. Basically, there's a ton of unnecessary technical debt because there isn't a more automatic way to set up/maintain redirects and instead often multiple manual entries are added for e.g. common misspellings of article names.
hk__2 6y ago
> After a year of development, when it was completed, wikipedia editors voted en-mass against integrating the visual editor. To be more accurate, those are the editors on the English Wikipedia (the largest one). The French Wikipedia, for instance, have this visual editor enabled.

103 points by sanqui 6y ago

I would be highly interested in Moving Wikipedia to Markdown in sphinx with own repo in git. But this poses issue to people who want to use a WYIWYG Editor and not git it away.

klodolph 6y ago
It also poses a major problem for advanced usage (templates, math, citations, etc.) Markdown is a decent language for basic stuff like Reddit, Stack Overflow, GitHub READMEs, etc. Stuff where the most important feature is making it easy for people to make content. Places like Wikipedia or blogs are where Markdown doesn’t work so well. Sure, you want it to be relatively easy for people to edit Wikipedia pages, but it’s worth sacrificing ease of editing because you want to improve the reading experience.
mhh__ 6y ago
They'd need to think a lot bigger and better than markdown. Honestly, wikipedia would be a great proving ground for a hypothetical evolution of LaTeX
- warmwaffles 6y ago
  The whole right to left and left to right orientation is definitely a difficult thing to solve. On top of that, you'd need to figure out a way to have translations live along side the article itself.
- BlueTemplar 6y ago
  LaTeX itself is a big mess...

kayodelycaon 6y ago

I think this article shoots itself in the foot by suggesting markdown. Wikipedia predates markdown by several years. The only truly “universal” markup language I know of at the time was bbcode. Even if markdown has been designed, it’s lacking support for templates and tables that wikis make heavy use of. Every wiki software I’ve used has had a custom language for one reason or another. Usually good reasons.

burnte 6y ago
And also that it says that WikiPedia articles are in an impenetrable, UNPARSABLE format that you can't run scripts against. This is absurd. If it's unparsable, how does Mediawiki manage? because it IS parsable! oh my, in order to work with the content you need to either use the MW parser or write your own. Well, literally everything is like that. just because there aren't 5 different MW parser engines out there doesn't mean it's unparsable. I stopped reading the article early on.
- detaro 6y ago
  Mediawiki manages by having defined "whatever our code does is how it is", and when they needed a second parser for their new WYSIWG editor (because the one in the main Mediawiki code base isn't really usable for other purposes) it added a lot of extra effort and pain. So no, it's not literally "impossible", but a large hurdle to working with it. When I had to deal with mediawiki markup I built all kinds of really ugly tricks that fed markup snippets into a mediawiki instance, parsed the HTML, adjusted the snippets based on that, repeat...
  - dredmorbius 6y ago
    POSIWID meet POMLIWIR https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
    - jwilk 6y ago
      What's POMLIWIR?
      - dredmorbius 6y ago
        The purpose of a markup language is what it renders.
- deleted 6y ago
  [deleted]

bzb6 6y ago

This blog is unreadable in mobile Safari.

cglong 6y ago
I had to zoom out on Chrome for Android to read it.
Hnrobert42 6y ago
Same for Firefox Focus on iPhone.

awinter-py 6y ago

> Work is being ravenously deleted all the time this reminds me of the library in the david brin uplift books, where the myth is that it contains perfect knowledge but in fact there are sinister memory holes (not saying it's sinister in this case)

ncmncm 6y ago
Yet, it is.

The_Colonel 6y ago

Most of those mentioned issues are quite minor - some of them (like talk pages containing copy pasted content) seem actually like non-issues. Or the category thing - yes, it has circular references which breaks the nice theory behind it, but what practical issues to the readers or editors does this present? The syntax is messy, yes and it's a problem. But converting it to a markdown/HTML is downright absurd. I think the current approach of improving it step by step is the most reasonable. Wikipedia is a project which can afford to sustain its own custom syntax (because of history, compatibility concerns and wikipedia's special needs). I think the biggest issue isn't technical in nature - it's the actual task of building the encyclopedia - to decide together on its scope, decide what is notable and what isn't. To fight against brigading from interest groups, fight against attempts to game their rules. All of these are intrinsically difficult tasks and of course wikipedia has made many mistakes. But even then it still fares pretty well in my book.

dalbasal 6y ago
Well said. Difficulties are what they are. Part of why things are difficult is because perfect, mechanical solutions aren't feasible. If they were, the problem would be solveable, commoditized and something else would be difficult instead. This article does a good job presenting difficulties, and they are all important areas where doing a good job is important. Eg "Or the category thing - yes, it has circular references which breaks the nice theory behind it, but what practical issues to the readers or editors does this present?" There probably isn't a good, legible solution that will "fix" this. It's an epistemological problem about knowledge itself. Wikipedia's approach is an approach. It isn't perfect because there are no perfect solutions to knowledge problems. It is however, a workable approach given the problem space. None of these need 100% solutions. They just need workable solutions.
fireattack 6y ago
> decide what is notable and what isn't. I feel like Wikipedia (at least English Wikipedia) has long past this, and pretty much keep everything that has even the slightest of notability. And I have no issue with it. I see this as one of the (big) advantage of Wikipedia over traditional encyclopedia. (Don't get me wrong, I knew https://en.wikipedia.org/wiki/Wikipedia:Notability https://en.wikipedia.org/wiki/Wikipedia:Notability still exists.)

artagnon 6y ago

Yes, this is the unfortunate reality. It happens to the best of software as it ages. The article doesn't address the elephant in the room: MediaWiki is really the _only_ solution to a difficult problem that we have today. Which other markup language is so comprehensive? Yes, it's unnecessarily complicated, but let's understand that the constraints are very tight: the data can't be separated out of the markup, and it's impossible to make backward-incompatible changes. Many of the really great pages on Wikipedia render beautifully. It's untrue that they haven't added features over the years: they have, but they do it at a glacial pace. Consider one of the recent ones: when you hover over a link, a box pops up and shows a preview. They used to render math on the client-side, but now, they just convert to SVG: as a result, it loads instantly, and renders consistently. The WYSIWIG editor rollout was slow, because people felt that it would attract low-quality low-commitment edits. They first released it as an optional feature, and then turned it on by default, when they were confident that it worked as intended. Oh, and my favorite? Allowing an article to start with a lower-cased word (say iOS); I remember that there were a bunch of redirects just to correct for this deficiency. Yes, it is a giant pain to edit some pages in that arcane syntax, but nothing else even comes close in terms of features. Yes, there are an enormous number of templates, but in practice, an infrequent contributor just finds a page that uses a similar template, and copies it out. Yes, there are lots of bots, and they try very hard to guard against spam, without making you sign up or even solve a CAPTCHA to edit. Plenty of bot edits are "good" edits: they revert rage-rewrites, rage-deletions, and all kinds of malicious user behavior. What you don't understand about redirects is that, the good ones can't be automated. It's not a string-matching problem. Yes, they could automate /some/ of the redirects, and they try. I've personally never run into a typo-redirect in recent years. Yes, it can get political at times, and it's _very_ difficult to have objective guidelines about which pages are worthy of existing. Politicians' pages often get locked, when there's an upcoming election, and this means that you need an account to edit. Again, MW has lots of great features. Wikipedia is aging, and nobody can deny that, but who would want to do the thankless work of parsing the markup and porting it to another system, AND correct the breakages? What commercial value does it have, and who's going to fund it?

fortran77 6y ago

It bothers me that he doesn't know "its" from "it's." It probably bothers others when he edits articles and spells it wrong. I think the biggest problem he mentioned is the "exclusionists." There's no harm in including a page on every local elected official, every Nobel prize winner, even every band that has a reference or two somewhere. Most likely it will remain a little article, but so what? The history of small local bands, for example, would be very interesting 100, 200 years from now. (Wouldn't it be fun to read about some town's local musical groups in 18th century America?)

rm445 6y ago
Oddly, this article made me understand the case that exists for deletionism (though it's still awful). It always seemed to be one of those weird things that keeps happening despite nobody being willing to stand up for it. I think I see now, that to the people who consider themselves the self-appointed guardians of Wikipedia's quality, each extra page spreads the work of guarding a little thinner. The counter to which is, of course, that Wikipedia doesn't have to be run by a relatively-tiny number of highly-dedicated editors. Over the hill somewhere just out of sight is a set of technologies, policies and social norms that would lead to still more people participating.

PaulHoule 6y ago

If you are extracting information from Wikipedia and trying to parse the markup you're doing it wrong. It's the difference between "screwed around with handwritten parsers for years and it still doesn't work 100% right" and "look this CSS selector extracts what you want." Freebase, DBpedia, and many others (me) have tried, but the reality is that the markup language is poorly defined and the only path that is really tested is the one that ends up rendering HTML. If you feed HTML from Wikipedia into a web parser that supports the DOM (say Beautiful Soup) you can generally parse out what you want pretty effectively. Once I switched from the "markup rabbithole" to "parsing standard HTML" I was able to turn my MediaWiki extractor into a Flickr extractor in about 15 minutes.

indymike 6y ago
This is exactly what I was thinking. Wikipedia articles carry semantic data (ld-json), so it should be pretty easy to just scrape away.
- PaulHoule 6y ago
  The LD-JSON data is quite limited. I looked at this page https://en.wikipedia.org/wiki/Thomas_Sowell https://en.wikipedia.org/wiki/Thomas_Sowell I went and used this https://addons.mozilla.org/en-US/firefox/addon/openlink-structured-data-sniff/ https://addons.mozilla.org/en-US/firefox/addon/openlink-stru... and the JSON-LD data comprised of "wrapper metadata" for the document, a photograph of that person, and a link to a concept in Wikidata where you would find some of the facts that you'd be looking for if you mistook the wikipedia page for the person. The page at Wikidata for that person https://www.wikidata.org/wiki/Q553449 https://www.wikidata.org/wiki/Q553449 has the birth date, gender and other facts such as he taught at both Cornell and Stanford and also links to identifiers for that person in many bibliographic databases.
- bawolff 6y ago
  There's also an alternate html output that has more semantic tags (and the html is consistent enough there are parsers to take this html and turn it back into wikitext): https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einstein https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einste...

thrower123 6y ago

None of the technical issues listed amount to even a molehill next to the mountain of cultural and political issues that plague Wikipedia.

indymike 6y ago
Sounds a lot like human knowledge. Its filled with cultural and political issues.

Simulacra 6y ago

I cannot recall the last time I used Wikipedia. The hyperaccurate wikis make even the most benign research task a daunting experience. It's like there are perfectionists out there that won't stop tweaking the wiki's. I think Wikipedia has served its usefulness and should be allowed to be absorbed or recreated into something better. For now it's just a trust fund for Jimmy Wales.

lacksconfidence 6y ago
How is it a trust fund for Jimmy Wales?
- 20after4 6y ago
  Simple: it isn't.
t0astbread 6y ago
Isn't that sort of accuracy natural for an encyclopedia? Of course, depending on your familiarity with a subject, articles might be hard to read but I think in general Wikipedia does a good job at distilling its sources down to a useful summary.
deleted 6y ago
[deleted]
belorn 6y ago
May I ask, do you ever look up trivial information online? I do believe there exist people who never look up information online. Such people must exist somewhere and thus I should not exclude them from existence. For the rest of people I strongly believe they either use a search engine, a virtual assistant, or go directly to Wikipedia. The search engine scrape Wikipedia, the virtual assistant use the search engine to find answers and Wikipedia is Wikipedia. In either case it is highly likely that one is using Wikipedia directly or indirectly.
- approxim8ion 6y ago
  Looking up trivial information is my #1 use case for wikipedia. For instance, I use it to look up movies/series, their cast and public reception all the time. It's presented much better on Wiki than Google/IMDB and just works for me.

8bitsrule 6y ago

Wikipedia is a complicated mess? So's the world, and for many of the same reasons. Look at the average town ... yours, for example. Where -wouldn't- you take visitors? What matters to Wikipedia visitors is content quality (and managed depth) and reliability. All the rest is froth. Layout? Are you kidding? Could it be better? Hell yes. It'd be great to see many articles, written by committee, visited by a professional editor with years of proven experience -in that category-. The Foundation's got the money to pay one per category. Each has to visit 5 articles per day. Each they finish is locked and marke 'pro-edited in (the year)'. Make that fact a search filter. ('Administrators' choosing? Brrrr.) Then there's Wikibestia.org. Every year, copy a limit of 1 million articles there. Chosen how? Poll the visitors, it's theirs! 'Wikibestia ... the people's choice!' 50 net downvotes? Guillotine! Big contest, media advertising, whatever. Articles that can only be read by subject-matter experts completely miss the point. They're just there for vanity or whatever. Flag them, give them one year to move to Wikiexpertia, while versions for non-experts are prepared (or not), then delete.

dredmorbius 6y ago
Wikipedia is a complicated mess? So's the world Map resembles terrain!
- 8bitsrule 6y ago
  Be careful, or the ghost of Korzybski might come looking for you! (I just learned today that the Institute of General Semantics he started in 1938 still exists!)
  - dredmorbius 6y ago
    Fun rabbit hole! https://en.wikipedia.org/wiki/Alfred_Korzybski https://en.wikipedia.org/wiki/Alfred_Korzybski I knew the expression (and was obviously alluding to it), not its origin. Thanks!

liminal 6y ago

I didn't know that Freebase offered their db contents for free and Wikimedia declined. What a stupid, short-sighted loss to the community.

zozbot234 6y ago
They did not 'decline' - the work is in progress, but there are lots of known issues wrt. the quality of existing data in Freebase. See e.g. https://www.wikidata.org/wiki/Wikidata:Freebase https://www.wikidata.org/wiki/Wikidata:Freebase for the details.

SmokeyHamster 6y ago

Yeah, I generally agree. Technologically, it's a dumpster fire. And even ideologically, it's just become a lefty political circle-jerk where any facts that are not endorsed, in some cases literally, by MSNBC are immediately purged by self-appointed gate keepers and admins. But that's fine. It'll create more innovation as people ditch it and create better tools. The one part of the article I disagree with is where they say it's "unparseable", which obviously it's not. It's parsed millions of times a day. Even by third party tools that harvest it for semantic data. That means it can be transformed and used to seed other databases using non-terrible description languages like Markdown.

approxim8ion 6y ago
Ah MSNBC, the world-renowned bastion of leftism.

ultimateocelot 6y ago

If you want to extract info from Wikipedia, you could use MediaWiki to parse the content and then scrape the (relatively) consistently formatted pages. Might be an easier angle of attack.

bawolff 6y ago
Especially if you use the alternate semantic html output: https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einstein https://en.wikipedia.org/api/rest_v1/page/html/Albert_Einste...

WorldPeas 6y ago

Could one not scrape the html-rendered pages, then transfer it to markdown? I see the bulk of the complication with the Einstein article is in citations, could one not simply write a script to track all citations, and create numeric links to them at the end of the document?

biryani_chicken 6y ago

Is there some kind of universal intermediate format that could be targeted by parsers and renders into html? maybe something from pandoc?

hateful 6y ago

The solution to all this seems simple to me. Have the site just accept both formats, the old wiki format and the new [not yet decided] format. Then each page will just have to have a format option. Then, slowly, pages can be converted, even if manually - perhaps just auto-converting and then fixing the manual stuff. And I'm sure there's a number of articles that are just simple links and they can be converted right away. You can put bounties or some kind of incentive for people to take on the conversion, but I think if the new format is better, a number of people will feel strongly enough about it to want to convert each article when it's updated. And even if 12 years from now some articles aren't converted, so what?

bawolff 6y ago
We have support for multiple formats already in mediawiki (Special:changeContentModel). For example you can mark a page ss only containing plaintext. There however is a lack of political interest in changing markup formats. Imo changing to markdown at this stage would be a bad idea.

4cao 6y ago

This is a good summary of some of the technical issues Wikipedia is facing. Those can be overcome and eventually will. The really serious problems are organizational in nature: the overgrowth of bureaucracy, the jungle of policies and "sort-of" policies that can be weaponized, the emergence of an inner clique skilled in navigating the quagmire, and the resulting impossibly high barrier to entry for new contributors. The experience of contributing to Wikipedia these days is inherently adversarial. People come and write something genuinely useful or at least do so in good faith, and then realize they unleashed on themselves a whole lengthy process when they have to defend what they did on numerous talk pages over an extended period of time, or succumb to deletionists. Even if they prevail, the experience is hardly pleasant, so they eventually leave and never look back. An artefact of the above is also that the editorial base suffers from an overrepresentation of people with vested interests, as they will always be the most highly-motivated to stay. There are of course many great editors too but the way things are going, the project is not sustainable, and its quality is being compromised due to an organizational failure. What Wikipedia needs is not Markdown (not that I have anything against it) but courage, fresh air, and less of the siege mentality.

samat 6y ago
Thank you for writing this. Some years ago I served as a CTO for a major news media org. Google approached us and asked to implement some 'structured web' stuff on our pages, so I dug in and realised there is a mandatory field for a link to a book or movie reviewers personal page in wikipedia. So I've found out that one of the most notable Russian book reviewers does not have an wikipedia page. No big deal, I've created one with some very basic info and some links to prove she's 'notable enough'. What ensued shortly after is kafkaesque story of some entitled shitheads who have power because they know some local lingo and have unlimited time resources to waste. Luckily, someone poured too much time and effort and made that page 'standard wikipedia article about a person' with all those 'templates and boxes' and used the right lingo to defend it's notability. I swear I tried to do it myself and failed. I am not stupid, the wikipedia editing is just too complicated. Thanks for reading, a few years passed and apparently I still have not got it fully out of my system. That feeling of complete helplessness, when those bureaucrats are clearly toying with you and enjoying it. It sucks big time.

bawolff 6y ago

I disagree with most of this (for context im a mediawiki developer. I used to work for WMF but dont anymore. My opinions are my own): -wikitext unparsable - wikitext is a bit insane but there exists a parser called parsoid. If we couldn't parse it it would be impossible to make a visual editor - most pages are redirects: not sure what the problem is. -boilerplate text - there is a template system to have repeated text in only one place. Not sure what the issue is. - he goes on a rant about no search without giving much context. There is a search feature based on elastic search (older versions were based on lucene directly). In my opinion its a pretty decent search engine (especially compared to most sites that make their own search). I'm not sure what the actual complaint is. -complaints about wikidata - this is more political than technical, however "If wikidata was a company, it would not exist anymore, and you wouldn't have heard of it." seems patently false. Wikidata is pretty popular even outside of academia, and is used quite extensively. - category tree being a graph not a tree - that's kind of unfortunate but what exactly is the problem here. Its a problem on commons, but i've never really seen how its an issue in practise on wikipedia. Complex categorization is being taken over by wikidata anyways. Template ecosystem is complex - it could certainly be better, but the complexity here is a trade-off allowing more flexibility and allowing the system to evolve. Inclusionist vs deletionist: no comment UI design: perhaps a fair point here, although i do kind of like the stability of the current design. Most of the modern web sucks imho. Moral failure: i dont want wikipedia to fix the world. That's not its role. Its job is to document, not to partake. Viz editor not being enabled by default: i agree, although i think it was pushed too hard in early days when there was still kinks, but its long past time now. To be clear, i definitely dont think wikipedia is perfect, i just disagree with some of these specific criticisms.

entropicdrifter 6y ago
- most pages are redirects: not sure what the problem is. With respect to this point, I think the author is saying that it's a dated system that introduces a ton of potentially unnecessary link rot. Basically, there's a ton of unnecessary technical debt because there isn't a more automatic way to set up/maintain redirects and instead often multiple manual entries are added for e.g. common misspellings of article names.

hk__2 6y ago

> After a year of development, when it was completed, wikipedia editors voted en-mass against integrating the visual editor. To be more accurate, those are the editors on the English Wikipedia (the largest one). The French Wikipedia, for instance, have this visual editor enabled.