7 ms·

That gives you Wikitext encapsulated in XML. How do you get at the content of the Wikitext? I work on a Wikitext parser [1]. So do many other people, in differ

by rspeer 9y ago

That gives you Wikitext encapsulated in XML. How do you get at the content of the Wikitext? I work on a Wikitext parser [1]. So do many other people, in different ways. Wikitext syntax is horrible and it mixes content and presentation indiscriminately (for example, it contains most of HTML as a sub-syntax). The problem is basically unsolvable, as the result of parsing a Wiki page is defined only by a complete implementation of MediaWiki (with all its recursively-evaluated template pages, PHP code, and Lua code), but if you run that whole stack what you get in the end is HTML -- just the presentation, not the content you presumably wanted. So people solve various pieces of the problem instead, creating approximate parsers that oversimplify various situations to meet their needs. One of these solutions is DBPedia [2], but if you use DBPedia you have to watch out for the parts that are false or misleading due to parse errors. [1] https://github.com/LuminosoInsight/wikiparsec https://github.com/LuminosoInsight/wikiparsec [2] http://wiki.dbpedia.org/ http://wiki.dbpedia.org/

shshhdhs 9y ago
Thanks for working on that! Didn't know it was so bad. The following is a possibly stupid idea, but I'd like to hear your thoughts: What if you just render the content into HTML and then "screen scraped" the text, and then convert into a more useful format (MarkDown, JSON, etc). Is that plausible?
- GhostVII 9y ago
  That would allow a basic UI change on Wikipedia to break your code. Sometimes it is necessary, but not usually the best option in my experience, and it's pretty annoying to do.
- 3131s 9y ago
  Wikipedia used to do HTML dumps but stopped a long time ago, unfortunately.
  - int_19h 9y ago
    You can get what amounts to an HTML dump (which is then indexed and compressed in a single huge archive) from Kiwix. Although they do them basically twice a year or so.
lacksconfidence 9y ago
You could download the search indexes, also on the dumps site, that has the text content among other things.
deleted 9y ago
[deleted]
10165 9y ago
"That gives you Wikitext encapsulated in XML." avar: "The goal of Wikipedia should be to spread the content as far & wide as possible, the way OpenStreetMap operates is a better model." I am confused. Doesn't OSM data come encapsulated in XML or some binary format? As for dispersion of content, I could have sworn I have seen Wikipedia content on non-Wikipedia websites. Is there some restriction that prohibits this? I have seen Wikipedia data offered in DNS TXT records as well.
- 3131s 9y ago
  For each article there is some metadata, but the entire text of an article is just a blob inside one XML element. For anyone who has not worked with the Wikipedia data dumps extensively before, trust us that it is not easily machine-readable and that even solutions like DBPedia / Wikidata are not yet suitable for many purposes.
  - WikipediasBad 9y ago
    As someone who contributes to many knowledge projects, including Wikipedia and Wikidata frequently, I'm curious about what you mean that Wikidata is not yet suitable for any purposes. Am I wasting my time contributing to it? I thought that it was helping a lot of machines understand data. Can you please explain further?
    - 3131s 9y ago
      Please reread, for many purposes! I love Wikipedia. The Wiki markup is extremely complicated and being user created, it is also inconsistent and error prone. I believe the MediaWiki parser itself is something like a single 5000 line PHP function! All of the alternate parsers I've tried are not perfect. There is a ton of information encoded in the semi-structured markup, but it's still not easy to turn that into actual structured data. That's where the problem lies.
      - lacksconfidence 9y ago
        > believe the MediaWiki parser itself is something like a single 5000 line PHP function! It's not. I'm on mobile so not easiest to link, but the PHP versio of the parser is nothing like a single function. There is also a nodejs version of the parser under active development with the goal of replacing the php parser.
Tpt 9y ago
You should have a look at [1] that outputs an HTML rendering of pages with a lot of metadata. [1] https://en.wikipedia.org/api/rest_v1/#!/Page_content/get_page_html_title https://en.wikipedia.org/api/rest_v1/#!/Page_content/get_pag...

by rspeer 9y ago

shshhdhs 9y ago

Thanks for working on that! Didn't know it was so bad. The following is a possibly stupid idea, but I'd like to hear your thoughts: What if you just render the content into HTML and then "screen scraped" the text, and then convert into a more useful format (MarkDown, JSON, etc). Is that plausible?

GhostVII 9y ago
That would allow a basic UI change on Wikipedia to break your code. Sometimes it is necessary, but not usually the best option in my experience, and it's pretty annoying to do.
3131s 9y ago
Wikipedia used to do HTML dumps but stopped a long time ago, unfortunately.
- int_19h 9y ago
  You can get what amounts to an HTML dump (which is then indexed and compressed in a single huge archive) from Kiwix. Although they do them basically twice a year or so.

lacksconfidence 9y ago

You could download the search indexes, also on the dumps site, that has the text content among other things.

deleted 9y ago

[deleted]

10165 9y ago

"That gives you Wikitext encapsulated in XML." avar: "The goal of Wikipedia should be to spread the content as far & wide as possible, the way OpenStreetMap operates is a better model." I am confused. Doesn't OSM data come encapsulated in XML or some binary format? As for dispersion of content, I could have sworn I have seen Wikipedia content on non-Wikipedia websites. Is there some restriction that prohibits this? I have seen Wikipedia data offered in DNS TXT records as well.

3131s 9y ago
For each article there is some metadata, but the entire text of an article is just a blob inside one XML element. For anyone who has not worked with the Wikipedia data dumps extensively before, trust us that it is not easily machine-readable and that even solutions like DBPedia / Wikidata are not yet suitable for many purposes.
- WikipediasBad 9y ago
  As someone who contributes to many knowledge projects, including Wikipedia and Wikidata frequently, I'm curious about what you mean that Wikidata is not yet suitable for any purposes. Am I wasting my time contributing to it? I thought that it was helping a lot of machines understand data. Can you please explain further?
  - 3131s 9y ago
    Please reread, for many purposes! I love Wikipedia. The Wiki markup is extremely complicated and being user created, it is also inconsistent and error prone. I believe the MediaWiki parser itself is something like a single 5000 line PHP function! All of the alternate parsers I've tried are not perfect. There is a ton of information encoded in the semi-structured markup, but it's still not easy to turn that into actual structured data. That's where the problem lies.
    - lacksconfidence 9y ago
      > believe the MediaWiki parser itself is something like a single 5000 line PHP function! It's not. I'm on mobile so not easiest to link, but the PHP versio of the parser is nothing like a single function. There is also a nodejs version of the parser under active development with the goal of replacing the php parser.

Tpt 9y ago

You should have a look at [1] that outputs an HTML rendering of pages with a lot of metadata. [1] https://en.wikipedia.org/api/rest_v1/#!/Page_content/get_page_html_title https://en.wikipedia.org/api/rest_v1/#!/Page_content/get_pag...