6 ms·

Can't you use their API? https://en.wikipedia.org/api/rest_v1/#!/Page_content/get_page https://en.wikipedia.org/api/rest_v1/#!/Page_content/get_pag...

squeaky-clean 8y ago
I haven't used their API in a long time, but I don't think there is a reliable way to get the sidebar that isn't worse than HTML. For example https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=xml&titles=Hacker+News&rvsection=0 https://en.wikipedia.org/w/api.php?action=query&prop=revisio... returns {{Infobox website | name = Hacker News | logo = File:hackernews_logo.png | logo_size = 100px | type = [[News aggregator]] | url = {{url|https://news.ycombinator.com/}} | screenshot = File:hn_screenshot.png | registration = Optional | programming_language = [[Arc (programming language)|Arc]] | founder = [[Paul Graham (computer programmer)|Paul Graham]] | launch date = {{start date and age|2007|02|19}} | current status = Online | owner = [[Y Combinator (company)|Y Combinator]] | language = [[English language|English]] }} '''Hacker News''' is a [[social news]] website focusing on [[Computer Science|computer science]] and [[Startup company|entrepreneurship]]. It is run by [[Paul Graham (computer programmer)|Paul Graham]]'s investment fund and startup incubator, [[Y Combinator (company)|Y Combinator]]. In general, content that can be submitted is defined as "anything that gratifies one's intellectual curiosity".<ref>{{cite news | first = Paul | last = Graham | title = Hacker News Guidelines | url = http://ycombinator.com/newsguidelines.html | accessdate = 2009-04-29 }}</ref> Which isn't very easy to parse either. From a cursory search, a better format doesn't seem possible without using a 3rd party like dbpedia. https://www.quora.com/Does-the-Wikipedia-API-give-structured-information-on-the-infobox https://www.quora.com/Does-the-Wikipedia-API-give-structured... I tried it for OSX, and it actually just returns a redirect statement to MacOS. So expect your tool consuming the API to break if you don't handle that in advance. https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=xml&titles=OSX&rvsection=0 https://en.wikipedia.org/w/api.php?action=query&prop=revisio... And then when trying it for MacOS, I can't actually find the version info anywhere in the response data. So you couldn't even get that data without scraping the page. https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=xml&titles=MacOS&rvsection=0 https://en.wikipedia.org/w/api.php?action=query&prop=revisio...

squeaky-clean 8y ago

I haven't used their API in a long time, but I don't think there is a reliable way to get the sidebar that isn't worse than HTML. For example https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=xml&titles=Hacker+News&rvsection=0 https://en.wikipedia.org/w/api.php?action=query&prop=revisio... returns {{Infobox website | name = Hacker News | logo = File:hackernews_logo.png | logo_size = 100px | type = [[News aggregator]] | url = {{url|https://news.ycombinator.com/}} | screenshot = File:hn_screenshot.png | registration = Optional | programming_language = [[Arc (programming language)|Arc]] | founder = [[Paul Graham (computer programmer)|Paul Graham]] | launch date = {{start date and age|2007|02|19}} | current status = Online | owner = [[Y Combinator (company)|Y Combinator]] | language = [[English language|English]] }} '''Hacker News''' is a [[social news]] website focusing on [[Computer Science|computer science]] and [[Startup company|entrepreneurship]]. It is run by [[Paul Graham (computer programmer)|Paul Graham]]'s investment fund and startup incubator, [[Y Combinator (company)|Y Combinator]]. In general, content that can be submitted is defined as "anything that gratifies one's intellectual curiosity".<ref>{{cite news | first = Paul | last = Graham | title = Hacker News Guidelines | url = http://ycombinator.com/newsguidelines.html | accessdate = 2009-04-29 }}</ref> Which isn't very easy to parse either. From a cursory search, a better format doesn't seem possible without using a 3rd party like dbpedia. https://www.quora.com/Does-the-Wikipedia-API-give-structured-information-on-the-infobox https://www.quora.com/Does-the-Wikipedia-API-give-structured... I tried it for OSX, and it actually just returns a redirect statement to MacOS. So expect your tool consuming the API to break if you don't handle that in advance. https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=xml&titles=OSX&rvsection=0 https://en.wikipedia.org/w/api.php?action=query&prop=revisio... And then when trying it for MacOS, I can't actually find the version info anywhere in the response data. So you couldn't even get that data without scraping the page. https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=xml&titles=MacOS&rvsection=0 https://en.wikipedia.org/w/api.php?action=query&prop=revisio...