3 ms·
"books" appears 3 times on http://en.wikipedia.org/wiki/History_of_Canada http://en.wikipedia.org/wiki/History_of_Canada
by 6ren 14y ago
"books" appears 3 times on http://en.wikipedia.org/wiki/History_of_Canada http://en.wikipedia.org/wiki/History_of_Canada
- robomartin 14y agoAh, yes. I didn't do a full cleanup job before processing and you caught one of those words. If you open that page with Chrome and do a "View Source" you'll see that "books" actually appears 597 times in a whole pile of links on the page. Some of them got through. If you only search what a human can see, yeah, it's somewhere in the order of 3 to 5 times. I might have a look at the code and see why the full <a> tag contents are not being removed. Funny enough, that's the only bit of code I did not write. I got lazy and grabbed something right out of the PHP manual pages. Lesson learned.
- dredmorbius 14y agoUsing a text-mode browser to render HTML to text, strip off headers, and avoid summarizing links (lynx works great for this) might be a fast route to cleaner results. You'd probably also want to exclude the "See also" and "References" sections. For Canada, not excluding stopwords ("the", "of", "and", etc.) or the country name itself, I get: 1 869 the 2 449 of 3 326 and 4 288 in 5 214 to 6 142 a 7 135 canada 8 106 was 9 87 by 10 72 as 11 71 british 12 70 with 13 69 s 14 62 war 15 61 on 16 53 canadian 17 52 from 18 49 for 19 44 were 20 42 new Which still puts "british" on top.
- robomartin 14y agoYup, in case you missed it, I posted updated code that really cleans-up the scraped page data very well. With the new code "british" is on top of the Canada list. It was a fun little exercise. There are some issues accessing Wikipedia this way but no big deal for a quick little test.