4 ms·

I was puzzled by this, particularly by the number of countries that were marked with "war". So, I did what any programmer might do, I wrote some code to check

• 6ren 14y ago
  "books" appears 3 times on http://en.wikipedia.org/wiki/History_of_Canada http://en.wikipedia.org/wiki/History_of_Canada
  • robomartin 14y ago
    Ah, yes. I didn't do a full cleanup job before processing and you caught one of those words. If you open that page with Chrome and do a "View Source" you'll see that "books" actually appears 597 times in a whole pile of links on the page. Some of them got through. If you only search what a human can see, yeah, it's somewhere in the order of 3 to 5 times. I might have a look at the code and see why the full <a> tag contents are not being removed. Funny enough, that's the only bit of code I did not write. I got lazy and grabbed something right out of the PHP manual pages. Lesson learned.
    • dredmorbius 14y ago
      Using a text-mode browser to render HTML to text, strip off headers, and avoid summarizing links (lynx works great for this) might be a fast route to cleaner results. You'd probably also want to exclude the "See also" and "References" sections. For Canada, not excluding stopwords ("the", "of", "and", etc.) or the country name itself, I get: 1 869 the 2 449 of 3 326 and 4 288 in 5 214 to 6 142 a 7 135 canada 8 106 was 9 87 by 10 72 as 11 71 british 12 70 with 13 69 s 14 62 war 15 61 on 16 53 canadian 17 52 from 18 49 for 19 44 were 20 42 new Which still puts "british" on top.
      • robomartin 14y ago
        Yup, in case you missed it, I posted updated code that really cleans-up the scraped page data very well. With the new code "british" is on top of the Canada list. It was a fun little exercise. There are some issues accessing Wikipedia this way but no big deal for a quick little test.
• robomartin 14y ago
  In the interest of completeness I updated the code based on some on and off-list feedback. The new results are much cleaner as they ignore all anchor, image, navigation and other text. I've updated the gist with the new code. Here's a sample output: United_States [ 118] war [ 108] states [ 96] american [ 78] united [ 42] first [ 37] president [ 35] rights [ 35] world [ 34] government [ 33] economic England [ 57] henry [ 57] king [ 54] english [ 43] government [ 41] britain [ 39] century [ 38] war [ 37] edward [ 34] local [ 34] kingdom Brazil [ 23] military [ 17] portugal [ 15] first [ 14] portuguese [ 13] years [ 13] war [ 13] president [ 12] political [ 12] country [ 10] called Argentina [ 51] president [ 43] military [ 41] argentine [ 40] government [ 37] war [ 36] political [ 30] buenos [ 30] aires [ 24] against [ 23] national Canada [ 71] british [ 62] war [ 53] canadian [ 38] north [ 37] french [ 35] quebec [ 34] government [ 34] first [ 32] american [ 27] france Spain [ 103] spanish [ 62] war [ 46] century [ 39] empire [ 38] king [ 34] under [ 29] peninsula [ 29] first [ 28] roman [ 27] country Portugal [ 99] portuguese [ 28] first [ 25] war [ 22] king [ 21] spanish [ 20] independence [ 19] east [ 19] republic [ 18] between [ 18] main Russia [ 105] soviet [ 105] russian [ 68] war [ 52] union [ 43] citation [ 41] century [ 39] europe [ 39] state [ 38] government [ 37] moscow Germany [ 240] german [ 88] war [ 74] empire [ 68] east [ 63] states [ 57] west [ 55] france [ 55] prussia [ 46] austria [ 42] between France [ 221] french [ 151] war [ 114] louis [ 95] king [ 68] paris [ 56] against [ 52] government [ 52] republic [ 49] would [ 49] empire Japan [ 89] japanese [ 58] war [ 50] emperor [ 40] main [ 37] military [ 37] china [ 35] imperial [ 35] ashikaga [ 35] power [ 34] government China [ 127] dynasty [ 62] chinese [ 29] han [ 26] emperor [ 26] shang [ 25] zhou [ 25] ming [ 22] empire [ 20] military [ 20] main Saudi_Arabia [ 83] saudi [ 48] arabia [ 28] oil [ 28] king [ 21] hejaz [ 21] arab [ 19] mecca [ 18] first [ 16] saud [ 16] state