3 ms·

Immensely useful which I am forever grateful that it exists. I've made use of it thoroughly during a search engine indexer test I ran back in 2013, which allowe

Immensely useful which I am forever grateful that it exists. I've made use of it thoroughly during a search engine indexer test I ran back in 2013, which allowed me to pretty much download and index (download + indexing in one go) an entire country (Romania) pages in less than 1 hour on a laptop with only 4GB ram and i7 Ivy Bridge processor (Zenbook UX21 a particularly). All the components I've wrote are optimized to the core (short of going assembly). It allowed me to scale my operations up to 10000 processes continuously for the duration of the test. All this was done on a 4G connection with roughly 40Mbps bandwidth at the time on a virtualized (VirtualBox) CentOS LAMP installation. It blew all my expectations, without parallel it would taken more than a month. Learned a lot from the experience developing it and maybe will make a post sharing what I've learned. In regards to being complicated, well it isn't really when you consider its use scenarios, for these sorts of tasks, it's a indispensable tool. Think of it this way: It is far less difficult than the difficulty of the tasks where you need to use it.

mkl 7y ago
How did you get the list of pages to download? Or was this spidering from some starting set? One hour on that connection means a maximum of 18GB of data, which seems surprisingly small for a whole country's internet (not that I have any point of comparison), small enough that you could easily do interesting analysis of the whole lot.
- AlexDragusin 7y ago
  I've started off a seed list (compiled from both Alexa and Romanian Wikipedia) and from there on spidering through links found on each of the pages. I needed only the pages written in Romanian language so right after download, a language checking would take place along with the ranking and finally indexing. Please note that only the HTML was downloaded, no images/css/js etc so the size would be considerably less than if you would normally browse the pages. The HTML/CSS/JS would be sripped off, leaving the content for analysis, ranking and indexing. True and after that I realized how small the Romanian internet really is in the big picture, surely allowed me to put things into perspective. It's small enough that it would be doable to have a near realtime update of the index since less than 10% of sites update daily and less than 1% on hourly basis such as news sites. It correlates with traffic details of top trafficked webs in Romania where one of the top sites gets around 200k uniques a day, a small amount when you consider the big picture. It is likely that today the amount of data would be larger but not by much.

mkl 7y ago

How did you get the list of pages to download? Or was this spidering from some starting set? One hour on that connection means a maximum of 18GB of data, which seems surprisingly small for a whole country's internet (not that I have any point of comparison), small enough that you could easily do interesting analysis of the whole lot.

AlexDragusin 7y ago
I've started off a seed list (compiled from both Alexa and Romanian Wikipedia) and from there on spidering through links found on each of the pages. I needed only the pages written in Romanian language so right after download, a language checking would take place along with the ranking and finally indexing. Please note that only the HTML was downloaded, no images/css/js etc so the size would be considerably less than if you would normally browse the pages. The HTML/CSS/JS would be sripped off, leaving the content for analysis, ranking and indexing. True and after that I realized how small the Romanian internet really is in the big picture, surely allowed me to put things into perspective. It's small enough that it would be doable to have a near realtime update of the index since less than 10% of sites update daily and less than 1% on hourly basis such as news sites. It correlates with traffic details of top trafficked webs in Romania where one of the top sites gets around 200k uniques a day, a small amount when you consider the big picture. It is likely that today the amount of data would be larger but not by much.