I've started off a seed list (compiled from both Alexa and Romanian Wikipedia) and from there on spidering through links found on each of the pages. I needed on

I've started off a seed list (compiled from both Alexa and Romanian Wikipedia) and from there on spidering through links found on each of the pages. I needed only the pages written in Romanian language so right after download, a language checking would take place along with the ranking and finally indexing. Please note that only the HTML was downloaded, no images/css/js etc so the size would be considerably less than if you would normally browse the pages. The HTML/CSS/JS would be sripped off, leaving the content for analysis, ranking and indexing. True and after that I realized how small the Romanian internet really is in the big picture, surely allowed me to put things into perspective. It's small enough that it would be doable to have a near realtime update of the index since less than 10% of sites update daily and less than 1% on hourly basis such as news sites. It correlates with traffic details of top trafficked webs in Romania where one of the top sites gets around 200k uniques a day, a small amount when you consider the big picture. It is likely that today the amount of data would be larger but not by much.