I guess this fits in here: Once upon a time I wrote my thesis on building a web crawler. The (tiny) blog post with an embedded preview: http://blog.marc-seege

I guess this fits in here: Once upon a time I wrote my thesis on building a web crawler. The (tiny) blog post with an embedded preview: http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blocks-of-a-scalable-webcrawler/ http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blo... The PDF itself: http://blog.marc-seeger.de/assets/papers/thesis_seeger-building_blocks_of_a_scalable_webcrawler.pdf http://blog.marc-seeger.de/assets/papers/thesis_seeger-build... It's mostly a "this is what I learned and the things I had to take into consideration" with a few "this is how you identify a CMS" bits sprinkled into it. These days I would probably change a thing or two, but people told me it's still an entertaining read. (Not a native speaker though, so the English might have some stylistic kinks)