3 ms·The Tale of Creating a Distributed Web Crawler6 points by kiyanwang 9y agoDamonHD 9y agoYes, how a software monster is really born! Those of us who solve non-trivial tasks on the Web or in other large distributed systems will feel this in our bones.