33 ms·
"Fast by current standards" is probably irrelevant if you're running a scraper: You'll be bandwidth-limited, and if you solve that you'll likely be disk I/O-lim
by mechanical_fish 14y ago
"Fast by current standards" is probably irrelevant if you're running a scraper: You'll be bandwidth-limited, and if you solve that you'll likely be disk I/O-limited or RAM-limited.
Getting back to the overall question: Yes, lots of us have done this before; fifteen years ago every geeky college student had your proposed setup.
The other responses are correct: From a business standpoint, you're crazy. Not only will you find that bandwidth and power cost you more at home than they do at a datacenter, but your inconsistent, unreliable homebrew infrastructure will consume maintenance time and replacement parts at a rate that slowly but surely eliminates all "savings". It's hard to appreciate that, because you've got little experience at keeping network infrastructure running long-term, and even less experience at billing yourself for your own work at market rates, so failure events feel rare and inexpensive to you. But I assure you that NOCs don't buy all that expensive capital equipment for no reason.
Having said that, I get the impression that you don't really care about uptime or reliability; you're running spare-time hobby projects. If so, then the question is whether assembling your own homebrew "distributed datacenter" out of PCs in various closets is educational. My feeling is that it isn't, not anymore. The ability to run production-quality systems on cheap consumer hardware is not a valuable skill and is no longer worth practicing - not when production hardware can be rented by the hour for pennies. You can practice installing Linux and tuning the LAMP stack on Linodes or AWS boxes and get all the same sysadmin skills, but in a realistic production environment. You can practice debugging disk throughput problems, or networking problems, or DNS headaches on real cloud infrastructure for a few bucks a day. Who would play with a toy bulldozer if a real bulldozer was available to rent for a dollar per hour?
(The counterargument is that homebrewed "infrastructure" is a great toy because it breaks all the time and you're forced to practice lots and lots of debugging. After all, I learned a lot of stuff this way - I had to; the "cloud" didn't exist. But my gut feeling is that this kind of "practice" feels more valuable than it actually is.)
But of course, hobbies are hobbies, and far be it from me to argue you out of your own happiness. If you just can't resist the fun of turning your home computers into a compute cluster, go for it! It's a better way to spend your time than MMORPGs.