4 ms·
you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale
by make3 12y ago
you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.
- jessaustin 12y agoYeah if you're Google. Most people are not, and wget is plenty. After all it's written in C.
- eru 12y agoOr use curl, for a slightly better engineered wget.
- copsarebastards 12y agoI'm pretty proficient in both, and I think that's a mischaracterization of the two tools. wget is more suited to pulling down large files, groups of files, etc. curl is more suited to API calls where you might need to do something complicated at the protocol level. Each has their use.
- untog 12y agoYou usually don't need that though, that's the point. If you're building an entire service around page crawling, sure. If you're doing a one-off task, don't bother.
- MrBuddyCasino 12y agoThat would be my first gut reaction too, but if it is as simple as downloading webpages, this is actually a really great solution. I suspect he used that when he built Milo, a now defunct startup sold to eBay where they had to update prices and inventory data regularly. A startup should make different choices than Google.