We typically use wget running on consumer machines. This isn't like the spider farm of a typical search engine, with hundreds of machines consuming gigabytes/se

We typically use wget running on consumer machines. This isn't like the spider farm of a typical search engine, with hundreds of machines consuming gigabytes/second of bandwidth. AT hits are specific and targeted.