4 ms·
My company runs a largish crawling infrastructure. We evaluated Amazon extensively but just could not make the price/performance calculation stack up. If you ha
by adamseabrook 13y ago
My company runs a largish crawling infrastructure. We evaluated Amazon extensively but just could not make the price/performance calculation stack up. If you have lots of "always on" gear dedicated works out significantly cheaper. We have boxes with 256GB of RAM, 300mbps network pipes for under $330 per month and we don't have to share any CPU or Disk I/O.
- MagicWishMonkey 13y agoLikewise. My company decided to lease a rack at a colo, unlimited bandwidth (practically speaking) and we can add capacity as necessary. We're planning to (eventually) utilize Amazon to temporarily add capacity when necessary, but it will never be our primary hosting solution.
- adamseabrook 13y agoThat is the one thing we do miss. Elastically scaling dedicated boxes that can often take 12 hours from order to go live means we end up with a few hot spares laying about just in case but overall it is still cheaper.
- beachstartup 13y agowith amazon you're paying for software, not hardware. big difference.
- druiid 13y agoYep. Many businesses should (and a lot don't...) make this cost comparison. Also, 256GB of memory in a server is surprisingly not that impressive (even for a 1U server) these days! Crazy to think there are 1U machines you can cram 1TB of memory into now.
- darkarmani 13y agoWow. That's a lot of bandwidth. I re-wrote the downloaders for siteadvisor a few years ago (hosted) and they didn't need anything close to that. They were just downloading EXEs and other content from pages the crawlers had already seen, but still.
- adamseabrook 13y agoThose servers run our ElasticSearch index so move TB of data around inside the network and out to customers. Our actual crawlers could happily sip from a 5mbps pipe.
- benmccann 13y agoWhat hosting provider are you using?
- adamseabrook 13y agoWe use OVH in Montreal.