4 ms·
Author here. I agree that pricing is highly negotiable for any large cloud provider, and there are even (capped) egress fee waivers that you can negotiate as a
by lebovic 4y ago
Author here. I agree that pricing is highly negotiable for any large cloud provider, and there are even (capped) egress fee waivers that you can negotiate as a part of your contract. There's also a place for using AWS; I used it for a smaller DNA sequencing facility, and I use it for my computational biology startup.
That said, I'll repeat something that I commented somewhere else: most of scientific computing (by % of compute) happens in a context that still doesn't make sense in AWS. There's often a physical machine within the organization that's creating data (e.g. a DNA sequencer, particle accelerator, etc), and a well-maintained HPC cluster that analyzes that data.
Spot instances are still pretty expensive for a steady queue (2x of Hetzer monthly costs, for reference), and you still have to pay AWS data transfer egress costs – which are at least 30x more expensive than a colo or on-prem, if you're saturating a 1 Gbps link. Data transfer to optimize for spot instance pricing becomes prohibitive when your job has 100 TB of raw data.
- lowbloodsugar 4y ago> 100 TB of raw data Surely the raw data is input, so ingress costs, which is free? The problem is if you have large amounts of intermediate data, and you want to transfer that somewhere else to continue analysis. Then it's "expensive". So the logical conclusion is to do all the work on the cloud, so you never have egress costs. That causes anxiety, sure. That said, 100TB costs $1000 to egress, assuming that your $50M/yr covers a Verizon 10GB/s line. These opinions are my own and not those of my employer or former employer.