4 ms·

I completely agree. I sometimes take flack for "hating" AWS (and I do hate some aspects of it), but I think it's a great platform for incubating your service or

by skuhn 12y ago

I completely agree. I sometimes take flack for "hating" AWS (and I do hate some aspects of it), but I think it's a great platform for incubating your service or proving a concept. Every company I've been at in the last five years has used AWS to an extent, despite operating their own datacenters. It's great for random analytics jobs or other tasks that aren't steady work tied to your core business. It quickly starts to not make sense for services that have usage-driven growth patterns. AWS's billing model just doesn't work for this -- even though higher usage is discounted, it isn't enough to overcome the skew of the model. Companies that have deployed on AWS often realize this too late: their business starts to take off, and they get crushed under giant AWS bills that would be disproportionate even at 50% off list. It's much harder to move off once you've hit scale.

acdha 12y ago
A lot of this also comes down to ops & the features you really need – e.g. your example above didn't include the costs of the geographic redundancy or strong data integrity features which are standard with S3. A lot of places simply don't have geographically redundant datacenters and either lack the support team to run something like a replicated cluster filesystem in heavy production usage or, more commonly, don't have enough usage to really push it into clear win. In the case above, the cost differential at 700TB is roughly the cost of a single sysadmin and so the running operation would probably be a wash versus S3 unless it had heavy data churn to maximize the S3 expenses or a good way to amortize the staffing needed for 24-hour-a-day support across other projects. Many places have budget processes which favor spending more on known upfront costs than getting permission to add full-time employees in the hopes of being able to beat those costs down the road. I'm hoping that something like OpenStack Swift matures to the point where some of the complexity at the software + ops level starts making the DIY option routine. You'll never get rid of the minimum sysadmin commitment but that becomes a lot easier if it can be 5% of the night-shift because the software doesn't require a lot of care and feeding.
- skuhn 12y ago
  Yeah, my solution is a gross oversimplification -- my math is just back-of-the-napkin stuff, you can actually do much better on component pricing (I certainly don't pay $5000 for a 1G top-of-rack switch) and there are other variables to consider. To quickly address some of your points: Geographic redundancy: not included. From a hardware perspective it's a pretty easy upgrade (build two datacenters in CA / VA, buy two racks, put one in each). From a service perspective, it's a bit more challenging. If storage is your business, these are the core components you should own though. The US Standard region in S3 does sort of provide geographic redundancy, but it's a question of whether it's good enough for your use case. S3 is a complete black box, and you have no way of knowing with certainty that every data blob is actually in both geographic regions. Besides that, it's eventually consistent, so if you need absolute assurances that data is not lost after a write succeeds, you have to implement logic on top of S3 -- and that logic must run in EC2 instances, or you'll pay a fortune in bandwidth to execute it. No other region in S3 provides geographic redundancy (or eventual consistency, the two are related), so if you wanted to deploy storage in Europe or APAC, you have to handle the geographic distribution yourself -- and it will cost double to store that data. The small cost difference: 1 rack over 1 year has a pretty small differential between the two. But you can optimize the datacenter numbers, and the S3 numbers are what they are unless you get massive. For example, over 3 years the datacenter cost goes to around $275,000 whereas S3 is $745,000. That's a serious difference now. Ownership is a powerful advantage. OpenStack Swift: I don't think it will ever be a good solution, based on the project's history. It's unfortunate, but hopefully something else will come along that does blob storage in a sane and scalable way. This is not rocket science, but there are some devils in those details. If a mature open source solution came along with immutable objects, erasure coding, background scrubbing and sane ops tools, it would blow Swift out of the water. Finally, I think it's a (common) mistake to think that you can get by without ops personnel if you deploy on AWS. The actual site / dc ops component is pretty minor if you build right. Either way, someone still has to run the service, do capacity / project / budget plans, and other things. And with AWS you need people who are capable of winning arguments with their support -- where you will burn substantial time proving where fault lies. S3 does a lot for you, but it's still a service that you build yours on top of and it's far from a perfect solution that never breaks. Buckets have to be primed before heavy usage. Your file naming scheme will impact your performance at scale. And so on. You need people to look at this, whether you call them ops engineers or sysadmins or developers.

by skuhn 12y ago

acdha 12y ago

A lot of this also comes down to ops & the features you really need – e.g. your example above didn't include the costs of the geographic redundancy or strong data integrity features which are standard with S3. A lot of places simply don't have geographically redundant datacenters and either lack the support team to run something like a replicated cluster filesystem in heavy production usage or, more commonly, don't have enough usage to really push it into clear win. In the case above, the cost differential at 700TB is roughly the cost of a single sysadmin and so the running operation would probably be a wash versus S3 unless it had heavy data churn to maximize the S3 expenses or a good way to amortize the staffing needed for 24-hour-a-day support across other projects. Many places have budget processes which favor spending more on known upfront costs than getting permission to add full-time employees in the hopes of being able to beat those costs down the road. I'm hoping that something like OpenStack Swift matures to the point where some of the complexity at the software + ops level starts making the DIY option routine. You'll never get rid of the minimum sysadmin commitment but that becomes a lot easier if it can be 5% of the night-shift because the software doesn't require a lot of care and feeding.

skuhn 12y ago
Yeah, my solution is a gross oversimplification -- my math is just back-of-the-napkin stuff, you can actually do much better on component pricing (I certainly don't pay $5000 for a 1G top-of-rack switch) and there are other variables to consider. To quickly address some of your points: Geographic redundancy: not included. From a hardware perspective it's a pretty easy upgrade (build two datacenters in CA / VA, buy two racks, put one in each). From a service perspective, it's a bit more challenging. If storage is your business, these are the core components you should own though. The US Standard region in S3 does sort of provide geographic redundancy, but it's a question of whether it's good enough for your use case. S3 is a complete black box, and you have no way of knowing with certainty that every data blob is actually in both geographic regions. Besides that, it's eventually consistent, so if you need absolute assurances that data is not lost after a write succeeds, you have to implement logic on top of S3 -- and that logic must run in EC2 instances, or you'll pay a fortune in bandwidth to execute it. No other region in S3 provides geographic redundancy (or eventual consistency, the two are related), so if you wanted to deploy storage in Europe or APAC, you have to handle the geographic distribution yourself -- and it will cost double to store that data. The small cost difference: 1 rack over 1 year has a pretty small differential between the two. But you can optimize the datacenter numbers, and the S3 numbers are what they are unless you get massive. For example, over 3 years the datacenter cost goes to around $275,000 whereas S3 is $745,000. That's a serious difference now. Ownership is a powerful advantage. OpenStack Swift: I don't think it will ever be a good solution, based on the project's history. It's unfortunate, but hopefully something else will come along that does blob storage in a sane and scalable way. This is not rocket science, but there are some devils in those details. If a mature open source solution came along with immutable objects, erasure coding, background scrubbing and sane ops tools, it would blow Swift out of the water. Finally, I think it's a (common) mistake to think that you can get by without ops personnel if you deploy on AWS. The actual site / dc ops component is pretty minor if you build right. Either way, someone still has to run the service, do capacity / project / budget plans, and other things. And with AWS you need people who are capable of winning arguments with their support -- where you will burn substantial time proving where fault lies. S3 does a lot for you, but it's still a service that you build yours on top of and it's far from a perfect solution that never breaks. Buckets have to be primed before heavy usage. Your file naming scheme will impact your performance at scale. And so on. You need people to look at this, whether you call them ops engineers or sysadmins or developers.