4 ms·
Yeah, my solution is a gross oversimplification -- my math is just back-of-the-napkin stuff, you can actually do much better on component pricing (I certainly d
by skuhn 12y ago
Yeah, my solution is a gross oversimplification -- my math is just back-of-the-napkin stuff, you can actually do much better on component pricing (I certainly don't pay $5000 for a 1G top-of-rack switch) and there are other variables to consider.
To quickly address some of your points:
Geographic redundancy: not included. From a hardware perspective it's a pretty easy upgrade (build two datacenters in CA / VA, buy two racks, put one in each). From a service perspective, it's a bit more challenging. If storage is your business, these are the core components you should own though.
The US Standard region in S3 does sort of provide geographic redundancy, but it's a question of whether it's good enough for your use case. S3 is a complete black box, and you have no way of knowing with certainty that every data blob is actually in both geographic regions. Besides that, it's eventually consistent, so if you need absolute assurances that data is not lost after a write succeeds, you have to implement logic on top of S3 -- and that logic must run in EC2 instances, or you'll pay a fortune in bandwidth to execute it.
No other region in S3 provides geographic redundancy (or eventual consistency, the two are related), so if you wanted to deploy storage in Europe or APAC, you have to handle the geographic distribution yourself -- and it will cost double to store that data.
The small cost difference: 1 rack over 1 year has a pretty small differential between the two. But you can optimize the datacenter numbers, and the S3 numbers are what they are unless you get massive. For example, over 3 years the datacenter cost goes to around $275,000 whereas S3 is $745,000. That's a serious difference now. Ownership is a powerful advantage.
OpenStack Swift: I don't think it will ever be a good solution, based on the project's history. It's unfortunate, but hopefully something else will come along that does blob storage in a sane and scalable way. This is not rocket science, but there are some devils in those details. If a mature open source solution came along with immutable objects, erasure coding, background scrubbing and sane ops tools, it would blow Swift out of the water.
Finally, I think it's a (common) mistake to think that you can get by without ops personnel if you deploy on AWS. The actual site / dc ops component is pretty minor if you build right. Either way, someone still has to run the service, do capacity / project / budget plans, and other things. And with AWS you need people who are capable of winning arguments with their support -- where you will burn substantial time proving where fault lies.
S3 does a lot for you, but it's still a service that you build yours on top of and it's far from a perfect solution that never breaks. Buckets have to be primed before heavy usage. Your file naming scheme will impact your performance at scale. And so on. You need people to look at this, whether you call them ops engineers or sysadmins or developers.
- toomuchtodo 12y agoIt seems we've traveled the same path at different times :)
- viraptor 12y agoI'm curious - since you went through trying out swift, when was that? Pre, or post diablo? That was the time of a pretty big change. Also, have you tried ceph at the same scale?
- skuhn 12y agoPost-Diablo, and it never got to a point where I deployed it at scale and found issues: it flunked out before that. Haven't tried ceph, although I look at it every few years. It sounds promising but it also sounds really complicated to setup and administer.
- vidarh 12y agoIt's funny how people thing AWS does away with ops. But doing ops work for people on AWS is highly lucrative - people have a really hard time getting it right.
- acdha 12y ago> Geographic redundancy: not included. … If storage is your business, these are the core components you should own though. Definitely – part of what I was thinking about is that it's also a good point to actually ask what you actually need. My experience has been in fields (science, digital preservation) where it turns out that people need large amounts of storage and strong guarantees against data corruption but online access is less critical since almost everyone works with more manageable derivatives than the original data. Unlike, say, a database there also aren't concerns with multiple writers since each batch of data originates in one place and should rarely if ever change, even though multiple locations might be generating other batches at the same time which everyone would want to read. That offers a ton of optimizations which are much harder to do if you're trying to maintain the same public contract as S3. > OpenStack Swift: I don't think it will ever be a good solution, based on the project's history Sadly, I've been coming to the same conclusion. I've been meaning to look at GlusterFS again as they've been adding things like erasure coding & the API portion of Swift but last time I checked they still haven't added strong integrity checks or scrubbing. > Finally, I think it's a (common) mistake to think that you can get by without ops personnel if you deploy on AWS. The actual site / dc ops component is pretty minor if you build right. Either way, someone still has to run the service, do capacity / project / budget plans, and other things. Agreed. The main thing I was referring to is that if you have to run a 24x7 service you have certain minimum staffing requirements. That's easily justified if you have enough demand but a small operation even inside a large company might find it easier to let AWS handle the basic ops so their staff can handle the higher-level ops work during normal business hours. (This is again a good opportunity to review the actual business needs to ask whether you really need immediate responses at 3am or can live with a delay while someone gets paged)