10 ms·
How AWS S3 serves 1 petabyte per second on top of slow HDDs
- EwanToo 1y agoI think a more interesting article on S3 is "Building and operating a pretty big storage system called S3" https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html https://www.allthingsdistributed.com/2023/07/building-and-op...
- giancarlostoro 1y agoReally nice read, thank you for that.
- enether 1y agoAuthor of the 2minutestreaming blog here. Good point! I'll add this as a reference at the end. I loved that piece. My goal was to be more concise and focus on the HDD aspect
- pjdesno 1y agoPlease fix the seek time numbers - they’re wildly in accurate; full platter seek is more like 20-25ms. I tried chasing down where the 8ms number came from a while back, and I think it applies to old sub-100GB 10K RPM high-speed drives from 25 or so years ago, which were purposely low density so they could swing the head faster and less precisely. Check out the Olmez et al paper from MSST 2024 - I linked it above, but here it is again: https://www.msstconference.org/MSST-history/2024/Papers/msst24-1.1.pdf https://www.msstconference.org/MSST-history/2024/Papers/msst...
- enether 1y ago!!! Thanks for calling that out. And sorry for falling prey to the numbers I saw (from the AWS talk) And for the 1/2 vs 1/3rd - I'm just dumb. Thanks again. Super cool paper too
- littlesnugblood 1y ago[flagged]
- gostsamo 1y agoCan you share some anecdotes?
- huntaub 1y agoWoah buddy, I worked with Andy for years and this is not my experience. Moving a large product like S3 around is really, really difficult, and I've always thought highly of Andy's ability to: (a) predict where he thought the product should go, (b) come up with novel ways of getting there, and (c) trimming down the product to get something in the hands of customers. Also, did you create this account for the express purpose of bashing Andy? That's not cool.
- Powdering7082 1y ago[flagged]
- wavesbelow 1y agoWell, the claim in question is not exactly a rebuttal of the original commenter's point despite the negative tone so I'd cut him some slack
- dang 1y agoDiscussed at the time: Building and operating a pretty big storage system called S3 - https://news.ycombinator.com/item?id=36894932 https://news.ycombinator.com/item?id=36894932 - July 2023 (160 comments)
- crabique 1y agoIs there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
- bayindirh 1y agoLustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.
- crabique 1y agoThanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast metadata store for Garage, the "special" device/small records store for the ZFS, the ZIL/SLOG device and so on. Currently it's a bit of a Frankenstein's monster: using XFS+OpenCAS as the backing storage for an old version of MinIO (containerized to run as 5 instances), I'm looking to replace it with a simpler design and hopefully get a better performance.
- foobarian 1y agoDo you know if some of these systems have components to periodically checksum the data at rest?
- bayindirh 1y agoZFS/OpenZFS can do scrub and do block-level recovery. I'm not sure about Lustre, but since Petabyte sized storage is its natural habitat, there should be at least one way to handle that.
- nerdjon 1y agoSo is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.
- MDGeist 1y agoI always assumed the really slow tiers were tape.
- hobs 1y agoNot even the higher tiers of Glacier were tape afaict (at least when it was first created), just the observation that hard drives are much bigger than you can reasonably access in useful time.
- temp0826 1y agoIn the early days when there were articles speculating on what Glacier was backed by, it was actually on crusty old S3 gear (and at the very beginning, it was just on S3 itself as a wrapper and a hand wavy price discount, eating the costs to get people to buy in to the idea!). Later on (2018 or so) they began moving to a home grown tape-based solution (at least for some tiers).
- luhn 1y agoDo you have any sources for that? I'm really curious about Glacier's infrastructure and AWS has been notoriously tight-lipped about it. I haven't found anything better than informed speculation.
- iamtedd 1y agoMy speculation: writes are to /dev/null, and the fact that reads are expensive and that you need to inventory your data before reading means Amazon is recreating your data from network transfer logs.
- wg0 1y agoDoes anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.
- jyscao 1y ago> conway’s law and how it shapes S3’s architecture (consisting of 300+ microservices)
- Twirrim 1y agoAmazon biases towards Systems Oriented Architecture approach that is in the middle ground between monolith and microservices. Biasing away from lots of small services in favour of larger ones that handle more of the work so that as much as possible you avoid the costs and latency of preparing, transmitting, receiving and processing requests. I know S3 has changed since I was there nearly a decade ago, so this is outdated. Off the top of my head it used to be about a dozen main services at that time. A request to put an object would only touch a couple of services en route to disk, and similar on retrieval. There were a few services that handled fixity and data durability operations, the software on the storage servers themselves, and then stuff that maintained the mapping between object and storage.
- hnexamazon 1y agoI was an SDE on the S3 Index team 10 years ago, but I doubt much of the core stack has changed. S3 is comprised primarily of layers of Java-based web services. The hot path (object get / put / list) are all served by synchronous API servers - no queues or workers. It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career. For a get call, you first hit a fleet of front-end HTTP API servers behind a set of load balancers. Partitioning is based on the key name prefixes, although I hear they’ve done work to decouple that recently. Your request is then sent to the Indexing fleet to find the mapping of your key name to an internal storage id. This is returned to the front end layer, which then calls the storage layer with the id to get the actual bits. It is a very straightforward multi-layer distributed system design for serving synchronous API responses at massive scale. The only novel bit is all the backend communication uses a home-grown stripped-down HTTP variant, called STUMPY if I recall. It was a dumb idea to not just use HTTP but the service is ancient and originally built back when principal engineers were allowed to YOLO their own frameworks and protocols so now they are stuck with it. They might have done the massive lift to replace STUMPY with HTTP since my time.
- dgllghr 1y agoI enjoyed this article but I think the answer to the headline is obvious: parallelism
- ffsm8 1y agoThat's like saying "how to get to the moon is obvious: traveling"
- crazygringo 1y agoI dunno, the article's tl;dr is just parallelism. Data gets split into redundant copies, and is rebalanced in response to hot spots. Everything in this article is the obvious answer you'd expect.
- deleted 1y ago[deleted]
- toolslive 1y agoIt's not really "redundant copies". It's erasure coding (ie, your data is the solution of an overdetermined system of equations).
- christina97 1y agoThat’s just fractional redundant copies.
- Dylan16807 1y agoAnd "fractional redundant copies" is way less obvious.
- crazygringo 1y agoThe fractional part isn't helping them serve data any faster. To the contrary, it actually reduces the speed from parallelism. E.g. a 5:9 scheme only achieves 1.8x throughput, whereas straight-up triple redundancy would achieve 3x. It just saves AWS money is all, by achieving greater redundancy with less disk usage.
- cramcgrab 1y ago[dead]
- ttfvjktesd 1y ago> tens of millions of disks If we assume enterprise HDDs in the double digit TB range then one can estimate that the total S3 storage volume of AWS is in the triple digit Exabyte range. That's propably the biggest storage system on planet earth.
- yanslookup 1y agoProduction scale enterprise HDDs are in the 30TB range, 50TB on the horizon...
- threeducks 1y agoA certain data center in Utah might top that, assuming that they have upgraded their hardware since 2013. https://www.forbes.com/sites/kashmirhill/2013/07/24/blueprints-of-nsa-data-center-in-utah-suggest-its-storage-capacity-is-less-impressive-than-thought/ https://www.forbes.com/sites/kashmirhill/2013/07/24/blueprin...
- ttfvjktesd 1y agoThis piece is interesting background, but worth noting that the actual numbers are highly speculative. The NSA has never disclosed hard data on capacity, and most of what's out there is inference from blueprints, water/power usage, or second-hand claims. No verifiable figures exist.
- sciencesama 1y agocan we replicate something similar to this for homelab ?
- ai-christianson 1y agogarage
- Datagenerator 1y agoAdd ZeroFS on top and get very low latency for frequently used data while bulk storage is remote S3.
- gregates 1y agoA few factual inaccuracies in here that don't affect the general thrust. For example, the claim that S3 uses a 5:9 sharding scheme. In fact they use many different sharding schemes, and iirc 5:9 isn't one of them. The main reason being that a ratio of 1.8 physical bytes to 1 logical byte is awful for HDD costs. You can get that down significantly, and you get wider parallelism and better availability guarantees to boot (consider: if a whole AZ goes down, how many shards can you lose before an object is unavailable for GET?).
- UltraSane 1y agoVAST data uses 146+4 https://www.vastdata.com/whitepaper/#similarity-reduction-in-practice https://www.vastdata.com/whitepaper/#similarity-reduction-in...
- bombela 1y agopage loads then quicky move up some video loads, and content is gone
- enether 1y agoSee timestamp 42:20 at https://youtu.be/NXehLy7IiPM?si=QQEOMCt7kOBTMaGK https://youtu.be/NXehLy7IiPM?si=QQEOMCt7kOBTMaGK The way it’s worded makes me understand that’s what scheme they’re using. Curious to hear what you know
- zeroimpl 1y agoNaively it seems difficult to decrease the ratio of 1.8x while simultaneously increasing availability. The less duplication, the greater risk of data loss if an AZ goes down? (I thought AWS promises you have a complete independent copy in all 3 AZs though?) To me though the idea that to read like a single 16MB chunk you need to actually read like 4MB of data from 5 different hard drives and that this is faster is baffling.
- Const-me 1y ago> full-platter seek time: ~8ms; half-platter seek time (avg): ~4ms Average distance between two points (first is current location, second is target location) when both are uniformly distributed in [ 0 .. +1 ] interval is not 0.5, it’s 1/3. If the full platter seek time is 8ms, average seek time should be 2.666ms.
- nick49488171 1y agoThere's acceleration of the read head to move it between the tracks. So it may well be 4ms because shorter distances are penalized by a lower peak speed of the read head as well as constant factors (settling at the end of motion)
- pjdesno 1y agoIT’S NOT 4 MS. That’s a bogus number from an ancient slide deck for a class in 2001 or so, that’s misled generations of folks googling for the answer. Note also that the outer tracks are longer (google ZCAV) and hold more data, so seeks across uniformly distributed block numbers do not generate uniformly distributed track numbers.
- pjdesno 1y agoFull seek on a modern drive is a lot closer to 25ms than 8ms. It's pretty easy to test yourself if you have a hard drive in your machine and root access - fire up fio with --readonly and feed it a handmade trace that alternates reading blocks at the beginning and end of disk. (--readonly does a hard disable of any code that could write to the drive) Here's a good paper that explains why the 1/3 number isn't quite right on any drives manufactured in the last quarter century or so - https://www.msstconference.org/MSST-history/2024/Papers/msst24-1.1.pdf https://www.msstconference.org/MSST-history/2024/Papers/msst... I'd be happy to answer any other questions about disk drive mechanics and performance.
- kvemkon 1y ago> Full seek on a modern drive is a lot closer to 25ms than 8ms Is "full seek" a synonymous for worst case time to reach a position occurring less than 1% of working time? From the article: max seek time is 15.2 ms + additionally 0 to 8.3 ms of rotational latency. Reordering of sector accesses by NCQ should reduce worst case scenario occurrences.
- pjdesno 1y agoNote that you can kind of infer that S3 is still using hard drives for their basic service by looking at pricing and calculating the IOPS rate that doubles the cost per GB per month. S3 GET and PUT requests are sufficiently expensive that AWS can afford to let disk space sit idle to satisfy high-performance tenants, but not a lot more expensive than that.
- rushi_agrawal 1y agoIt is interesting that even after falling prices of HDDs, S3 costs have remained the same for at least 8 years. There's just not enough competition to push them to reduce costs. But imagine money it brings in in AWS because of this.
- layoric 1y agoSame with every other aspect of their offerings. Look at EC2 even with instances like m7a.medium, 1 vCPU (not core) and 4GB memory for ~$50 USD/month on demand or ~$35/month reserve 1 year. It isn't even close to be competitive outside other big cloud providers. EDIT: clarity on monthly pricing.
- master_crab 1y agoThere is inflation, so it has effectively dropped in price. But your point is taken: inflation’s effect on prices is most assuredly slower than the progress of technology’s effect.
- Pxtl 1y agoHow much have hdd prices really fallen? AFAIK the incredible improvements in price per byte in HDD had slowed so much that they'll be eclipsed by SSDs in a few years.
- rbranson 1y agoMagnetic hard drives are 100X cheaper per GB than when S3 launched, and are about 3X cheaper than in 2016 when the price last dropped. Magnetic prices have actually ticked up recently due to supply chain issues, but HAMR is expected to cause a significant drop (50-75%/GB) in magnetic storage prices as it rolls out in next few years. SSDs are ~$120/T and magnetic drives are ~$18/T. This hasn't changed much in the last 2 years.
- pjdesno 1y agoFlash went from within 2x the price of DRAM in 2012 or so to maybe 40-50x cheaper today, driven somewhat by shrinking feature sizes, but mostly by the shift from SLC (1 bit/cell) to TLC (3 bits) and QLC (4 bits) and from planar to 300+ layer 3D flash. Flash is near the end of the “S-curve” of those technologies being rolled out. During that time HDD technology was pretty stagnant, with a mere 2x increase due to higher platter count with the use of helium. New HDD technologies (HAMR) are just starting their rollout, promising major improvements in $/GB over the next few years as they roll out. You can’t just look at a price curve on a graph and predict where it’s going to go. The actual technologies responsible for that curve matter.
- sylware 1y agoAnd still serving/protecting those 'internet scanners' and hacking script kiddies? Because as a shelf-hosted, I had to block a ton of AWS ipv4s because of those.
- finalhacker 1y agoconsider the read/write ratio. most of time s3 service are serving memory buffer. and memory is quite a lot faster than disk.
- limingzju 1y agoDose aws s3 pack small files into a large file? It seems hard to do it with erasue code and shardstore.
- neiesc 1y agohttps://github.com/rustfs/rustfs https://github.com/rustfs/rustfs performs?
- nightshift1 1y agoHere is another very interesting post on the subject from 2023: https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html https://www.allthingsdistributed.com/2023/07/building-and-op...