8 ms·
This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' techn
by pclmulqdq 3y ago
This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions.
SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, which was the backing technology when a lot of these network attached storage systems were invented, because they are fundamentally slow compared to networks, but it is a problem for SSD.
- brucethemoose2 3y agoYeah this was my impression. I am but an end user, but I noticed that disk IO for a certain app was glacial compared to a local test deployment, and I chalked it up to networking/VM overhead
- vlovich123 3y agoWhy do they fundamentally need to be network attached storage instead of local to the VM?
- Filligree 3y agoThey don't. Some cloud providers (i.e. Hetzner) let you rent VMs with locally attached NVMe, which is dramatically faster than network-attached even factoring in the VM tax. Of course then you have a single point of failure, in the PCIe fabric of the machine you're running on if not the NVMe itself. But if you have good backups, which you should, then the juice really isn't worth the squeeze for NAS storage.
- ssl-3 3y agoA network adds more points of failure. It does not reduce them.
- supriyo-biswas 3y agoA network attached, replicated storage hedges against data loss but increases latency; however most customers usually prefer higher latency to data loss. As an example, see the highly upvoted fly.io thread[1] with customers complaining about the same thing. [1] https://news.ycombinator.com/item?id=36808296 https://news.ycombinator.com/item?id=36808296
- ssl-3 3y agoLocally-attached, replicated storage also hedges against data loss.
- supriyo-biswas 3y agoRAID rebuild times make it an unviable option and customers typically expect problematic VMs to be live-migrated to other hosts with the disks still having their intended data. The self hosted version of this is GlusterFS and Ceph, which have the same dynamics as EBS and its equivalents in other cloud providers.
- mike_hearn 3y agoWith NVMe SSDs? What makes RAID unviable in that environment?
- dijit 3y agoThis depends, like all things. When you say RAID, what level? Software-raid or hardware raid? What controller? Let's take best-case: RAID10, small enough (but many) NVMe drives and an LVM/Software RAID like ZFS, which is data aware so only rebuilds actual data: rebuilds will degrade performance enough potentially that your application can become unavailable if your IOPS are 70%+ of maximum. That's an ideal scenario, if you use hardware raid which is not data-aware then your rebuild times depend entirely on the size of the drive being rebuilt and it can punish IOPs even more during the rebuild. But it will affect your CPU less. There's no panacea. Most people opt for higher latency distributed storage where the RAID is spread across an enormous amount of drives, which makes rebuilds much less painful.
- Retric 3y agoRedundancy, local storage is a single point of failure. You can use local SSD’s as slow RAM, but anything on it can go away at any moment.
- cduzz 3y agoI've seen SANs get nuked by operator error or by environmental issues (overheated DC == SAN shuts itself down). Distributed clusters of things can work just fine on ephemeral local storage (aka local storage). A kafka cluster or an opensearch cluster will be fine using instance local storage, for instance. As with everything else.... "it depends"
- pclmulqdq 3y agoReliability. SSDs break and screw up a lot more frequently and more quickly than CPUs. Amazon has published a lot on the architecture of EBS, and they go through a good analysis of this. If you have a broken disk and you locally attach, you have a broken machine. RAID helps you locally, but fundamentally relies on locality and low latency (and maybe custom hardware) to minimize the time window where you get true data corruption on a bad disk. That is insufficient for cloud storage.
- vlovich123 3y agoSure, but there's plenty of software that's written to use distributed unreliable storage similar to how cloud providers write their own software (e.g. Kafka). I can understand if many applications are just need something like EBS that's durable but looks like a normal disk, but not so sure it's a fundamentally required abstraction.
- SteveNuts 3y agoBecause even if you can squeeze 100TB or more of SSD/NVMe in a server, and there are 10 tenants using the machine, you're limited to 10TB as a hard ceiling. What happens when one tenant needs 200TB attached to a server? Cloud providers are starting to offer local SSD/NVMe, but you're renting the entire machine, and you're still limited to exactly what's installed in that server.
- jalk 3y agoHow is that different from how cores, mem and network bandwidth is allotted to tenants?
- pixl97 3y agoBecause a fair number of customers spin up another image when cores/mem/bandwidth run low. Dedicated storage breaks that paradigm. Also, adding, if I am on an 8 core machine and need 16, network storage can be detached from host A and connected to host B. In dedicated storage it must be fully copied over first.
- baq 3y agoIt isn't. You could ask for network-attached CPUs or RAM. You'd be the only one, though, so in practice only network-attached storage makes sense business-wise. It also makes sense if you need to provision larger-than-usual amounts like tens of TB - these are usually hard to come by in a single server, but quite mundane for storage appliances.
- vel0city 3y agoGiven AWS and GCP offer multiple sizes for the same processor version with local SSDs, I don't think you have to rent the entire machine. Search for i3en API names and you'll see: i3en.large, 2x CPU, 1250GB SSD i3en.xlarge, 4x CPU, 2500GB SSD i3en.2xlarge, 8x CPU, 2x2500GB SSD i3en.3xlarge, 12x CPU, 7500GB SSD i3en.6xlarge, 24x CPU, 2x7500GB SSD i3en.12xlarge, 48x CPU, 4x7500GB SSD i3en.24xlarge, 96x CPU, 8x7500GB SSD i3en.metal, 96x CPU, 8x7500GB SSD So they've got servers with 96 CPUs and 8x7500GB SSDs. You can get a slice of one, or you can get the whole one. All of these are the ratio of 625GB of local SSD per CPU core. https://instances.vantage.sh/ https://instances.vantage.sh/ On GCP you can get a 2-core N2 instance type and attach multiple local SSDs. I doubt they have many physical 2-core Xeons in their datacenters.
- drewda 3y agoThe major clouds do offer VMs with fast local storage, such as SSDs connected by NVMe connections directly to the VM host machine: - https://cloud.google.com/compute/docs/disks/local-ssd https://cloud.google.com/compute/docs/disks/local-ssd - https://learn.microsoft.com/en-us/azure/virtual-machines/enable-nvme-interface https://learn.microsoft.com/en-us/azure/virtual-machines/ena... - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-inst... They sell these VMs at a higher cost because it requires more expensive components and is limited to host machines with certain configurations. In our experience, it's also harder to request quota increases to get more of these VMs -- some of the public clouds have a limited supply of these specific types of configurations in some regions/zones. As others have noted, instance storage isn't as dependable. But it can be the most performant way to do IO-intense processing or to power one node of a distributed database.
- _Rabs_ 3y agoSo much of this. The amount of times I've seen someone complain about slow DB performance when they're trying to connect to it from a different VPC, and bottlenecking themselves to 100Mbits is stupidly high. Literally depending on where things are in a data center... If you're looking for closely coupled and on a 10G line on the same switch, going to the same server rack. I bet you performance will be so much more consistent.
- bugbuddy 3y agoAren’t 10G and 100G connections standard nowadays in data centers? Heck, I thought they were standard 10 years ago.
- geerlingguy 3y agoDatacenters are up to 400 Gbps and beyond (many places are adopting 1+ Tbps on core switching). However, individual servers may still operate at 10, 25, or 40 Gbps to save cost on the thousands of NICs in a row of racks. Alternatively, servers with multiple 100G connections split that bandwidth allocation up among dozens of VMs so each one gets 1 or 10G.
- nixass 3y ago400G is fairly normal thing in DCs nowadays
- pixl97 3y agoBandwidth delay product does not help serialized transactions. If you're reaching out to disk for results, or if you have locking transactions on a table the achievable operations drops dramatically as latency between the host and the disk increases.
- bee_rider 3y agoThe typical way to trade bandwidth away for latency would, I guess, be speculative requests. In the CPU world at least. I wonder if any cloud providers have some sort of framework built around speculative disk reads (or maybe it is a totally crazy trade to make in this context)?
- ejb999 3y agoHow much faster would the network need to get, in order to meet (or at least approach) the speed of a local SSD? are we talking about needing to 2x or 3x the speed, or by factors of hundreds or thousands?
- deleted 3y ago[deleted]
- Filligree 3y agoThe Samsung 990 in my desktop provides ~3.5 GB/s streaming reads, ~2 GB/s 4k random-access reads, all at a latency measured at around 20-30 microseconds. My exact numbers might be a little off, but that's the ballpark you're looking at, and a 990 is a relatively cheap device. 10GbE is about the best you can hope for from a local network these days, but that's 1/5th the bandwidth and many times the latency. 100GbE would work, except the latency would still mean any read dependencies would be far slower than local storage, and I'm not sure there's much to be done about that; at these speeds the physical distance matters. In practice I'm having to architecture the entire system around the SSD just to not bottleneck it. So far ext4 is the only filesystem that even gets close to the SSD's limits, which is a bit of a pity.
- ants_a 3y agoNetworking doesn't have to have high latency. You can buy network hardware that is able to provide sub-microsecond latency. Physical distance still matters, but 10% of typical NVMe latency gets you through a kilometer of fiber.
- wmf 3y agoAround 4x-10x depending on how many SSDs you want. A single SSD is around the speed of a 100 Gbps Ethernet link.
- selectodude 3y agoSATA3 is 6 Gbit, so each VM on a machine multiplied by 6 Gbit. For NVMe, probably closer to 4-5x that. You’d need some serious interconnects to get a server rack access to un-bottlenecked SSD storage.
- zokier 3y ago> SSDs in the cloud are attached over a network, and fundamentally have to be Not on AWS. Instance stores (what the article is about) are physical local disks.
- jsolson 3y agoSame for Google (the GP is incorrect about how GCE Local SSD works).
- mkoubaa 3y agoDumb question. Why does the network have to be slow? If the SSDs are two feet away from the motherboard and there's an optical connection to it, shouldn't it be fast? Are data centers putting SSDs super far away from motherboards?
- bugbuddy 3y ago> One theory is that EC2 intentionally caps the write speed at 1 GB/s to avoid frequent device failure, given the total number of writes per SSD is limited. This is the theory that I would bet on because it lines up with their bottom line.
- Dylan16807 3y agoBut the sentence right after undermines it. > However, this does not explain why the read bandwidth is stuck at 2 GB/s. Faster read speeds would give them a more enticing product without wearing drives out.
- bugbuddy 3y agoThey may be limiting the read artificially to increase your resource utilization else where. If you have disk bottleneck then you would be more likely to use more instances. It is still about the bottom line.
- Dylan16807 3y agoThat could be. But it's a completely different reason. If you summarize everything as "bottom line", you lose all the valuable information.
- formercoder 3y agoWhat happens when your vm is live migrated 1000 feet away or to a different zone?
- supriyo-biswas 3y ago
- jsnell 3y agoAccording to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.
- pclmulqdq 3y agoIt's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.
- zokier 3y agoWhat makes you think that?
- ddorian43 3y agoDo you have a link to explain this? I dont think its true.
- candiddevmike 3y agoDepends on the cloud provider. Local SSDs are physically attached to the host on GCP, but that makes them only useful for temporary storage.
- pclmulqdq 3y agoIf you're at G, you should read the internal docs on exactly how this happens and it will be interesting.
- rfoo 3y agoWhy would I lose all data on these SSDs when I initiate a power off of the VM on console, then? I believe local SSDs are definitely attached to the host. They are just not exposed via NVMe ZNS hence the performance hit.
- wmf 3y agoNo, the i3/i4 VMs discussed in the blog have local SSDs. The network isn't the reason local SSDs are slow.
- fngjdflmdflg 3y agoSo do cloud vendors simply not use fast SSDs? If so I would expect the SSD manufacturers themselves to work on this problem. Perhaps they already are.
- adrr 3y agoIf the local drives are network drives(eg: SAN) then why are they ephemeral?
- baq 3y agolive vm migrations, perhaps
- deleted 3y ago[deleted]
- Dylan16807 3y agoEven assuming that "local" storage is a lie, hasn't the network gotten a lot faster? The author is only asking for a 5x increase at the end of the post.
- boulos 3y agoI'm not sure which external or internal product you're talking about, but there are no networks involved for Local SSD on GCE: https://cloud.google.com/compute/docs/disks/local-ssd https://cloud.google.com/compute/docs/disks/local-ssd Are you referring to PD-SSD? Internal storage usage?
- nostrademons 3y agoMakes me wonder if we're on the crux of a shift back to client-based software. Historically changes in the relative cost of computing components have driven most of the shifts in the computing industry. Cheap teletypes & peripherals fueled the shift from batch-processing mainframes to timesharing minicomputers. Cheap CPUs & RAM fueled the shift from minicomputers to microcomputers. Cheap and fast networking fueled the shift from desktop software to the cloud. Will cheap SSDs & TPU/GPUs fuel a shift back toward thicker clients? There are a bunch of supporting social trends toward this as well. Renewed emphasis on privacy. Big Tech canceling beloved products, bricking devices, and generally enshittifying everything - a lot of people want locally-controlled software that isn't going to get worse at the next update. Ever-rising prices which make people want to lock in a price for the device and not deal with increasing rents for computing power.
- davkan 3y agoI think a major limiting factor here for many applications is that mobile users are a huge portion of the user base. In that space storage, and more importantly battery life, are still at a premium. Granted the storage cost just seems to be gouging from my layman’s point of view, so industry needs might force a shift upwards.
- mike_hearn 3y agoMobile devices are the desktop computers of the 2010s though. They are mostly used with very thick clients.
- tw04 3y ago>The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. *As implemented in the public cloud providers. You can absolutely get better than local disk speeds from SAN devices and we've been doing it for decades. To do it on-prem with flash devices will require NVMe over FC or Ethernet and an appropriate storage array. Modern all-flash array performance is measured in millions of IOPS. Will there be a slight uptick in latency? Sure, but it's well worth it for the data services and capacity of an external array for nearly every workload.
- folmar 3y agoA quick comparison of marketing slides lands you with 0.5 MOPS for FC-NVMe for Qlogic 2770 and 0.7 MOPS for PCIe Micron 9100 PRO, so the better speeds are not quite there, although spending quite a lot on server gear lands you near the performance of a workstation-grade drive from 2017. Which is still not bad, when I was shopping around in 2018 no money could buy performance comparable to a locally-attached NVMe in a more professional/datacenter-ready form. [1] https://media-www.micron.com/-/media/client/global/documents/products/technical-marketing-brief/brief_business_case_pcie_ssd.pdf?la=en&rev=ce226f742a3741209f98d5f0bb152f2c https://media-www.micron.com/-/media/client/global/documents... [2] https://www.marvell.com/content/dam/marvell/en/public-collateral/fibre-channel/marvell-fibre-channel-nvme-over-fabrics-white-paper.pdf https://www.marvell.com/content/dam/marvell/en/public-collat...
- scottlamb 3y ago> The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, which was the backing technology when a lot of these network attached storage systems were invented, because they are fundamentally slow compared to networks, but it is a problem for SSD. Certainly true that SSD bandwidth and latency improvements are hard to match, but I don't understand why intra-datacenter network latency in particular is so bad. This ~2020-I-think version of the "Latency Numbers Everyone Should Know" says 0.5 ms round trip (and mentions "10 Gbps network" on another line). [1] It was the same thing in a 2012 version (that only mentions "1 Gbps network"). [2] Why no improvement? I think that 2020 version might have been a bit conservative on this line, and nice datacenters may even have multiple 100 Gbit/sec NICs per machine in 2024, but still I think the round trip actually is strangely bad. I've seen experimental networking stuff (e.g. RDMA) that claims significantly better latency, so I don't think it's a physical limitation of the networking gear but rather something at the machine/OS interaction area. I would design large distributed systems significantly differently (be much more excited about extra tiers in my stack) if the standard RPC system offered say 10 µs typical round trip latency. [1] https://static.googleusercontent.com/media/sre.google/en//static/pdf/rule-of-thumb-latency-numbers-letter.pdf https://static.googleusercontent.com/media/sre.google/en//st... [2] https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832
- dekhn 3y agoModern data center networks don't have full cross connectivity. Instead they are built using graphs and hierarchies that provide less than the total bandwidth required for all pairs of hosts to be communicating. This means, as workloads start to grow and large numbers of compute hosts demand data IO to/from storage hosts, the network eventually gets congested, which typically exhibits as higher latencies and more dropped packets. Batch jobs are often relegated to "spare" bandwidth while serving jobs often get dedicated bandwidth At the same time, ethernetworks with layered network protocols on top typically have a fair amount of latency overhead, that makes it much slower than bus-based direct-host-attached storage. I was definitely impressed at how quickly SSDs reached and then exceeded SATA bandwidth. nvme has made a HUGE difference here.
- 3y ago
- AlgorithmicTime 3y ago> SSDs in the cloud are attached over a network, and fundamentally have to be. No they don't. I work for a cloud provider and I can guarantee that your SSD is local to your VM.
- jsolson 3y agoThis is also true for GCE Local SSD: https://cloud.google.com/local-ssd https://cloud.google.com/local-ssd The GP is incorrect.
- throwawaaarrgh 3y ago> SSDs in the cloud are attached over a network, and fundamentally have to be SANs can still be quite fast, and instance storage is fast, both of which are available in cloud providers
- samstave 3y agoForgive if stupid, but when netflix was doing all their edge content boxes, where they were putting machines much more latency-close to customers... is/does/can this model kinda work for SSDs in a SScDN type of network (client---->CDN->SScDN-------<>SSD?
- dan-robertson 3y agoI can see network attached SSDs having poor latency, but shouldn’t the networking numbers quoted in the article allow for higher throughput than observed?
- paulddraper 3y ago> and fundamentally have to be Can you expound?
- PaulHoule 3y agoThey don’t have to be. Architecturally there are many benefits to storage area networks but I have built plenty of systems which are self-contained, download a dataset to a cloud instance with a direct attached SSD, load it into a database and provide a different way.
- waynesonfire 3y agoIn other words, they're saving money. This is a fundamental problem with cloud providers. Value created from technological innovation is captured by the cloud provider and bare minimum is shared to reduce prices. The margins are ridiculous.
- nimbius 3y agoOnce again getting a sensible chuckle on hn listening to the cloud crowd whine and complain about what we used to call "the SAN being slow" with the architects argument "the ssds will make things faster"
- deleted 3y ago[deleted]
- ec109685 3y agoThis is an interesting benchmark that compares disk read latency across different clouds across various disk configurations, including SSD's and EBS: https://github.com/scylladb/diskplorer https://github.com/scylladb/diskplorer Google's disk perform quite poorly. And how Discord worked around it: https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency https://discord.com/blog/how-discord-supercharges-network-di...
- amir734jj 3y agoKind of funny but we use similar idea in Azure.
- jsolson 3y agoThis is untrue of Local SSD (https://cloud.google.com/local-ssd https://cloud.google.com/local-ssd) in Google Cloud. Local SSDs are PCIe peripherals, not network attached. There are also multiple Persistent Disk (https://cloud.google.com/persistent-disk https://cloud.google.com/persistent-disk) offerings that are backed by SSDs over the network. (I'm an engineer on GCE. I work directly on the physical hardware that backs our virtualization platform.)
- jiggawatts 3y agoIt's notable that your second link has a screenshot for 24(!) NVMe SSDs totalling 9 terabytes, but the aggregate performance is 2.4M IOPS and 9.3 GB/s for reads. In other words, just 100K/400MB per individual SSD, which is very low these days. For comparison, a single 1 TB consumer SSD can deliver comparable numbers (lower IOPS but higher throughput). If I plugged 24 consumer SSDs into a box, I would expect over 30M IOPS and near the memory bus limit for throughput (>50 GB/s).
- jsolson 3y agoI should've aimed for more clarity in my original comment -- the first link is to locally attached storage. The second is network attached storage (what the GP was likely referring to, but not what is described in the article). Persistent Disk is not backed by single devices (even for a single NVMe attachment), but by multiple redundant copies spread across power and network failure domains. Those volumes will survive the failure of the VM to which they are attached as well as the failure of any individual volume or host.
- barrkel 3y agoThere's a quadrant of the market which is poorly served by the Cloud model of elastic compute: persistent local SSDs across shutdown and restart. Elastic compute means you want to be able to treat compute hardware as fungible. Persistent local storage makes that a lot harder because the Cloud provider wants to hand out that compute to someone else after shutdown, so the local storage needs to be wiped. So you either get ephemeral local SSDs (and have to handle rebuild on restart yourself) or network-attached SSDs with much higher reliability and persistence, but a fraction of the performance. Active instances can be migrated, of course, with sufficient cleverness in the I/O stack.
- vidarh 3y agoIt does not fundamentally have to be. That's an architectural choice driven by cloud providers backing off from the "instances can die on you" choice that AWS started with and then realized customers struggled with, towards attempting to keep things running in the face of many types of hardware failures. Which is ironic given that when building on-prem/colo'ed setups you'll replicate things to be prepared for unknown lengths of downtime while equipment is repaired or replaced, so this was largely cloud marketing coming bak to bite cloud providers' collective asses. Not wanting instances to die "randomly" for no good reason does not always mean wanting performance sacrifices for the sake of more resilient instances. But AWS at least still offers plenty of instances with instance storage. If I'm setting up my own database cluster, while I don't want it running on cheap consumer-grade hardware without dual power supplies and RAID, I also don't want to sacrifice SSD speed for something network-attached to survive a catastrophic failure when I'm going to have both backups, archived shipped logs and at least one replica anyway. I'd happily pick network-attached storage for many things if it gets me increased resilience, but selling me a network-attached SSD, unless it replicates local SSD performance characteristics, is not competitive for applications where performance matters and I'm set up to easily handle system-wide failures anyway.
- sitkack 3y agoYou are wrong, even on GCP. PD, persistent disk are all remote, but actual Local SSD are, local. There is still a bit of code that runs between the guest and the device. All of this is public information.