8 ms·
According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutio
by jsnell 3y ago
According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions.
Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.
- pclmulqdq 3y agoIt's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.
- zokier 3y agoWhat makes you think that?
- ddorian43 3y agoDo you have a link to explain this? I dont think its true.
- candiddevmike 3y agoDepends on the cloud provider. Local SSDs are physically attached to the host on GCP, but that makes them only useful for temporary storage.
- pclmulqdq 3y agoIf you're at G, you should read the internal docs on exactly how this happens and it will be interesting.
- rfoo 3y agoWhy would I lose all data on these SSDs when I initiate a power off of the VM on console, then? I believe local SSDs are definitely attached to the host. They are just not exposed via NVMe ZNS hence the performance hit.
- manquer 3y agoIt is because on reboot you may not get the same physical server . They are not rebooting the physical server for you , just the VM Same VM is not allocated for a variety of reasons , scheduled maintenance, proximity to other hosts on the vpc , balancing quiet and noisy neighbors so on. It is not that the disk will always wiped , sometimes the data is still there on reboot just that there is no guarantee allowing them to freely move between hosts
- mr_toad 3y agoData persists between reboots, but not shutdowns: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-reboot.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst... https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html#lifecycle-differences https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...
- res0nat0r 3y agoYour EC2 instance with instance-store storage when stopped can be launched on any other random host in the AZ when you power it back on. Since your rootdisk is an EBS volume attached across the network, so when you start your instance back up you're going to be launched likely somewhere else with an empty slot, and empty local-storage. This is why there is always a disclaimer that this local storage is ephemeral and don't count on it being around long-term.
- mrcarrot 3y agoI think the parent was agreeing with you. If the “local” SSDs _weren’t_ actually local, then presumably they wouldn’t need to be ephemeral since they could be connected over the network to whichever host your instance was launched on.
- jsolson 3y agoIn most cases, they're physically plugged into a PCIe CEM slot in the host. There is no network in the way, you are either misinformed or thinking of a different product.
- seedless-sensat 3y agoWhy are you protecting Google's internal architecture onto to AWS? Your Google mental model is not correct here
- amluto 3y agoWhich is a weird sort of limitation. For any sort of you-own-the-hardware arrangement, NVMe disks are fine for long term storage. (Obviously one should have backups, but that’s a separate issue. One should have a DR plan for data on EBS, too.) You need to migrate that data if you replace an entire server, but this usually isn’t a very big deal.
- supriyo-biswas 3y agoThis is Hyrum’s law at play: AWS wants to make sure that the instance stores aren’t seen as persistent, and therefore enforce the failure mode for normal operations as well. You should also see how they enforce similar things for their other products and APIs, for example, most of their services have encrypted pagination tokens.
- throwawaaarrgh 3y agoYes, that's what their purpose is in cloud applications: temporary high performance storage only. If you want long term local storage you'll have to reserve an instance host.
- mike_hearn 3y agoThey do this because they want SSDs to be in a physically separate part of the building for operational reasons, or what's the point in giving you a "local" SSD that isn't actually plugged into the real machine?
- ianburrell 3y agoThe reason for having most instances use network storage is that it makes possible migrating instances to other hosts. If the host fails, the network storage can be pointed at the new host with a reboot. AWS sends out notices regularly when they are going to reboot or migrate instances. Their probably should be more local instance storage types for using with instances that can be recreated without loss. But it is simple for them to have a single way of doing things. At work, someone used fast NVMe instance storage for Clickhouse which is a database. It was a huge hassle to copy data when instances were going to be restarted because the data would be lost.
- mike_hearn 3y agoSure, I understand that, but this user is claiming that on GCP even local SSDs aren't really local, which raises the question of why not. I suspect the answer is something to do with their manufacturing processes/rack designs. When I worked there (pre GCP) machines had only a tiny disk used for booting and they wanted to get rid of that. Storage was handled by "diskful" machines that had dedicated trays of HDDs connected to their motherboards. If your datacenters and manufacturing processes are optimized for building machines that are either compute or storage but not both, perhaps the more normal cloud model is hard to support and that pushes you towards trying to aggregate storage even for "local" SSD or something.
- colechristensen 3y agoFor AWS there are EBS volumes attached through a custom hardware NVMe interface and then there's Instance Store which is actually local SSD storage. These are different things. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Instance...
- kwillets 3y agoEBS is also slower than local NVMe mounts on i3's. Also, both features use Nitro SSD cards, according to AWS docs. The Nitro architecture is all locally attached -- instance storage to the instance, EBS to the EBS server.
- jrullman 3y agoI can attest to the fact that on EC2, "instance store" volumes are actually physically attached.
- jsnell 3y agoI think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing either. GCP has "local SSD disks"[1], which I'm going to claim are likewise local, not over the network block storage. (Though the language isn't as explicit as for AWS.) [0] https://aws.amazon.com/ec2/instance-types/ https://aws.amazon.com/ec2/instance-types/ [1] https://cloud.google.com/compute/docs/disks#localssds https://cloud.google.com/compute/docs/disks#localssds
- 20after4 3y agoIf the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.
- discodave 3y agoIf you have one of the metal instance types, then you get the whole host, e.g. i4i.metal: https://aws.amazon.com/ec2/instance-types/i4i/ https://aws.amazon.com/ec2/instance-types/i4i/
- aeyes 3y agoOn AWS yes, the older instances which I am familiar with had 900GB drives and they sliced that up into volumes of 600, 450, 300, 150, 75GB depending on instance size. But they also tell you how much IOPS you get: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/storage-optimized-instances.html#storage-instances-diskperf https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/stora...
- ownagefool 3y agoPCI bus, etc too
- 3y ago
- dekhn 3y agoI suspect you must be conflating several different storage products. Are you saying https://cloud.google.com/compute/docs/disks/local-ssd https://cloud.google.com/compute/docs/disks/local-ssd devices talk to the host through a network (say, ethernet with some layer on top)? Because the documentation very clearly says otherwise, "This is because Local SSD disks are physically attached to the server that hosts your VM. For this same reason, Local SSD disks can only provide temporary storage." (at least, I'm presuming that by physically attached, they mean it's connected to the PCI bus without a network in between). I suspect you're thinking of SSD-PD. If "local" SSDs are not actually local and go through a network, I need to have a discussion with my GCS TAM about truth in advertising.
- op00to 3y ago> physically attached Believe it or not, superglue and a wifi module! /s
- mint2 3y agoI don’t really agree with assuming the form of physical attachment and interaction unless it is spelled out. If that’s what’s meant it will be stated in some fine print, if it’s not stated anywhere then there is no guarantee what the term means, except I would guess they may want people to infer things that may not necessarily be true.
- dekhn 3y ago"Physically attached" has had a fairly well defined meaning and i don't normally expect a cloud provider to play word salad to convince me a network drive is locally attached (like I said, if true, I would need to have a chat with my TAM about it). Physically attached for servers, for the past 20+ years, has meant a direct electrical connection to a host bus (such as the PCI bus attached to the front-side bus). I'd like to see some alternative examples that violate that convention.
- adgjlsfhk1 3y agoEthernet cables are physical...
- crotchfire 3y agoThis is incorrect. Amazon offers both locally-attached storage devices as well as instance-attached storage devices. The article is about the latter kind.
- bfncieezo 3y agoinstances can have block storage which is network attached, or local attached ssd/nvme. its 2 separate things.
- choppaface 3y agoNope! Well not as advertised. There are instances, usually more expensive ones, where there are supposed to be local NVME disks dedicated to the instance. You're totally right that providing good I/O is a big problem! And I have done studies myself showing just how bad Google Cloud is here, and have totally ditched Google Cloud for providing crappy compute service (and even worse customer service).
- yolovoe 3y agoYou’re wrong. Instance local means SSD is physically attached to the droplet and is inside the server chassis, connected via PCIe. Sourece: I work on nitro cards.
- hathawsh 3y agoThat seems like a big opportunity for other cloud providers. They could provide SSDs that are actually physically attached and boast (rightfully) that their SSDs are a lot faster, drawing away business from older cloud providers.
- solardev 3y agoFor what kind of workloads would a slower SSD be a significant bottleneck?
- lolc 3y agoI tend some workloads that transform data grids of varying sizes. The grids are anon mmaps so that when mem runs out, they get paged out. This means processing stays mostly in-mem yet won't abort when mem runs tight. The processes that get hit by paging slow to a crawl though. Getting faster SSD means they're still crawling but crawling faster. Doubling SSD throughput would pretty much half the tail latency.
- solardev 3y agoI see. Thanks for explaining!
- ReflectedImage 3y agoPretty much all work loads, work loads that are not affected would be the exception
- jandrewrogers 3y agoI run very large database-y workloads. Storage bandwidth is by far the throughput rate limiting factor. Cloud environments are highly constrained in this regard and there is a mismatch between the amount of CPU you are required to buy to get a given amount of bandwidth. I could saturate a much faster storage system with a fraction of the CPU but that isn’t an option. Note that latency is not a major concern here. This has an enormous economic impact. I once did a TCO study with AWS to run data-intensive workload running on purpose-built infrastructure on their cloud. AWS would have been 3x more expensive per their own numbers, they didn’t even argue it. The main difference is that we had highly optimized our storage configuration to provide exceptional throughput for our workload on cheap hardware. I currently run workloads in the cloud because it is convenient. At scale though, the cost difference to run it on your own hardware is compelling. The cloud companies also benefit from a learned helplessness when it comes to physical infrastructure. Ironically, it has never been easier to do a custom infrastructure build, which companies used to do all the time, but most people act like it is deep magic now.
- Salgat 3y agoAt first you'd think maybe they can do a volume copy from a snapshot to a local drive on instance creation but even at 100gbps you're looking at almost 3 minutes for a 2TB drive.
- crazygringo 3y ago> It's about SSDs being two orders of magnitude slower than datacenter networks. Could that have to do with every operation requiring a round trip, rather than being able to queue up operations in a buffer to saturate throughput? It seems plausible if the interface protocol was built for a device it assumed was physically local and so waited for confirmation after each operation before performing the next. In this case it's not so much the throughput rate that matters, but the latency -- which can also be heavily affected by buffering of other network traffic.
- Nextgrid 3y agoUnderlying protocol limitations wouldn't be an issue - the cloud provider's implementation can work around that. They're unlikely to be sending sequential SCSI/NVMe commands over the wire - instead, the hypervisor pretends to be the NVME device, but then converts to some internal protocol (that's less chatty and can coalesce requests without waiting on individual ACKs) before sending that to the storage server. The problem is that ultimately your application often requires the outcome of a given IO operation to decide which operation to perform next - let's say when it comes to a database, it should first read the index (and wait for that to complete) before it knows the on-disk location of the actual row data which it needs to be able to issue the next IO operation. In this case, there's no other solution than to move that application closer to the data itself. Instead of the networked storage node being a dumb blob storage returning bytes, the networked "storage" node is your database itself, returning query results. I believe that's what RDS Aurora does for example, every storage node can itself understand query predicates.
- karmakaze 3y agoI've run CI/CD pipelines on EC2 machines with local storage, typically running Raid-0, btrfs, noaccestime. I didn't care if the filesystem got corrupt or whatever, I had a script that would rebuild it in under 30mins. In addition to the performance you're not paying by IOPs.