4 ms·

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storag

by jsnell 3y ago

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing either. GCP has "local SSD disks"[1], which I'm going to claim are likewise local, not over the network block storage. (Though the language isn't as explicit as for AWS.) [0] https://aws.amazon.com/ec2/instance-types/ https://aws.amazon.com/ec2/instance-types/ [1] https://cloud.google.com/compute/docs/disks#localssds https://cloud.google.com/compute/docs/disks#localssds

20after4 3y ago
If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.
- discodave 3y ago
  If you have one of the metal instance types, then you get the whole host, e.g. i4i.metal: https://aws.amazon.com/ec2/instance-types/i4i/ https://aws.amazon.com/ec2/instance-types/i4i/
- aeyes 3y ago
  On AWS yes, the older instances which I am familiar with had 900GB drives and they sliced that up into volumes of 600, 450, 300, 150, 75GB depending on instance size. But they also tell you how much IOPS you get: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/storage-optimized-instances.html#storage-instances-diskperf https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/stora...
- ownagefool 3y ago
  PCI bus, etc too
- throwawaaarrgh 3y ago
  Instance storage is not networked. That's why it's there.
- queuebert 3y ago
  How do these machines manage the sharing of one local SSD across multiple VMs? Is there some wrapper around the I/O stack? Does it appear as a network share? Geniuinely curious...
  - felixg3 3y ago
    Probably NVME namespaces [0]? [0]: https://nvmexpress.org/resource/nvme-namespaces/ https://nvmexpress.org/resource/nvme-namespaces/
    - bravetraveler 3y ago
      Less fancy, quite often... at least on VPS providers [1]. They like to use reflinked files off the base images. This way they only store what differs. 1: Which is really a cloud without a certain degree of software defined networking/compute/storage/whatever.
  - dan-robertson 3y ago
    AWS have custom firmware for at least some of their SSDs, so could be that
  - magicalhippo 3y ago
    In say VirtualBox you can create a file backed on the physical disk, and attach it to the VM so the VM sees it as a NVMe drive. In my experience this is also orders of magnitude slower that true direct access, ie PCIe pass-through, as all access has to pass through the VM storage driver and so could explain what is happening.
    - bravetraveler 3y ago
      The storage driver may have more impact on VBox. You can get very impressive results with 'virtio' on KVM
      - magicalhippo 3y ago
pclmulqdq 3y ago
That's the abstraction they want you to work with, yes. That doesn't mean it's what is actually happening - at least not in the same way that you're thinking. As a hint for you, I said "a network", not "the network." You can also look at public presentations about how Nitro works.
- jng 3y ago
  Nitro "virtual NVME" device are mostly (only?) for EBS -- remote network storage, transparently managed, using a separate network backbone, and presented to the host as a regular local NVME device. SSD drives in instances such as i4i, etc. are physically attached in a different way -- but physically, unlike EBS, they are ephemeral and the content becomes unavaiable as you stop the instance, and when you restart, you get a new "blank slate". Their performance is 1 order of magnitude faster than standard-level EBS, and the cost structure is completely different (and many orders of magnitude more affordable than EBS volumes configured to have comparable I/O performance).
  - rcarmo 3y ago
    This is the way Azure temporary volumes work as well. They are scrubbed off the hardware once the VM that accesses them is dead. Everything else is over the network.
- jsnell 3y ago
  I've linked to public documentation that is pretty clearly in conflict with what you said. There's no wiggle room in how AWS describes their service without it being false advertising. There's no "ah, but what if we define the entire building to be the host computer, then the networked SSDs really are inside the host computer" sleight of hand to pull off here. You've provided cryptic hints and a suggestion to watch some unnamed presentation. At this point I really think the burden of proof is on you.
  - stingraycharles 3y ago
    You are correct, and the parent you’re replying to is confused. Nitro is for EBS, not the i3 local NVMe instances. Those i3 instances lose your data whenever you stop and start them again (ie migrate to a different host machine), there’s absolutely no reason they would use network. EBS itself uses a different network than the “normal” internet, if I were to guess it’s a converged Ethernet network optimized for iSCSI. Which is what Nitro optimizes for as well. But it’s not relevant for the local NVMe storage.
wstuartcl 3y ago
the tests were for these local (metal direct connect ssds). The issue is not network overhead -- its that just like everything else in cloud the performance of 10 years ago was used as the baseline that carries over today with upcharges to buy back the gains. there is a reason why vcpu performance is still locked to the typical core from 10 years ago when every core on a machine today in those data scenters is 3-5x or more speed basis. Its cause they can charge you for 5x the cores to get that gain.
- wmf 3y ago
  vcpu performance is still locked to the typical core from 10 years ago No. In some cases I think AWS actually buys special processors that are clocked higher than the ones you can buy.
  - gowld 3y ago
    You are talking about real CPU not virtual cpu
    - wmf 3y ago
      Generally each vCPU is a dedicated hardware thread, which has gotten significantly faster in the last 10 years. Only lambdas, micros, and nanos have shared vCPUs and those have probably also gotten faster although it's not guaranteed.
      - jandrewrogers 3y ago
        In fairness, there are a not insignificant number of workloads that do not benefit from hardware threads on CPUs [0], instead isolating processes along physical cores for optimal performance. [0] Assertion not valid for barrel processors.
  - phanimahesh 3y ago
    The parent claims that though aws uses better hardware, they bill in vcpus whose benchmarks are from a few years ago, so that they can sell more vcpu units per performant physical cpu. This does not contradict your claim that aws buys better hardware.
Hewitt821 3y ago
Local SSD is part of the machine, not network attached.
reactordev 3y ago
AWS is so large, every concept of hardware is virtualized over a software layer. “Instance storage” is no different. It’s just closer to the edge with your node. It’s not some box in a rack where some AWS tech slots in an SSD. AWS has a hardware layer, but you’ll never see it.

by jsnell 3y ago

20after4 3y ago

If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.

discodave 3y ago
If you have one of the metal instance types, then you get the whole host, e.g. i4i.metal: https://aws.amazon.com/ec2/instance-types/i4i/ https://aws.amazon.com/ec2/instance-types/i4i/
aeyes 3y ago
On AWS yes, the older instances which I am familiar with had 900GB drives and they sliced that up into volumes of 600, 450, 300, 150, 75GB depending on instance size. But they also tell you how much IOPS you get: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/storage-optimized-instances.html#storage-instances-diskperf https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/stora...
ownagefool 3y ago
PCI bus, etc too
throwawaaarrgh 3y ago
Instance storage is not networked. That's why it's there.
queuebert 3y ago
How do these machines manage the sharing of one local SSD across multiple VMs? Is there some wrapper around the I/O stack? Does it appear as a network share? Geniuinely curious...
- felixg3 3y ago
  Probably NVME namespaces [0]? [0]: https://nvmexpress.org/resource/nvme-namespaces/ https://nvmexpress.org/resource/nvme-namespaces/
  - bravetraveler 3y ago
    Less fancy, quite often... at least on VPS providers [1]. They like to use reflinked files off the base images. This way they only store what differs. 1: Which is really a cloud without a certain degree of software defined networking/compute/storage/whatever.
- dan-robertson 3y ago
  AWS have custom firmware for at least some of their SSDs, so could be that
- magicalhippo 3y ago
  In say VirtualBox you can create a file backed on the physical disk, and attach it to the VM so the VM sees it as a NVMe drive. In my experience this is also orders of magnitude slower that true direct access, ie PCIe pass-through, as all access has to pass through the VM storage driver and so could explain what is happening.
  - bravetraveler 3y ago
    The storage driver may have more impact on VBox. You can get very impressive results with 'virtio' on KVM
    - magicalhippo 3y ago

pclmulqdq 3y ago

That's the abstraction they want you to work with, yes. That doesn't mean it's what is actually happening - at least not in the same way that you're thinking. As a hint for you, I said "a network", not "the network." You can also look at public presentations about how Nitro works.

jng 3y ago
Nitro "virtual NVME" device are mostly (only?) for EBS -- remote network storage, transparently managed, using a separate network backbone, and presented to the host as a regular local NVME device. SSD drives in instances such as i4i, etc. are physically attached in a different way -- but physically, unlike EBS, they are ephemeral and the content becomes unavaiable as you stop the instance, and when you restart, you get a new "blank slate". Their performance is 1 order of magnitude faster than standard-level EBS, and the cost structure is completely different (and many orders of magnitude more affordable than EBS volumes configured to have comparable I/O performance).
- rcarmo 3y ago
  This is the way Azure temporary volumes work as well. They are scrubbed off the hardware once the VM that accesses them is dead. Everything else is over the network.
jsnell 3y ago
I've linked to public documentation that is pretty clearly in conflict with what you said. There's no wiggle room in how AWS describes their service without it being false advertising. There's no "ah, but what if we define the entire building to be the host computer, then the networked SSDs really are inside the host computer" sleight of hand to pull off here. You've provided cryptic hints and a suggestion to watch some unnamed presentation. At this point I really think the burden of proof is on you.
- stingraycharles 3y ago
  You are correct, and the parent you’re replying to is confused. Nitro is for EBS, not the i3 local NVMe instances. Those i3 instances lose your data whenever you stop and start them again (ie migrate to a different host machine), there’s absolutely no reason they would use network. EBS itself uses a different network than the “normal” internet, if I were to guess it’s a converged Ethernet network optimized for iSCSI. Which is what Nitro optimizes for as well. But it’s not relevant for the local NVMe storage.

wstuartcl 3y ago

the tests were for these local (metal direct connect ssds). The issue is not network overhead -- its that just like everything else in cloud the performance of 10 years ago was used as the baseline that carries over today with upcharges to buy back the gains. there is a reason why vcpu performance is still locked to the typical core from 10 years ago when every core on a machine today in those data scenters is 3-5x or more speed basis. Its cause they can charge you for 5x the cores to get that gain.

wmf 3y ago
vcpu performance is still locked to the typical core from 10 years ago No. In some cases I think AWS actually buys special processors that are clocked higher than the ones you can buy.
- gowld 3y ago
  You are talking about real CPU not virtual cpu
  - wmf 3y ago
    Generally each vCPU is a dedicated hardware thread, which has gotten significantly faster in the last 10 years. Only lambdas, micros, and nanos have shared vCPUs and those have probably also gotten faster although it's not guaranteed.
    - jandrewrogers 3y ago
      In fairness, there are a not insignificant number of workloads that do not benefit from hardware threads on CPUs [0], instead isolating processes along physical cores for optimal performance. [0] Assertion not valid for barrel processors.
- phanimahesh 3y ago
  The parent claims that though aws uses better hardware, they bill in vcpus whose benchmarks are from a few years ago, so that they can sell more vcpu units per performant physical cpu. This does not contradict your claim that aws buys better hardware.

Hewitt821 3y ago

Local SSD is part of the machine, not network attached.

reactordev 3y ago

AWS is so large, every concept of hardware is virtualized over a software layer. “Instance storage” is no different. It’s just closer to the edge with your node. It’s not some box in a rack where some AWS tech slots in an SSD. AWS has a hardware layer, but you’ll never see it.