5 ms·

How do these machines manage the sharing of one local SSD across multiple VMs? Is there some wrapper around the I/O stack? Does it appear as a network share?

• felixg3 3y ago
  Probably NVME namespaces [0]? [0]: https://nvmexpress.org/resource/nvme-namespaces/ https://nvmexpress.org/resource/nvme-namespaces/
  • bravetraveler 3y ago
    Less fancy, quite often... at least on VPS providers [1]. They like to use reflinked files off the base images. This way they only store what differs. 1: Which is really a cloud without a certain degree of software defined networking/compute/storage/whatever.
• dan-robertson 3y ago
  AWS have custom firmware for at least some of their SSDs, so could be that
• magicalhippo 3y ago
  In say VirtualBox you can create a file backed on the physical disk, and attach it to the VM so the VM sees it as a NVMe drive. In my experience this is also orders of magnitude slower that true direct access, ie PCIe pass-through, as all access has to pass through the VM storage driver and so could explain what is happening.
  • bravetraveler 3y ago
    The storage driver may have more impact on VBox. You can get very impressive results with 'virtio' on KVM
    • magicalhippo 3y ago
      Yeah I've yet to try that. I know I get a similar lack of performance with Bhyve (FreeBSD) using VirtIO, so it's not a given it's fast. I have no idea how AWS run their VMs, was just saying a slow storage driver could give such results.
      • bravetraveler 3y ago
        > just saying a slow storage driver could give such results Oh, absolutely - not to contest that! There's a whole lot of academia on 'para-virtualized' and so on in this light. That's interesting to hear about FreeBSD; basically all of my experience has been with Linux/Windows.
• icedchai 3y ago
  With Linux and KVM/QEMU, you can map an entire physical disk, disk partition, or file to a block device in the VM. For my own VM hosts, I use LVM and map a logical volume to the VM. I assumed cloud providers did something conceptually similar, only much more sophisticated.
  • bravetraveler 3y ago
    Files with reflinks are a common choice, the main benefit being: only storing deltas. The base OS costs basically nothing LVM/block like you suggest is a good idea. You'd be surprised how much access time is trimmed by skipping another filesystem like you'd have with a raw image file
  • jethro_tell 3y ago
    Heh, you'd probably be surprised, there's some really cool cutting edge stuff being done in those data centers but a lot of what is done is just plan old standard server management without much in the way of tricks. Its just someone else does it instead of you and the billing department is counting milliseconds.
    • icedchai 3y ago
      Do cloud providers document these internals anywhere? I'd love to read about that sort of thing.
      • jethro_tell 3y ago
        Not generally, especially not the super generic stuff. Where they really excel is having the guy that wrote the kernel driver or hypervisor on staff. But a lot of it is just an automated version of what you'd do on a smaller scale