4 ms·

I feel there is some slight mis-information in this article that might confuse people and give them the wrong impression of 'Local SSDs' vs the network based on

I feel there is some slight mis-information in this article that might confuse people and give them the wrong impression of 'Local SSDs' vs the network based ones and about the nvme protocol and made it seem like "local ssd == nvme". NVMe - 'non-volatile memory express'. This is a _protocol_ that storage devices like SSDs can use to communicate with the external world. It is orthogonal to whether the disk is attached to your motherboard via pcie, or it lives somewhere else in the datacenter and uses a different transport layer. For example, local NVME SSDs will use PCIe transport. But you can also have NVME-over-TCPIP, or NVME-over-fiber-channel. Or NVME-over-Fabrics. Many of these are able to provide significantly lower latencies than a millisecond. As a concrete example, AWS `io2-express` has latencies of ~0.25-0.5ms, though i'm not sure which technology it's using. nvme over fabrics: https://www.techtarget.com/searchstorage/definition/NVMe-over-Fabrics-Nonvolatile-Memory-Express-over-Fabrics https://www.techtarget.com/searchstorage/definition/NVMe-ove... many interesting presentations on the official nvme website: https://nvmexpress.org/education/documents-and-videos/presentations/ https://nvmexpress.org/education/documents-and-videos/presen... aws: https://aws.amazon.com/blogs/storage/achieve-higher-database-performance-using-amazon-ebs-io2-block-express-volumes/ https://aws.amazon.com/blogs/storage/achieve-higher-database...

tyingq 4y ago
It seems like expected confusion. The protocol is well suited for a wide/fast path, so people associate it with something local that uses a lot of PCI lanes, which is much faster/wider/lower-latency than your NIC.
shrubble 4y ago
But why then would the article mention that they had problems with reliability? That seems ... odd; what have you experienced in your working with these (assuming you have)?
- themoonisachees 4y ago
  These disks aren't typically meant for database server usage. They're fine for consumers and even stuff like webservers, but the amount of data discord moves around on the daily means that they are bound to make some disks fail pretty quickly. The normal course of action for this is usually to have a raid array over all your nvme disks, but since google just migrates your VM to a machine that has good disks, doing that is useless. Really this whole article is "we are going to keep using google cloud despite their storage options being unfit for our purpose and here's how".
  - jart 4y ago
    > Really this whole article is "we are going to keep using google cloud despite their storage options being unfit for our purpose and here's how". And that's called hacking. Welcome to Hacker News. Discord's engineers are gods among mortals for squeezing this kind of latency and reliability out of something Google intended for consumers. Yes I know the way you're supposed to do something like this is to use something like Cloud BigTable where you probably have to call a salesperson and pay $20,000 before you're even allowed to try the thing. But Discord just ignored the enterprisey solution and took what they needed from the consumer tools instead. It reminds me of how early Google used to build data centers out of cheap personal computers.
    - stym06 4y ago
      google might see this and add it as an offering with cost lower than bigtable
    - prvit 4y ago
      Gods among mortals, because … they set up RAID?
      - srer 4y ago
        Consider it is not an unheard of experience to join the workforce, use a preconfigured macbook with precisely 1 SSD, and spend one's career building on SaaS platforms like AWS Lambda. In such a world, perhaps those who remember there's a world of flexibility and power within the OS could be seen as welding some supernatural power.
    - mwcampbell 4y ago
      More charitably, the GP could be suggesting that Discord should have gone to a lower layer in the stack by using their own hardware, which would let them choose a more appropriate storage substrate instead of layering on top of an inappropriate one.
  - Aeolun 4y ago
    > Really this whole article is "we are going to keep using google cloud despite their storage options being unfit for our purpose and here's how". Hmm, I read it more like they figured out a way to scale their existing storage a bit better while making Google eat the cost. It’s not like they pay extra if they wear out the disks sooner.
jon-wood 4y ago
Thanks for the clarification, I consider myself fairly well informed about server hardware and had no idea that there were transport methods for NVMe other than PCI Express.
- PeterCorless 4y ago
  There were a slew of new NVMe specs that came out last year, such as the separation of storage and transport. So now NVMe has the following transport protocols: • PCIe Transport specification • Fibre Channel Transport specification (NVMe-oF) • RDMA Transport specification • TCP Transport specification Separation of storage from transport is a huge game changer. I am really hoping NVMe-over-fibre really takes off. But I'd suggest people would first see that in on-prem deployments before you see it in the cloud-hosting hyperscalers. More on the 80,000 foot view of what's going on in NVMe world is covered in link below. But there are tech specs you can read over if you are so interested in how exactly it works. https://nvmexpress.org/nvm-express-announces-the-rearchitected-nvme-2-0-library-of-specifications/ https://nvmexpress.org/nvm-express-announces-the-rearchitect... [EDIT: Also, the GCP persistent disks are not NVMe-oF as far as we know. They seem to be iSCSI based off Colossus/D. see: https://news.ycombinator.com/item?id=21732387 https://news.ycombinator.com/item?id=21732387]

tyingq 4y ago

It seems like expected confusion. The protocol is well suited for a wide/fast path, so people associate it with something local that uses a lot of PCI lanes, which is much faster/wider/lower-latency than your NIC.

shrubble 4y ago

But why then would the article mention that they had problems with reliability? That seems ... odd; what have you experienced in your working with these (assuming you have)?

themoonisachees 4y ago
These disks aren't typically meant for database server usage. They're fine for consumers and even stuff like webservers, but the amount of data discord moves around on the daily means that they are bound to make some disks fail pretty quickly. The normal course of action for this is usually to have a raid array over all your nvme disks, but since google just migrates your VM to a machine that has good disks, doing that is useless. Really this whole article is "we are going to keep using google cloud despite their storage options being unfit for our purpose and here's how".
- jart 4y ago
  > Really this whole article is "we are going to keep using google cloud despite their storage options being unfit for our purpose and here's how". And that's called hacking. Welcome to Hacker News. Discord's engineers are gods among mortals for squeezing this kind of latency and reliability out of something Google intended for consumers. Yes I know the way you're supposed to do something like this is to use something like Cloud BigTable where you probably have to call a salesperson and pay $20,000 before you're even allowed to try the thing. But Discord just ignored the enterprisey solution and took what they needed from the consumer tools instead. It reminds me of how early Google used to build data centers out of cheap personal computers.
  - stym06 4y ago
    google might see this and add it as an offering with cost lower than bigtable
  - prvit 4y ago
    Gods among mortals, because … they set up RAID?
    - srer 4y ago
      Consider it is not an unheard of experience to join the workforce, use a preconfigured macbook with precisely 1 SSD, and spend one's career building on SaaS platforms like AWS Lambda. In such a world, perhaps those who remember there's a world of flexibility and power within the OS could be seen as welding some supernatural power.
  - mwcampbell 4y ago
    More charitably, the GP could be suggesting that Discord should have gone to a lower layer in the stack by using their own hardware, which would let them choose a more appropriate storage substrate instead of layering on top of an inappropriate one.
- Aeolun 4y ago
  > Really this whole article is "we are going to keep using google cloud despite their storage options being unfit for our purpose and here's how". Hmm, I read it more like they figured out a way to scale their existing storage a bit better while making Google eat the cost. It’s not like they pay extra if they wear out the disks sooner.

jon-wood 4y ago

Thanks for the clarification, I consider myself fairly well informed about server hardware and had no idea that there were transport methods for NVMe other than PCI Express.

PeterCorless 4y ago
There were a slew of new NVMe specs that came out last year, such as the separation of storage and transport. So now NVMe has the following transport protocols: • PCIe Transport specification • Fibre Channel Transport specification (NVMe-oF) • RDMA Transport specification • TCP Transport specification Separation of storage from transport is a huge game changer. I am really hoping NVMe-over-fibre really takes off. But I'd suggest people would first see that in on-prem deployments before you see it in the cloud-hosting hyperscalers. More on the 80,000 foot view of what's going on in NVMe world is covered in link below. But there are tech specs you can read over if you are so interested in how exactly it works. https://nvmexpress.org/nvm-express-announces-the-rearchitected-nvme-2-0-library-of-specifications/ https://nvmexpress.org/nvm-express-announces-the-rearchitect... [EDIT: Also, the GCP persistent disks are not NVMe-oF as far as we know. They seem to be iSCSI based off Colossus/D. see: https://news.ycombinator.com/item?id=21732387 https://news.ycombinator.com/item?id=21732387]