4 ms·

TLDR: 25GbE >23 drives we can then RAID 0 umm, is this satire? So we replace "expensive and complicated" storage controller with expensive consumables (ssds

by rasz 5y ago

TLDR: 25GbE >23 drives we can then RAID 0 umm, is this satire? So we replace "expensive and complicated" storage controller with expensive consumables (ssds have limited lifespan) and move complicated part to the server. What are the upsides?

boxingrock 5y ago
software-defined bare-metal, that's pretty much the gist of it.
- rasz 5y ago
  So exactly like you can do the right now on storage controller. Backblaze storage pods are "software-defined bare-metal".
gravypod 5y ago
Right now the bottleneck is memory bandwidth & CPU lanes. NVMEoE lets you access storage from a VM that you can move around and, depending on what you are doing, you are probably already serving traffic over your NIC. If you only need to do a minimal amount of I/O to disk all of the lanes used for I/O when you're idle is useless. If you are doing things like cache servers, DBs that manage replication for themselves, etc you can really optimize storage using this approach. Basically the difference between 25GB/s between your server and it's storage and 25GB/s between your server and it's clients is just where the packets are going.
- rasz 5y ago
  2x2.5GB/s for the whole system. Everything you can do with this you can do in classic system using cheaper COTS components. I simply dont see the point other than inventing new proprietary drives to cut off third party suppliers.
  - paranoidrobot 5y ago
    > 2x2.5GB/s for the whole system. Pretty sure that's per drive.
    - deleted 5y ago
      [deleted]
  - rektide 5y ago
    Its 2x25Gbps per drive. 25Gbe is quite cheap, not hard to do, low power & especially so over short backplanes. This chassis has 1.2Tbps switching, half for drives, half for attached computers. It's not proprietary. The drives can plug into ethernet. They speak industry standard NVMeoF; you could not get more standard if you tried.
paranoidrobot 5y ago
I might be misreading it, but it's a 2x25GbE connection to each drive. So 24x 25GbE in that one switch chassis. The network-switch-chassis has 6x 100GbE (or is it actually 200GbE) for external connections. Since the drives are dual-port, the other chassis has another 6x100/200GbE if you wanted to split traffic over the second disk port + network-switch-chassis. Upsides seem to be: - Can scale up storage by plugging in another switch/disk shelf - No storage-enforced limits on the number or type of arrays - Can define namespaces on drives so you can adjust storage to clients based on need. - Can scale up performance for a single client by making more drives available and/or faster client NICs - Not limited to the number of PCI lanes in the client so you could have fast access to FAR more disks than you could physically plug into one host. Performance would be limited to the client's NIC(s). At the end of the video Patrick talks about that while they demo it being done directly on a host, they can also have DPU cards that could do all of the mounting/array management bits. There's a video on that coming soon.
- rasz 5y ago
  "Marvell 98EX5630 Ethernet switch, 6x 100GbE on the rear of the chassis = 600Gbps. 25GbE for each drive * 24 drives = 600Gbps providing a 1:1 ratio" Whats the appeal of proprietary drives talking to fiber switch directly versus NVME chassis with 3 2x100G PCIe NICs on the back? From logical point of view there is no difference. You can parturition, provision, bond it up just the same. You just move PCIE-fiber conversion from central point (cheaper) to bespoke solution inside proprietary drives (sounds expensive vendor lock-in).
  - paranoidrobot 5y ago
    I don't think these are proprietary drives or connectors. I can't see anything there to indicate it is proprietary. I think Marvell just happens to be the first out with the NVMEoF drive adapter and switch chips. Kioxia and the switch vendor seem to be first out with the drives and switches using Marvell chips/boards. I wouldn't be surprised to hear in a year or two that you can buy a NVMEoF switch from a bunch of vendors, and then NVMEoF drives from your choice of drive vendor. The specifics of advantages over existing solutions I can't speak to except repeating much of what is in the article and video. I might be placing too much faith in Patrick + STH, but they seem excited, and other times there have been proprietary solutions for things they have pointed it out.

by rasz 5y ago

boxingrock 5y ago

software-defined bare-metal, that's pretty much the gist of it.

rasz 5y ago
So exactly like you can do the right now on storage controller. Backblaze storage pods are "software-defined bare-metal".

gravypod 5y ago

Right now the bottleneck is memory bandwidth & CPU lanes. NVMEoE lets you access storage from a VM that you can move around and, depending on what you are doing, you are probably already serving traffic over your NIC. If you only need to do a minimal amount of I/O to disk all of the lanes used for I/O when you're idle is useless. If you are doing things like cache servers, DBs that manage replication for themselves, etc you can really optimize storage using this approach. Basically the difference between 25GB/s between your server and it's storage and 25GB/s between your server and it's clients is just where the packets are going.

rasz 5y ago
2x2.5GB/s for the whole system. Everything you can do with this you can do in classic system using cheaper COTS components. I simply dont see the point other than inventing new proprietary drives to cut off third party suppliers.
- paranoidrobot 5y ago
  > 2x2.5GB/s for the whole system. Pretty sure that's per drive.
  - deleted 5y ago
    [deleted]
- rektide 5y ago
  Its 2x25Gbps per drive. 25Gbe is quite cheap, not hard to do, low power & especially so over short backplanes. This chassis has 1.2Tbps switching, half for drives, half for attached computers. It's not proprietary. The drives can plug into ethernet. They speak industry standard NVMeoF; you could not get more standard if you tried.

paranoidrobot 5y ago

I might be misreading it, but it's a 2x25GbE connection to each drive. So 24x 25GbE in that one switch chassis. The network-switch-chassis has 6x 100GbE (or is it actually 200GbE) for external connections. Since the drives are dual-port, the other chassis has another 6x100/200GbE if you wanted to split traffic over the second disk port + network-switch-chassis. Upsides seem to be: - Can scale up storage by plugging in another switch/disk shelf - No storage-enforced limits on the number or type of arrays - Can define namespaces on drives so you can adjust storage to clients based on need. - Can scale up performance for a single client by making more drives available and/or faster client NICs - Not limited to the number of PCI lanes in the client so you could have fast access to FAR more disks than you could physically plug into one host. Performance would be limited to the client's NIC(s). At the end of the video Patrick talks about that while they demo it being done directly on a host, they can also have DPU cards that could do all of the mounting/array management bits. There's a video on that coming soon.

rasz 5y ago
"Marvell 98EX5630 Ethernet switch, 6x 100GbE on the rear of the chassis = 600Gbps. 25GbE for each drive * 24 drives = 600Gbps providing a 1:1 ratio" Whats the appeal of proprietary drives talking to fiber switch directly versus NVME chassis with 3 2x100G PCIe NICs on the back? From logical point of view there is no difference. You can parturition, provision, bond it up just the same. You just move PCIE-fiber conversion from central point (cheaper) to bespoke solution inside proprietary drives (sounds expensive vendor lock-in).
- paranoidrobot 5y ago
  I don't think these are proprietary drives or connectors. I can't see anything there to indicate it is proprietary. I think Marvell just happens to be the first out with the NVMEoF drive adapter and switch chips. Kioxia and the switch vendor seem to be first out with the drives and switches using Marvell chips/boards. I wouldn't be surprised to hear in a year or two that you can buy a NVMEoF switch from a bunch of vendors, and then NVMEoF drives from your choice of drive vendor. The specifics of advantages over existing solutions I can't speak to except repeating much of what is in the article and video. I might be placing too much faith in Patrick + STH, but they seem excited, and other times there have been proprietary solutions for things they have pointed it out.