3 ms·

Network attached HDDs were killed by ethernet. In their infinite wisdom, ethernet decided there was no need for speeds between 1GbE and 10GbE. They finally cave

Network attached HDDs were killed by ethernet. In their infinite wisdom, ethernet decided there was no need for speeds between 1GbE and 10GbE. They finally caved to reality with 40GbE, 25GbE, and 50GbE. They eventually ratified 2.5GbE, but it was too late for network drives. The transfer rate of an HDD is well over 1Gbps.

oarsinsync 7y ago
> In their infinite wisdom, ethernet decided there was no need for speeds between 1GbE and 10GbE. They finally caved to reality with 40GbE, 25GbE, and 50GbE. I’m not sure that’s accurate. There were no intermediate steps to get to 10GbE, it was a straight jump from 1GbE and it was relatively easy. To get to 40GbE, it involved running four lanes of 10GbE. To get to 100GbE, it involved running four lanes of 25GbE. 50GbE is just two lanes of 25GbE. We’ve gone from a single serialised stream to multiple parallel streams in order to reach next order speeds. This is magnified when you start looking at 400GbE and 800GbE services that operate on 8 lanes (QSFP-DD or the confusingly named OSFP).
- jdnenej 7y ago
  10gbit networking gear is massively more expensive than 1gbit
  - amiga-workbench 7y ago
    Its finally beginning to fall in price, Microtik are doing relatively inexpensive routers with 10gig SFP ports.
    - gitgudnubs 7y ago
      If I were going to sell a network attached spinning disk at scale, then I would probably target 2.5GBASE-T. The problem there is that no routers support it.
- gitgudnubs 7y ago
  >it was a straight jump from 1GbE and it was relatively easy. It wasn't relatively easy, because you can't run 10GBASE-T over Cat5e, and because a 10GbE NIC was as expensive as the rest of the drive put together. Today you can run 2.5GBASE-T over Cat5e, but the standard was too late, and there was very little demand between 1GbE and 10GbE, so there are virtually no controllers. What controllers you can find are usually 10GbE controllers that can also do 2.5GbE. Until 2Gbps home internet and LANs become popular, there's no reason to expect change. >There were no intermediate steps to get to 10GbE That's the problem. Disk transfer rates can approach 1.5Gbps, so a 1GbE is a serious bottleneck. But 10 GbE hardware was significantly more expensive. What am I going to do, build a JBOD with a dozen disks and 1x10GbE, or put a dozen disks onto the network with 10GbE interfaces? So instead you need 2x1GbE on every disk, which complicates management and doubles cabling, switches, and cost. >We’ve gone from a single serialised stream to multiple parallel streams in order to reach next order speeds. Which is hilarious when you consider PATA.
MertsA 7y ago
3 years back WD put out a special variant of their Ultrastar He8 drives that ran a Ceph OSD on the drive itself that supported 2.5 Gbe to each drive over a SATA port with an odd and incompatible pinout. https://ceph.com/geen-categorie/500-osd-ceph-cluster/ https://ceph.com/geen-categorie/500-osd-ceph-cluster/ One thing to note, in that test they were using a chassis that limited the drives to two 1 Gbe connections but that's not a limitation of the drives. Now that WDLabs has fizzled out who knows what became of that platform, it was actually quite interesting but it wasn't really directly using the drive to natively support object storage, internally it was still presenting itself as a block device to the Ceph daemon running on it.
- gitgudnubs 7y ago
  Exactly. It's a beautiful concept: the server is roughly half the TCO per gigabyte. That can decrease significantly with better architectures, as shown by this. But making it 2x1GbE is a huge complicating factor. And 2.5GbE is way out of the cost curve per bps.

oarsinsync 7y ago

> In their infinite wisdom, ethernet decided there was no need for speeds between 1GbE and 10GbE. They finally caved to reality with 40GbE, 25GbE, and 50GbE. I’m not sure that’s accurate. There were no intermediate steps to get to 10GbE, it was a straight jump from 1GbE and it was relatively easy. To get to 40GbE, it involved running four lanes of 10GbE. To get to 100GbE, it involved running four lanes of 25GbE. 50GbE is just two lanes of 25GbE. We’ve gone from a single serialised stream to multiple parallel streams in order to reach next order speeds. This is magnified when you start looking at 400GbE and 800GbE services that operate on 8 lanes (QSFP-DD or the confusingly named OSFP).

jdnenej 7y ago
10gbit networking gear is massively more expensive than 1gbit
- amiga-workbench 7y ago
  Its finally beginning to fall in price, Microtik are doing relatively inexpensive routers with 10gig SFP ports.
  - gitgudnubs 7y ago
    If I were going to sell a network attached spinning disk at scale, then I would probably target 2.5GBASE-T. The problem there is that no routers support it.
gitgudnubs 7y ago
>it was a straight jump from 1GbE and it was relatively easy. It wasn't relatively easy, because you can't run 10GBASE-T over Cat5e, and because a 10GbE NIC was as expensive as the rest of the drive put together. Today you can run 2.5GBASE-T over Cat5e, but the standard was too late, and there was very little demand between 1GbE and 10GbE, so there are virtually no controllers. What controllers you can find are usually 10GbE controllers that can also do 2.5GbE. Until 2Gbps home internet and LANs become popular, there's no reason to expect change. >There were no intermediate steps to get to 10GbE That's the problem. Disk transfer rates can approach 1.5Gbps, so a 1GbE is a serious bottleneck. But 10 GbE hardware was significantly more expensive. What am I going to do, build a JBOD with a dozen disks and 1x10GbE, or put a dozen disks onto the network with 10GbE interfaces? So instead you need 2x1GbE on every disk, which complicates management and doubles cabling, switches, and cost. >We’ve gone from a single serialised stream to multiple parallel streams in order to reach next order speeds. Which is hilarious when you consider PATA.

MertsA 7y ago

3 years back WD put out a special variant of their Ultrastar He8 drives that ran a Ceph OSD on the drive itself that supported 2.5 Gbe to each drive over a SATA port with an odd and incompatible pinout. https://ceph.com/geen-categorie/500-osd-ceph-cluster/ https://ceph.com/geen-categorie/500-osd-ceph-cluster/ One thing to note, in that test they were using a chassis that limited the drives to two 1 Gbe connections but that's not a limitation of the drives. Now that WDLabs has fizzled out who knows what became of that platform, it was actually quite interesting but it wasn't really directly using the drive to natively support object storage, internally it was still presenting itself as a block device to the Ceph daemon running on it.

gitgudnubs 7y ago
Exactly. It's a beautiful concept: the server is roughly half the TCO per gigabyte. That can decrease significantly with better architectures, as shown by this. But making it 2x1GbE is a huge complicating factor. And 2.5GbE is way out of the cost curve per bps.