4 ms·

I have one of these in a new Ryzen 5000 build (that also has 3200Mhz ECC RAM! In a workstation!). The latency and throughput are impressive. What a time to be a

I have one of these in a new Ryzen 5000 build (that also has 3200Mhz ECC RAM! In a workstation!). The latency and throughput are impressive. What a time to be alive where SSDs are doing 5-8 GB/s with low double digit microseconds. At home the bottleneck is obviously my network, but even in a data center you’d need 100Gb networking in order for the network to not be the bottleneck. I really hope these hardware trends continue and we start to see a paradigm shift to more local processing/storage.

Railsify 6y ago
10G is quite affordable these days. I'm working on 10G LAN with 3 R430s I recently purchased on ebay.
ComputerGuru 6y ago
This isn’t the right drive for that, though. Outside of provisioned storage for VMs, these SSDs are simply too small to have much use in a network setting.
stordoff 6y ago
> At home the bottleneck is obviously my network I've started to upgrade parts of my network to 10Gbps, and it was surprisingly affordable. My NAS and server now have a dedicated 10GB Gbps link between them (the NAS also has 4 1Gbps connections to my primary network), and it only cost about £50. Obviously it still could be a bottleneck, but it's a decent step in the right direction.
- ngcc_hk 6y ago
  I did it one year ago and have to turned it off to use trunk 4 gbps ... too noisy. Except macmini and the nas ... the router and switch just annoying noise.
  - Daho0n 6y ago
    What are you on about? Did you buy something with a fan?
- rafaelm 6y ago
  I've been looking at doing this at home with the Mikrotik CRS305-1G-4S+IN. I honestly don't have a need for it right now, but just knowing that it's possible makes me want to go for it...
  - MrFoof 6y ago
    I just started doing a large overhaul (first in 10 years), and moving to 10GbE was a pretty big deal. When your storage is fast, waiting for the network is the real bottleneck. Uploading things on a WAN link (I have 1Gbps down, but only 40-50Mbps up) is the new drag. For video editing I can use the SSDs on the workstation, though I can also saturate the 10GbE connection to an iSCSI target on a Synology full of hard disks in RAID 10. Want to provision a new virtual machine? I have some 1L form factor computers (Some 175x175x35mm HPs with 8-core Ryzen 7 PRO APUs that sip 12-15W at idle) in an XCP-NG cluster. Build an image and provision it quickly. I'm a bit limited here as USB 3 NICs tend to cap out at 3.5Gbps because the top of the controller's block diagram is a 3.2 Gen 1, but when USB Type-C Gen 2 NICs hit they'll rip along at around 7-8Gbps, pulling large built images down from the Synology and writing it to their SSDs (980 Pros) in well under 10 seconds. Scary stuff though is the virtualized Spark cluster running on my workstation. Four Intel D7-P5510s in RAID10. With ~25GB/sec of sequential reads you're not held up by storage anymore. Just the network when you want to write out results, and any suboptimal code to ensure you don't starve the CPU cores because you're not ripping through memory fast enough. The fast new reality is great. Join in. :)
  - katbyte 6y ago
    Switched my core network for 10g last summer with no regrets, can imagine life without it now tbh
    - Hamuko 6y ago
      For what cost?
      - moonbug 6y ago
        hardly anything, old mellanox NICs are practically free on eBay, DACs and 10g optics are a pittance. Even DC 10g switching is cheap. Franky, 100ge is within home budgets now.
- jlgaddis 6y ago
  I'm right in the middle of a huge home network/storage upgrade myself, although I'm making a slightly bigger jump. I've got a fairly extensive home lab network but I've been "stuck" at 1 GbE for several years now, mostly because I couldn't justify the cost of upgrading to 10 GbE. See, I'd need a switch with a minimum of five 10 GbE ports -- just for the FreeNAS box, my workstation, and three of the VMware nodes; if I were gonna make the jump to 10 GbE, I'd want to upgrade those machines, at the least. A couple years ago I bought a pair of cheap, old Mellanox 10 GbE cards and a short direct-attach cable (DAC) to directly connect the FreeNAS machine and the "giant" VMware box (which has roughly the same "horsepower" of three of the others combined!) that I ran most things on. For the most part, everything else has a single 1 GbE link, except for my workstation (2 x 1 GbE) and the other VMware servers (4 x 1 GbE). Anyways, I recently started thinking about upgrading and/or replacing some things. While doing my research, I came across recommendations for some specific (enterprise) network gear that -- having been mostly a Cisco / Juniper guy for the last 15 years or so -- I wouldn't have normally considered (and, no, MikroTik is NOT an option, for several reasons). I also learned a few "interesting" pieces of information, like how to magically turn some specific (relatively cheap) 10 GbE NICs into (not so cheap!) 40 GbE NICs just by flashing different firmware on them. I also ended up getting a set of server rails and a small box full of dual-port 10 GbE NICs "free" when I went to buy the set of rails from a guy off Craigslist. They were Broadcom NICs but they'll work just fine in the other VMware servers! So, by the time I'm done with everything (I'm also in the process of building a new room in the basement just for the lab), my FreeNAS box and workstation will both be connected at 40 GbE and the half-dozen or so other servers will all be connected at 10 GbE (although, to be honest, I will very likely end up connecting the second 10 GbE ports on them all, too; I'll have enough 10 GbE ports left on the switch.). Before any asks, I'm not sure how much I've spent on the upgrade (and I'm halfway afraid to total it all up, honestly!). I've almost certainly spent more than I had originally planned when first starting to think about doing all of this but: 1) the additional cost to go to 40 GbE on the storage box and my workstation was really that much more so it was easy to justify, 2) once I'm all done with everything, I'm gonna get rid of the "giant" VMware box and, quite likely, a few of the others -- and maybe even some of the other things I've forgotten I have and/or don't use anymore; that should make the total overall cost fairly neglible, and 3) I've got 40 gigabit at my house! Of course, at this point, the storage server won't be able to come close to saturating the 40 GbE link... which means I'll need to add more, faster storage... rinse and repeat, it never ends! Just say no to home labs, kids!
p1mrx 6y ago
> 3200Mhz ECC RAM! In a workstation! If 3200 is your RAM's stock speed, you can probably run it at least 10% faster using "DRAM Calculator for Ryzen".
- GordonS 6y ago
  Not guaranteed tho. I have a Ryzen 2600 with an Asus B450-Plus motherboard, and recently tried to upgrade to 3200MHz DDR4 (from 2666MHz) - I used the DRAM calculator (which is quite confusing TBH), but no matter I tried, it wouldn't even boot at anything higher than 2666MHz @CL16. A bit disappointing, but I still ended up with more RAM overall, so not a disaster.
baruch 6y ago
I work on an SSD storage system with multiple SSDs in any server and 100Gbps NICs are the low end for us. We see significant use for 200 Gbps.

Railsify 6y ago

10G is quite affordable these days. I'm working on 10G LAN with 3 R430s I recently purchased on ebay.

ComputerGuru 6y ago

This isn’t the right drive for that, though. Outside of provisioned storage for VMs, these SSDs are simply too small to have much use in a network setting.

stordoff 6y ago

> At home the bottleneck is obviously my network I've started to upgrade parts of my network to 10Gbps, and it was surprisingly affordable. My NAS and server now have a dedicated 10GB Gbps link between them (the NAS also has 4 1Gbps connections to my primary network), and it only cost about £50. Obviously it still could be a bottleneck, but it's a decent step in the right direction.

ngcc_hk 6y ago
I did it one year ago and have to turned it off to use trunk 4 gbps ... too noisy. Except macmini and the nas ... the router and switch just annoying noise.
- Daho0n 6y ago
  What are you on about? Did you buy something with a fan?
rafaelm 6y ago
I've been looking at doing this at home with the Mikrotik CRS305-1G-4S+IN. I honestly don't have a need for it right now, but just knowing that it's possible makes me want to go for it...
- MrFoof 6y ago
  I just started doing a large overhaul (first in 10 years), and moving to 10GbE was a pretty big deal. When your storage is fast, waiting for the network is the real bottleneck. Uploading things on a WAN link (I have 1Gbps down, but only 40-50Mbps up) is the new drag. For video editing I can use the SSDs on the workstation, though I can also saturate the 10GbE connection to an iSCSI target on a Synology full of hard disks in RAID 10. Want to provision a new virtual machine? I have some 1L form factor computers (Some 175x175x35mm HPs with 8-core Ryzen 7 PRO APUs that sip 12-15W at idle) in an XCP-NG cluster. Build an image and provision it quickly. I'm a bit limited here as USB 3 NICs tend to cap out at 3.5Gbps because the top of the controller's block diagram is a 3.2 Gen 1, but when USB Type-C Gen 2 NICs hit they'll rip along at around 7-8Gbps, pulling large built images down from the Synology and writing it to their SSDs (980 Pros) in well under 10 seconds. Scary stuff though is the virtualized Spark cluster running on my workstation. Four Intel D7-P5510s in RAID10. With ~25GB/sec of sequential reads you're not held up by storage anymore. Just the network when you want to write out results, and any suboptimal code to ensure you don't starve the CPU cores because you're not ripping through memory fast enough. The fast new reality is great. Join in. :)
- katbyte 6y ago
  Switched my core network for 10g last summer with no regrets, can imagine life without it now tbh
  - Hamuko 6y ago
    For what cost?
    - moonbug 6y ago
      hardly anything, old mellanox NICs are practically free on eBay, DACs and 10g optics are a pittance. Even DC 10g switching is cheap. Franky, 100ge is within home budgets now.
jlgaddis 6y ago
I'm right in the middle of a huge home network/storage upgrade myself, although I'm making a slightly bigger jump. I've got a fairly extensive home lab network but I've been "stuck" at 1 GbE for several years now, mostly because I couldn't justify the cost of upgrading to 10 GbE. See, I'd need a switch with a minimum of five 10 GbE ports -- just for the FreeNAS box, my workstation, and three of the VMware nodes; if I were gonna make the jump to 10 GbE, I'd want to upgrade those machines, at the least. A couple years ago I bought a pair of cheap, old Mellanox 10 GbE cards and a short direct-attach cable (DAC) to directly connect the FreeNAS machine and the "giant" VMware box (which has roughly the same "horsepower" of three of the others combined!) that I ran most things on. For the most part, everything else has a single 1 GbE link, except for my workstation (2 x 1 GbE) and the other VMware servers (4 x 1 GbE). Anyways, I recently started thinking about upgrading and/or replacing some things. While doing my research, I came across recommendations for some specific (enterprise) network gear that -- having been mostly a Cisco / Juniper guy for the last 15 years or so -- I wouldn't have normally considered (and, no, MikroTik is NOT an option, for several reasons). I also learned a few "interesting" pieces of information, like how to magically turn some specific (relatively cheap) 10 GbE NICs into (not so cheap!) 40 GbE NICs just by flashing different firmware on them. I also ended up getting a set of server rails and a small box full of dual-port 10 GbE NICs "free" when I went to buy the set of rails from a guy off Craigslist. They were Broadcom NICs but they'll work just fine in the other VMware servers! So, by the time I'm done with everything (I'm also in the process of building a new room in the basement just for the lab), my FreeNAS box and workstation will both be connected at 40 GbE and the half-dozen or so other servers will all be connected at 10 GbE (although, to be honest, I will very likely end up connecting the second 10 GbE ports on them all, too; I'll have enough 10 GbE ports left on the switch.). Before any asks, I'm not sure how much I've spent on the upgrade (and I'm halfway afraid to total it all up, honestly!). I've almost certainly spent more than I had originally planned when first starting to think about doing all of this but: 1) the additional cost to go to 40 GbE on the storage box and my workstation was really that much more so it was easy to justify, 2) once I'm all done with everything, I'm gonna get rid of the "giant" VMware box and, quite likely, a few of the others -- and maybe even some of the other things I've forgotten I have and/or don't use anymore; that should make the total overall cost fairly neglible, and 3) I've got 40 gigabit at my house! Of course, at this point, the storage server won't be able to come close to saturating the 40 GbE link... which means I'll need to add more, faster storage... rinse and repeat, it never ends! Just say no to home labs, kids!

p1mrx 6y ago

> 3200Mhz ECC RAM! In a workstation! If 3200 is your RAM's stock speed, you can probably run it at least 10% faster using "DRAM Calculator for Ryzen".

GordonS 6y ago
Not guaranteed tho. I have a Ryzen 2600 with an Asus B450-Plus motherboard, and recently tried to upgrade to 3200MHz DDR4 (from 2666MHz) - I used the DRAM calculator (which is quite confusing TBH), but no matter I tried, it wouldn't even boot at anything higher than 2666MHz @CL16. A bit disappointing, but I still ended up with more RAM overall, so not a disaster.

baruch 6y ago

I work on an SSD storage system with multiple SSDs in any server and 100Gbps NICs are the low end for us. We see significant use for 200 Gbps.