6 ms·

> At home the bottleneck is obviously my network I've started to upgrade parts of my network to 10Gbps, and it was surprisingly affordable. My NAS and server n

> At home the bottleneck is obviously my network I've started to upgrade parts of my network to 10Gbps, and it was surprisingly affordable. My NAS and server now have a dedicated 10GB Gbps link between them (the NAS also has 4 1Gbps connections to my primary network), and it only cost about £50. Obviously it still could be a bottleneck, but it's a decent step in the right direction.

ngcc_hk 6y ago
I did it one year ago and have to turned it off to use trunk 4 gbps ... too noisy. Except macmini and the nas ... the router and switch just annoying noise.
- Daho0n 6y ago
  What are you on about? Did you buy something with a fan?
rafaelm 6y ago
I've been looking at doing this at home with the Mikrotik CRS305-1G-4S+IN. I honestly don't have a need for it right now, but just knowing that it's possible makes me want to go for it...
- MrFoof 6y ago
  I just started doing a large overhaul (first in 10 years), and moving to 10GbE was a pretty big deal. When your storage is fast, waiting for the network is the real bottleneck. Uploading things on a WAN link (I have 1Gbps down, but only 40-50Mbps up) is the new drag. For video editing I can use the SSDs on the workstation, though I can also saturate the 10GbE connection to an iSCSI target on a Synology full of hard disks in RAID 10. Want to provision a new virtual machine? I have some 1L form factor computers (Some 175x175x35mm HPs with 8-core Ryzen 7 PRO APUs that sip 12-15W at idle) in an XCP-NG cluster. Build an image and provision it quickly. I'm a bit limited here as USB 3 NICs tend to cap out at 3.5Gbps because the top of the controller's block diagram is a 3.2 Gen 1, but when USB Type-C Gen 2 NICs hit they'll rip along at around 7-8Gbps, pulling large built images down from the Synology and writing it to their SSDs (980 Pros) in well under 10 seconds. Scary stuff though is the virtualized Spark cluster running on my workstation. Four Intel D7-P5510s in RAID10. With ~25GB/sec of sequential reads you're not held up by storage anymore. Just the network when you want to write out results, and any suboptimal code to ensure you don't starve the CPU cores because you're not ripping through memory fast enough. The fast new reality is great. Join in. :)
- katbyte 6y ago
  Switched my core network for 10g last summer with no regrets, can imagine life without it now tbh
  - Hamuko 6y ago
    For what cost?
    - moonbug 6y ago
      hardly anything, old mellanox NICs are practically free on eBay, DACs and 10g optics are a pittance. Even DC 10g switching is cheap. Franky, 100ge is within home budgets now.
      - NullPrefix 6y ago
        Dollars, please express your opinions in monetary values.
        katbyte 6y ago
        i personally spent about 700$ - but i got a fancy managed switch. NICs for were about 200$
    - magicalhippo 6y ago
      I've got some Mellanox cards for about 20-30 USD each, some direct attach cable can be had for a few dollars as well. Later I picked up some cheap optical gear from FS[1] for when I moved my NAS into another room. About $20 for the transceivers and fiber cable was not expensive either. Regular 1GbE is sooo low now. Not got the switch yet, just running point to point. Using a different IP range so I'm sure I'm hitting that 10G goodness. [1]: https://www.fs.com https://www.fs.com
jlgaddis 6y ago
I'm right in the middle of a huge home network/storage upgrade myself, although I'm making a slightly bigger jump. I've got a fairly extensive home lab network but I've been "stuck" at 1 GbE for several years now, mostly because I couldn't justify the cost of upgrading to 10 GbE. See, I'd need a switch with a minimum of five 10 GbE ports -- just for the FreeNAS box, my workstation, and three of the VMware nodes; if I were gonna make the jump to 10 GbE, I'd want to upgrade those machines, at the least. A couple years ago I bought a pair of cheap, old Mellanox 10 GbE cards and a short direct-attach cable (DAC) to directly connect the FreeNAS machine and the "giant" VMware box (which has roughly the same "horsepower" of three of the others combined!) that I ran most things on. For the most part, everything else has a single 1 GbE link, except for my workstation (2 x 1 GbE) and the other VMware servers (4 x 1 GbE). Anyways, I recently started thinking about upgrading and/or replacing some things. While doing my research, I came across recommendations for some specific (enterprise) network gear that -- having been mostly a Cisco / Juniper guy for the last 15 years or so -- I wouldn't have normally considered (and, no, MikroTik is NOT an option, for several reasons). I also learned a few "interesting" pieces of information, like how to magically turn some specific (relatively cheap) 10 GbE NICs into (not so cheap!) 40 GbE NICs just by flashing different firmware on them. I also ended up getting a set of server rails and a small box full of dual-port 10 GbE NICs "free" when I went to buy the set of rails from a guy off Craigslist. They were Broadcom NICs but they'll work just fine in the other VMware servers! So, by the time I'm done with everything (I'm also in the process of building a new room in the basement just for the lab), my FreeNAS box and workstation will both be connected at 40 GbE and the half-dozen or so other servers will all be connected at 10 GbE (although, to be honest, I will very likely end up connecting the second 10 GbE ports on them all, too; I'll have enough 10 GbE ports left on the switch.). Before any asks, I'm not sure how much I've spent on the upgrade (and I'm halfway afraid to total it all up, honestly!). I've almost certainly spent more than I had originally planned when first starting to think about doing all of this but: 1) the additional cost to go to 40 GbE on the storage box and my workstation was really that much more so it was easy to justify, 2) once I'm all done with everything, I'm gonna get rid of the "giant" VMware box and, quite likely, a few of the others -- and maybe even some of the other things I've forgotten I have and/or don't use anymore; that should make the total overall cost fairly neglible, and 3) I've got 40 gigabit at my house! Of course, at this point, the storage server won't be able to come close to saturating the 40 GbE link... which means I'll need to add more, faster storage... rinse and repeat, it never ends! Just say no to home labs, kids!

ngcc_hk 6y ago

I did it one year ago and have to turned it off to use trunk 4 gbps ... too noisy. Except macmini and the nas ... the router and switch just annoying noise.

Daho0n 6y ago
What are you on about? Did you buy something with a fan?

rafaelm 6y ago

I've been looking at doing this at home with the Mikrotik CRS305-1G-4S+IN. I honestly don't have a need for it right now, but just knowing that it's possible makes me want to go for it...

MrFoof 6y ago
I just started doing a large overhaul (first in 10 years), and moving to 10GbE was a pretty big deal. When your storage is fast, waiting for the network is the real bottleneck. Uploading things on a WAN link (I have 1Gbps down, but only 40-50Mbps up) is the new drag. For video editing I can use the SSDs on the workstation, though I can also saturate the 10GbE connection to an iSCSI target on a Synology full of hard disks in RAID 10. Want to provision a new virtual machine? I have some 1L form factor computers (Some 175x175x35mm HPs with 8-core Ryzen 7 PRO APUs that sip 12-15W at idle) in an XCP-NG cluster. Build an image and provision it quickly. I'm a bit limited here as USB 3 NICs tend to cap out at 3.5Gbps because the top of the controller's block diagram is a 3.2 Gen 1, but when USB Type-C Gen 2 NICs hit they'll rip along at around 7-8Gbps, pulling large built images down from the Synology and writing it to their SSDs (980 Pros) in well under 10 seconds. Scary stuff though is the virtualized Spark cluster running on my workstation. Four Intel D7-P5510s in RAID10. With ~25GB/sec of sequential reads you're not held up by storage anymore. Just the network when you want to write out results, and any suboptimal code to ensure you don't starve the CPU cores because you're not ripping through memory fast enough. The fast new reality is great. Join in. :)
katbyte 6y ago
Switched my core network for 10g last summer with no regrets, can imagine life without it now tbh
- Hamuko 6y ago
  For what cost?
  - moonbug 6y ago
    hardly anything, old mellanox NICs are practically free on eBay, DACs and 10g optics are a pittance. Even DC 10g switching is cheap. Franky, 100ge is within home budgets now.
    - NullPrefix 6y ago
      Dollars, please express your opinions in monetary values.
      - katbyte 6y ago
        i personally spent about 700$ - but i got a fancy managed switch. NICs for were about 200$
  - magicalhippo 6y ago
    I've got some Mellanox cards for about 20-30 USD each, some direct attach cable can be had for a few dollars as well. Later I picked up some cheap optical gear from FS[1] for when I moved my NAS into another room. About $20 for the transceivers and fiber cable was not expensive either. Regular 1GbE is sooo low now. Not got the switch yet, just running point to point. Using a different IP range so I'm sure I'm hitting that 10G goodness. [1]: https://www.fs.com https://www.fs.com

jlgaddis 6y ago

I'm right in the middle of a huge home network/storage upgrade myself, although I'm making a slightly bigger jump. I've got a fairly extensive home lab network but I've been "stuck" at 1 GbE for several years now, mostly because I couldn't justify the cost of upgrading to 10 GbE. See, I'd need a switch with a minimum of five 10 GbE ports -- just for the FreeNAS box, my workstation, and three of the VMware nodes; if I were gonna make the jump to 10 GbE, I'd want to upgrade those machines, at the least. A couple years ago I bought a pair of cheap, old Mellanox 10 GbE cards and a short direct-attach cable (DAC) to directly connect the FreeNAS machine and the "giant" VMware box (which has roughly the same "horsepower" of three of the others combined!) that I ran most things on. For the most part, everything else has a single 1 GbE link, except for my workstation (2 x 1 GbE) and the other VMware servers (4 x 1 GbE). Anyways, I recently started thinking about upgrading and/or replacing some things. While doing my research, I came across recommendations for some specific (enterprise) network gear that -- having been mostly a Cisco / Juniper guy for the last 15 years or so -- I wouldn't have normally considered (and, no, MikroTik is NOT an option, for several reasons). I also learned a few "interesting" pieces of information, like how to magically turn some specific (relatively cheap) 10 GbE NICs into (not so cheap!) 40 GbE NICs just by flashing different firmware on them. I also ended up getting a set of server rails and a small box full of dual-port 10 GbE NICs "free" when I went to buy the set of rails from a guy off Craigslist. They were Broadcom NICs but they'll work just fine in the other VMware servers! So, by the time I'm done with everything (I'm also in the process of building a new room in the basement just for the lab), my FreeNAS box and workstation will both be connected at 40 GbE and the half-dozen or so other servers will all be connected at 10 GbE (although, to be honest, I will very likely end up connecting the second 10 GbE ports on them all, too; I'll have enough 10 GbE ports left on the switch.). Before any asks, I'm not sure how much I've spent on the upgrade (and I'm halfway afraid to total it all up, honestly!). I've almost certainly spent more than I had originally planned when first starting to think about doing all of this but: 1) the additional cost to go to 40 GbE on the storage box and my workstation was really that much more so it was easy to justify, 2) once I'm all done with everything, I'm gonna get rid of the "giant" VMware box and, quite likely, a few of the others -- and maybe even some of the other things I've forgotten I have and/or don't use anymore; that should make the total overall cost fairly neglible, and 3) I've got 40 gigabit at my house! Of course, at this point, the storage server won't be able to come close to saturating the 40 GbE link... which means I'll need to add more, faster storage... rinse and repeat, it never ends! Just say no to home labs, kids!