3 ms·

IMO one of the biggest frustration I currently have with homelab-scale gear (and edge compute in general) is how everything has massively regressed to only offe

IMO one of the biggest frustration I currently have with homelab-scale gear (and edge compute in general) is how everything has massively regressed to only offering 2.5gbit connectivity at best. Try to find one of these hundreds of small form factor products that has anything faster than a 2.5Gbit nic and despair! What good are these machines if you cant even get data in and out of them?! The list of hardware with 2x10GBe or better connectivity is amazingly short: - Minisforum MS-01 - GoWin RS86S*, GW-BS-1U-* - Quotom Q20331G\* From major manufacturers, only the following: - Mac Mini (1x10gbaseT option, ARM, ) - HPE Z2 Mini\* (1x10GbaseT option) Please someone, anyone, build a SFF box with the following specs: Minimum of 2 NVMe Minimum of 2 additional storage devices (SATA or more NVMe) Minimum of 2 10GbE Dedicated or shared IMPI/OOBM Minimum of 8 Core x86 CPU Minimum of 64GB RAM Make sure the nics have enough PCIe lanes to hit their throughput limits, give the rest to storage. Stop with the proliferation of USB and thunderbolt and 5 display outputs on these things, please already!

Borealid 3y ago
I understand that more is better in some sense, but why the emphasis on having two (2) 10GbE instead of one for a single device that isn't a router? What's the second link for? Just more bandwidth, or is there some crucial use case I'm missing?
- gorkish 3y ago
  In my experience, a second nic is important to build anything with resilience; it's not just for routing. I see it as the most important hardware to "double up" on. If you are building a small cluster, for instance, you generally want to have one interface for the backend interconnect and one interface for client traffic. Or maybe you want to be able to upgrade or change your networking without taking traffic offline -- you'll need two physical connections for that, too. Of course you can still build routers and everything with a single interface (which is why I noted the single port hardware in the first place), but if you are going to design something specific to the mini-server market, the cost difference between a single port and dual port controller is negligible enough that there's no reason not to include 2 or even 4 ports. The GoWin machines simply toss in an OCP nic slot and call it a day. That would be a fantastic solution to see adopted by other vendors.
- jmbwell 3y ago
  If a home lab is for tinkering, it’s something to play with first of all, but it also helps facilitate re-patching things or moving things around, which in many cases you can do one port/cable at a time without interrupting services. I once moved a server from one room to another by moving its redundant power cables one at a time to a UPS on a cart, then swapping its short network cables for longer ones one a time, wheeling the thing to the other room, then “reversing” the procedure in the new location. Zero downtime! Zero point really, but fun in the way I reckon a home lab is meant to be.
CharlesW 3y ago
> Stop with the proliferation of USB and thunderbolt… If you don't need it built-in, external adapters are a good way to get to 10GbE and even 25GbE. https://www.qnap.com/en-us/product/qna-t310g1s https://www.qnap.com/en-us/product/qna-t310g1s https://www.sonnettech.com/product/thunderbolt/ethernet-networking.html https://www.sonnettech.com/product/thunderbolt/ethernet-netw...
- gorkish 3y ago
  By "good" I assume you mean "possible" because there is nothing really good about a connectivity solution that is both technically inferior and vastly more expensive than it ought to be. I'm not opposed to Thunderbolt though; let me know when someone ships a 8+ port thunderbolt interdomain/multi-root PCIe switch. That would be a good solution.
skazazes 3y ago
You should check out the Xeon D offerings from Supermicro. They make some M-ITX motherboards with embedded Xeon D chips that sport dual 10GB Ethernet. Used one in a freenas system at one point for years Looks like the X10SDV-TLN4F[1] has everything you mentioned besides a second m.2 slot. Although its 8 cores they are a few generations old and low power (45w TDP on x86). For true compute its not exactly fast but for something like a high performance file server in 1U connected to a disk shelf, they are really nice. [1] https://www.supermicro.com/en/products/motherboard/x10sdv-tln4f https://www.supermicro.com/en/products/motherboard/x10sdv-tl...
- gorkish 3y ago
  I actually do run a production edge cluster with these exact boards; they are just too old. The price/performance is so far off the mark WRT modern hardware they are simply not worth using anymore. I have started hitting CPU instruction set issues with current software due to the ancient microarchitectures in these lowend parts. On one hand they are able to still sell them because nobody is competing, but what is the reason for the lack of competition? I have a theory about Broadcom buying and blocking up the low end PCIe switch market to purposely hold back this segment, but I'm not sure how significant that is.
- jmbwell 3y ago
  I use one of these. For a system that’s honestly mostly idle, it’s plenty for proxmox with Plex and pihole and whatever else, and it’s good on power and heat and noise. I made the mistake of getting Seagate 10TB rock tumblers, but they’re the only source of real noise. WD would probably have been quieter. I have this and my switches and a ham radio base station all racked up on top of the cabinets in my laundry/utility room, out of the way but not hidden away, so I can still notice if the fans are too loud or the wrong lights are blinking or not blinking or there’s weird smells etc.

Borealid 3y ago

I understand that more is better in some sense, but why the emphasis on having two (2) 10GbE instead of one for a single device that isn't a router? What's the second link for? Just more bandwidth, or is there some crucial use case I'm missing?

gorkish 3y ago
In my experience, a second nic is important to build anything with resilience; it's not just for routing. I see it as the most important hardware to "double up" on. If you are building a small cluster, for instance, you generally want to have one interface for the backend interconnect and one interface for client traffic. Or maybe you want to be able to upgrade or change your networking without taking traffic offline -- you'll need two physical connections for that, too. Of course you can still build routers and everything with a single interface (which is why I noted the single port hardware in the first place), but if you are going to design something specific to the mini-server market, the cost difference between a single port and dual port controller is negligible enough that there's no reason not to include 2 or even 4 ports. The GoWin machines simply toss in an OCP nic slot and call it a day. That would be a fantastic solution to see adopted by other vendors.
jmbwell 3y ago
If a home lab is for tinkering, it’s something to play with first of all, but it also helps facilitate re-patching things or moving things around, which in many cases you can do one port/cable at a time without interrupting services. I once moved a server from one room to another by moving its redundant power cables one at a time to a UPS on a cart, then swapping its short network cables for longer ones one a time, wheeling the thing to the other room, then “reversing” the procedure in the new location. Zero downtime! Zero point really, but fun in the way I reckon a home lab is meant to be.

CharlesW 3y ago

> Stop with the proliferation of USB and thunderbolt… If you don't need it built-in, external adapters are a good way to get to 10GbE and even 25GbE. https://www.qnap.com/en-us/product/qna-t310g1s https://www.qnap.com/en-us/product/qna-t310g1s https://www.sonnettech.com/product/thunderbolt/ethernet-networking.html https://www.sonnettech.com/product/thunderbolt/ethernet-netw...

gorkish 3y ago
By "good" I assume you mean "possible" because there is nothing really good about a connectivity solution that is both technically inferior and vastly more expensive than it ought to be. I'm not opposed to Thunderbolt though; let me know when someone ships a 8+ port thunderbolt interdomain/multi-root PCIe switch. That would be a good solution.

skazazes 3y ago

You should check out the Xeon D offerings from Supermicro. They make some M-ITX motherboards with embedded Xeon D chips that sport dual 10GB Ethernet. Used one in a freenas system at one point for years Looks like the X10SDV-TLN4F[1] has everything you mentioned besides a second m.2 slot. Although its 8 cores they are a few generations old and low power (45w TDP on x86). For true compute its not exactly fast but for something like a high performance file server in 1U connected to a disk shelf, they are really nice. [1] https://www.supermicro.com/en/products/motherboard/x10sdv-tln4f https://www.supermicro.com/en/products/motherboard/x10sdv-tl...

gorkish 3y ago
I actually do run a production edge cluster with these exact boards; they are just too old. The price/performance is so far off the mark WRT modern hardware they are simply not worth using anymore. I have started hitting CPU instruction set issues with current software due to the ancient microarchitectures in these lowend parts. On one hand they are able to still sell them because nobody is competing, but what is the reason for the lack of competition? I have a theory about Broadcom buying and blocking up the low end PCIe switch market to purposely hold back this segment, but I'm not sure how significant that is.
jmbwell 3y ago
I use one of these. For a system that’s honestly mostly idle, it’s plenty for proxmox with Plex and pihole and whatever else, and it’s good on power and heat and noise. I made the mistake of getting Seagate 10TB rock tumblers, but they’re the only source of real noise. WD would probably have been quieter. I have this and my switches and a ham radio base station all racked up on top of the cabinets in my laundry/utility room, out of the way but not hidden away, so I can still notice if the fans are too loud or the wrong lights are blinking or not blinking or there’s weird smells etc.