9 ms·

10Gb/s Ethernet: what I did to get it working in my home

232 points by gpjt 5mo ago

hapless 5mo ago
10g-base-T is a sick joke high latency, high error rates, and terrifying heat output from SFPs (which the author noted for himself) the only cat6 left in my home network is the link to verizon's ont, because in their infinite wisdom the ONLY connectivity offered was 10g-base-t
- zamadatix 5mo ago
  10GBase-T can still be nice if you're going to do native PoE (which the author did not) or you expect a mix of devices with 1/2.5/5 which you don't want to or can't upgrade/replace immediately (which is where it sounds like the author was situated). The new 10GBASE-T SFPs are actually not too bad - you can get the full 100 meters at half the wattage it takes for the old space heater generation ones to reach 30 meters. Based on the article, the author did not know there were newer cooler options for the ~the same price.
mikepurvis 5mo ago
That's pretty wild. I have 1.5/900 fibre to my house, and I bring a 2.5 line from the modem to my home office where a 2.5 switch delivers it to my workstation, laptop, and unraid NAS. But those devices are all themselves just gigE I think, and I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.
- rhplus 5mo ago
  10Gbps is enough bandwidth for 500 concurrent Netflix streams in 4K/UHD (15Mbps) AND 500 concurrent video calls (4Mbps). Home users don’t need more bandwidth to improve their internet experiences, they need lower latency, less congestion and less loss. https://help.netflix.com/en/node/306 https://help.netflix.com/en/node/306 https://learn.microsoft.com/en-us/microsoftteams/prepare-network https://learn.microsoft.com/en-us/microsoftteams/prepare-net...
  - wpm 5mo ago
    Home users only do video calls and watch Netflix? More and more regular people are getting network storage appliances. More and more people have laptops with SSDs that can write at 4 or 5 GB/s. Why shouldn't they get to use all of it?
    - afavour 5mo ago
      To quote the previous post: > I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available. There are many things along the way that would get in the way of a home user downloading something from the internet that would hit that 5GB/s speed. It's not that people should be "banned" from it or something, more that the investment cost isn't worth it.
      - mlyle 5mo ago
        We have a 5gbps pipe; routinely download games from Steam at >3gbps; when I had to reinitialize my cloud backup it was >4gbps. All of this without impacting anyone else on the pipe. Yah, our P95 bandwidth is just a few megabits per second. But it's not that expensive and routinely saves me a few minutes here and there. 10gbps on the LAN is more broadly useful. Pegging it for a file share is a daily occurrence.
        ProfessorLayton 5mo ago
        Also storage has gotten super expensive lately, and rather than upgrading my machines/consoles I've been offloading games and downloading them as needed and now am routinely downloading dozens of GB just to play a game. My gaming time is limited so the faster the better.
      - fmajid 5mo ago
        I regularly saturate my 1G home and 1G office connection syncing ~6GB files between the two. It's also nice to be able to download a 100G or so game quickly. Remote backups to cloud storage also benefit from fast upload speeds (and more importantly, restores).
- IshKebab 5mo ago
  Yeah I foolishly paid for 500 Mb/s but the only things I ever get over even 200 Mb/s for in the UK are Steam downloads (and speed tests). Everything else seems to be roughly throttled at that rate.
  - mikepurvis 5mo ago
    Telus gave me a good price and I'd already invested in a Unifi gateway that could accept the ONT directly, so that was fun to play with.
    - lostlogin 5mo ago
      How else would you connect? Some ISP box? Sorry, I’m not familiar with any other way.
      - mikepurvis 5mo ago
        Yeah the default is they supply a "modem" with a regular ethernet port, so a familiar config to someone coming from cable or DSL, but inside it's a QSFP that you can move to your own equipment if you want.
- zamadatix 5mo ago
  I regularly hit 4.7 gbps on a 5 Gbps line pulling files (usenet is usually faster than torrent, but the latter can be equally as fast depending on the torrent & how good the client software is). It's great to just grab an entire movie series in 4k Blu Ray remux quality in 5 minutes and go. No real need to plan ahead for anything.
  - HDBaseT 5mo ago
    Yeah, Usenet is crazy quick. I've saturated 10Gbps on Usenet before, pretty easily. Torrents (as you mention) can be equally as fast. Well seeded, private tracker torrents can easily download at above 1Gbps (depending on piece size, etc).
    - lostlogin 5mo ago
      Can’t get much over a gigabit here I’d nz.
- throwforfeds 5mo ago
  And here I am sitting in Brooklyn and haven't had one apartment that has had fiber as an option. I get to pay Spectrum $90/month for "400/20" and in reality get 100/10.
  - fuomag9 5mo ago
    I was paying 21.9€ per month to TIM for a 10/2Gbit connection Welcome to Italy :D
    - throwforfeds 5mo ago
      But Brooklyn is an old city that's hard to wire, unlike those new Italian cities... /s
rrevi 5mo ago
The device nomenclature alone is worth the read! (otherwise an impressive feat to see)
apelapan 5mo ago
I might have been lucky, but in the one home and one office were I've connected 10gbit switches and PCIe cards, it has just worked. Especially the office was a nice surprise, because it is at least 20 meters (probably more) of unknown cabling and at least one unknown patch panel between the utility closet where the NAS lives and the desk area. The cables were run 15 years ago, so I expected it to be cat 5, but clearly not. It is nice moving/streaming large files across the network at 10 gbit. It really is ten times less waiting than with plain old gigabit. Of course, most of the time I'm working with lots of small files and then the spinning disk array in the NAS has no chance to saturated the this giant pipe, or even a normal gigabit connection...
- saltcured 5mo ago
  I know when I was doing some custom wiring in a house around 2005-6, it was clear that cat-6e was the thing to use if you wanted any future-proofing. So I bought a reel of that even though I was only going to be using 1000-BaseT. I don't remember there being too much premium on the wire itself.
  - loeg 5mo ago
    Do you mean Cat 6 or Cat 5e? Cat 6e isn't a thing and 6a didn't exist in 2006. 6 was certainly more future-proof at the time, although arguably 5e is still fine even today. (Super-gigabit consumer equipment didn't really exist until the last five years and it's till notably more expensive and less common than gigabit, which runs on 5e just fine.)
    - saltcured 5mo ago
      Cat 6 for sure. I thought there was some extra tweak beyond that, but I probably misremembered after all this time. Perhaps it was just that it was plenum-rated..
      - globular-toast 5mo ago
        You might be thinking of cat6a which officially supports 10G over long lengths, unlike cat6.
afavour 5mo ago
Both impressive and surprising that thermals were the biggest barrier! Meanwhile I'm sat here wishing I could justify running any ethernet in my apartment, but improving wi-fi tech means I never can...
- jauntywundrkind 5mo ago
  I wonder what the idle vs at load difference is for power draw/heat. Would really love to see this feature in reviews some! I wonder if you could negotiate down to 1gbit until you see some level of activity, if that would help at all? I'm still eyeing 10Gb, but if my home needs +30w for three computers, I don't feel like it's really worth it. Would love to see more details on the power consumption from folks, especially tuning for idle.
Boss0565 5mo ago
What's the point?
- snihalani 5mo ago
  for the funs
gigel82 5mo ago
I was surprised that the old Cat5e in my home supports 10Gbps without any issue, so went ahead and upgraded the rest of the network with 10Gbps switches (expensive Ubiquiti gear, but worth it to talk at 10Gbps between all my machines, even though the internet is only 5Gbps Fiber).
- rconti 5mo ago
  Yeah, my home is cat5e (I think!) and I was only able to get 2.5 or 5Gbps to my _single_ 10GbE capable machine I had at the time. So I ended up just buying a switch that supports 2.5Gbps over copper since it's such a huge expense to step up. Anyway, my devices are pretty much 2.5Gbps (APs, flex mini switches, etc).
TexanFeller 5mo ago
I'm extremely happy after upgrading my network to 10gbit copper ethernet. It was much more expensive than I thought it should be, but worth it even if I only max it occasionally. Now I can easily fully saturate my 10gbit ethernet doing a first Time Machine backup or transferring files to my M.2 SSD NAS which saves me waiting rime and is satisfying to watch. It's wild to me that 10gbit isn't the norm by now and tech people who should know better seem to think WiFi matches or even exceeds even 1gbit ethernet. My MBP connects to my WiFi7 setup(Ubiquiti E7) at a nominal 1.5-1.9gbit but Time Machine backups and file transfers are slower than plugging into 1gbit ethernet, probably in large part due to latency and retransmissions. Not to mention that ethernet works with near 100% reliability with dramatically less variation in speed and error rate.
- ocdtrekkie 5mo ago
  Heh it's honestly wild to me anyone needs over a gig. My work has a one gig fiber line supporting hundreds of employees and usage generally remains below 10%. The high expense of 10gig is, in part, because it isn't widely necessary and the people buying it are willing to pay extra.
  - godzillabrennus 5mo ago
    I put 5Gbit internet into my home (fiber) to build my startup. I'm processing terabytes of data. I have over 100TB of storage in my basement. I can regularly saturate my internet connection. That said, I remember well when a 1Gbit connection provided enough bandwidth for a 500-person call center for daily workloads (back about 10 years ago).
    - ocdtrekkie 5mo ago
      That puts you in an extreme minority, even amongst enterprise businesses. Many medium sized enterprises have storage that looks like "a couple dozen TB total" for hundreds of staff. Having 100 TB of storage in your home basement is an even more extreme minority than that. ;) A gigabit connection is more than enough for a 500-person call center today.
cyberax 5mo ago
The Mikrotik switch is awesome, and it's still the most compact 10G switch available. You can fix the thermal issue either by adding a small fan (Noctua is great) or by adding more radiators: https://pics.ealex.net/share/UxeSf_AWHLIuc-qzK5zl7JIgQvQDAZhy4-8sseK5bNVYVR7M83lxkBnXejQI___-dEw https://pics.ealex.net/share/UxeSf_AWHLIuc-qzK5zl7JIgQvQDAZh... I've been running it like this in a closed comm box for the last 3 years without any issues. SFP+ modules actually do not use that much power, it's just that it's concentrated into a small package, resulting in high temps.
- bot403 5mo ago
  Cripes. When possible just do fiber and DACs. Faster and much cooler than 10Gbit. 10Gbit uses an absurd amount of power per port thus the need for all that cooling.
  - m463 5mo ago
    yeah, the article though says... "The apartment has structured cabling -- each room has one or more RJ45 sockets in the wall," ... Which is the main problem most folks face. wish the standard was "conduit" instead of "bake-this-years-tech-into-the-wall" which doesn't always last...
    - ssl-3 5mo ago
      Folks have been saying that conduit is the way, and the fiber is the future, and all kinds of things like that for decades so far. But the simple truth for all those decades is this: When there's already cat-whatever cable in the wall, it generally still works. Decently-installed conduit (ie, actually-usable conduit) adds a ton of time and expense, which is why it is very seldom used for data circuits in residential structures. The cable that exists is a lot better than the conduit that doesn't. And copper ethernet is bog-standard like MP3 is: It isn't the best in any technical sense at all, but everything supports it. Universal compatibility is pretty nice. --- So the ongoing cost of copper 10gbe is electricity. Someone else here in the comments says that a copper 10GBe SFP+ module can use ~3 Watts, or that a newer one can use about 1.5 Watts. We can be generous by using the larger figure of 3 Watts, and 8 devices.. With 4 ports, eight 10-gig endpoints @ 3 Watts each, and $0.19 per kWh [delivered]: That's $3.28 per month, or about $400 per decade. If we assume 1.5 Watt endpoints, then that number halves. If we subtract the power consumption of fiber SFP+ modules (or media converters or whatever) to make the number a relative comparison instead of an absolute, then that figure goes down further. Not so bad, compared to conduit.
glitchc 5mo ago
One thing I'll add that I learnt in the process of doing my own house: It's not just the cable type but also the SFP module that can limit the distance. I used MicroTik hardware and their S+R10J modules are limited to ~30m for 10Gbps speeds.
xxpor 5mo ago
Unfortunately the blog didn't link to the SFP+ module they're using, but everyone should know there's effectively 2 different generations of 10gbit sfp+ to ethernet^H10BASE-T modules. The old gen, labeled as 30 meters, draws ~3 W, and gets extremely hot (to the point it'll usually cause link flaps), and the newer gen, usually labeled as 100m or 80m, draws ~1.5 W, and runs much, much cooler. Example of the new gen: https://www.amazon.com/Wiitek-Transceiver-Compatible-UF-RJ45-10G-Consumption/dp/B0CF9YSQDK https://www.amazon.com/Wiitek-Transceiver-Compatible-UF-RJ45... Old gen: https://www.amazon.com/10Gtek-SFP-10G-T-S-Compatible-10GBase-T-Transceiver/dp/B01KFBFL16 https://www.amazon.com/10Gtek-SFP-10G-T-S-Compatible-10GBase... Typically the old gen uses a Marvell AQR113C, and the new gen uses a Broadcom chip that I forget the number of off hand.
- CSSer 5mo ago
  Wow, and at essentially the same price!
  - xxpor 5mo ago
    Yeah, the new ones have gotten much cheaper it seems. About a year ago they were ~2x.
    - selectodude 5mo ago
      I think a lot of patents have recently expired for 10GbaseT. I’ve noticed it getting a lot more accessible.
- thefz 5mo ago
  Thanks! 10Gb Eth is insane for exactly this reason (optical SFP+ modules are way cheaper and more reliable)
  - oakwhiz 5mo ago
    Minor nitpick but they are both considered Ethernet. The 4 pair copper one is 10GBASE-T.
thefz 5mo ago
For me the threshold has always been "can I stream a 4K movie from the NAS downstairs or from my seed box". No real need for anything above. Still I ran 10Gb single mode fiber in all the ducts.
- TacticalCoder 5mo ago
  > For me the threshold has always been "can I stream a 4K movie from the NAS downstairs or from my seed box". So basically, and this is not a snark, you're happy with 100 Mbit/s?
  - thefz 5mo ago
    Ran the fiber for future proofing a new house, still have 200/50 Mbit internet connection.
zamadatix 5mo ago
The only thing I'd caution anyone else looking to do the same is doing a software router/fw like the Portectli is it's usually not hard to get the raw bandwidth to look nice with big flows but the new connection latency, connections per second, jitter, and QoS handling tend to suffer vs something with hw offloads (which is what most are used to even with cheapo gigabit home AP+router+switch combos). It's also not usually the cheapest way to get the 10G class NAT/L4 FW bandwidth, but it is usually the cheapest way to get "full" FW functionality if you don't care as much about the performance. If you want a full FW solution that can actually FW+NAT at 10G bidirectional without breaking a sweat then something like the FortiGate 90G is the cheapest thing I've found that performs really well across the board. Great QoS, great latency, amazing throughput performance (does well with even small packet sizes in a single stream), easy enough to use UI (once you get oriented), low power. If you want to enable all of the NGFW stuff (e.g. AV and IPS) then it'll dip below line rate though. If you just want something that NATs/connection direction oriented filtering like a "normal" home router then something like the MikroTik CCR2004 can get you better than the performance they got on the VP2440 + give you 12 ports of 10G SFP+ to work with. If you were planning to do "fancy" FWing/functionality beyond a normal home NAT FW (with decent managed switching built in) then the feature set will be a bit limiting, of course.
globular-toast 5mo ago
I put cat6a in my last house and plan to in my current house "just in case". The cable isn't that much more expensive and nobody wants to do cabling again so why not. But I'll keep using a gigabit switch because I have absolutely no idea what I'd use 10G for. It's crazy that gigabit was affordable for me as a student in the early 00s and between then and now we've gone from DVDs to 4K and it's still plenty fast enough. In fact, most people are happy with WiFi (not me, though).
tamimio 5mo ago
Regarding the proxmox cluster, I would advise not to set up a cluster unless you really need to, it adds complexity in order of shutting down the server, HA, Quorum votes, ceph monitor, among many others, and if one server goes offline, or even the wrong order, it will impact the others and sometimes some tough time in recovery, or data loss.
xenadu02 5mo ago
> The most important question was the structured cabling in the walls; was it CAT-5E or CAT-6, or even CAT-6A? Remember from the last post, 10GBASE-T might work over short runs of -5E (even though officially it's not meant to be able to). This is not quite correct. The primary problem is cross-talk. Copper wire itself will carry the relevant frequencies up to 100m without issue but even with balanced pairs the balancing is not perfect and the "dirty paper precoding" is not perfect so some cross-talk will occur. How long you can go with Cat-5e depends on how well the wire is twisted, how many wires are bundled together, are there any loops or tight bends, and other factors. Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out. Bottom line is: for almost any normal home or apartment any quality Cat-5e cable properly terminated will carry 10GBase-T without issue. In fact if you have problems I would first re-terminate the cable before assuming you need to run new cable. Cat-6 or 6A just isn't necessary. As a PSA: beware of "CCA". I've noticed Amazon and eBay are absolutely flooded with cheap chinese electrical and networking cable that shows nice shiny copper in the pictures but is actually "copper clad aluminum". If they mention anything at all they code it as "CCA" cable without explaining what that means. CCA cable cannot, by definition, be ethernet cable. I won't get into the full technical details but the standard was amended to clarify that only pure copper wires are acceptable for ethernet. Personally I would not dare use CCA for anything. It has lower performance, lower current-carrying capability for the same wire diameter (inherent in aluminum), and introduces the risk of oxidation and loosening of connections as people will treat them as copper connections when aluminum needs special installation procedures and connections to avoid them coming loose over time. For electrical connections especially this not only can but absolutely will lead to a fire over time if not treated with the appropriate care. All it takes is a little bit of mechanical action scraping off the thin copper layer and you now have an effectively aluminum wire - a time bomb ticking away.
- globular-toast 5mo ago
  > Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out. This isn't quite right. Cat6A guarantees the frequency and therefore effective bandwidth that is available. But it doesn't talk about physical cable type. A common structured cable for Cat6A is U/FTP which has no plastic separator, no outer shield, but rather each pair is wrapped in foil and is actually less tightly twisted than even Cat5e. Anyone can buy U/FTP cable, but to actually be Cat6A it has to be installed correctly and the installation certified to support Cat6A speeds.
DeathArrow 5mo ago
I wired my home with single mode fiber. It works like a charm and I can always upgrade the speed.
tristor 5mo ago
Using SFP+ modules for 10GbE is interesting, I deployed a full 10GbE network in my home connected by 5Gbit/s symmetric fiber from AT&T and a 1Gbit/s x 100mbit/s DOCSIS 3.1 connection from Spectrum as backup, I did this around August of last year and used UniFi/Ubiquiti for everything. Their new XG and XGS products all support 10GbE natively, without needing SFP+ modules to connect. I'm using the UDM Pro Max with an XGSPON ONU flashed with the community firmware from PON Wiki (see https://pon.wiki/category/att/ https://pon.wiki/category/att/ ) and the 2.5GbE port as my second WAN link to the Spectrum modem. I'm not doing any sort of bonding/load balancing, just failover so I'm almost always just using AT&T. There's a short DAC cable connecting the UDM Pro Max to a Switch Pro XG 24-POE which serves 10GbE to the rest of the house. The only SFP+ module involved is the XGSPON ONU. Most of the devices in the house connect to WiFi served by the UniFi U7 Pro XGS APs, which take 10GbE w/ POE and give off WiFi 7. I pretty much always get at least 4600Mbit/s both directions over AT&T, and generally cap it out. Spectrum typically gives me at least 800Mbit/s down. The ISPs are definitely the bottleneck because neither provides dedicated infrastructure from each house to the CO, instead you have some sort of aggregation point which has a shared backhaul and that means you compete for resources, but having the largest plan on each at least gives you traffic priority. Realistically, I don't need any of this, I was doing just fine with normal gigabit fiber with a 2.5GbE network before I moved last year, but it's nice knowing everything is as fast as possible in my path and that eliminates congestion and local network resource contentions as causes when a problem arises, I know it's pretty much always upstream of me somewhere.
rconti 5mo ago
I'm surprised they got a reliable 10Gbps USB adapter. I've tried 3 of them and they're all trash. They cannot sustain multi-gigabit speeds and end up doing like 400Mbps when you hit them hard. I think I have sabrent and ugreen and one other. They all just sit in a drawer, if I need a USB-C ethernet adapter i just grab my trusty 1 gig one.
theYipster 5mo ago
Yep... Did this 3 years ago in my home and came to the same conclusion. If I were to do it again, I'd run fiber through the walls instead of Cat-6a. It took forever to find SFP+ modules that would work with my Unifi setup... (not wanting to pay for more than one router or switch with native 10g RJ-45 ports, which are still very expensive.) I loose POE but, on the whole, it would've been much easier and much less costly--I think--to have just run fiber.
- marshray 5mo ago
  Same. I wish I had seen this before pulling all that Cat-6a.
TYPE_FASTER 5mo ago
I was just reading this thread today: https://www.reddit.com/r/homelab/comments/1sz9li4/invisible_bend_insensitive_bidi_fiber_is_amazing/ https://www.reddit.com/r/homelab/comments/1sz9li4/invisible_...
bhouston 5mo ago
My 10G experience is documented here, mine was TP-Link-based: https://ben3d.ca/blog/home-network-lessons https://ben3d.ca/blog/home-network-lessons
shmerl 5mo ago
That Mikrotik switch must be heating up like crazy if you fill all 4 ports, since it's using copper . How about this one? https://mikrotik.com/product/crs305_1g_4s_in https://mikrotik.com/product/crs305_1g_4s_in You can use fiber optic SFP+ in it which should not run as hot. Run fiber everywhere for the fun of it, instead of copper Ethernet. Here is an interesting post about it: https://sschueller.github.io/posts/wiring-a-home-with-fiber/ https://sschueller.github.io/posts/wiring-a-home-with-fiber/ You can simplify it and avoid splicing which is the most complex thing about it.
ziofill 5mo ago
What’s the use case for 10Gb/s Ethernet over 2.5?
- gh02t 5mo ago
  It's 4x faster? If you have a storage server, its easy to exceed 2.5 or even 5 gigabit with SATA spinning disks if they're in an array. 10 gig is fast enough to work with network storage like it's local: do your work, store your steam library, edit videos, etc, but store it all somewhere else and share it with multiple computers.
Lwrless 5mo ago
I went a slightly different route. My switches are linked with 10Gbps SFP+ across the apartment, but it was way too late (and too much hassle) to pull proper in-wall fiber. Instead I used one of those ultra-thin (around 0.1mm), unshielded fiber cables, and just snaked it through door frames and taped it along the walls. I'm genuinely impressed this stuff exists, it makes retrofitting fiber into a finished space so much less painful. Most of my edge devices are still on 2.5GbE though, and I'm increasingly aware that for anything with plain SATA disks, the drives are the real bottleneck. Once I LAG'd 2×2.5GbE to get a 5Gbps pipe, it became obvious the network wasn't the slow part anymore in a lot of cases. And yeah, the 10GbE SFP+ modules run hot, so hot that I would not lay my fingers on them for more than 2 seconds. I stuck 2 copper heatsinks on my module, not sure they do much but the module runs smoothly. Even so, I'm pretty happy with the overall setup: from my 10GbE-equipped Mac I can saturate multiple machines at once and I no longer think about the network most of the time, which was the goal.
- Lwrless 5mo ago
  And after getting 10Gbps working at home I was getting greedy, and looked at InfiniBand as well, 40Gbps and proper RDMA is very tempting compared to Ethernet. The catch for me was the practical side: IB needs PCIe slots and those chunky, inflexible cables. With most of my stuff being laptops, mini PCs and Macs, I just couldn’t see a clean way to route those or even plug cards in everywhere, so in the end the "door-frame‑friendly" skinny SFP+ fiber still won out for this apartment.
- hju22_-3 5mo ago
  What kind of fiber cables? If you have a link, I'd appreciate it. I can only find ones at 0.5mm.
  - Lwrless 5mo ago
    Good catch — that was actually my mistake. I mixed up cm and mm from the marketing material, and when I actually measured just now it came out to around 0.9mm. So not quite as thin as I claimed, and I probably wouldn't have noticed if you hadn't asked. I did find some Kevlar-reinforced options that are supposedly ~0.3mm, but they seem to be raw fiber without connectors, purposed for drones, and I'm not sure about global availability.
Arubis 5mo ago
If you’re going to go to all this trouble, please do seriously consider pulling singlemode fibre instead. It is more planning up front, but when you’re ready for your next upgrade cycle, you can reuse the same fibre and change out the hardware on each end.
- lostlogin 5mo ago
  They are much smaller to post though holes etc too.
- denkmoon 5mo ago
  Why recommend single mode out of interest? I've just done some fibre in my house with OM3 because that's what I had, I couldn't really work out why you'd prefer one over the other for residential lans.
  - Arubis 5mo ago
    Multimode is probably fine, provided it's OM3 or higher. You might need to think about it down the road, 5+y out. Having a spool just lying around would make it a perfectly reasonable choice. The fibre itself is a little more expensive; the optics are cheaper. But multimode does run into physical limitations a little faster, so when you decide you want to go to 100G, or 400G, or beyond, you may need to pull fresh media. Singlemode you just never have to think about again. It's likely to be more future-proof than your actual physical walls.
Jamesits 5mo ago
I don't understand the decisions to use RJ45 copper cable for anything higher than 1GbE, unless a) You like small computers and these only comes with RJ45 ports and 0 PCIe slots b) You enjoy fighting with super rigid cables, port flaps and hot modules I upgraded my home to full 10Gbps network at 2018. At that time: * 10Gbps SFP+ PCIe NICs (Mellanox CX3 and similar cards) are already dirt cheap ($15 for OCP adapted ones, $50 for native PCIe cards) because datacenters are upgrading to CX5 * 10KM SM SFP+ modules are already dirt cheap (~$6) because 4G BBU-RRU communication uses these * Silent 10Gbps SFP+ switches start appearing (still a little pricey); 10Gbps RJ45 switches are not available anywhere Those 10KM SM modules will happily run off <20m MM fibers so I just used whatever fiber I have at hand. The price is so cheap that I think the RJ45 upgrade path (2.5G has not been universal, 5G still very far away, 10G is so pricey) looks like a scam. After 2024: * 25G/100G SFP+ PCIe NICs are dirt cheap (~$80 for single port 100G NIC) * 100G QSFP28 modules are in acceptable price; 25G SM/CWDM SFP+ modules area already dirt cheap (~$50 for 6-slot CWDM ones) because 5G BBU-RRU communication uses these * Silent 100Gbps switches start appearing (you can get 24*25G + 4*100G under 100W; still a little pricey); 100Gbps RJ45 switches are not available anywhere I already have an experimental 6*25G CWDM fiber connection between my 2 rooms. Given that most of my devices wouldn't need anything higher than 10Gbps, I'm thinking of giving each room a single 100G port, split them into 4 lanes in the room with passive wavelength splitters for different devices, which further reduces cost on switches.
- p_ing 5mo ago
  Power usage is a good reason to stick with non-SFP ports. And you can use CAT5e with 10Gbps networks given the run is short enough. I run 2.5Gbps over CAT5e with no issues throughout my house (I don't need that much bandwidth except when downloading Steam games).
  - scq 5mo ago
    10Gbps fibre uses significantly less power than 10Gbps copper ethernet.
    - lostlogin 5mo ago
nl 5mo ago
Has anyone tried the InvisiLight kits? They give you a 600 micron fiber optical link in the kit which you can hide down the side of carpet etc. I haven't tried it but a few YT vids made it look interesting. https://www.amazon.com/stores/InvisiLight/page/DFF3C042-0D2E-4935-9F11-1D574786B361 https://www.amazon.com/stores/InvisiLight/page/DFF3C042-0D2E...
- hamdingers 5mo ago
  Looks like a complicated and expensive way to avoid running copper. "Up to 1Gbps" is not confidence inspiring, if you really can't put holes in your walls I'd try a powerline kit first for 1/5th the price
  - nl 5mo ago
    This is the YT video I saw about it: https://www.youtube.com/watch?v=Z2FbzCyiNr4 https://www.youtube.com/watch?v=Z2FbzCyiNr4 He used different transceivers and got 10Gbs
    - hamdingers 5mo ago
      That is an ad, I'm not surprised he had good results.
  - Galanwe 5mo ago
    I used powerline before. From my experience its really bad, unstable, slow. What did work pretty well though is MoCa, basically reusing coax/antenna cabling. I get a pretty stable connection roughly equivalent to 2.5G.
    - hamdingers 5mo ago
      It depends on the topology of your circuits. If you can get both adapters on the same circuit that is ideal, same phase is often okay at reduced speed. Different phases or having other complicated stuff in the way like subpanels is probably what you were dealing with. MoCa is fine, but if the coax was retrofit replacing it with proper ethernet cable isn't too hard.
fulafel 5mo ago
It's interesting how slow the uptake has been. 10GbE is from 2002, 24 years old. It's much worse than IPv6, at least we're up to 50% uptake there. 100 GbE is 15 years old too depending on which standard you think of. (The most common physical layer, SR4, is 11 yrs old) Gigabit ethernet came much faster after 100 M.
- randunel 5mo ago
  I can install 1Gbps using copper cables + RJ45 and household tools. I can't install 10Gbps using a $10 crimper. Therefore, I have 1Gbps all through my house and 10Gbps between equipment next to the access point.
  - lostlogin 5mo ago
    You can’t crimp cat 6?
    - randunel 5mo ago
      Crimping implies RJ45, so 1Gbps. You can't crimp SFP+.
      - gh02t 5mo ago
        Even CAT5e can do 10 Gbps if you do a decent job making the cable, even though it's out of spec. 6A can do it easily, in spec. I used it in my previous home. I much prefer to use fiber, but copper with good old RJ45 works fine for 10 gig.
Drdiamond 5mo ago
10Gb Ethernet is incredible for this exact reason but optical SFP+ modules are far more affordable and reliable.

232 points by gpjt 5mo ago

10g-base-T is a sick joke high latency, high error rates, and terrifying heat output from SFPs (which the author noted for himself) the only cat6 left in my home network is the link to verizon's ont, because in their infinite wisdom the ONLY connectivity offered was 10g-base-t

zamadatix 5mo ago
10GBase-T can still be nice if you're going to do native PoE (which the author did not) or you expect a mix of devices with 1/2.5/5 which you don't want to or can't upgrade/replace immediately (which is where it sounds like the author was situated). The new 10GBASE-T SFPs are actually not too bad - you can get the full 100 meters at half the wattage it takes for the old space heater generation ones to reach 30 meters. Based on the article, the author did not know there were newer cooler options for the ~the same price.

mikepurvis 5mo ago

That's pretty wild. I have 1.5/900 fibre to my house, and I bring a 2.5 line from the modem to my home office where a 2.5 switch delivers it to my workstation, laptop, and unraid NAS. But those devices are all themselves just gigE I think, and I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.

rhplus 5mo ago
10Gbps is enough bandwidth for 500 concurrent Netflix streams in 4K/UHD (15Mbps) AND 500 concurrent video calls (4Mbps). Home users don’t need more bandwidth to improve their internet experiences, they need lower latency, less congestion and less loss. https://help.netflix.com/en/node/306 https://help.netflix.com/en/node/306 https://learn.microsoft.com/en-us/microsoftteams/prepare-network https://learn.microsoft.com/en-us/microsoftteams/prepare-net...
- wpm 5mo ago
  Home users only do video calls and watch Netflix? More and more regular people are getting network storage appliances. More and more people have laptops with SSDs that can write at 4 or 5 GB/s. Why shouldn't they get to use all of it?
  - afavour 5mo ago
    To quote the previous post: > I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available. There are many things along the way that would get in the way of a home user downloading something from the internet that would hit that 5GB/s speed. It's not that people should be "banned" from it or something, more that the investment cost isn't worth it.
    - mlyle 5mo ago
      We have a 5gbps pipe; routinely download games from Steam at >3gbps; when I had to reinitialize my cloud backup it was >4gbps. All of this without impacting anyone else on the pipe. Yah, our P95 bandwidth is just a few megabits per second. But it's not that expensive and routinely saves me a few minutes here and there. 10gbps on the LAN is more broadly useful. Pegging it for a file share is a daily occurrence.
      - ProfessorLayton 5mo ago
        Also storage has gotten super expensive lately, and rather than upgrading my machines/consoles I've been offloading games and downloading them as needed and now am routinely downloading dozens of GB just to play a game. My gaming time is limited so the faster the better.
    - fmajid 5mo ago
      I regularly saturate my 1G home and 1G office connection syncing ~6GB files between the two. It's also nice to be able to download a 100G or so game quickly. Remote backups to cloud storage also benefit from fast upload speeds (and more importantly, restores).
IshKebab 5mo ago
Yeah I foolishly paid for 500 Mb/s but the only things I ever get over even 200 Mb/s for in the UK are Steam downloads (and speed tests). Everything else seems to be roughly throttled at that rate.
- mikepurvis 5mo ago
  Telus gave me a good price and I'd already invested in a Unifi gateway that could accept the ONT directly, so that was fun to play with.
  - lostlogin 5mo ago
    How else would you connect? Some ISP box? Sorry, I’m not familiar with any other way.
    - mikepurvis 5mo ago
      Yeah the default is they supply a "modem" with a regular ethernet port, so a familiar config to someone coming from cable or DSL, but inside it's a QSFP that you can move to your own equipment if you want.
zamadatix 5mo ago
I regularly hit 4.7 gbps on a 5 Gbps line pulling files (usenet is usually faster than torrent, but the latter can be equally as fast depending on the torrent & how good the client software is). It's great to just grab an entire movie series in 4k Blu Ray remux quality in 5 minutes and go. No real need to plan ahead for anything.
- HDBaseT 5mo ago
  Yeah, Usenet is crazy quick. I've saturated 10Gbps on Usenet before, pretty easily. Torrents (as you mention) can be equally as fast. Well seeded, private tracker torrents can easily download at above 1Gbps (depending on piece size, etc).
  - lostlogin 5mo ago
    Can’t get much over a gigabit here I’d nz.
throwforfeds 5mo ago
And here I am sitting in Brooklyn and haven't had one apartment that has had fiber as an option. I get to pay Spectrum $90/month for "400/20" and in reality get 100/10.
- fuomag9 5mo ago
  I was paying 21.9€ per month to TIM for a 10/2Gbit connection Welcome to Italy :D
  - throwforfeds 5mo ago
    But Brooklyn is an old city that's hard to wire, unlike those new Italian cities... /s

rrevi 5mo ago

The device nomenclature alone is worth the read! (otherwise an impressive feat to see)

apelapan 5mo ago

I might have been lucky, but in the one home and one office were I've connected 10gbit switches and PCIe cards, it has just worked. Especially the office was a nice surprise, because it is at least 20 meters (probably more) of unknown cabling and at least one unknown patch panel between the utility closet where the NAS lives and the desk area. The cables were run 15 years ago, so I expected it to be cat 5, but clearly not. It is nice moving/streaming large files across the network at 10 gbit. It really is ten times less waiting than with plain old gigabit. Of course, most of the time I'm working with lots of small files and then the spinning disk array in the NAS has no chance to saturated the this giant pipe, or even a normal gigabit connection...

saltcured 5mo ago
I know when I was doing some custom wiring in a house around 2005-6, it was clear that cat-6e was the thing to use if you wanted any future-proofing. So I bought a reel of that even though I was only going to be using 1000-BaseT. I don't remember there being too much premium on the wire itself.
- loeg 5mo ago
  Do you mean Cat 6 or Cat 5e? Cat 6e isn't a thing and 6a didn't exist in 2006. 6 was certainly more future-proof at the time, although arguably 5e is still fine even today. (Super-gigabit consumer equipment didn't really exist until the last five years and it's till notably more expensive and less common than gigabit, which runs on 5e just fine.)
  - saltcured 5mo ago
    Cat 6 for sure. I thought there was some extra tweak beyond that, but I probably misremembered after all this time. Perhaps it was just that it was plenum-rated..
    - globular-toast 5mo ago
      You might be thinking of cat6a which officially supports 10G over long lengths, unlike cat6.

afavour 5mo ago

Both impressive and surprising that thermals were the biggest barrier! Meanwhile I'm sat here wishing I could justify running any ethernet in my apartment, but improving wi-fi tech means I never can...

jauntywundrkind 5mo ago
I wonder what the idle vs at load difference is for power draw/heat. Would really love to see this feature in reviews some! I wonder if you could negotiate down to 1gbit until you see some level of activity, if that would help at all? I'm still eyeing 10Gb, but if my home needs +30w for three computers, I don't feel like it's really worth it. Would love to see more details on the power consumption from folks, especially tuning for idle.

Boss0565 5mo ago

What's the point?

snihalani 5mo ago
for the funs

gigel82 5mo ago

I was surprised that the old Cat5e in my home supports 10Gbps without any issue, so went ahead and upgraded the rest of the network with 10Gbps switches (expensive Ubiquiti gear, but worth it to talk at 10Gbps between all my machines, even though the internet is only 5Gbps Fiber).

rconti 5mo ago
Yeah, my home is cat5e (I think!) and I was only able to get 2.5 or 5Gbps to my _single_ 10GbE capable machine I had at the time. So I ended up just buying a switch that supports 2.5Gbps over copper since it's such a huge expense to step up. Anyway, my devices are pretty much 2.5Gbps (APs, flex mini switches, etc).

TexanFeller 5mo ago

I'm extremely happy after upgrading my network to 10gbit copper ethernet. It was much more expensive than I thought it should be, but worth it even if I only max it occasionally. Now I can easily fully saturate my 10gbit ethernet doing a first Time Machine backup or transferring files to my M.2 SSD NAS which saves me waiting rime and is satisfying to watch. It's wild to me that 10gbit isn't the norm by now and tech people who should know better seem to think WiFi matches or even exceeds even 1gbit ethernet. My MBP connects to my WiFi7 setup(Ubiquiti E7) at a nominal 1.5-1.9gbit but Time Machine backups and file transfers are slower than plugging into 1gbit ethernet, probably in large part due to latency and retransmissions. Not to mention that ethernet works with near 100% reliability with dramatically less variation in speed and error rate.

ocdtrekkie 5mo ago
Heh it's honestly wild to me anyone needs over a gig. My work has a one gig fiber line supporting hundreds of employees and usage generally remains below 10%. The high expense of 10gig is, in part, because it isn't widely necessary and the people buying it are willing to pay extra.
- godzillabrennus 5mo ago
  I put 5Gbit internet into my home (fiber) to build my startup. I'm processing terabytes of data. I have over 100TB of storage in my basement. I can regularly saturate my internet connection. That said, I remember well when a 1Gbit connection provided enough bandwidth for a 500-person call center for daily workloads (back about 10 years ago).
  - ocdtrekkie 5mo ago
    That puts you in an extreme minority, even amongst enterprise businesses. Many medium sized enterprises have storage that looks like "a couple dozen TB total" for hundreds of staff. Having 100 TB of storage in your home basement is an even more extreme minority than that. ;) A gigabit connection is more than enough for a 500-person call center today.

cyberax 5mo ago

The Mikrotik switch is awesome, and it's still the most compact 10G switch available. You can fix the thermal issue either by adding a small fan (Noctua is great) or by adding more radiators: https://pics.ealex.net/share/UxeSf_AWHLIuc-qzK5zl7JIgQvQDAZhy4-8sseK5bNVYVR7M83lxkBnXejQI___-dEw https://pics.ealex.net/share/UxeSf_AWHLIuc-qzK5zl7JIgQvQDAZh... I've been running it like this in a closed comm box for the last 3 years without any issues. SFP+ modules actually do not use that much power, it's just that it's concentrated into a small package, resulting in high temps.

bot403 5mo ago
Cripes. When possible just do fiber and DACs. Faster and much cooler than 10Gbit. 10Gbit uses an absurd amount of power per port thus the need for all that cooling.
- m463 5mo ago
  yeah, the article though says... "The apartment has structured cabling -- each room has one or more RJ45 sockets in the wall," ... Which is the main problem most folks face. wish the standard was "conduit" instead of "bake-this-years-tech-into-the-wall" which doesn't always last...
  - ssl-3 5mo ago
    Folks have been saying that conduit is the way, and the fiber is the future, and all kinds of things like that for decades so far. But the simple truth for all those decades is this: When there's already cat-whatever cable in the wall, it generally still works. Decently-installed conduit (ie, actually-usable conduit) adds a ton of time and expense, which is why it is very seldom used for data circuits in residential structures. The cable that exists is a lot better than the conduit that doesn't. And copper ethernet is bog-standard like MP3 is: It isn't the best in any technical sense at all, but everything supports it. Universal compatibility is pretty nice. --- So the ongoing cost of copper 10gbe is electricity. Someone else here in the comments says that a copper 10GBe SFP+ module can use ~3 Watts, or that a newer one can use about 1.5 Watts. We can be generous by using the larger figure of 3 Watts, and 8 devices.. With 4 ports, eight 10-gig endpoints @ 3 Watts each, and $0.19 per kWh [delivered]: That's $3.28 per month, or about $400 per decade. If we assume 1.5 Watt endpoints, then that number halves. If we subtract the power consumption of fiber SFP+ modules (or media converters or whatever) to make the number a relative comparison instead of an absolute, then that figure goes down further. Not so bad, compared to conduit.

glitchc 5mo ago

One thing I'll add that I learnt in the process of doing my own house: It's not just the cable type but also the SFP module that can limit the distance. I used MicroTik hardware and their S+R10J modules are limited to ~30m for 10Gbps speeds.

xxpor 5mo ago

Unfortunately the blog didn't link to the SFP+ module they're using, but everyone should know there's effectively 2 different generations of 10gbit sfp+ to ethernet^H10BASE-T modules. The old gen, labeled as 30 meters, draws ~3 W, and gets extremely hot (to the point it'll usually cause link flaps), and the newer gen, usually labeled as 100m or 80m, draws ~1.5 W, and runs much, much cooler. Example of the new gen: https://www.amazon.com/Wiitek-Transceiver-Compatible-UF-RJ45-10G-Consumption/dp/B0CF9YSQDK https://www.amazon.com/Wiitek-Transceiver-Compatible-UF-RJ45... Old gen: https://www.amazon.com/10Gtek-SFP-10G-T-S-Compatible-10GBase-T-Transceiver/dp/B01KFBFL16 https://www.amazon.com/10Gtek-SFP-10G-T-S-Compatible-10GBase... Typically the old gen uses a Marvell AQR113C, and the new gen uses a Broadcom chip that I forget the number of off hand.

CSSer 5mo ago
Wow, and at essentially the same price!
- xxpor 5mo ago
  Yeah, the new ones have gotten much cheaper it seems. About a year ago they were ~2x.
  - selectodude 5mo ago
    I think a lot of patents have recently expired for 10GbaseT. I’ve noticed it getting a lot more accessible.
thefz 5mo ago
Thanks! 10Gb Eth is insane for exactly this reason (optical SFP+ modules are way cheaper and more reliable)
- oakwhiz 5mo ago
  Minor nitpick but they are both considered Ethernet. The 4 pair copper one is 10GBASE-T.

thefz 5mo ago

For me the threshold has always been "can I stream a 4K movie from the NAS downstairs or from my seed box". No real need for anything above. Still I ran 10Gb single mode fiber in all the ducts.

TacticalCoder 5mo ago
> For me the threshold has always been "can I stream a 4K movie from the NAS downstairs or from my seed box". So basically, and this is not a snark, you're happy with 100 Mbit/s?
- thefz 5mo ago
  Ran the fiber for future proofing a new house, still have 200/50 Mbit internet connection.

zamadatix 5mo ago

The only thing I'd caution anyone else looking to do the same is doing a software router/fw like the Portectli is it's usually not hard to get the raw bandwidth to look nice with big flows but the new connection latency, connections per second, jitter, and QoS handling tend to suffer vs something with hw offloads (which is what most are used to even with cheapo gigabit home AP+router+switch combos). It's also not usually the cheapest way to get the 10G class NAT/L4 FW bandwidth, but it is usually the cheapest way to get "full" FW functionality if you don't care as much about the performance. If you want a full FW solution that can actually FW+NAT at 10G bidirectional without breaking a sweat then something like the FortiGate 90G is the cheapest thing I've found that performs really well across the board. Great QoS, great latency, amazing throughput performance (does well with even small packet sizes in a single stream), easy enough to use UI (once you get oriented), low power. If you want to enable all of the NGFW stuff (e.g. AV and IPS) then it'll dip below line rate though. If you just want something that NATs/connection direction oriented filtering like a "normal" home router then something like the MikroTik CCR2004 can get you better than the performance they got on the VP2440 + give you 12 ports of 10G SFP+ to work with. If you were planning to do "fancy" FWing/functionality beyond a normal home NAT FW (with decent managed switching built in) then the feature set will be a bit limiting, of course.

globular-toast 5mo ago

I put cat6a in my last house and plan to in my current house "just in case". The cable isn't that much more expensive and nobody wants to do cabling again so why not. But I'll keep using a gigabit switch because I have absolutely no idea what I'd use 10G for. It's crazy that gigabit was affordable for me as a student in the early 00s and between then and now we've gone from DVDs to 4K and it's still plenty fast enough. In fact, most people are happy with WiFi (not me, though).

tamimio 5mo ago

Regarding the proxmox cluster, I would advise not to set up a cluster unless you really need to, it adds complexity in order of shutting down the server, HA, Quorum votes, ceph monitor, among many others, and if one server goes offline, or even the wrong order, it will impact the others and sometimes some tough time in recovery, or data loss.

xenadu02 5mo ago

> The most important question was the structured cabling in the walls; was it CAT-5E or CAT-6, or even CAT-6A? Remember from the last post, 10GBASE-T might work over short runs of -5E (even though officially it's not meant to be able to). This is not quite correct. The primary problem is cross-talk. Copper wire itself will carry the relevant frequencies up to 100m without issue but even with balanced pairs the balancing is not perfect and the "dirty paper precoding" is not perfect so some cross-talk will occur. How long you can go with Cat-5e depends on how well the wire is twisted, how many wires are bundled together, are there any loops or tight bends, and other factors. Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out. Bottom line is: for almost any normal home or apartment any quality Cat-5e cable properly terminated will carry 10GBase-T without issue. In fact if you have problems I would first re-terminate the cable before assuming you need to run new cable. Cat-6 or 6A just isn't necessary. As a PSA: beware of "CCA". I've noticed Amazon and eBay are absolutely flooded with cheap chinese electrical and networking cable that shows nice shiny copper in the pictures but is actually "copper clad aluminum". If they mention anything at all they code it as "CCA" cable without explaining what that means. CCA cable cannot, by definition, be ethernet cable. I won't get into the full technical details but the standard was amended to clarify that only pure copper wires are acceptable for ethernet. Personally I would not dare use CCA for anything. It has lower performance, lower current-carrying capability for the same wire diameter (inherent in aluminum), and introduces the risk of oxidation and loosening of connections as people will treat them as copper connections when aluminum needs special installation procedures and connections to avoid them coming loose over time. For electrical connections especially this not only can but absolutely will lead to a fire over time if not treated with the appropriate care. All it takes is a little bit of mechanical action scraping off the thin copper layer and you now have an effectively aluminum wire - a time bomb ticking away.

globular-toast 5mo ago
> Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out. This isn't quite right. Cat6A guarantees the frequency and therefore effective bandwidth that is available. But it doesn't talk about physical cable type. A common structured cable for Cat6A is U/FTP which has no plastic separator, no outer shield, but rather each pair is wrapped in foil and is actually less tightly twisted than even Cat5e. Anyone can buy U/FTP cable, but to actually be Cat6A it has to be installed correctly and the installation certified to support Cat6A speeds.

DeathArrow 5mo ago

I wired my home with single mode fiber. It works like a charm and I can always upgrade the speed.

tristor 5mo ago

Using SFP+ modules for 10GbE is interesting, I deployed a full 10GbE network in my home connected by 5Gbit/s symmetric fiber from AT&T and a 1Gbit/s x 100mbit/s DOCSIS 3.1 connection from Spectrum as backup, I did this around August of last year and used UniFi/Ubiquiti for everything. Their new XG and XGS products all support 10GbE natively, without needing SFP+ modules to connect. I'm using the UDM Pro Max with an XGSPON ONU flashed with the community firmware from PON Wiki (see https://pon.wiki/category/att/ https://pon.wiki/category/att/ ) and the 2.5GbE port as my second WAN link to the Spectrum modem. I'm not doing any sort of bonding/load balancing, just failover so I'm almost always just using AT&T. There's a short DAC cable connecting the UDM Pro Max to a Switch Pro XG 24-POE which serves 10GbE to the rest of the house. The only SFP+ module involved is the XGSPON ONU. Most of the devices in the house connect to WiFi served by the UniFi U7 Pro XGS APs, which take 10GbE w/ POE and give off WiFi 7. I pretty much always get at least 4600Mbit/s both directions over AT&T, and generally cap it out. Spectrum typically gives me at least 800Mbit/s down. The ISPs are definitely the bottleneck because neither provides dedicated infrastructure from each house to the CO, instead you have some sort of aggregation point which has a shared backhaul and that means you compete for resources, but having the largest plan on each at least gives you traffic priority. Realistically, I don't need any of this, I was doing just fine with normal gigabit fiber with a 2.5GbE network before I moved last year, but it's nice knowing everything is as fast as possible in my path and that eliminates congestion and local network resource contentions as causes when a problem arises, I know it's pretty much always upstream of me somewhere.

rconti 5mo ago

I'm surprised they got a reliable 10Gbps USB adapter. I've tried 3 of them and they're all trash. They cannot sustain multi-gigabit speeds and end up doing like 400Mbps when you hit them hard. I think I have sabrent and ugreen and one other. They all just sit in a drawer, if I need a USB-C ethernet adapter i just grab my trusty 1 gig one.

theYipster 5mo ago

Yep... Did this 3 years ago in my home and came to the same conclusion. If I were to do it again, I'd run fiber through the walls instead of Cat-6a. It took forever to find SFP+ modules that would work with my Unifi setup... (not wanting to pay for more than one router or switch with native 10g RJ-45 ports, which are still very expensive.) I loose POE but, on the whole, it would've been much easier and much less costly--I think--to have just run fiber.

marshray 5mo ago
Same. I wish I had seen this before pulling all that Cat-6a.

TYPE_FASTER 5mo ago

I was just reading this thread today: https://www.reddit.com/r/homelab/comments/1sz9li4/invisible_bend_insensitive_bidi_fiber_is_amazing/ https://www.reddit.com/r/homelab/comments/1sz9li4/invisible_...

bhouston 5mo ago

My 10G experience is documented here, mine was TP-Link-based: https://ben3d.ca/blog/home-network-lessons https://ben3d.ca/blog/home-network-lessons

shmerl 5mo ago

That Mikrotik switch must be heating up like crazy if you fill all 4 ports, since it's using copper . How about this one? https://mikrotik.com/product/crs305_1g_4s_in https://mikrotik.com/product/crs305_1g_4s_in You can use fiber optic SFP+ in it which should not run as hot. Run fiber everywhere for the fun of it, instead of copper Ethernet. Here is an interesting post about it: https://sschueller.github.io/posts/wiring-a-home-with-fiber/ https://sschueller.github.io/posts/wiring-a-home-with-fiber/ You can simplify it and avoid splicing which is the most complex thing about it.

ziofill 5mo ago

What’s the use case for 10Gb/s Ethernet over 2.5?

gh02t 5mo ago
It's 4x faster? If you have a storage server, its easy to exceed 2.5 or even 5 gigabit with SATA spinning disks if they're in an array. 10 gig is fast enough to work with network storage like it's local: do your work, store your steam library, edit videos, etc, but store it all somewhere else and share it with multiple computers.

Lwrless 5mo ago

I went a slightly different route. My switches are linked with 10Gbps SFP+ across the apartment, but it was way too late (and too much hassle) to pull proper in-wall fiber. Instead I used one of those ultra-thin (around 0.1mm), unshielded fiber cables, and just snaked it through door frames and taped it along the walls. I'm genuinely impressed this stuff exists, it makes retrofitting fiber into a finished space so much less painful. Most of my edge devices are still on 2.5GbE though, and I'm increasingly aware that for anything with plain SATA disks, the drives are the real bottleneck. Once I LAG'd 2×2.5GbE to get a 5Gbps pipe, it became obvious the network wasn't the slow part anymore in a lot of cases. And yeah, the 10GbE SFP+ modules run hot, so hot that I would not lay my fingers on them for more than 2 seconds. I stuck 2 copper heatsinks on my module, not sure they do much but the module runs smoothly. Even so, I'm pretty happy with the overall setup: from my 10GbE-equipped Mac I can saturate multiple machines at once and I no longer think about the network most of the time, which was the goal.

Lwrless 5mo ago
And after getting 10Gbps working at home I was getting greedy, and looked at InfiniBand as well, 40Gbps and proper RDMA is very tempting compared to Ethernet. The catch for me was the practical side: IB needs PCIe slots and those chunky, inflexible cables. With most of my stuff being laptops, mini PCs and Macs, I just couldn’t see a clean way to route those or even plug cards in everywhere, so in the end the "door-frame‑friendly" skinny SFP+ fiber still won out for this apartment.
hju22_-3 5mo ago
What kind of fiber cables? If you have a link, I'd appreciate it. I can only find ones at 0.5mm.
- Lwrless 5mo ago
  Good catch — that was actually my mistake. I mixed up cm and mm from the marketing material, and when I actually measured just now it came out to around 0.9mm. So not quite as thin as I claimed, and I probably wouldn't have noticed if you hadn't asked. I did find some Kevlar-reinforced options that are supposedly ~0.3mm, but they seem to be raw fiber without connectors, purposed for drones, and I'm not sure about global availability.

Arubis 5mo ago

If you’re going to go to all this trouble, please do seriously consider pulling singlemode fibre instead. It is more planning up front, but when you’re ready for your next upgrade cycle, you can reuse the same fibre and change out the hardware on each end.

lostlogin 5mo ago
They are much smaller to post though holes etc too.
denkmoon 5mo ago
Why recommend single mode out of interest? I've just done some fibre in my house with OM3 because that's what I had, I couldn't really work out why you'd prefer one over the other for residential lans.
- Arubis 5mo ago
  Multimode is probably fine, provided it's OM3 or higher. You might need to think about it down the road, 5+y out. Having a spool just lying around would make it a perfectly reasonable choice. The fibre itself is a little more expensive; the optics are cheaper. But multimode does run into physical limitations a little faster, so when you decide you want to go to 100G, or 400G, or beyond, you may need to pull fresh media. Singlemode you just never have to think about again. It's likely to be more future-proof than your actual physical walls.

Jamesits 5mo ago

I don't understand the decisions to use RJ45 copper cable for anything higher than 1GbE, unless a) You like small computers and these only comes with RJ45 ports and 0 PCIe slots b) You enjoy fighting with super rigid cables, port flaps and hot modules I upgraded my home to full 10Gbps network at 2018. At that time: * 10Gbps SFP+ PCIe NICs (Mellanox CX3 and similar cards) are already dirt cheap ($15 for OCP adapted ones, $50 for native PCIe cards) because datacenters are upgrading to CX5 * 10KM SM SFP+ modules are already dirt cheap (~$6) because 4G BBU-RRU communication uses these * Silent 10Gbps SFP+ switches start appearing (still a little pricey); 10Gbps RJ45 switches are not available anywhere Those 10KM SM modules will happily run off <20m MM fibers so I just used whatever fiber I have at hand. The price is so cheap that I think the RJ45 upgrade path (2.5G has not been universal, 5G still very far away, 10G is so pricey) looks like a scam. After 2024: * 25G/100G SFP+ PCIe NICs are dirt cheap (~$80 for single port 100G NIC) * 100G QSFP28 modules are in acceptable price; 25G SM/CWDM SFP+ modules area already dirt cheap (~$50 for 6-slot CWDM ones) because 5G BBU-RRU communication uses these * Silent 100Gbps switches start appearing (you can get 24*25G + 4*100G under 100W; still a little pricey); 100Gbps RJ45 switches are not available anywhere I already have an experimental 6*25G CWDM fiber connection between my 2 rooms. Given that most of my devices wouldn't need anything higher than 10Gbps, I'm thinking of giving each room a single 100G port, split them into 4 lanes in the room with passive wavelength splitters for different devices, which further reduces cost on switches.

p_ing 5mo ago
Power usage is a good reason to stick with non-SFP ports. And you can use CAT5e with 10Gbps networks given the run is short enough. I run 2.5Gbps over CAT5e with no issues throughout my house (I don't need that much bandwidth except when downloading Steam games).
- scq 5mo ago
  10Gbps fibre uses significantly less power than 10Gbps copper ethernet.
  - lostlogin 5mo ago

nl 5mo ago

Has anyone tried the InvisiLight kits? They give you a 600 micron fiber optical link in the kit which you can hide down the side of carpet etc. I haven't tried it but a few YT vids made it look interesting. https://www.amazon.com/stores/InvisiLight/page/DFF3C042-0D2E-4935-9F11-1D574786B361 https://www.amazon.com/stores/InvisiLight/page/DFF3C042-0D2E...

hamdingers 5mo ago
Looks like a complicated and expensive way to avoid running copper. "Up to 1Gbps" is not confidence inspiring, if you really can't put holes in your walls I'd try a powerline kit first for 1/5th the price
- nl 5mo ago
  This is the YT video I saw about it: https://www.youtube.com/watch?v=Z2FbzCyiNr4 https://www.youtube.com/watch?v=Z2FbzCyiNr4 He used different transceivers and got 10Gbs
  - hamdingers 5mo ago
    That is an ad, I'm not surprised he had good results.
- Galanwe 5mo ago
  I used powerline before. From my experience its really bad, unstable, slow. What did work pretty well though is MoCa, basically reusing coax/antenna cabling. I get a pretty stable connection roughly equivalent to 2.5G.
  - hamdingers 5mo ago
    It depends on the topology of your circuits. If you can get both adapters on the same circuit that is ideal, same phase is often okay at reduced speed. Different phases or having other complicated stuff in the way like subpanels is probably what you were dealing with. MoCa is fine, but if the coax was retrofit replacing it with proper ethernet cable isn't too hard.

fulafel 5mo ago

It's interesting how slow the uptake has been. 10GbE is from 2002, 24 years old. It's much worse than IPv6, at least we're up to 50% uptake there. 100 GbE is 15 years old too depending on which standard you think of. (The most common physical layer, SR4, is 11 yrs old) Gigabit ethernet came much faster after 100 M.

randunel 5mo ago
I can install 1Gbps using copper cables + RJ45 and household tools. I can't install 10Gbps using a $10 crimper. Therefore, I have 1Gbps all through my house and 10Gbps between equipment next to the access point.
- lostlogin 5mo ago
  You can’t crimp cat 6?
  - randunel 5mo ago
    Crimping implies RJ45, so 1Gbps. You can't crimp SFP+.
    - gh02t 5mo ago
      Even CAT5e can do 10 Gbps if you do a decent job making the cable, even though it's out of spec. 6A can do it easily, in spec. I used it in my previous home. I much prefer to use fiber, but copper with good old RJ45 works fine for 10 gig.

Drdiamond 5mo ago

10Gb Ethernet is incredible for this exact reason but optical SFP+ modules are far more affordable and reliable.