14 ms·

High-speed 10Gbps full-mesh network based on USB4 for just $47.98

• johng 3y ago
  This is neat.
  • deleted 3y ago
    [deleted]
• riku_iki 3y ago
  15c/kwh estimate in California looks very far from the current situation.
  • jms703 3y ago
    Yeah, I think the website the author linked to has old data. It's 3x that in California.
    • kube-system 3y ago
      It is 26.72 cents per kwh as of the latest official numbers which are from October 2023. https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...
      • riku_iki 3y ago
        Not clear who are those "ultimate customers". Also, in case of server hardware, it is excessive over basic tier electricity consumption, so it will hit those 50c/kwh.
        • kube-system 3y ago
          > Not clear who are those "ultimate customers". The first column is residential customers.
          • riku_iki 3y ago
            It is residential "ultimate" customers. My current bill for initial tier is: 42c/kwh delivery + 15c/kwh generation, and even higher over initial consumption. I guess I am not "ultimate" customer.
            • kube-system 3y ago
              'ultimate customer' means the customer using the power, which you are. The reason you don't pay the price listed on that page is because you are only one customer. You are not the average customer.
              • riku_iki 3y ago
                The fact that I live in the center of densely populated area and serviced by major provider who also holds monopoly, makes me think there is a chance that table's numbers are just not what you think. They may include just generation cost, and not delivery, or there is something about "ultimate customer" definition. Also, there were several hikes since Oct in my understanding.
                • kube-system 3y ago
                  I read form EIA-861, which is where the data comes from. Looks like sales to "ultimate customers" means it excludes electricity that was not sold, electricity that was sold to resellers, energy lost, etc. The form also collects information about revenue from delivery "and other related charges" 42c/kwh delivery sounds insane. I couldn't find much data about average delivery rates, but I plugged in a few counties here and it looks like many areas have delivery rates significantly lower than that: https://www.cpuc.ca.gov/RateComparison https://www.cpuc.ca.gov/RateComparison
• nickstinemates 3y ago
  Incredible. Traditional 10Gb network gear is very expensive to buy and to operate. $50 is a lot cheaper.
  • Palomides 3y ago
    not really true, a similar setup with 40Gbe (three nodes, no switch) would run you less than $100 for pcie cards and cables
    • hamandcheese 3y ago
      The only NICs I see in that price range are old Mellanox cards. Intel NICs are 5-10x that price. I'm not sure why, but my suspicion is that it has to do with driver support and/or interoperability.
      • Palomides 3y ago
        I run mellanox connectx3 cards, they work immediately with no extra drivers on windows 10/11 and every linux I've tried mellanox is/was quite good at getting code upstreamed maybe I need to do my own blog post about my pile of computers...
        • TacticalCoder 3y ago
          > maybe I need to do my own blog post about my pile of computers... Yes, several of us love to read about that! I haven't switched to 10 Gbit/s yet...
        • thefz 3y ago
          > I run mellanox connectx3 cards, they work immediately with no extra drivers on windows 10/11 and every linux I've tried Own three, can confirm for Windows and Linux.
        • gregors 3y ago
          would love to read this!
• zamadatix 3y ago
  This is a great use of the Thunderbolt hardware that comes built into consumer devices. For throughput try sticking "--parallel 4" or shorthand "-P 4" on the iPerf3 command and see if total throughput changes.
• toyg 3y ago
  > I don’t understand why it can only hit 11Gbps [...] other people building similar networks were able to hit 20Gbps. [...] So it could simply be that the machine only supports up to this speed [...] it might compete with [Intel's] network controllers, so they capped the speed I'm bad at low-level networking, but could it be a routing issue? Effectively every machine is also a router, so there might be some wasting going on.
  • arghwhat 3y ago
    The machines are not configured as "routers" in that example. It is just a point-to-point layer 2 network, where each network link serves exactly one possible destination. As simple as it can be.
  • r1ch 3y ago
    It's measuring single TCP connection performance which is already difficult to optimize. With jumbo frames and tuned buffer sizes I'd expect it to get higher, but it will likely be serialized to a single core's worth of CPU. Using multiple connections should give a better representation of available link bandwidth.
  • adrian_b 3y ago
    IIRC the specification of Thunderbolt from Intel (which was inherited by USB 4) limits the Ethernet emulation mode to 10 Gb/s. Why did they set a limit so low is not known, but the supposition made by another poster that this is a market segmentation feature may be right, because such policies have always been typical for Intel.
    • wmf 3y ago
      This post is getting almost 12 Gbps and someone mentioned getting 16 Gbps on Mac so I don't think there's any intentional limit. Thunderbolt is really 22 Gbps anyway and networking over Thunderbolt is just inefficient.
  • DannyBee 3y ago
    It's more likely to be offloads and processing time - none of these have any of the offloads a regular network card does. Even if the CPU can handle it, it increases latency, and with a single stream like this, that affects throughput badly. At 10gigabit/second, with 1518 byte packets, it's 823451.9104 packets per second. So in a single stream, you have to process each packet within 1.21microseconds to keep up At 20gbps, you have 600nanoseconds per packet. There are also almost certainly timing/synchronization issues between different stacks like this. It's horribly inefficient. Network cards achieve >10gbps in part by offloading a lot of stuff. Even if the CPU can handle the load, just going through different stacks like they are may add enough latency to throw single stream throughput off. The posited reason of "not compete with network cards" is beyond stupid. It can't because you can only do a few meters this way, max. That's not interesting at all. 25gbps network cards are cheap and for a few meters, a $10-15 25gbps DAC will suffice For more than that, 25gbps transceivers are 25 bucks a pop for 25gbps-SR, which will do 100meters. With none of the problems of trying to use longer thunderbolt cables, too. Intel's 25gbps SKUs are not where they make their money anyway.
• jms703 3y ago
  Neat. I stopped reading at "California’s average residential electricity rate is 15.34 cents per kWh" and checked my electricity bill. Here in California, I'm paying 40.82 cents per kWh. The website https://www.electricitylocal.com/ https://www.electricitylocal.com/ seems wildly off. Edit: I did read the entire article. Just stopped to check what I was paying.
  • alchemist1e9 3y ago
    At that rate how are there any data centers in California?
    • kube-system 3y ago
      Industrial and commercial users of power almost always get different pricing than residential rates in the US.
      • alchemist1e9 3y ago
        Different yes but not 4x different. I know in Illinois standard generic commercial is around 9 cents and residential is like 12 cents.
    • diggan 3y ago
      Not everyone has the same contracts and contacts.
  • zamadatix 3y ago
    Probably just older data based on looking at the trends https://data.bls.gov/timeseries/APUS49E72610?amp%253bdata_tool=XGtable&output_view=data&include_graphs=true https://data.bls.gov/timeseries/APUS49E72610?amp%253bdata_to... and https://data.bls.gov/timeseries/APUS49A72610?amp%253bdata_tool=XGtable&output_view=data&include_graphs=true https://data.bls.gov/timeseries/APUS49A72610?amp%253bdata_to... That said, mentioning it is one thing (it makes the case for low power devices even stronger) but why stop reading anything after the first error you identify? That doesn't guarantee you only get accurate information it just guarantees you'll miss out on good information, like the cheap high speed connectivity the article is actually demonstrating.
• bhouston 3y ago
  I saw this mini-computer recently, the UM790 Pro, and I almost bought one as well. The AMD Ryzen CPU, AMD Ryzen 9 7940HS CPU, has a rating of ~30,000 on the CPU Benchmarks page (https://www.cpubenchmark.net/high_end_cpus.html https://www.cpubenchmark.net/high_end_cpus.html) which is quite high end for the size/cost. The only caveat is that I am running a 10Gbit ethernet network at home and it wasn't that costly to setup. A 10GigE switch costs around $500 CAD right now and that is all you need.
  • Liftyee 3y ago
    > A 10GigE switch costs around $500 CAD right now and that is all you need. Surely your machines all need 10GbE NICs as well? Admittedly my hardware isn't the newest (2.5GbE at most), but from a quick search 10Gig PCIe cards are around $150 each. Meanwhile 10-20 Gbps USB3.0 is reasonably common already. Though, using Ethernet still has many other advantages over running USB C cables everywhere.
    • adrian_b 3y ago
      There are also adapters from USB4 to 10 Gb/s Ethernet, for the case when it is desired to use a switch or long cables.
    • bhouston 3y ago
      My MacMini M2 has one built in. My other machines, yeah, I did buy some, which go for around $100 CAD on Amazon (e.g. TX401). Ethernet is advantageous because my house is wired, so I can have my switching a central location.
    • robes 3y ago
      server-grade 2x SFP+ 10G NIC pulls are pretty easy to pick up from eBay for about $20 each With a multigig capable SFP+ module it can handle 2.5G/5G copper as well.
• mrkstu 3y ago
  On actual network gear you'd typically drop the interfaces into a subnet/broadcast domain, but the blogger here has put everything in /32s and it looks like it's creating a peer connection without a shared domain. Any particular advantage/disadvantage to doing it this way?
  • ComputerGuru 3y ago
    It doesn’t scale. Number of connections/hardware is exponential to the number of nodes.
    • cma 3y ago
      Isn't it just N^2?
      • ComputerGuru 3y ago
        Absolutely. Sorry, I meant quadratic but had a brain fart.
  • dfox 3y ago
    In this case the primary reason for configuring it as point to point links is that it matches the hardware topology. In theory there is a standards compliant way to create an ethernet-style network from this, but there are two critical software components to that that are not implemented in mainline kernels (namely raw Ethernet over TB and SPB). The main disadvantage inherent in the physical topology is that you are always going to do switching/routing decisions on the CPU. But as long as you use that as a virtualization platform or run k8s on that you are going to do that anyway and the additional overhead is probably irrelevant. (This assumes the full mesh topology, which with this hardware is not scalable over 3 nodes, might be to like 5 with somewhat more expensive consumer grade HW and is not really scalable to more than 8 nodes due to the sheer amount of cabling required)
• gaudat 3y ago
  USB4/thunderbolt is a magical protocol. Turns out the fastest way to move data between 2 modern PCs is to connect their thunderbolt ports with a USB-C cable. The connection shows up as a ethernet port on Windows and I can easily saturate a SSD with its 1GB/s+ transfer rate. And it just works (tm). Reminds me of firewire on these 20 yo Macs. And What happen if you wire both ports together on the same PC?.. Do you get a broadcast (thunder) storm?
  • jwells89 3y ago
    Target disk mode over FireWire was magical back in the day. Nothing like turning someone’s laptop into an oversized external hard drive to rescue data or get a borked OS X installation booting again.
    • xattt 3y ago
      FireWire had IP networking stacks in Windows and OS X in 2007. You could daisy chain a bunch of devices together to share a network connection.
      • FirmwareBurner 3y ago
        Sure but it was very expensive compared to USB and Ethernet so Firewire never caught on with mainstream conumers other than some niche cases like camcorders. Thunderbolt was also expensive, which is why adoption was limited, but it's becoming more maisntream since Intel and Apple have been pushing it in the last years, adn piggibacking over USB-C makes it an easy sell comapred to requireing a separate connector like firewire. Still, thunderbolt peripherals are way more expensive than USB ones, so like Firewire before, use is still more in the enthusianst/professional space.
        • GeekyBear 3y ago
          > it was very expensive compared to USB and Ethernet so Firewire never caught on Compared to Gigabit Ethernet back in that time period? Firewire was a huge bargain.
• ThinkBeat 3y ago
  My problem with setups like these is storage. Not much room for expansion in the small boxes and I am not thrilled with external drives. Still had a QNAP or other NAS and hope to get one with compatible USB4 port
  • jmrm 3y ago
    AFAIK those can have 2/3 NVMe drives, and you can expand them even more with a PCI Express card to add even more NVMe drives
    • ThinkBeat 3y ago
      This is true but adding 20GB Nvm drives costs a lot more than the spinnings cusins.
• notorandit 3y ago
  Maybe he can get 20 Gbps between 2 servers. You get half because of USB limitations (2 ports on the same hub).
• rahimnathwani 3y ago
  average residential electricity rate is 15.34 cents per kWh This didn't seem right, as I pay more than double that here in San Francisco. (I calculated $0.35/kWh by dividing the total I paid for electricity generation and delivery, and dividing it by the number of kWh consumed.) The linked page cites data from over a decade ago (2012).
  • mlyle 3y ago
    https://www.globalenergyinstitute.org/2022-average-us-electricity-prices https://www.globalenergyinstitute.org/2022-average-us-electr... Within the state, there's huge variation. The average is around 25 cents. E.g. if you live in Santa Clara, you pay 16.6 cents per kilowatt hour to Silicon Valley Power, while all surrounding cities pay 45 cents-ish. https://www.siliconvalleypower.com/residents/rates-and-fees https://www.siliconvalleypower.com/residents/rates-and-fees
  • krallja 3y ago
    Mine is currently 9.4¢/kWh, no TOU, net metered. PG&E is the problem, not the data.
    • delfinom 3y ago
      32 cents / kWh here in NY.
    • throwaway-blaze 3y ago
      Where in CA roughly is this?
    • rahimnathwani 3y ago
      I was responding to what OP said about California rates.
      • krallja 3y ago
        My mistake, sorry. Still seems atrocious to me. It’s not like we have better power generation equipment here in NC.
• Fb24k 3y ago
  I still remember connecting two MS-DOS computers together with a parallel cable, that was the easiest way to do "large" file transfers there for a while (used a program called LapLink, iirc)...
  • farkanoid 3y ago
    Laplink! That part ot my brain hasn't activated in decades... My favourite was bootstrapping using 'COPY /B COM1: C:LL3.EXE' to get the Laplink executable to the target machine over Serial, when you didn't have a spare floppy.
  • myself248 3y ago
    You could also run PLIP over the same cable. It's like SLIP (which is like PPP) but much faster. Where Laplink isn't really a network, just a file transfer thing that requires Laplink to be running on both PCs, PLIP is a network driver that lets you do all the usual network things over the connection. And since PCs can have up to 3 parallel ports before things start getting stupid, it's pretty straightforward to have a row of machines with PLIP links going both ways, bridging or routing the interfaces. Or, do PLIP-SLIP-PLIP-SLIP without adding any ports, and you could have a functional-but-brittle-and-slow network for pennies.
    • TacticalCoder 3y ago
      > You could also run PLIP over the same cable. I was running that in the nineties. My main desktop, running Linux, and an ultra old, ultra crappy laptop running Linux too. They'd be connected using PLIP and the desktop, more powerful, was running its own X server but also applications for that were running on the laptop's X server. So my brother and I could both be using Netscape to surf the net (we'd call it that back then) at the same time, over the 33.6 modem connection. It was really easy to run PLIP and was saving me the trouble to try to get network card running under Linux on my desktop and most importantly saving me the trouble to try to get the PCMCIA crap to work on my laptop. Fun times... P.S: and, yup, back then laptops had a full parallel port!
      • 3y ago
• z3t4 3y ago
  I suspect there will be some extra network latency using usb-c compared to a RJ hardware switch.
  • dfox 3y ago
    It is almost certainly the other way around. Even 1000-base-T PHY is complex block of analog magic that draws significant power and adds measurable latency over direct SGMII/optical link. In comparison USB4 is even more directly connected to CPU than the whateverMII interface of PCIe NIC.
    • johnwalkr 3y ago
      For a while at work I was trying to use ethernet for embedded stuff because it seemed more modern than CAN, i2c etc. I couldn't figure out why so few microcontrollers support it. It turns out just normal copper 100Mbps ethernet uses about 0.25W per port (0.5W for 1 point-point ethernet link) so it doesn't make sense for embedded applications. Gigbit uses more but I forget how much more.
• Sweepi 3y ago
  8 Zen4 cores dont beat 16 Zen2 cores in multicore. Either Geekbench 6 does not utilize all cores, or the 3950x setup is borked.
  • diamondlovesyou 3y ago
    GB6 will use the Zen4's AVX512, which Zen2 doesn't support.
  • jltsiren 3y ago
    Geekbench 6 multi-core is a single-task benchmark. It runs one task at a time, using multiple threads when possible. This makes it scale worse and favor single-core performance more than benchmarks that run multiple independent tasks in parallel.
  • wtallis 3y ago
    Geekbench 5 pretended that all multi-core workloads were embarrassingly parallel by running N independent copies of the single-core test across N cores, with zero communication between threads. Geekbench 6 takes a more realistic approach so Amdahl's Law applies and it doesn't scale linearly with more CPU cores.
• daxfohl 3y ago
  This would only work for close range, right? To do a whole-house thing you'd need to get a bunch of USB-cat6 adapters which would end up being more expensive than a switch?
  • manmal 3y ago
    Yes a couple meters max, at full speed. The full speed cables are also quite expensive.
• robocat 3y ago
  The full-mesh is three Mini-PCs where each PC connects to the other two PCs by USB4: unfortunately the image crops out the USB4 cable that links the top PC to the bottom PC which makes it a bit unclear.
  • langarus 3y ago
    it's linked in the article. It's this cable https://www.amazon.com/dp/B0BLXLSDN5?th=1 https://www.amazon.com/dp/B0BLXLSDN5?th=1
• thekombustor 3y ago
  10Gbps mesh for only $48, assuming you already have very modern (expensive) USB4 capable machines. The article also seems to make the assumption that a server would be pulling 1000W all the time, 24/7, which is rarely the case (of course, I can't comment on what their workload might be, but it would be quite unlikely) Still, I like the direct networking being done here. But saying "you can build a 10Gbps mesh for $50" when you have 3x $750+ machines seems a bit disingenuous. It is not unreasonable to get 10Gb SFP+ NICs on ebay for ~$50 a pop ($150 for 3)
  • E39M5S62 3y ago
    $50 would be roughly all-in for the SFP+ NIC, two optics and a generous length of multi-mode fiber. I just did this, and here's the breakdown of my costs from eBay: 1x Juniper EX3300-24p - $75 7x SFP+ optics - $7/each 3x Intel X520-DA2 NIC - $20/each 4x 3 meter OM3 LC-LC fiber - $6/each 1x 30 meter OM3 LC-LC fiber - $24 ----- Total: $232 The EX3300-24p has 24x 1gb copper ports with PoE+ on them, and 4x SFP+ ports. If you need more SFP+ ports you'll want to find a different switch - but for a small multi-use home network the EX3300-24p nicely matched my requirements.
    • justsomehnguy 3y ago
      > If you need more SFP+ ports you'll want to find a different switch CRS309-1G-8S+IN, Suggested price $269.00 >> Desktop switch with one Gigabit Ethernet port and eight SFP+ 10Gbps ports https://mikrotik.com/product/crs309_1g_8s_in https://mikrotik.com/product/crs309_1g_8s_in
      • E39M5S62 3y ago
        Mikrotik switches/products are pretty intriguing. Do you have any experience with that model? I'm interested in how stable it is and if the known bugs are anything to be concerned about.
        • justsomehnguy 3y ago
          Nope, but if you saw/used one Mikrotik then you saw them all. Personally I have a love/hate relationship with MT, with a little love and a lot of hate, but at their price range they are unbeatable and works 99% of time. https://www.servethehome.com/mikrotik-crs309-1g-8sin-review-inexpensive-8x-10gbe-switch/ https://www.servethehome.com/mikrotik-crs309-1g-8sin-review-...
• mcoliver 3y ago
  Re speed, looks like the cables are the right ones. Would be nice to find a wiring diagram of the motherboard to see how the pcie lanes are allocated but hard to find on these consumer devices. Perhaps each usb4 port is a single lane gen2 which would top out around 10Gbps. You could try paralleling iperf3 to give you a bit more info. Also check the tx_speed and rx_speed in /sys/bus/thunderbolt/devices to see what it is negotiating. Company specific TB driver/firmware updates can be found here https://www.thunderbolttechnology.net/updates https://www.thunderbolttechnology.net/updates
  • ralonso 3y ago
    I believe you might be right regarding the port itself being the issue here. I have a Minisforum UM775 (basically the previous gen of the 790 Pro mentioned in the OP) and despite having two "USB4" ports, only one is rated for 40Gbps while the second one is 10Gbps. I cannot find any specifics regarding the ports on this 790 Pro one, but I'm certain that's ultimately the issue: the bottleneck is on that second (the one on the right) USB4 port.
• stoltzmann 3y ago
  >While the machine itself isn’t expensive, it is not cheap if you consider the cost to operate. A machine like this is very power-hungry. Suppose the power consumption is 1000W per hour. Alright, a server will be more power hungry than e.g. your desktop, but... This specific Dell R630 has 2x 750W PSUs in it. That 750W is the maximum rating of one power supply, and there's 2 of them for redundancy - not for increased power intake. That server will run at 750W maximum - but that is the absolute, absolute maximum power it should draw. It's when you have all the rails loaded to the limit and running the server to the ground. A more realistic scenario would be e.g. 100W or so on average. The worse problem if running this server at home would be the terrible small high-RPM fans they have in 1RU servers. The loud high-pitched whine of them will drive you nuts. A better idea would be to get either a lower-power 1RU pizza box, or something larger that can take larger fans - replacing the fans with something quieter and adjusting the fan controller to spin at lower RPMs. >1000W per hour This is just wrong.
  • livueta 3y ago
    Yeah, I have a bunch of fully loaded R720XDs and with 12 3.5" SATA drives they sit at about 120-140W, both according to the lights-out console and wall wart meters.
  • november84 3y ago
    An addition, you have the ability to chain the PSUs for increased total power at the loss of redundancy.
  • strobe 3y ago
    noise actually not that bad with some bios configuration tricks - I used to run dell r720 and r420 together with ~20 HDDs near my desk table and most of the day noise was close to some non-silent gaming PC/workstation. Only problem was if is some need to reboot it, in that case it just blow fans to full power and it's loud like vacum cleaner that you for sure don't wanna to use at middle of the night.
  • mmastrac 3y ago
    I swapped out the screamer fans for Noctuas on a 1U server and it's significantly better. Needed a custom mini PCB and a bios setting to ignore fan speed feedback but it's extremely quiet and runs cool. I don't think it uses much power when idle either. I think that rack servers being expensive is a myth.
• Yukonv 3y ago
  Related, Intel was showing off Thunderbolt Share at CES[1]. Allows Thunderbolt 4/5 device-to-device transfer of files. Theoretical speeds in the 20Gbps and 40Gbps for Thunderbolt four and five respective. One idea for why they were only able to reach 11Gbps is having only one Thunderbolt/USB4 controller[2], meaning the two USB4 ports split the 40Gbps PCIe lane. Throw in a full-duplex connection and you get 10Gbps in one direction. [1] https://youtu.be/GqCwLjhb4YY?t=81 https://youtu.be/GqCwLjhb4YY?t=81 [2] Just a theory but seems like a sane assumption.
• myself248 3y ago
  > I recall reading some networking books that mentioned interesting ancient network structures a long time ago, such as ring topology networks or daisy chain networks. IP-over-SCSI was great, you could throw 8 PCs on one SCSI chain. Put two SCSI controllers in each machine, do rows-and-columns, and you could have 64 hosts a maximum of 2 hops from each other, at U320 speeds, in the 1990s. https://www.linuxjournal.com/article/2344 https://www.linuxjournal.com/article/2344 Imagine a beowulf cluster of hot grits! Er, sorry...
  • yonatan8070 3y ago
    > IP-over-SCSI I've never heard of it, but given that it's possible, could you do theoretically do IP-over-SATA?
    • myself248 3y ago
      It's been talked about but as far as I'm aware, nobody's ever gotten it working: https://lore.kernel.org/linux-ide/4D6A5B72.6040600@teksavvy.com/T/ https://lore.kernel.org/linux-ide/4D6A5B72.6040600@teksavvy....
• Havoc 3y ago
  Would it not make sense to put the k8s network on 2.5gbe and give the three nodes a usb4 connection to a storage NAS instead? Assuming you can find a nas with 3x usb4 I suppose
  • alphabettsy 3y ago
    A NAS typically doesn’t use or support connecting to a PC over USB, that would be DAS(direct attached storage), but even then they don’t support connecting to multiple PCs simultaneously.
    • Havoc 3y ago
      The PCs in the article are connected to multiple other devices each so seems possible at least in principle. I don’t see why the same wouldn’t be possible for a NAS. Stuff like truenas can serve across multiple interfaces
    • robes 3y ago
      Assuming the NAS can do the same networking over USB4 that the author used it could work.
• BloodyIron 3y ago
  Didn't read the article, but the math they did about the R730 power draw is NOWHERE near realistic. A 1000Watt power supply unit (PSU) will only draw what is actually being used, not the full capacity 100% of the time. I have a fleet of R720's (generation before the example R730's) and their typical at-wall draw when running many VMs on them is about 150 Watts. So their math for that aspect is 100% WRONG. That being said, neat concept IMO.
  • deleted 3y ago
    [deleted]
  • armhole2625 3y ago
    I also have a fleet of R720s and can confirm BloodyIron's numbers. Average draw is around 150 watts.
• dgacmu 3y ago
  Somewhat related, I recently wanted to fix a slow nfs problem (1 server, 1 client -- basically, a file server and a machine with some GPUs). It turns out that used previous generation mellanox 100gig NICs are quite cheap now, and you can direct-connect two machines for about $600. (What I really wanted was the low latency, but the bandwidth is handy to have sometimes)
• theogravity 3y ago
  > Given California’s average residential electricity rate is 15.34 cents per kWh In the bay area it's 2.5x - 3x the rate. I stopped using my rack server for that reason. I was dumb and didn't consider the electricity costs when buying mine at the time years ago. It sits around collecting dust now. The author's alternative idea is really good considering 8w idle and 80w peak and I doubt it hits peak for what the author wants to do.
  • matmatmatmat 3y ago
    It's actually even worse in SDG&E territory. A kWh costs around 32 cents, but "transmission" and "distribution" are again twice that. The end result is about $1.00 / kWh.
    • thinkerswell 3y ago
      I’m convinced SDGE is the most hated company in the country. The southeast is so much cheaper for electricity.
• johnwalkr 3y ago
  I use a shorter, similar cable when I travel with 2 macbooks (work and personal). I like to use barrier (software kind-of KVM) to share my mouse and keyboard between them and work on one, but use the other one for music and reference material. I get work/personal separation with the convenience of using one mouse and keyboard. I simply set up static IPs on each one and also setup internet sharing from one to the other. Surprisingly barrier, internet sharing and even charging always works with this arrangement. I only have to connect 1 macbook to wifi and power.
• arghwhat 3y ago
  Nit: This is not a mesh. With only 3 nodes, it's not even a ring but just a direct connection between servers. A "real" ring is formed when you have more than 3 nodes and so some destinations require more than one hop. A mesh implies a network formed by somewhat arbitrary point-to-point connections with multiple possible routes between two points.
  • deleted 3y ago
    [deleted]
  • toast0 3y ago
    Full-mesh typically means each node connects directly to each other node (for some value of directly), so IMHO, a three node system with each node connected to the other two fits the name. I do agree that it's not very interesting with only three nodes.
    • arghwhat 3y ago
      A network where every node has a direct connection is a point-to-point network. Yes, it is a special-case called a fully-connected mesh, but these networks are rarely made (a fully-connected mesh of 32 servers require 32 NICs in each server and 496 cables) and it is not a common term. What is usually described as a mesh network (technically: "partially connected mesh") is a network where every node has a route to every other node, and that every node may have a direct connection to any number of nodes, but where there is no guarantee that any two nodes have a direct connection and may need to route over other nodes. The ability to handle this sort of completely arbitrary network topology is the core to mesh networks. See https://en.wikipedia.org/wiki/Mesh_networking https://en.wikipedia.org/wiki/Mesh_networking. There's also the old WiFi mesh spec (https://en.wikipedia.org/wiki/IEEE_802.11s https://en.wikipedia.org/wiki/IEEE_802.11s, what the OLPC used), not to mention Zigbee and Bluetooth Mesh.
  • tssva 3y ago
    This may not be a ring, but a “real” ring doesn’t require more than 3 nodes. A ring can be formed with as little as 2 nodes. If routing was enabled this would definitely be a mesh network. It would be a full mesh network where every node is directly connected to each other and a secondary path through another node is available should the direct path fail.
• nabilt 3y ago
  This is a pretty cool solution. I didn't know the capabilities of USB4 before this. The comparison with the Dell r630 power numbers got me interested since I just purchased a Dell r430 to host my site so I decided to benchmark mine. Specs: * 2x Xeon E5-2680 v3 (same CPU) * 64GB RAM * 2x power supply (can't remember if it's 500W or 750W each and too lazy to look) * 1 SSD & 1 7300 RPM HDD * Ubuntu server 22.04.3 LTS Using a Kill-A-Watt meter I measure ~100 watts after boot. Running sysbench (sysbench --test=cpu --threads=12 --cpu-max-prime=100000 --time=300 run) I get up to ~220 watts. If my calculations are correct that's 72 kW per day or $11.05 per month at idle: 0.1 kW * 24 hours * 30 days = 72 kWh 72 kWh * 15.34 cents/kWh = $11.05 and 158.4 kW or $24.3 per month during load: 0.22 kW * 24 hours * 30 days = 158.4 kWh 158.4 kWh * 15.34 cents/kWh = $24.3 I'm not sure of OP's use case, but these numbers are probably more realistic than using the max wattage of the power supply for most people. I will still be hosting in a co-location for the reliable internet and so I can sleep without the sound of a jet engine taking off. Those fans are loud!
  • justsomehnguy 3y ago
    > I'm not sure of OP's use case Lack of understanding. Even comparing 65W to 1000W should had ring some bells, but. > but these numbers are probably more realistic Almost, depends on the load (hardware) and load (software), as someone who manages a fleet of 720/730/630, a standby server eats around 150W and under the load up to 300-350W, depending on the package. > Using a Kill-A-Watt meter You can use built-in measuring in iDRAC.
    • nabilt 3y ago
      Thanks, good to know. I may need to request a 3A 1U in that case. I didn't know iDRAC could measure real time power usage. Pretty amazing.
      • buffington 3y ago
        I inherited a few R420's from my work, and the coolest thing about it is the iDRAC. I'm not a SRE or anything of the sort, so don't get to see stuff like that much, but the utility of the iDRAC is fantastic. The machines each have 192GB of ram, so I thought I'd set them up as LLM inference machines. I figured that with that much ram, I could load just about any model. Then I discovered how slow the CPUs on these older machines is. It was so utterly slow. I have a machine I bought from Costco a few years ago that was under $1k and came with a RTX 3060 with 12GB of GPU ram. That machine can run around 20+/tokens per second on 13B models (I actually don't know - I stream the text, and cap it at 9 tokens per second so I can actually read it). The R420? Its tokens per second were in the 0.005 to 0.01 range. So, yeah, not a good CPU for that sort of task. For other stuff, sure. I thought I'd setup a small file server with one instead, but the fans are so jet engine loud that it's intolerable to have in any part of the house, even when managing fan speeds with software.
• neilalexander 3y ago
  > But still, I don’t understand why it can only hit 11Gbps at this moment The interface/bridge MTUs might want increasing to a larger value. I notice a pretty big difference when connecting Macs together using Thunderbolt with an MTU of 9000 vs 1500.
• amelius 3y ago
  Does USB4 have the galvanic isolation that ethernet has?
  • ApolIllo 3y ago
    Thunderbolt 3 had a fiber option for longer runs, but I don't see it for Tbolt4.
• nope96 3y ago
  How good is error handling on USB 4? The rare times I've had data corruption it was almost always when transferring lots of data using USB (this happened to me on multiple drives, computers, usb ports or cables. this was in the early USB 3 days)
  • TexasMick 3y ago
    Bulk has bus error detection and so you shouldn't get errors
• AceJohnny2 3y ago
  This was a selling point for the tras^H^H cylinder Mac Pros (2013), with Thunderbolt 2. I hear some teams constructed compute clusters based on that...
• jauntywundrkind 3y ago
  The speed here is concerning to me. This is way short of what I'd expect or hope for. I hope in the future reviewers start testing this sort of thing when doing in depth reviews; this should be very visible high pressure an objective. Hopefully there's wins available & this isn't some silly unadvertised hardware gotcha. Higher mtu, trying some parallelism are two good suggestions I heard. One other thing I'd note: it's only been very recently that Linux has learned how to let the USB and DisplayPort negotiate/allocate bandwidth. It was evenly split between the two until March. Linux 6.3 Adds Thunderbolt/USB4 DisplayPort Bandwidth Allocation Mode. Linux 6.3 Adds Thunderbolt/USB4 DisplayPort Bandwidth Allocation Mode It's unclear how automatic this bandwidth management is under what systems; users might need to use the new thunderbolt-utils from July to adjust it manually. Intel Rolls Out thunderbolt-utils To Manage USB4/Thunderbolt Devices On Linux. https://www.phoronix.com/news/Intel-Linux-thunderbolt-utils https://www.phoronix.com/news/Intel-Linux-thunderbolt-utils I really want to hope much better is already possible. I don't own any USB4 systems though! How awesome they are. I hope we see some cool all-in-one's with multiple USB4 in; upcoming Minisforum V3 tablet for example has at least one usb4 that can do DisplayPort In, if I understand, for example, and that capability feels like it should just be coming for free now on PC's USB4 ports!!!
  • TexasMick 3y ago
    It's a problem with anything after USB3 in general, bandwidth is very unpredictable.
• lathiat 3y ago
  I recently wanted to do this and was surprised to learn that USB3 did not have a point-to-point mode like thunderbolt - some controllers had "dual mode" ability but the vast majority don't. I was doing this on Macs with thunderbolts 10 years ago but still couldn't do it on a PC today. Despite being great in most ways, AMD desktop systems in general rarely have thunderbolt (unlike Intel) - a few niche motherboards have it. You also can't just add it with a PCIe card, it requires special support from the motherboard. Hopefully this might change with USB4 not sure.. be interesting to see what the lower end motherboards end up shipping support for.
• PeterStuer 3y ago
  Side note about the author's distrust about cheap switches 'from China': i have installed many switches from different brands over the years, and the most reliable by a large margin where from TP-Link, easily beating the likes of HP, Netgear, Linksys and Cisco.