24 ms·

25 Gigabit Linux internet router PC build

• deleted 5y ago
  [deleted]
• uvesten 5y ago
  Great :) I’ve been wondering what equipment to get when I upgrade to the 25 gbit connection too. Maybe a custom build is the way to go after all!
• j1elo 5y ago
  I'm always amazed by people who don't bat an eye on the perspective of having a home server, sucking up electricity 24/7. In this case replacing a typical router (which consumes like a lightbulb), with a full-fledged PC (probably consuming like x100 the power). I guess some people around the world have quite cheap utility bills! For me, it's either a Raspberry Pi type of power consumption, or else a server that only powers on when needed. But I haven't learned yet how to do the latter, if possible at all. In fact this is a nice place to ask: how would you build a "something" that monitors the network for packets sent to powered-off machines, then somehow caches the request, powers the destination machine On, and finally lets the request continue to its target? Has this been tackled anywhere? There must be tons of people wanting a homeserver but living in places where electricity has a considerable cost...
  • philjohn 5y ago
    There is a middle ground - either an Intel Atom or Intel Celeron based server - typically use around 10w and you still get niceties like SATA and even PCIe
    • easygenes 5y ago
      Also plenty of slightly more capable laptop class SKUs that are in this ballpark.
      • vladvasiliu 5y ago
        Can you still find "desktop" motherboards to go with them? A long time ago, I remember there being desktop MBs for the Pentium M.
        • londons_explore 5y ago
          For an always running thing, choosing a laptop motherboard is a nice way to guarantee all the power saving features will work properly. Most desktop motherboards won't allow the PCI express bus to sleep for example, using an extra ~4 watts, $8 per year in eco-friendly countries. Across the ~10 years your NAS will probably sit in a cupboard, that's $80, which isn't an awful lot, but probably would have let you buy a bigger SSD or hard drive which would have had more utility.
          • vladvasiliu 5y ago
            That's true, but then again I wouldn't be comfortable with a single-disk NAS... However, for running random VMs, etc, I think a laptop is quite a good proposition.
    • ttsiodras 5y ago
      If you are OK with external USB drives, I did just that with my Atomic PI [1] (Atom-based, 35$ SBC, 3.5W idle, running Debian on its internal EMMC, with two external USB drivers in ZFS-mirror configuration). [1] https://www.thanassis.space/atomicpi.html https://www.thanassis.space/atomicpi.html
      • philjohn 5y ago
        You can also get rackmount servers in 1U format with Atom CPU's that support 4 3.5" drives, e.g. https://www.broadberry.co.uk/intel-atom-rackmount-servers/cyberserve-atom-104s https://www.broadberry.co.uk/intel-atom-rackmount-servers/cy...
  • baybal2 5y ago
    What a lot of other people don't realise: - There is no such thing as a 25gbps home router. What's currently on the market is a very serious overkill ever for 10gbps.
    • great-potential 5y ago
      In terms of CPU it is not totally overkill if you're using an IPS/IDS, bare in mind you'll also be disabling most of the network card offloading in a full fledged firewall and that will ultimately result in consuming slightly more CPU cycles.
  • hocuspocus 5y ago
    > I guess some people around the world have quite cheap utility bills! Or they work at Google and don't really have to care. Electricity is not particularly cheap in Switzerland, but not particularly expensive either (nothing like Germany for instance). If running a home lab is your hobby, why not. There are plenty of hobbies that are a lot more expensive.
    • ianai 5y ago
      The machine's clearly massive overkill for routing. Another commenter points out it typically runs around 50w - and the OP says they're also using it as a server in this thread.
      • dijit 5y ago
        FWIW: raspberry pis (the 4 series) sucks in 15w which is why it’s so bloody difficult to power them through standard USB power adapters (which go up to 12w). So, not hundreds of times more power hungry, but definitely 2-3x
        • ajsnigrutin 5y ago
          yep, which makes them a pain in the ass to efficiently cool down without active cooling (=fans)
          • larschdk 5y ago
            My passively cooled RPi4 with 8GiB RAM + 1GiB/USB ethernet dongle working as a 1Gb router never goes above 50 degrees C (all-metal case from aliexpress).
        • geerlingguy 5y ago
          Just to note that's peak power consumption. They idle between 1-4W, or if you're running a PoE HAT, 4-6W.
          • 5y ago
  • great-potential 5y ago
    I think if the OP uses this only as a router it is indeed a waste of power. What I would probably do is also use it as nas/workstation by using virtualization, SR-IOV is now pretty standard on these cards.
    • secure 5y ago
      I’m using the machine also as a server, so it replaces 2 existing devices. In terms of power consumption, it’s likely only a small increase, if at all.
      • Datagenerator 5y ago
        Also, the user can use his freedom to build and do whatever wished for. This tone of you shouldn't do this because some COTS is available sounds very commercially driven. People have the power to create another Google, we are not powerless.
      • ianai 5y ago
        My only rub with that is - shouldn't the router be only a router from a security perspective? Definitely combining servers for home use does make sense though.
        • secure 5y ago
          Ideally yes. But with the resources needed for 25 Gbps, not using one machine for multiple purposes seems wasteful. The server only stores publically available data, though, so not a big deal from a security perspective.
          • ianai 5y ago
            What do you mean by resources required? expansion ports on the mb?
        • great-potential 5y ago
          Yeah running your nas-bittorrent/firewall-router on the same OS is clearly not something I would do, especially that now you can use somrthing like proxmox for example and virtualize your pfsense/opnsense instance.
  • IgorPartola 5y ago
    What kind of light bulb? :) A typical consumer router will take up something like 5-10 watts. A PC will suck in 20-50. Remember that the PC will be a lot more powerful so it’ll spend most of its time with low CPU usage. Say the worst case scenario: 45W difference. 45W * (24 * 30) / 1000 = 32.4 kW*h/month. At $0.10 kWh rate that’s $3.24/month, less than a cup of Starbucks coffee.
    • mixermachine 5y ago
      Add HDDs and 10 GBit/s hardware and you surely reach 60 watts or more idling. Many home labs also don't use the newest hardware but old server hardware (add 10 - 40 watt idle). A NUC or other Laptop hardware on the other hand would be OK to run. Maybe SSD raids are (financially) feasible for everyday files in the near future. You are lucky to pay only $0.10. Here (Germany) I pay 0.30 Euro per kW/h. About $0.36 dollars. I turned off my home NAS a few months back and partially switched to VPS services (also because of the better connection).
      • iforgotpassword 5y ago
        > Many home labs also don't use the newest hardware but old server hardware (add 10 - 40 watt idle). Yes, but switching to newer, more efficient hardware is not free, so you need to factor this in when considering a switch. Also, in case you don't only care about your wallet but also environmental impact it's getting even more complicated, since manufacturing your new shiny toy definitely is polluting the environment somewhere in Asia, gets shipped via ship burning horribly dirty fuel in international waters etc. If you don't have any use for the old hardware, chances are it will end up burning on some open field somewhere in Africa after local recyclers already extracted the good stuff...
        • ClumsyPilot 5y ago
          "your new shiny toy definitely is polluting the environment somewhere in Asia" Thats true whether I use dropbox or my own server
    • foolmeonce 5y ago
  • deleted 5y ago
    [deleted]
  • melling 5y ago
    He claims one of his goals is to be power efficient “The PC consumes about 48W of power when idle (only management …” https://www.eia.gov/tools/faqs/faq.php?id=97&t=3 https://www.eia.gov/tools/faqs/faq.php?id=97&t=3 “ How much electricity does an American home use? In 2019, the average annual electricity consumption for a U.S. residential utility customer was 10,649 kilowatthours (kWh), an average of about 877 kWh per month”
    • iforgotpassword 5y ago
      Ok so what am I missing here? In Germany a typical four person household is 2000kwh a year. Surely air conditioning can't make up for such a difference? Everybody already switched to electric cars?
      • BenjiWiebe 5y ago
        I don't have the numbers, but air conditioning can suck up a LOT of energy. I'm pretty sure air conditioning would be (in our household) the single biggest energy user.
      • tzs 5y ago
        That's about 170 kWh/month. I'm in the US and use around 450-500 kWh/month in the summer, which is my time of minimal use because it is warm enough to not need heating and usually not so warm as to need air conditioning. I'm curious why such a big difference. Here's what I'm using in a household of one. 1. Kitchen appliances: Fridge, microwave, toaster, oven, dishwasher. The first three are used daily. The oven and dishwasher once per week. I'll also occasionally use a bread machine and an electric kettle, maybe a couple times a month. 2. Washing machine and dryer. 4-6 times per month. 3. A 2017 27" iMac, a Raspberry Pi 3, cable modem, TP-Link A7 router, and two TP-Link SG108E switches. A USB hub and a case for a couple external SSDs. An external monitor. 4. A 55" LCD TV which is used a couple hours a day, and an A/V receiver which is always on but usually idle. A couple streaming boxes (Fire Stick 4K, Xfinity Flex). 5. iPhone, iPad, Apple Watch, and Surface Pro 4 chargers charging those devices. 6. Indoor lighting. It is almost all LED. The only exceptions are the lights in the fridge and dryer, one bulb each in the attic and crawl space that I forgot about when switching to LED. 7. An outdoor security light in back that is on overnight. I'm not sure about its power consumption, but these are typically under 100 W, which would be 24 kWh/month during the summer. (I'm a bit puzzled by that light. I'm still using the same bulb that was there when I bought the house, giving the bulb on "on" time since I've owned it of around 56000 hours. That's quite a bit longer than expected for every kind of light bulb that I can think of that this might be. Only LED should approach that and definitely is not LED). 8. Well pump and water heater. The well pump runs maybe a couple times a day for maybe 5 minutes at a time, and would account for at most a few kWh/month. I know the water heater is a beast--but I think it only runs a little more frequently than once a day. 9. A couple box fans in windows at night to blow out hot air. They are 50 W each. Maybe 50 kWh/month. 10. Miscellaneous. Charger for rechargeable AA and AAA NiMH batteries. Electric toothbrush. A Google Home Mini as a kitchen timer. An Echo Dot to control lights. a Hue hub for the lights. Charger for the batteries for some cordless tools (drill, string trimmer, hedge trimmer) that are all used rarely.
  • FpUser 5y ago
    >"I'm always amazed by people who don't bat an eye on the perspective of having a home server, sucking up electricity 24/7" Some people simply run business from home and it is legitimate business expense. For example in Toronto server consuming 200 Watt 24x7 comes to about $20/month. Not much to dwell about if you are making money as a business.
  • easygenes 5y ago
    I have a computer in a room that’s generally otherwise too cold… so all that power is just useful heat dumped into the room.
    • londons_explore 5y ago
      If you pay a lot for electricity, you could get heat cheaper with a heat pump, gas furnace, or simply adding insulation on the roof/walls so any heat you do add lasts longer.
      • varjag 5y ago
        Re-insulating a house is a substantial cost that can take a decade to pay back in utility bills.
        • londons_explore 5y ago
          Yet spending $6 on draught excluders to keep the wind out can pay back in just a day or two. It all depends on your baseline...
    • midasuni 5y ago
      Albeit not as efficient as say a heat pump
      • deadbunny 5y ago
        Significantly cheaper though.
        • yjftsjthsd-h 5y ago
          Cheaper to buy, but not to run
        • easygenes 5y ago
          Commodity window-mount heat pumps which run off household power outlets can be had for a few hundred dollars, and are good for a minimum of a couple kilowatts of heat output. Any computing solution which dumps a couple kilowatts of heat and didn’t fall off a truck would probably cost more than that. Computers are fairly efficient heaters, but they’re a lot more expensive than heaters.
  • rzzzt 5y ago
    I do remember a "smart NIC" making its rounds in the news a few years ago -- although "few" in this case might be >10. It was essentially a low-power machine in itself (with no Raspberry Pis in sight yet) that could finish an HTTP download or receive e-mails without waking up the host computer. But that's more high-level than the buffering/relaying of packets you are thinking of. Edit: a bit weird, I wrote a very similar comment a year ago about this item. Now I feel slightly compelled to find the news source for it...
    • rasz 5y ago
      asus "killer nic", almost pure marketing wankery with little benefit https://www.pcgamer.com/motherboards-with-killer-network-adapters-arent-worth-your-money/ https://www.pcgamer.com/motherboards-with-killer-network-ada...
      • rzzzt 5y ago
        "Application offloading" was one of the terms I tried when searching for it, and got this as a result - not the same thing, but similar: https://www.academia.edu/10225597/Putting_it_on_the_NIC_A_case_study_on_application_offloading_to_a_network_interface_card_NIC https://www.academia.edu/10225597/Putting_it_on_the_NIC_A_ca...
        • rasz 5y ago
          https://pcper.com/2007/03/bigfoot-networks-killer-nic-k1-and-fna-bit-torrent-client-review/6/ https://pcper.com/2007/03/bigfoot-networks-killer-nic-k1-and...
  • teekert 5y ago
    My home server is about 70W, meaning about 150€ a year (~0.20 eur/kWH). So about 12€ a month. For one of my biggest hobbies. It’s fun, but it also runs WireGuard, 3 NextCloud installs, Home Assistant and Mosquitto, 2 MineCraft servers, a FoundryVTT instance, samba, sabnzbd, Unify Controller, an Nginx static site, LibreSpeed, VaultWarden and soon a Django site. Oh and a virtual desktop (vnc) I can always leave running with stuff open. Do I need all of that? Meh, but it’s not bad value for money-wise IMHO. Perhaps mostly because I just enjoy it. Edit: of course the thing itself was also quite expensive but not much more than a decent NAS which I think is a must for many people anyway.
    • binkHN 5y ago
      > but it also runs ... MineCraft servers At least you have your priorities straight :P !
    • josephcsible 5y ago
      > (~0.20 eur/kWH) Are such high electricity prices common in Europe? That's higher than it is in 49 of the 50 US states (assuming 0.20 EUR = 0.24 USD, and with Hawaii being the exception).
      • cinntaile 5y ago
        In Western Europe at least this is quite common. Mostly because politicians have discovered this as a great tax resource.
      • selectodude 5y ago
        Electricity in the EU is really really expensive. That person is actually paying below average rates.
      • zbrozek 5y ago
        That's cheaper than California!
      • mixermachine 5y ago
        0.30 Euro per kW/h in Germany
  • gravypod 5y ago
    You can dynamically turn on and off things with MaaS and monitor power though most UPSs.
  • rasz 5y ago
    let me introduce you to "My PlayHouse" https://www.youtube.com/watch?v=Jhwj_aCEYc0&list=PLS2odYzlaogJuAvN2dGRz-oNuBSA27UHp&index=9 https://www.youtube.com/watch?v=Jhwj_aCEYc0&list=PLS2odYzlao... fun stuff starts at 5:00
  • iforgotpassword 5y ago
    > who don't bat an eye on the perspective of having a home server, sucking up electricity 24/7. Maybe they need it? A router isn't a home server. But even if, why do you keep your power efficient router running 24/7? You could turn it off when you're not home to save even more energy. > a full-fledged PC (probably consuming like x100 the power Maybe do some research before claims like these. He states this pc idles at 48W. Please show me a router capable of handling 25gbit/s that consumes 0,5W at idle. > how would you build a "something" that monitors the network for packets sent to powered-off machines, then somehow caches the request, powers the destination machine On, and finally lets the request continue to its target? That's what "wake on LAN" is for.
    • yakubin 5y ago
      > But even if, why do you keep your power efficient router running 24/7? You could turn it off when you're not home to save even more energy. You jest, but I turn off my router for the night and each time I leave home for more than a day. Not just router to be exact, everything that's connected to a power strip goes down, as I turn off all the power strips. (Not OP.)
      • 83457 5y ago
        Why not just go all in and shut off at breakers?
        • oriolid 5y ago
          Fridge.
          • herewego 5y ago
            Then don’t flip the fridge breaker.
            • detaro 5y ago
              Plenty places don't have a separate breaker for that.
  • naikrovek 5y ago
    Have you measured how much electricity a computer doing routing consumes? I bet it's a lot less than you think. Also measure how much a good, high-bandwidth router uses. I bet that is a lot more than you think. the "why do this" question for me comes about entirely because of the continual manual intervention that is needed for solutions like this. just not worth it, to me.
    • jacoblambda 5y ago
      I mean I have and my router is on average drawing around 15W but it also isn't remotely close to being able to hit 25Gbps. I don't think 50W (apparently the draw of this router) is unrealistic for a 25Gbps router by any means.
  • unixhero 5y ago
    Maybe they have money and they find that spending money on homelabbing is worth the effort. And really it isn't THAT expensive either. I for one don't bat an eye on my USD200 yearly homelab electricity cost. Of course if one wishes to perfectly optimize everything then having a homelab might well wasteful.
    • sponaugle 5y ago
      Indeed every case is unique. My home lab runs about ~4kw all the time, which end up being about 2.8mWh/month. Here in Oregon that ends up costing $340/month. This is a brand new house, and as soon as I get permits approved I'm adding 21kw of solar which will help offset that.
  • tyingq 5y ago
    His starting requirements include one pci card for 25gb and another for 4 port 10gb. I don't think there's any low power way to do that. Any motherboard with enough slots, CPU power, and PCI lanes to handle all that aggregate bandwidth isn't going to be low power setup. If you underpowered it, then you might as well back off of 25gb/10gb. So it's not really a home server. It just happens to be in a home.
  • mirekrusin 5y ago
    45W is not really "sucking up electricity", it's really nothing compared to other things in the house. I don't think PC is consuming 100x more as well, less than 10x and the same or nothing when not used.
  • ClumsyPilot 5y ago
    I put together a tiny server with power consumption in mind, it consumes 15W and has i3-8100, that's enough to run just about everything I can be bothered to run: kubernetes, owncast, homeassistant, pihole, etc. Generally, a laptop or Intel Nuc will give you a good low power server, much better platforms for development than a PI. Then if you decide you need a RAID array, that's a different ball game.
  • ab3rC1te 5y ago
    You know you could just go buy an alternator and rig yourself up your own infinite power supply. Depends on the Amps of the alternator.
  • vbezhenar 5y ago
    > In fact this is a nice place to ask: how would you build a "something" that monitors the network for packets sent to powered-off machines, then somehow caches the request, powers the destination machine On, and finally lets the request continue to its target? Has this been tackled anywhere? There must be tons of people wanting a homeserver but living in places where electricity has a considerable cost... IP protocol does not guarantee delivery. So you don't have to cache request. Just power on the machine, client will retry sending the packets until the machine is powered on and can respond. Just make sure that there's no gap between networking available and http server is still starting on.
    • secure 5y ago
      I have built such a gateway for SSH: it accepts the connection, powers on the target machine if needed, then forwards traffic. https://github.com/stapelberg/zkj-nas-tools/blob/master/wolgw/wolgw.go https://github.com/stapelberg/zkj-nas-tools/blob/master/wolg... The advantage over just catching any packets is that an SSH connection is authorized, so less noisy in terms of undesired wakeups.
      • secabeen 5y ago
        Apple has this. It's called Boujour sleep proxy, and it allows a always-on device like a Airport or AppleTV to claim the IP address of a sleeping Mac, waking it with a WoL packet when traffic for that device comes in. It's probably pretty useful, but is annoying on a managed network, as it spams my arpwatch system every time the IP is transferred between devices.
  • johndhowell 5y ago
    I personally run two servers with a NAS 24/7 acting as media, ftp, and web server. My electric bill maybe increased a couple of dollars, but otherwise I haven't really noticed any significant increase. I'm based in the southern US
  • tibbon 5y ago
    I’ve got a handful of computers running in a rack at home. Their total power? Around 400w. I also have a dehumidifier running all the time in the basement. It’s power usage is 500w. No one bats and eye at a heater or dehumidifier, but a computer and people get worried
    • sponaugle 5y ago
      I agree with your sentiment. It is interesting to look at total power usage and power usage by device. I'm using IotaWatts to monitor every individual circuit in my house (over 200 of them), and it is easy to miss things that add up. With computers it is quite amazing the difference in power load based on cpu/disk load. When I run my PI calculator (which not only pegs CPU, but hits my SAN very hard) I see over a 1kw difference in my homelab power draw.
      • 542458 5y ago
        Sorry, your house has 200 circuits? Is every outlet on its own circuit or something? Maybe I’m biased because my house is old, but I’ve never heard of having that much copper in the walls.
        • sponaugle 5y ago
          A few different things drove up the circuit count: (1) It is a large house ( over 10,000 sq ft ). (2) There are two power paths throughout the entire house - generator and non generator, and most panel locations have two panels for that reason. (3) There are two incoming services from the power company - A standard 400 amp service, and a second 200 amp TOU (Time of use) circuit that is primary used for EV charging. And there are 4 EV chargers, two in the upper garage and two in the lower garage. (4) There is a 16kVa UPS and circuits behind that UPS that go to a few places in the house including the server room. Lots of cool pictures and info here: https://www.garagejournal.com/forum/threads/jeffs-mountain-side-shop-portland.409988/ https://www.garagejournal.com/forum/threads/jeffs-mountain-s...
          • 5y ago
  • adrian_b 5y ago
    "probably consuming like x100 the power" is greatly exaggerated. A typical PC might have an idle power consumption of less than 50 W, while a very small computer with ARM little cores might have an idle power of 2 ... 3 W, so at most the power consumption ratio would be 20. However even that is not realistic because a computer or appliance with less than 3 W power consumption will not be able to route 10 Gb/s or faster links and it will struggle even with multiple 1 Gb/s ports. A dedicated router appliance able to route 10 Gb/s or faster links will probably have an idle power of around 10 W or even more. For routing only 1 Gb/s links, you can use among the standard PC's a NUC-like model, which will have an idle power consumption between 5 W and 10 W, quite close to a dedicated appliance. This is actually what I am using for my own Internet router/firewall (which also runs many other services, e.g. DNS server and proxy, NTP, e-mail, HTTP server and proxy etc.) with 1 external 1 Gb/s port and 4 internal 1 Gb/s ports (4 of the 5 ports are made with USB to Ethernet adapters). Some of the more recent NUC or similar computers have multiple Thunderbolt ports or 10 Gb/s USB ports, so those can be used to route multiple 10 Gb/s links (using Ethernet adapters), with an idle power of around 10 W, similar to any equivalent commercial routers. For 25 Gb/s Ethernet links, a commercial router is unlikely to be much cheaper or to consume much less than a standard PC.
    • tomnipotent 5y ago
      SFP requires less energy than RJ45, only 0.7W/port. The MikroTik 4-port CRS305-1G-4S+IN draws 10-18W.
      • bcrl 5y ago
        That's not entirely true. SFP power consumption depends on the type of SFP. A 10Gbps DWDM SFP+ might draw 1.8W of power. The why for this is actually quite interesting: the lasers used in DWDM SFPs have much more stringent requirements for temperature stability to ensure the light emitted doesn't drift out of spec. In order to achieve that temperature stability, they use a built in peltier to pump heat away from the laser and control the temperature. They're quite the marvel of modern engineering!
        • 5y ago
  • Zenst 5y ago
    For some perspective, this is a nice list of many household appliances and their power usage: https://www.daftlogic.com/information-appliance-power-consumption.htm https://www.daftlogic.com/information-appliance-power-consum...
  • michael1999 5y ago
    Yes. In fact, apple’s AirPort Extreme units do that for macs in the house though their zeroconf networking. On sleep, the airport borrows the address and ARPs it. On traffic to that IP, it holds it, sends a special wake packet to the sleeping nic, and then re-sends the packet to with the real MAC, and the waking machine picks it up.
  • flatiron 5y ago
    I run just plex and pi hole. Money I save with plex that I don’t need a million different services and having no ads at home is well worth the price. One day my friends went out and I couldn’t go. They got all drunk and shared how much they rely on my plex and by the time I woke up I had close to $1k in PayPal and a note that said “for an upgrade and your time” so honestly I probably making out on the deal. Went from a haswell pentium to a ninth gen i3. 8 gigs to 16. SSD for the root drive. 8 TB staging server before going to google drive. $20 a month for unlimited google drive is way more than the power to run the server. But I have about 80TB on there. Can’t store that for less than $20 a month no matter how you slice it.
    • j1elo 5y ago
      I've been looking at the offers of big providers and the average seems to be pretty much standard: 2 TB for $10/month. All across Dropbox, Google, Apple, even Microsoft (the individual plan) are in the same ballpark.
      • flatiron 5y ago
        Google drive enterprise for $20 a month for unlimited is the best I’ve found. Rclone and encfs to mount and send the content. Almost 0 issues.
        • alternize 5y ago
          sounds intriguing for backup purposes. I was not able to find this pricing online (without contacting sales), would you have a link for this?
      • _huayra_ 5y ago
        I've been using pcloud which has a lifetime plan which is a pretty good deal. I recommend waiting until black friday or August 1 (Swiss national day; they're a CH-based company) when they severely drop the price.
      • flatiron 5y ago
        I’ve read that it costs them $2 to store a single TB a month. Which makes sense. $4 -> $10 is a reasonable markup. I store 80 TB for $20. They lose money on me.
  • setBoolean 5y ago
    I have a HomeKit power plug that is able to measure the power usage. It sits before most of my living room devices including: 2x OG HomePods (Standby) - 55" Philips OLED (Standby) - Hue Bridge 2. Gen (Running) - ATV 4K 1.Gen (Standby) - Netatmo Weather Station (Running) - Linksys MR8300 OpenWrt 21.02 (Running) - Home Server (i5-6600, 16GB DDR3, 2x 256GB 850 EVO, 1Gb USB3 Ethernet Adapter as second NIC, Running) Reading with all devices combined is 35-40W.
  • deleted 5y ago
    [deleted]
  • antonzabirko 5y ago
    I mean if it's a problem just solve it. Yes, the folks in texas who have to pay 10x for electricity probably shouldn't get in if thuy can't afford it. Also, get solar/wind if you are concerned aboit the impact.
  • swiley 5y ago
    Many of these things are really competing with a $20/mo VPS so a full PC that's on 24/7 is still cheaper. Yes you could go further: use portable slim apps that fit on ARM SBCs but then you have to re-do your server config and learn a new app. Plus, at least for me, my PC is on 24/7 anyway.
  • specialist 5y ago
    Why not? Every one needs a hobby. Another benefit to Stapelberg's noodling is setting expectations. Imagine you're bootstrapping a municipal ISP or mid-sized org. Dealing with vendors and products for the first time. What's reasonable? Who knows? Having these projects be a reality check is awesome.
  • zbrozek 5y ago
    I have a symmetric 10 gbps connection at home. I have an EPYC machine that plays host to a number of virtual machine guests for various tasks. One of them is running opnsense to be my router. It's not fast enough to route at line speed, but it's close enough that I don't care. A consumer router or a pi or something would be far slower, and I would start to care. Having a tolerably powerful computer doing this means that it's also my web server for several sites, stores local backups and handles offsite backups, acts as my print server, and hosts a Windows virtual machine for using proprietary software (e.g., for my label printers or firmware updates for random widgets like my Lutron light system). Quiescent appears to be something in the neighborhood of 40 watts. It's not nothing, but it's acceptable, especially for the utility. I don't pay for a VPS because I have the bandwidth and the capacity to self-host everything I want. I spend $8/mo in electricity to run that machine, and that will drop to zero when my solar array and battery become functional. If the power consumption were 5x worse I would probably not have gone this route.
  • hatware 5y ago
    I have decade old hardware that is certainly inefficient, but my services hosted have taught me so much it's not even worth the comparison. Yes, electricity is cheap in America but we all choose to spend money on our hobbies as we please. I'll bet there's a few nitpicks with your hobbies where you trade a lot of time/money/something that I don't necessarily understand. And that's okay.
  • allenrb 5y ago
    When I was younger, 100W was a light bulb. We’ve come a long way. :-)
  • juancb 5y ago
    I've used managed power strips for data center applications in the past. The strips have individually addressable power ports and a web interface, telnet, as well as SNMP support. They've been around for decades and are the solution to your problem of needing to remotely manage the power state of power hungry devices. If you set your servers to always power on after power is restored you can control them with that device. There's also Wake On Lan (WoL) support in a lot of systems, where you can use a correctly crafted "magic frame" to wake up any machine that received it.
  • harikb 5y ago
    There is such a thing as wake-on-lan https://en.wikipedia.org/wiki/Wake-on-LAN https://en.wikipedia.org/wiki/Wake-on-LAN
  • AdrianB1 5y ago
    My home server using a very similar configuration (same CPU, a lot less potent network) is what enables me to work from home, making a living, so the cost of electricity is compensated by the gas saved on commuting. I am also in process of adding solar to my house, covering a lot more than this server. If people use old servers in the kilowatt range consumption for fooling around having a server in the basement is questionable, building something with low power in mind is nothing to complain about.
  • cptskippy 5y ago
    My home has a 0.5-0.7kWh idle load, with the varying 0.2kWh being from the refrigerators cycling. I tried shutting down my NAS and Home server and it was less than 0.1kWh. I'm at a loss to explain exactly what is drawing 0.4kWh of power but my neighbor has a similar load on his. I'm suspecting all of the fancy and useless motion/astro light switches installed are partially to blame but there's no easy way to verify that.
  • cortesoft 5y ago
    My solar panels on my roof generate about 10KW/h more than I use. I can bank credits with the grid, but can never cash them out, so electricity is basically free for me.
  • f3d46600-b66e 5y ago
    My Netgear access point consumes 7-8watts. My home server, with 2*12TB drives, 2*3TB, and 1nvme, which also acts as NVR (for POE cameras), recording 24x7, and which also acts as owncloud server (and few other things) consumes 40-50watts (and it includes a 10gbps SFP+ fiber). It also includes a wireless card and acts as an access point. It also runs a few VMS, continuous integration server, pihole and other stuff. This is not x100, it's x7 times. And the utility is much much higher. Before, i was using amd Apu, and it consumed 20-30 watts, but did not support AES-NI, which made the disk access limited to 60MB/s :-( It's way cheaper to run everything at home than paying for the cloud, even if u include electricity cost. It's about $50/year for electricity (1 watt 24/7 =~ $1 per year)
  • Aeolun 5y ago
    It costs about the same as a Netflix subscription? The people that do this see it as a hobby, so it’s worth spending some money on.
  • insulanus 5y ago
    A Netgear Nighthawk R7000 idles at 8W, max 34W. That will get you 1 Gigabit. Bleeding edge CPUs from Intel and AMD do guzzle power when run at full load, but they also improve in power per IPS, and have added dynamic underclocking, so they also run Compare the TDP of a Pentium 4 from 2000 to a Ryzen 5 3600 or an i7 7700K (These TDPs are supposedly max power output, but take them with a nugget of salt) https://en.wikipedia.org/wiki/List_of_CPU_power_dissipation_figures https://en.wikipedia.org/wiki/List_of_CPU_power_dissipation_... Gotta update that power intuition!
• sliken 5y ago
  Was hoping for similar but 10 gbit. Anyone know a nice small/quiet linux box with 2x10 gbit for use as a router? I have a 8 port x 1gbit ubiquiti router, but ubiquiti seems to be going downhill, and 8 port x 10gbit routers are pretty expensive. My plan was a 2 port router and use vlan tagging to a 8+ port 10g switch. That way I can have separate networks for trusted ethernet, trusted wifi, untrusted ethernet, and untrusted wifi.
  • opencl 5y ago
    AsRock makes an X570 ITX board with 2x10gig ethernet, the X570D4I-2T. Though at $500 for the board alone the resulting system is not going to be particularly cheap.
  • zamadatix 5y ago
    Look for an old but not outdated enterprise switch and grab a refurb. Most very large enterprises redo the network on roughly a 7 year cycle and in 2014 10g uplinks with tons of 1g devices was very standard. E.g. for less than 100$ you can usually pick up an ers 4826gts on ebay which has 24 ports of 10/100/1000 copper with POE+ capability and 2 1/10G SFP+ ports. It has hardware switching and routing which will perform better than software, particularly on latency. As far as noise 1u devices are usually a bit whiney even when not pushing much air but that's usually fixed with 1 or 2 200mm PC fans. You'll still likely need something with NAT, preferably hardware NAT, for the actual internet handoff. Some of this class enterprise device have NAT support (not the 4826) but that'll usually be in CPU and perform not that great especially if you have like a gig connection. Or if you want to go pure software here any cheap mini PC with 2 gigabit Ethernet ports should be able to handle NAT reasonably for gig internet or less since it doesn't also have to do the 10g internal route/switch on top.
  • tyingq 5y ago
    I would go with a used SuperMicro A1SAI-2750F motherboard. It comes with an Atom 8-core C2750 (20w TDP) and a single 8 lane PCIe slot, but doesn't need active cooling. Supports ECC ram too. There's a passive heatsink on the CPU that works fine. I believe that would keep up with 2x10gb, be quiet, and power efficient. They are $200 or so used on eBay. So not terribly expensive for something that can push 10g. And it will fit in any Mini-ITX case. Don't be put off by the "Atom" branding. The C2750 was no slouch for its heyday.
• rwmj 5y ago
  Here I am having just had FTTH installed, but capped to a mere 160 Mbps. The main problem I am having is not the router (which handles such a "slow" speed fine) but the wiring throughout the house. It's all copper ethernet and because of the lengths and the fact that the runs must go parallel to some mains electricity cables, it tops out at 100 Mbps. I hope the poster has already put fibre around his house.
  • secure 5y ago
    “The poster” (hey!) has put fiber around their house :) https://michael.stapelberg.ch/posts/2020-08-09-fiber-link-home-network/ https://michael.stapelberg.ch/posts/2020-08-09-fiber-link-ho... https://michael.stapelberg.ch/posts/2021-05-16-home-network-fiber-10-gbits-upgrade/ https://michael.stapelberg.ch/posts/2021-05-16-home-network-...
• great-potential 5y ago
  Well, hate to tell you but you'll be limited 10Gb/s with these Intel cards because they require FEC.
  • secure 5y ago
    You mean 10 Gbit/s per link? Even on the 25 Gbit/s card? Where can I read more about FEC (forward error correction?) and how that affects link speeds?
    • great-potential 5y ago
      Yes the transceivers recommended by Init7 (bidi-LR) do not support FEC and you'll be running in degraded mode (according to the controller datasheet), and I don't think this is something that can be achieved by coding of the transceivers.
      • secure 5y ago
        init7 requirements are at https://www.init7.net/en/support/router-information/ https://www.init7.net/en/support/router-information/ The specific 25G optic is https://www.flexoptix.net/en/p-b1625g-10-ad.html?co10426=97205 https://www.flexoptix.net/en/p-b1625g-10-ad.html?co10426=972... and indeed does not support FEC. The remote end is a Cisco C9500-48Y4C, on which one can turn off FEC. On the Intel side, I found https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-ethernet-controller-feature-matrix.pdf https://www.intel.com/content/dam/www/public/us/en/documents..., which mentions that while No-FEC might result in “poor link quality” (sure, that’s why people use FEC), it does not say anything about degraded mode or limits to 10G. By controller datasheet, I take it you’re referring to https://www.mouser.cn/datasheet/2/612/710_series_datasheet_v_3_6_5_08_28_2019_final-1140607.pdf https://www.mouser.cn/datasheet/2/612/710_series_datasheet_v... That document contains a state machine diagram listing 25G AUI‐RS, 25G AUI‐FC and 25G AUI‐N (in order) before the fallback to 10G SFI. Did you misread or am I missing something? In either case, I’ll try the card. If it won’t link at 25G, I can get a different one. Any recommendations? :)
• adamcharnock 5y ago
  What is the practical consumer (or even SME) use of a 10 or 25G connection? As a consumer I’d definitely get it for the fun of it, but what is the point in reality? We’ve just launched an ISP a couple of weeks ago here in Europe. Peak bandwidth use is about 3mbps per customer, and customers will download the same amount of data regardless of their plan. I acknowledge this could be lack of imagination on my part, but the progression of bandwidth availability seems to be wildly outstripping demand (at least when it comes to fibre deployments). Perhaps someone living in SF with a 25G connection can disabuse me of this notion.
  • baybal2 5y ago
    > What is the practical consumer (or even SME) use of a 10 or 25G connection? Remote file storage, leasing servers for things like icecream, or basic HPC, self-hosting for a small business, even a small DC can run off it.
    • adamcharnock 5y ago
      Remote file storage I can just about see. But most consumers are streaming content (certainly TV, but possibly increasingly games), so where are all these multi-gigabyte files coming from on average 20/30-year-olds laptop/iPad? I don’t think people are shooting that much video, are they?
      • baybal2 5y ago
        Renting storage, and accessing it over 10gbps at home may be cheaper than building your own multi-tb storage rack.
        • adamcharnock 5y ago
          Absolutely. But from what I’m seeing, the average consumer’s need for storage space is decreasing not increasing.
          • 5y ago
• r1ch 5y ago
  One thing I've run into when building my own 10gbps router that I didn't see mentioned - you probably need to disable power saving / frequency scaling. The handling of packets is done in software, so when you start a download and the flood of initial data comes in at 10gbps, your CPU will still be running at 600 MHz and unable to keep up. It ramps up within 5-50 milliseconds depending on CPU, but for a TCP download the sender will have already interpreted the dropped packets as congestion and reduced their upload rate. You can monitor this with ethtool stats to see how many packets the NIC dropped due to host buffers being full.
  • secure 5y ago
    In none of my tests did I need to change power saving settings. My iperf3 transfers stayed at 10 Gbps throughout.
    • r1ch 5y ago
      You need to check your dropped packet statistics to know for sure. iperf3 TCP over LAN will not see throughput affected by a few dropped packets, a stream 100ms away may take longer to recover. High bandwidth UDP testing with iperf3 should also expose this, the first measurement will usually see dropped packets.
    • AdamJacobMuller 5y ago
      You won't see the impact because you're testing over something with very low latency.
  • bluedino 5y ago
    He was only using 15% of the CPU at max speed
    • r1ch 5y ago
      You can't measure the CPU at 1 second granularity to see this problem, in fact measuring the CPU at the sampling rate required may be enough to raise the CPU frequency :).
• timvdalen 5y ago
  I spent a couple of hours last week trying to set up a fiber connection in our new office. Since I didn't want to use consumer hardware in an office setting (and I already had a router that was compatible), I opted for getting my own hardware (a media converter). I had _some_ prior knowledge of networking, but had never messed around with VLAN and MTUs before. Luckily I found this[1] gist for a comparable setup that saved me. I'm still only getting 350Mbps where I should be getting 1Gbps, but I _think_ that's just due to the bad networking chip in my cheap chromebook. When I move all my stuff to the office, we'll see what we can top out at... All this is to say, I definitely respect the effort the author put into this and it pains me to find out that my new SFP setup is already obsolete :). [1]: https://gist.github.com/Ruben-E/abb9a4a872a7c4ffff058ae291ef2627 https://gist.github.com/Ruben-E/abb9a4a872a7c4ffff058ae291ef...
• clktmr 5y ago
  The most interesting part for me came at the end: It runs router7+gokrazy, a self-written pure-Go userland instead of the traditional GNU userland.
• cedricgle 5y ago
  I wish P4 fpga boards weren't so expensive for hobbyist. You can do fun stuff with them and create a custom network pipeline. It's totally overkill for a home network (they are used to create datacenter fabric) but it's nice and shiny :)
• formerly_proven 5y ago
  > I read on many different sites that AMD’s current CPUs beat Intel’s CPUs in terms of performance per watt. We can better achieve goals 2 and 3 (low noise and low power usage) by using fewer watts, so we’ll pick an AMD CPU and mainboard for this build. Unfortunately many reviews are very misleading here. Zen 2/3 CPUs have good performance per Watt, that's true. But for a machine like this, which will be mostly idle, this is not the interesting metric and Zen 2/3 systems show that you can combine good perf/W with poor idle power consumption (which is not true for their monolithic APU brethren, which are used in laptops). One of the biggest idle power hogs for these is the IO die, so make sure that XMP is disabled and the memory uses one of the slow JEDEC timings. This should be fine for a router. Check that the SoC/NB voltage is set to 1 V or less. Some boards set this higher. In the AMD CBS section of the board firmware there should be an item "SoC OC Mode" somewhere. Disable it. Some boards allow you to set a new PPT (package power target), but it's worth pointing out that values which are too low will make the CPU very slow because it essentially forces all cores to very low power states in order to meet the PPT since the CPU can't influence the baseline power (due to fabric and I/O die). The upside of using a reasonable PPT of e.g. 50-60 W is that you reduce power consumption if some errant task hogs the CPU. These settings make a big difference, but only if the CPU is really idle. Even fairly light loads (e.g. on a desktop, moving the mouse on the background) has everything rev up. In deep idle (nothing running at all, no user interaction on a desktop) you might get a Zen 2/3 CPU down to around 20 Watts, but as soon as anything is happening at all we're straight back to the 40-70 W region. Using an Intel system for this would have likely saved 10-20 W.
  • iforgotpassword 5y ago
    Yes I can unfortunately confirm this. Don't know how much has changed, but about a year ago my current employer evaluated about 10 desktop pcs from dell, lenovo, hp and the likes as all staff was supposed to get new machines. One of the important criteria was power consumption. The few AMD systems that were among the contestants had absolutely ridiculous idle consumption and weren't even considered any further.
    • ta988 5y ago
      At work people leave their computers and screens on (and unlocked most of the time)... (edit): I tried to educate but nobody really care...
• blackcat201 5y ago
  If anyone is interested in building 10Gb router on a budget you can buy Mellanox OCP NIC with an adapter from OCP to PCIe for around 5~30 USD ( depend on your location )[0]. I recently build one 10G router with 4x10Gb, 2x1Gb ports for around 200 USD using second hand PC. [0] https://www.youtube.com/watch?v=hkf3PkuKdOA https://www.youtube.com/watch?v=hkf3PkuKdOA
• great-potential 5y ago
  Not sure why you buy at fs.com when you have online stores in CH that are selling much cheaper: https://www.microspot.ch/de/computer-gaming/pc-komponenten/steckkarten--c589100/intel-netzwerkadapterkarte-pci-express-3-0-2-x-sfp28--p0001320718 https://www.microspot.ch/de/computer-gaming/pc-komponenten/s...
• daneel_w 5y ago
  At some point the idea of "use more case fans, then all of them run at lower RPM" simply turns on you and instead produce more noise. For the low wattages of this build a single case fan would've been sufficient to keep PSU and CPU fan at silent levels. I have a few more watts in my PC case, but not a single case fan, and the machine still hovers at a comfortably quiet level at 1 meter's distance. No temperature issues.
  • secure 5y ago
    Yeah, I’ll probably turn off some fans to see how the temperatures react. That’s for some time in the future, though… :)
• phreeza 5y ago
  One day init7 is going to poach you :)
• pmorici 5y ago
  Kind of surprised there wasn't much attention paid to the software. My understanding is that with stuff like DPDK and fd.io you can get much better performance without going to extremes with your hardware. Netgate TNSR is one product that puts all the opensource pieces together to make it easy but it is all opensource software so should be usable by the average home user.
  • virtuallynathan 5y ago
    I was going to say that... with fd.io this should be pretty easy these days.
  • h43k3r 5y ago
    Author has written his own router software and probably wants to continue to develop it. They are much more pro than your normal average home user.
  • floatinglotus 5y ago
    With a Xeon, a SmartNIC, and DPDK you can hit 100 Gbps.
• bastardoperator 5y ago
  I have a very similar setup using a desktop as a router (opnsense, 2 WAN, fibre/switch) and it has a single, yet fatal flaw. The power button. When you lose power, and the power comes back on, your router does not. If you are away, you must physically turn it back on.
  • fuster 5y ago
    Most (all?) PCs should have a BIOS option to always restore power after loss.
  • alexymik 5y ago
    You can usually change this in the BIOS (or equivalent). Look for "Power on after AC loss" or similar.
  • secure 5y ago
    I describe the relevant UEFI option in https://michael.stapelberg.ch/posts/2021-07-10-linux-25gbit-internet-router-pc-build/#asrock-b550-taichi-mainboard-uefi-setup https://michael.stapelberg.ch/posts/2021-07-10-linux-25gbit-... :)
• tohmasu 5y ago
  > Our network cards need PCIe 3.0, so that disqualifies 5 chipsets right away: only the A520, B550 and X570 chipsets remain. No, the linked Wikipedia page only shows the PCIe lanes connected through the chipset and doesn't account for PCIe lanes provided directly from supported CPUs. X470 supports CPUs which have PCIe 3.0 and a board with x8/x8 mode like the ROG Strix X470-F https://www.asus.com/microsite/motherboard/AMD-X470/ https://www.asus.com/microsite/motherboard/AMD-X470/ should work just fine (and has no fan).
  • secure 5y ago
    Thank you. I have added a note to the article with your correction :)
• Mave83 5y ago
  why no just buy a good nvidia/mellanox nic and route directly on it? Modern NIC's are amazing and you can offload everything if you like.
  • ju-st 5y ago
    but it is only offloaded to some ARM cores and not done in hardware? can they really route and NAT 25gbit? and the manuals for connectx-5 or nvidia bluefield don't even mention ipv6?
• specialist 5y ago
  Very cool post. Thank you. This article, and its many links, are helping noob me learn how to ask about I/O perf. TechEmpower benchmarks HTTP servers. While contestants continue to improve, I've long been curious what the theoretical fastest HTTP server could be. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ Basically, an update to the "C10K problem" for the year 2021. https://en.wikipedia.org/wiki/C10k_problem https://en.wikipedia.org/wiki/C10k_problem A bit like how Daniel Lemire initially thought about parsing JSON. Paraphrasing: Why can't a JSON parser run at wire speed? "Parsing JSON quickly: early comparisons in the wild" https://lemire.me/blog/2019/03/02/parsing-json-quickly-early-comparisons-in-the-wild/ https://lemire.me/blog/2019/03/02/parsing-json-quickly-early... -- Also, I really appreciate including the sound and cooling considerations. It's just great seeing the process of system design accommodating (balancing) multiple goals. Bravo.
  • goodpoint 5y ago
    > Basically, an update to the "C10K problem" for the year 2021. https://en.wikipedia.org/wiki/C10k_problem https://en.wikipedia.org/wiki/C10k_problem No, routing, natting and terminating connections are very different problems. Especially if we consider different implementations.
• tw04 5y ago
  I’d strongly recommend people take a look at the supermicro (or your vendor of choice) Xeon-d or AMD embedded 3000 series. Lower power draw, very quiet and more than powerful enough to push 25gbit with cpu left over for VMs.
• InTheArena 5y ago
  I have a Unifi UDMP, which while I think is probably the best prosumer option right now, falls way short due to a PPPoE problem that limits fiber connections to under 500mb/s, and the lack of load balancing across multiple link networks (which I totally admit, is a first world problem). I am thinking about building my own, but then comes the maintaince of all of the hardware / software. For me that would be fine, but for my family, it would be a total PITA to manage, and any downtime would be horrible. So, stuck with UDMP for the moment, until they either fix their problems, or alternatively I decide to bite the bullet and build this on top of linux.
  • subhro 5y ago
    Odroid N2?
    • InTheArena 5y ago
      Need a lot more power then that to handle 2x 1GB/s connections.
      • coder543 5y ago
        What? The UDM Pro[0] has a quad-core Cortex A57 CPU running at 1.7GHz, according to the spec sheet I'm looking at right now. It has 4GB of RAM. The ODROID-N2+[1] has a quad-core Cortex A73 CPU running at 2.4GHz. It also has 4GB of RAM. The UDM Pro is like half the performance of the ODROID-N2+, accounting for architecture and frequency differences, so... I'm not sure what you're getting at? The ODROID should be more than enough to handle a gigabit connection if the UDM Pro is even halfway capable of it, from a power perspective. Connecting more ethernet ports to it without an exposed PCIe connector is going to be clunky, but that's not the issue you pointed out. [0]: https://dl.ubnt.com/ds/udm-pro https://dl.ubnt.com/ds/udm-pro [1]: https://www.hardkernel.com/shop/odroid-n2-with-4gbyte-ram-2/ https://www.hardkernel.com/shop/odroid-n2-with-4gbyte-ram-2/
        • InTheArena 5y ago
          There is far more power in hardware than just the instruction set times the GHZ. Part of the UDM's problem is that there is evidently no offload for needed for PPPoE. It should also be self-evident, but I also need more the one interface to route traffic to two seperate networks, ignoring that the requirement is actually to load balance multiple systems to a internal 10gb/s network. The USB is nowhere near what you need to run a 10gb/s or even a 2.5GB/e (based on my experience with other ARM hardware).
• gigel82 5y ago
  Damn, $70 / month for 25Gbps symmetric; meanwhile, we're paying $130 / month for 1Gbps (400Mbps really) down / 40Mbps up in Comcast monopoly-land in PNW.
• spicyramen 5y ago
  Good post, bring me back old memories where I used to setup my own PCs back in high school. I think the graphics card and GPU are really not required for a router. Few years ago there was a Linux project called zebra router which was deprecated in favor of https://www.quagga.net/ https://www.quagga.net/
• Bud 5y ago
  The real story here is how great the internet service can be now in a civilized country if you don't let Comcast have monopolies and ban local internet cooperatives and coast along selling the same service for 15 years while raising prices. 25Gbps symmetric for about $70/month. That's significantly less than I pay Comcast for its crappy 900Mbps down/40Mbps up service.
  • nixgeek 5y ago
    At least in PNW the service isn’t 900/40 any more, it’s advertised as “1.2Gbps” and the profile actually seems to be 20% over that, I’ve seen 1.5Gbps or 170-180MB/sec downstream performance recently. Upstream is still <= 50Mbps and otherwise agree with your comment that the U.S. is falling behind in price-throughput on domestic internet services versus many other developed nations.
    • wormslayer666 5y ago
      There have always been regional variations in this; I've had the same 1.2GBps plan in a few states now and it only ever got close to that number in the one where Fios was an option. Comcast's only innovation in the last few years has been to add data caps to residential fiber service (my 300mb/s "1.2 GBps" is capped at 1.4TB/mo).
      • nixgeek 5y ago
        We’ve just been paying a $30/mo upcharge for unlimited data (removal of overage charges). Basically because it’s more predictable than the $10 for every 50GB overages. A few months we’ve had 4-5TB through the service and Comcast doesn’t seem to have slowed us down or charged more.
      • Bud 5y ago
        Let's please not further the idea that it's acceptable to quote only downstream speed when citing Internet speeds. Doing that just leads to 20 more years of Comcast killing the concept of modern Internet service for Americans. Cite both numbers.
• nimish 5y ago
  Without PPS measurements its hard to say how good this is. 1.488MPPS is needed to saturate a gigabit connection and that usually needs some tricks like poll mode drives/dpdk/etc on commodity hardware.
• louwrentius 5y ago
  I really respect this build, the only thing that I don’t like is the power consumption at idle of ~50 Watt. I understand that this is probably due to the used NICs but still. It might almost be interesting to setup a second router (a Pi4 might do) for regular casual internet usage + VRRP and only turn on this beast when more bandwidth is required. The 25Gbit machine would be the VRRP master and the Pi4 the slave.
• sydney6 5y ago
  To get a grip around the numbers, i quote from George Neville-Neil's Talk at BSDCan '15 "Measure Twice, Code Once" [1]: - 10 Gbps is 14.8 million 64 byte packets per second - 67.5 ns per packet or 200 cycles at 3 GHz - Cache miss is 32 ns [1] https://www.youtube.com/watch?v=LE4wMsP7zeA https://www.youtube.com/watch?v=LE4wMsP7zeA
  • eqvinox 5y ago
    You're not running a continuous stream of 64-byte packets in a home or SME setup. Also, assuming a 1:1 mapping to packet processing is a false dichotomy these days, NICs are doing an unbelievable amount of preprocessing, particularly grouping related packets together.
    • sydney6 5y ago
      No, of course not. A good starting point for real world performance benchmarking could be e.g. IMIX [1]. The example above represents the solely theroretical worst case as a means to establish a baseline for performance benchmarking. Anyway, if you are referring to HW offloading capabilities of "modern" NIC's, using techniques like LRO would break the "end-to-end"-principle of a router. [1] https://en.wikipedia.org/wiki/Internet_Mix https://en.wikipedia.org/wiki/Internet_Mix
• kortilla 5y ago
  The article leaves out the actual network packet processing. Is this being done by the kernel or is dpdk being used?
  • secure 5y ago
    It’s done by the Linux kernel with default settings.
• ksec 5y ago
  The most surprising details was [1] from init7. >Compatibility requirements There is no obligation on you to procure the hardware through us, and the hardware shown here is not the only possible hardware for you to use. There are also other compatible products, as long as the requisite «bi-di» fibre optic technology conforms to the following specifications (recommended: Flexoptix, more router information): No Modem / ONT. Just a Router with compatible SFP optic. I wish more ISP do that. And not force me to use your crappy ONT or Wifi Router. They could of course go another router and provide actual decent ONT or WiFi Router. But the chance of happening, or they care about quality is slim. [1] https://www.init7.net/en/internet/hardware/ https://www.init7.net/en/internet/hardware/
  • pabs3 5y ago
    A lot of fibre is GPON (a type of PON, passive optical network), a scheme where each customer receives data from multiple other customers as well as their own data, so it would be very insecure for them to allow customers to use their own ONT. If you can get AON (active optical networking), then likely they will allow BYO ONT. Note that AON requires more hardware that is more expensive on the ISP side.
    • ksec 5y ago
      I know there are lots of GPON Network that allows you to use your own ONT, and I think that is even by law in Germany where you could log in with username and password just like DSL. I know in GPON they are all shared but I never thought about the security aspect of it, so thanks for pointing that out I will dig a little bit deeper in to the subject.
    • yrro 5y ago
      Isn't the downstream traffic in GPON encrypted?!
      • pabs3 5y ago
        I just looked that up, and you seem to be correct indeed, AES-128 is used.
• unethical_ban 5y ago
  This is not related to the article, but to networking in general: I just discovered Brocade switches and how cheap they can be for what you get. https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/ https://forums.servethehome.com/index.php?threads/brocade-ic... Between wanting to dabble with a 10G file server and thinking of hosting 40-60 person LANs, this is incredible.
• kiririn 5y ago
  This build would be simpler, better and probably cheaper overall with a server grade motherboard and ECC RAM
• Aissen 5y ago
  @secure: I'm a fan and love what you did with router7, gokrazy and your homelab. Note that this might be seen as "How I replaced my off-the-shelf 600CHF router with a 1700CHF custom PC router". I hope you'll update this once Fiber7 updates their POP to show measured performance :-)