11 ms·

Show HN: A low power 1U Raspberry Pi cluster server for inexpensive colocation

175 points by postpawl 5y ago

LeoPanthera 5y ago
I'm astounded to learn that colo goes as low as $30/month. I had no idea.
- postpawl 5y ago
  When I started the build, NextArray was also $30/month for 5x IPs. They've increased their prices recently though. You might occasionally find deals on the webhostingtalk forums though.
- franga2000 5y ago
  Yeah I was really surprised by that as well. Unfortunately, that doesn't seem to be the case where I live. 90 € / month (before tax) for 2U with 4 IPv4 and 100Mbps "best effort". You can almost rent a small office for that price and just throw the server in there. I've run the numbers and if you need the standard 3U worth of space (a 2U storage and 1U compute server), you're better off renting an office and a residential 600/300Mbps fibre connection.
  - truckerbill 5y ago
    Where are you based? In a capital city that doesn't sound right but I haven't run the numbers like you have.
    - quickthrower2 5y ago
      90eu a month is a fraction of the rent I was paying for a shared house room in 1999 in a cheap city. 90eu a month office must be in south east asia or something like that.
    - franga2000 5y ago
      It's not a capital city and we've had a decline in business lately (even before covid, but especially now). 150€/mo gets you a 17m2 office pretty much in the centre and there's currently a listing for 15m2 outside the centre for 100€/mo. Both of those are too small for most things and so there isn't much of a market for them (if all you need is a desk and computer, most people can work from home). Power is fairly cheap as well and you can get a very good Internet connection for 40€ or so. It would probably be a bit over the ~200€ for collocation, but you have an office in a secure building, not just two servers in a rack somewhere, so it might be worth it.
Mave83 5y ago
if you use a 2U and much deeper System, you could fit them standing instead of laying down. It should be possible to add a huge amount by that. May I also suggest to just use a small 3d printed part to hold them in place. How about the raspberry pi m2 shield? Or how about using a 24x 2.5" chassis to support Hot swap drive replacement?
- postpawl 5y ago
  If I was to do a V2, I'm really interested in Pine64's clusterboard and the Turing Pi 2. However, the clusterboard is out of stock, the Turing Pi 2 isn't out yet, and it's very difficult to get Raspberry Pi CM4's right now. Fitting all the Pi's on one board would make this way easier. The 3d printed part, 24x drives, and m2 shield all sound like awesome ideas. If I had an use for 2U worth of Pis (that would be a ton!), I'd definitely consider it.
coverband 5y ago
I used to do colo, but surprisingly it’s usually a better deal to pay monthly for an unmanaged server, with standard hardware replacement warranty.
- postpawl 5y ago
  I’m interested to see what breaks first. I’m mainly hoping it’s not the switch or the power supply.
- LeoPanthera 5y ago
  This is only true if you don't need a lot of storage. Stuffing a few U full of hard disks is still cheaper.
  - gruez 5y ago
    How much storage are we talking about? Hezner storage servers are pretty cost effective.
    - xur17 5y ago
      Are there any providers similar to Hetzner that are a bit closer to the US? I have a view vps instances with them, but the high latency is a major bummer.
      - LeoPanthera 5y ago
        Well there's rsync.net but they're something like 5x more expensive.
    - LeoPanthera 5y ago
      Well the multi-TB range, but Hetzner has no servers, or competition, in the USA as far as I can tell.
lukevp 5y ago
This is a cool idea, and I was really interested to see it be done, but I wouldn’t actually do this. There are a lot of parts to this just to have cheap colo, and I would be worried about failure of the components like the usb hubs taking down the cluster. you can get a Mac Mini colocated for $54/mo with 24/7 remote hands, a dedicated ip, remote power management, and gigabit uncapped bandwidth, which is much better hosting than that $30/mo offering, so you’re not paying more for nothing. An M1 Mac mini with 16 gb of ram and a 1 tb SSD is $1300 not on sale. From a cursory google of benchmarks, an M1 on geek bench is about 11x more powerful than a raspberry pi 4b, so you are getting much more compute. Also, the 16 gb of ram is directly integrated with the processor, there is no need to manage multiple nodes, the hardware is battle-tested, it has true gigabit Ethernet, and the people at the data center also understand the setup and can repair it for you easily. I guess what I’m saying is - super cool that you did this (and I have a cluster of pis at home as well to play with, I love them) but if I was going to pay for colo, I would just colo a Mac mini.
- postpawl 5y ago
  Is it possible to run linux (debian?) like normal on an M1 Mac yet? Agreed an M1 Mac mini is great hardware and power for the price. Yeah it looks like you can rent them for $75/month.
  - jbverschoor 5y ago
    Since 5.13 yes http://lkml.iu.edu/hypermail/linux/kernel/2105.1/00457.html http://lkml.iu.edu/hypermail/linux/kernel/2105.1/00457.html
    - kzrdude 5y ago
      I guess the initial support is not "Linux as normal"? Do any regular install medium work?
- 14u2c 5y ago
  > Mac Mini colocated for $54/mo Interesting, who is the provider for this?
neilv 5y ago
This is a nice build. One tiny change I'd make is because I'm paranoid about the pokey power switch lever. It might be at eye height, when someone is wrestling to get a heavy server on rails, or similar. You can get aircraft-like guards that fit over that design of switch[1], or you can get a less-pokey rocker or pushbutton switches. (Note that rocker and pushbutton switches can be easier to actuate accidentally, such as when a neighboring server is being racked/unracked, but you can get those switches with guards, too.) [1] Example: https://www.ebay.com/itm/292009956841 https://www.ebay.com/itm/292009956841
- postpawl 5y ago
  Great suggestion! Added it to the list of things for V2.
  - accountofme 5y ago
    better yet a push button and a latching flipflop?
    - p0d 5y ago
      I accidentally powered down a san once by knocking it's rocker switch with my elbow :-)
      - mysterydip 5y ago
        - redundant power supplies - redundant ethernet connections - data across redundant disks ... and all of it bypased by an accidental elbow :)
    - simcop2387 5y ago
      A latching relay would be perfect for this kind of thing, good current capacity and they're pretty reliable.
- traceroute66 5y ago
  > when someone is wrestling to get a heavy server on rails, or similar Don't do that !!!! Save your back, reduce risk of damage to server and rails .... use a server lift. Seriously. Trust me, when I made the change from wrestling to lift, it was a real "what the hell was I doing before" moment. If your employer or facility does not have a lift, do what you need to do to get one. At its most basic, it is, without doubt, a serious health & safety at work issue for your employer or the facility if they don't provide one.
neilv 5y ago
Note (for anyone interested but new to colocation facilities) that they're trying to fit within a 1A power limit (and maybe short-depth rack limit). If anyone wants to build a bigger SBC cluster in a 1U height, you might be able to go 3x to 4x as deep. Though, one downside of the full-depth hobby servers is that they're less flexible for also using at home, since they don't fit nicely in most furniture (and the fans tend to be crazy-loud). In the past, I've wanted this flexibility for evolving needs, and when I kept a cold spare of a deep 1U colo server at home (in case it needed to be run over to the colo to swap in for a hardware failure).
- nirui 5y ago
  For home use cases, a even lower-budget option is to just give up full-depth server racks, and buy a bigger power/Internet box then throw the Pi(s) in instead. That way you don't need a chassis or many other components listed in the article. And since usually most of these boxes are installed outside the range of "living space", the fan noises should not be a big concern. I have a super low-budget setup for my two SBCs (not Pi) in my parents place [0]. It's been there for more than 3 years now, running a Docker cluster. I'd say it's fairly stable for non-production use. Of course, the box in my case does not allow fan installations. So the cluster runs fairly hot sometimes during summer time. Currently it runs around 60°C (140.00°F) under light~medium load. In the winter, the temperature could drop to as low as around 20°C. [0]: https://imgur.com/a/bfBlUb4 https://imgur.com/a/bfBlUb4
2Gkashmiri 5y ago
with high speed internet even at homes, what benefit is "10 Mbit Ethernet" in todays terms? what kind of projects can you use this speed with? i would be interested to know... i use a 30mbps home internet connection with somewhat dynamic ipv6. i use duckdns to bypass that and it works wonderfully. i have a couple of pi 3b+ and and zeros in the router USB ports itself from where they get their power from. they get internet via wifi or a small patch cable. this means as long as wifi router has internet, my machines will run. granted they are all dangling and flopping around but still... i have started into self hosting and other similar stuff and i find the pis on the local network with this power work great. i would be interested in this. oh, the next big thing^(tm), are there any hosting providers who do managed pi as a service? like they use similar to your setup and like amazon does mac mini, does someone do pi? you get a managed pi for cheap and they can host a tonne of users in a small space for cheap
- mfashby 5y ago
  I've seen https://www.mythic-beasts.com/article/about https://www.mythic-beasts.com/article/about do pi hosting
  - 2Gkashmiri 5y ago
    nice. so you get to use a pi 4 8gb which costs £81.90 on amazon uk and they charge you 125 for a year of storage, power, bandwidth, device. nice
- hutrdvnj 5y ago
  If you have a CDN in front of your website and a good packet filter / firewall / DDOS protection and you mostly serve cachable (static) content, then 10 Mbit Ethernet can do more than one might think.
  - xur17 5y ago
    I ran an image / file hosting service off an old computer in my parent's basement when I was in high school. You can do more than you might think with ~1mbps upload even.
geerlingguy 5y ago
I'm really hoping a blade like the Uptime.Lab concept [1] makes it to production, because you could cram 15-20 Compute Module 4s, with native NVMe boot, into a 1U chassis, for around 100 ARM cores, plus over 100 GB of RAM and whatever amount of SSD storage you can afford. I have been in contact with Ivan, the creator of the board, and I'm currently testing his beta '0.6' version. It really makes racking Pis for simple clusters easy, compared to trying to mash Pi 4s into a rack. There's also the Scalenode [2], but I haven't heard much about it since the first post to GitHub. [1] https://www.instagram.com/p/CRZMXmRLRma/ https://www.instagram.com/p/CRZMXmRLRma/ [2] https://github.com/antmicro/scalenode https://github.com/antmicro/scalenode
- postpawl 5y ago
  I’ve really enjoyed your videos about all the new boards coming out for the CM4. I can’t wait until it’s easier to get CM4s! I checked newark today and it said they’re backordered out to 2022-02-28.
- robbiet480 5y ago
  Been watching progress of the Blade for the last few weeks and am very excited for it to hit Kickstarter soon. My only wish would be for it to include IPMI like functions, or at least remote PSU capabilities (although I guess if you’re doing PoE you have remote PSU). And Jeff, thanks for your videos, they are always a highlight of my day whenever a new one comes out. Happy to be a patron!
  - rainbowzootsuit 5y ago
    If your wish is not granted and need an add-on kvm, check out one of these: https://tinypilotkvm.com/ https://tinypilotkvm.com/ https://pikvm.org/download.html https://pikvm.org/download.html
- laumars 5y ago
  Is there any more content on the RPi rack that’s not on Instagram? Seems I can’t even view odd pictures on their mobile site without signing up :(
elcritch 5y ago
Some Intel NUC's look to be 38mm tall, short enough to possibly fit in a 1U rack at 44.5mm. There's also the Minisforum for Ryzen's or Odroids H2+.
- postpawl 5y ago
  The NUCs and Minisforum would be fewer cores for price (with 4 cores per $400-500), but I'm sure they'd be way faster. Excellent idea.
  - elcritch 5y ago
    They're pretty fast, with a few hexacores. Quite a bit more than a RPi4 but still under 25W each. There's a few NUCs for $350 or so.
    - vetinari 5y ago
      They have a small fan, which is quite annoying when the NUC is under load. Not a problem in colo, but annoying in homelab. Not a deal breaker, but something to think about.
- jpdb 5y ago
  NUCs are a bit difficult to source right now, but I'd be super interested in coloing ~4 of them + a switch + a nas of some sort in a 1/2U chassis running k3s.
- vetinari 5y ago
  Yes, the slim -DNKE NUCs do fit into 1U, three of them next to each other; you can find mounts like these to do that: https://www.myelectronics.nl/us/nuc-minipc-19-rackmount-kit-1-3-nucs.html https://www.myelectronics.nl/us/nuc-minipc-19-rackmount-kit-.... If you have 3U to work with, you can fit 12 of them. For the thick (-DNHE) NUCs, you either need 1.5U for three of them, or you can fit 8 of them into 3U.
  - postpawl 5y ago
techsupporter 5y ago
I remember when there were several providers offering "Pi Colo." I had a Pi at PCextreme in Amsterdam for years before they ended the service. This is pretty clever, I like it.
jbverschoor 5y ago
I dunno.. a macmini M1 idles 7watt. This cluster does 36watt idle, 43watt normal use, or 48watt idle with the fan, and probably 62watt peak load with high fan. The M1 uses 26 under load and 39watt peak load (16GB with the 2TB ssd version according to apple). Measured with kill-a-watt, I've read 4watt idle and 20watt full load. So I think the 26watt is probably a more realistic value. 16GB version: $899 1TB usb-c ssd: $109 1U rackmount: $199 This will give you 16GB ram and 1.2TB ssd for $1207. It can fit two mac minis. As the purpose was 1U with max 1A/120V, you'd be able to run 3 mac minis in 1U (or 4 if you make sure they don't exceed 30watt). In terms of performance the pi does 295single and 830multi. 5 rpis is 295single (does not increase with 5 rpis!) and 4150multi. The M1 does 1748single and 7660multi. So you get about twice the cpu performance (ignoring the super fast 256gb ssd, and other hardware features the M1 has, and the overhead of running a 5 node cluster). A rackshelf for 4 mac-minis is $359. If you'd actually put in 3 (because of the 1A limit), you'd have 6x the cpu performance for somewhere between $3000 and $3300 (depending on the storage you want). All still in 1U. If power usage can be higher, you could actually fit up to 8 mac minis (36" depth) for 2.6A @120V. The comparison table in the article compares this setup with a Dell using 2.08A ("not at peak?"). But if you consider a max of about 20watt per M1, you can host 4-6 M1s with 1A. Also, you won't have to deal with breaking sd cards, the heat, noise, the wiring mess, and it has many hardware features that an rpi doesn't have. You won't have to deal with the storage, memory and cpu overhead of running something distributed. Still it's fun to tinker around with rpis. Although I stopped using them, as storage is really flakey because of the SSDs. Other SBCs offer built-on ssd storage.
- hutrdvnj 5y ago
  That's why I hope that the pi 5 has built-on ssd storage + m.2 slot and gets a significant performance boost to come more close to M1 speeds.
  - the_biot 5y ago
    The point is that the Raspberry Pi -- any version -- has always been underpowered and overhyped. Wishing for the next version to be better is nonsense. Pick any recent SBC, and it'll be better bang for the buck. This has been true every since the first RPi.
spoonjim 5y ago
Raspberry Pis are not the right hardware for this. At a minimum you would want an SBC with onboard storage instead of paying this much to run your storage on thumb drives or SD cards.
andyjohnson0 5y ago
This is really neat, and I appreciate how much work the author did to describe the build process in so much detail. A couple of general questions: 1. Wouldn't compute modules allow a higher density of pi's? I appreciate that the power management would be more tricky though. I don't know whether there are backplanes available that would address this. 2. Do colos commonly accept self-built kit like this? With single strand wire and electrical tape on the gpios, etc. (No disrespect intended to the author. As I said, I think this is a great piece of work.)
- gnfargbl 5y ago
  > 2. Do colos commonly accept self-built kit like this? My (UK) colo agreement says: In exercising the rights granted in this Agreement, the Customer agrees to maintain the Apparatus to a standard which ensures that at all times it is safe and complies with all applicable health and safety standards. So I guess it depends on how you interpret those kinds of provisions. Personally I wouldn't choose to risk installing a homebrew design, because the colo provider would likely attempt to hold me liable for consequential losses if it started a fire.
  - andyjohnson0 5y ago
    That's kind of what I thought. > Personally I wouldn't choose to risk installing a homebrew design, because the colo provider would likely attempt to hold me liable for consequential losses if it started a fire. Even if a fire occurred and your kit didn't start it, it might be difficult to prove that. And thinking about what happened at ovh, the other colo customers might pile in too.
    - genewitch 5y ago
      we put a dishrack with GPUs ziptied to it in our colo and they let it go for several months, then decided it was "a fire hazard" - probably the power usage was the issue more than anything else, if you have a cage/rack(s), what's the difference?
- BackBlast 5y ago
nippoo 5y ago
It's worth figuring out what your colo provider's power usage policy is. Mine were pretty lax (nominally 0.5A at 240V, but I had a PowerEdge 1950 there which would definitely draw m ore at startup!) I think they were mostly concerned about average rather than maximum current draw, and they certainly wouldn't power anything off if I drew a bit more current...
whalesalad 5y ago
I hope your cage is fireproof. Shoving all that consumer gear in a 1U seems… risky.
- mikecoles 5y ago
  Why would a 1U be different than a 2U or 6U?
Saris 5y ago
I have some passively cooled i3-7100u boxes I recently got, each with an M.2 slot and 2 RAM modules, and they idle at around 2W each when booted into Debian. I've never felt like Pis are that great for power vs performance for most applications, and make more sense in places where you can make use of all that GPIO and compact size.
johnklos 5y ago
I've built several 1U servers to colocate. One nice thing when you only have one feed and one static IPv4 address is that you can use IPv6 for management. For example, here's a server I build several years ago (pre Raspberry Pi 4): https://www.klos.com/~john/anath.jpg https://www.klos.com/~john/anath.jpg It has an NVIDIA Jetson TK2 as the main system plus a Raspberry Pi 3 that provides serial console and has a relay connected to a GPIO for remote reset for the TK2. The Pi can be reached directly via a static IPv6, so no additional IPv4 required. I also have a 1U server with an 8 gig Raspberry Pi 4 and two 8 TB drives set up as a mirror. This system is also significantly below 100 watts, both peak and average: https://twitter.com/AnachronistJohn/status/1287628712981049345 https://twitter.com/AnachronistJohn/status/12876287129810493...
- neerlent 5y ago
  Can you explain your combined use of IPv4 and IPv6? Is the Pi only reachable directly via IPv6, or can IPv4 requests be tunneled through the Jetson to reach the Pi?
exabrial 5y ago
Last time I checked, the RPI to really be unlocked needs an additional PCIe line. IIRC, right now, I think there's just one, and the consumer board uses it for USB. The blade edition makes it available on the connector, but then you have to choose between network or storage. It'd be ideal if there were three PCIe lines, one for usb, one for network, one for storage.
- wwarren 5y ago
  I think the Pi 4 and above don't have that issue any more? I could be wrong
- myself248 5y ago
  Or you need a PCIe switch on the base-board, and those are practically impossible to get without a giant NDA. My kingdom for a PCIe switch chip that's as easy to use as the TUSB series of hub chips.
bserge 5y ago
Dear God, those USB controllers are ticking time bombs! I believe it should be possible to make a sub-120W machine using an ITX or laptop board with more reliable PCIe and SATA connections for the drives.
- BackBlast 5y ago
  So much this. I've used rPi 4 as a home server and it croaks under load. Burns through SD cards if you go that route. The USB, while better, similarly does not operate well under load and I've similarly killed USB controllers when I subject them to too many IOPS. This thing is a maintenance nightmare. If you're just doing a one off server systems you can pick up older HP thin clients with better specs than a pi 4 and real IO for a similar price point. T630s can be had for around $60 shipped off eBay with 4GB ram and 8-32GB M.2 SATA upgradable to 64GB RAM (spec sheet says 32GB but some have made 64 work) CPU supports ECC but I haven't tried that yet and as much storage as you want to stuff into the SATA slots (1 22x80 and 1 22x47). You're not going to be able to cram as many in a U1. But you will have an actually stable system. I'd put the CPU at the near equivalent of 2 pi 4s - better if you're doing lots of AES.
ehutch79 5y ago
What is this actually useful for?
pacman2 5y ago
You better have a good liability insurance if someone hosts this for you.
alexellisuk 5y ago
A super cool project, but it's so easy to self-host a Raspberry Pi at home now with a public IP - even behind NAT and firewalls. I'm the creator of inlets (self-hosted tunnels - L4/L7) https://docs.inlets.dev/#/ https://docs.inlets.dev/#/ But Argo Tunnels from Cloudflare, and reverse SSH (for those with too much time on their hands) also work. For more on a teeny K3s cluster: https://blog.alexellis.io/state-of-netbooting-raspberry-pi-in-2021/ https://blog.alexellis.io/state-of-netbooting-raspberry-pi-i...

175 points by postpawl 5y ago

LeoPanthera 5y ago

I'm astounded to learn that colo goes as low as $30/month. I had no idea.

postpawl 5y ago
When I started the build, NextArray was also $30/month for 5x IPs. They've increased their prices recently though. You might occasionally find deals on the webhostingtalk forums though.
franga2000 5y ago
Yeah I was really surprised by that as well. Unfortunately, that doesn't seem to be the case where I live. 90 € / month (before tax) for 2U with 4 IPv4 and 100Mbps "best effort". You can almost rent a small office for that price and just throw the server in there. I've run the numbers and if you need the standard 3U worth of space (a 2U storage and 1U compute server), you're better off renting an office and a residential 600/300Mbps fibre connection.
- truckerbill 5y ago
  Where are you based? In a capital city that doesn't sound right but I haven't run the numbers like you have.
  - quickthrower2 5y ago
    90eu a month is a fraction of the rent I was paying for a shared house room in 1999 in a cheap city. 90eu a month office must be in south east asia or something like that.
  - franga2000 5y ago
    It's not a capital city and we've had a decline in business lately (even before covid, but especially now). 150€/mo gets you a 17m2 office pretty much in the centre and there's currently a listing for 15m2 outside the centre for 100€/mo. Both of those are too small for most things and so there isn't much of a market for them (if all you need is a desk and computer, most people can work from home). Power is fairly cheap as well and you can get a very good Internet connection for 40€ or so. It would probably be a bit over the ~200€ for collocation, but you have an office in a secure building, not just two servers in a rack somewhere, so it might be worth it.

Mave83 5y ago

if you use a 2U and much deeper System, you could fit them standing instead of laying down. It should be possible to add a huge amount by that. May I also suggest to just use a small 3d printed part to hold them in place. How about the raspberry pi m2 shield? Or how about using a 24x 2.5" chassis to support Hot swap drive replacement?

postpawl 5y ago
If I was to do a V2, I'm really interested in Pine64's clusterboard and the Turing Pi 2. However, the clusterboard is out of stock, the Turing Pi 2 isn't out yet, and it's very difficult to get Raspberry Pi CM4's right now. Fitting all the Pi's on one board would make this way easier. The 3d printed part, 24x drives, and m2 shield all sound like awesome ideas. If I had an use for 2U worth of Pis (that would be a ton!), I'd definitely consider it.

coverband 5y ago

I used to do colo, but surprisingly it’s usually a better deal to pay monthly for an unmanaged server, with standard hardware replacement warranty.

postpawl 5y ago
I’m interested to see what breaks first. I’m mainly hoping it’s not the switch or the power supply.
LeoPanthera 5y ago
This is only true if you don't need a lot of storage. Stuffing a few U full of hard disks is still cheaper.
- gruez 5y ago
  How much storage are we talking about? Hezner storage servers are pretty cost effective.
  - xur17 5y ago
    Are there any providers similar to Hetzner that are a bit closer to the US? I have a view vps instances with them, but the high latency is a major bummer.
    - LeoPanthera 5y ago
      Well there's rsync.net but they're something like 5x more expensive.
  - LeoPanthera 5y ago
    Well the multi-TB range, but Hetzner has no servers, or competition, in the USA as far as I can tell.

lukevp 5y ago

This is a cool idea, and I was really interested to see it be done, but I wouldn’t actually do this. There are a lot of parts to this just to have cheap colo, and I would be worried about failure of the components like the usb hubs taking down the cluster. you can get a Mac Mini colocated for $54/mo with 24/7 remote hands, a dedicated ip, remote power management, and gigabit uncapped bandwidth, which is much better hosting than that $30/mo offering, so you’re not paying more for nothing. An M1 Mac mini with 16 gb of ram and a 1 tb SSD is $1300 not on sale. From a cursory google of benchmarks, an M1 on geek bench is about 11x more powerful than a raspberry pi 4b, so you are getting much more compute. Also, the 16 gb of ram is directly integrated with the processor, there is no need to manage multiple nodes, the hardware is battle-tested, it has true gigabit Ethernet, and the people at the data center also understand the setup and can repair it for you easily. I guess what I’m saying is - super cool that you did this (and I have a cluster of pis at home as well to play with, I love them) but if I was going to pay for colo, I would just colo a Mac mini.

postpawl 5y ago
Is it possible to run linux (debian?) like normal on an M1 Mac yet? Agreed an M1 Mac mini is great hardware and power for the price. Yeah it looks like you can rent them for $75/month.
- jbverschoor 5y ago
  Since 5.13 yes http://lkml.iu.edu/hypermail/linux/kernel/2105.1/00457.html http://lkml.iu.edu/hypermail/linux/kernel/2105.1/00457.html
  - kzrdude 5y ago
    I guess the initial support is not "Linux as normal"? Do any regular install medium work?
14u2c 5y ago
> Mac Mini colocated for $54/mo Interesting, who is the provider for this?

neilv 5y ago

This is a nice build. One tiny change I'd make is because I'm paranoid about the pokey power switch lever. It might be at eye height, when someone is wrestling to get a heavy server on rails, or similar. You can get aircraft-like guards that fit over that design of switch[1], or you can get a less-pokey rocker or pushbutton switches. (Note that rocker and pushbutton switches can be easier to actuate accidentally, such as when a neighboring server is being racked/unracked, but you can get those switches with guards, too.) [1] Example: https://www.ebay.com/itm/292009956841 https://www.ebay.com/itm/292009956841

postpawl 5y ago
Great suggestion! Added it to the list of things for V2.
- accountofme 5y ago
  better yet a push button and a latching flipflop?
  - p0d 5y ago
    I accidentally powered down a san once by knocking it's rocker switch with my elbow :-)
    - mysterydip 5y ago
      - redundant power supplies - redundant ethernet connections - data across redundant disks ... and all of it bypased by an accidental elbow :)
  - simcop2387 5y ago
    A latching relay would be perfect for this kind of thing, good current capacity and they're pretty reliable.
traceroute66 5y ago
> when someone is wrestling to get a heavy server on rails, or similar Don't do that !!!! Save your back, reduce risk of damage to server and rails .... use a server lift. Seriously. Trust me, when I made the change from wrestling to lift, it was a real "what the hell was I doing before" moment. If your employer or facility does not have a lift, do what you need to do to get one. At its most basic, it is, without doubt, a serious health & safety at work issue for your employer or the facility if they don't provide one.

neilv 5y ago

Note (for anyone interested but new to colocation facilities) that they're trying to fit within a 1A power limit (and maybe short-depth rack limit). If anyone wants to build a bigger SBC cluster in a 1U height, you might be able to go 3x to 4x as deep. Though, one downside of the full-depth hobby servers is that they're less flexible for also using at home, since they don't fit nicely in most furniture (and the fans tend to be crazy-loud). In the past, I've wanted this flexibility for evolving needs, and when I kept a cold spare of a deep 1U colo server at home (in case it needed to be run over to the colo to swap in for a hardware failure).

nirui 5y ago
For home use cases, a even lower-budget option is to just give up full-depth server racks, and buy a bigger power/Internet box then throw the Pi(s) in instead. That way you don't need a chassis or many other components listed in the article. And since usually most of these boxes are installed outside the range of "living space", the fan noises should not be a big concern. I have a super low-budget setup for my two SBCs (not Pi) in my parents place [0]. It's been there for more than 3 years now, running a Docker cluster. I'd say it's fairly stable for non-production use. Of course, the box in my case does not allow fan installations. So the cluster runs fairly hot sometimes during summer time. Currently it runs around 60°C (140.00°F) under light~medium load. In the winter, the temperature could drop to as low as around 20°C. [0]: https://imgur.com/a/bfBlUb4 https://imgur.com/a/bfBlUb4

2Gkashmiri 5y ago

with high speed internet even at homes, what benefit is "10 Mbit Ethernet" in todays terms? what kind of projects can you use this speed with? i would be interested to know... i use a 30mbps home internet connection with somewhat dynamic ipv6. i use duckdns to bypass that and it works wonderfully. i have a couple of pi 3b+ and and zeros in the router USB ports itself from where they get their power from. they get internet via wifi or a small patch cable. this means as long as wifi router has internet, my machines will run. granted they are all dangling and flopping around but still... i have started into self hosting and other similar stuff and i find the pis on the local network with this power work great. i would be interested in this. oh, the next big thing^(tm), are there any hosting providers who do managed pi as a service? like they use similar to your setup and like amazon does mac mini, does someone do pi? you get a managed pi for cheap and they can host a tonne of users in a small space for cheap

mfashby 5y ago
I've seen https://www.mythic-beasts.com/article/about https://www.mythic-beasts.com/article/about do pi hosting
- 2Gkashmiri 5y ago
  nice. so you get to use a pi 4 8gb which costs £81.90 on amazon uk and they charge you 125 for a year of storage, power, bandwidth, device. nice
hutrdvnj 5y ago
If you have a CDN in front of your website and a good packet filter / firewall / DDOS protection and you mostly serve cachable (static) content, then 10 Mbit Ethernet can do more than one might think.
- xur17 5y ago
  I ran an image / file hosting service off an old computer in my parent's basement when I was in high school. You can do more than you might think with ~1mbps upload even.

geerlingguy 5y ago

I'm really hoping a blade like the Uptime.Lab concept [1] makes it to production, because you could cram 15-20 Compute Module 4s, with native NVMe boot, into a 1U chassis, for around 100 ARM cores, plus over 100 GB of RAM and whatever amount of SSD storage you can afford. I have been in contact with Ivan, the creator of the board, and I'm currently testing his beta '0.6' version. It really makes racking Pis for simple clusters easy, compared to trying to mash Pi 4s into a rack. There's also the Scalenode [2], but I haven't heard much about it since the first post to GitHub. [1] https://www.instagram.com/p/CRZMXmRLRma/ https://www.instagram.com/p/CRZMXmRLRma/ [2] https://github.com/antmicro/scalenode https://github.com/antmicro/scalenode

postpawl 5y ago
I’ve really enjoyed your videos about all the new boards coming out for the CM4. I can’t wait until it’s easier to get CM4s! I checked newark today and it said they’re backordered out to 2022-02-28.
robbiet480 5y ago
Been watching progress of the Blade for the last few weeks and am very excited for it to hit Kickstarter soon. My only wish would be for it to include IPMI like functions, or at least remote PSU capabilities (although I guess if you’re doing PoE you have remote PSU). And Jeff, thanks for your videos, they are always a highlight of my day whenever a new one comes out. Happy to be a patron!
- rainbowzootsuit 5y ago
  If your wish is not granted and need an add-on kvm, check out one of these: https://tinypilotkvm.com/ https://tinypilotkvm.com/ https://pikvm.org/download.html https://pikvm.org/download.html
laumars 5y ago
Is there any more content on the RPi rack that’s not on Instagram? Seems I can’t even view odd pictures on their mobile site without signing up :(

elcritch 5y ago

Some Intel NUC's look to be 38mm tall, short enough to possibly fit in a 1U rack at 44.5mm. There's also the Minisforum for Ryzen's or Odroids H2+.

postpawl 5y ago
The NUCs and Minisforum would be fewer cores for price (with 4 cores per $400-500), but I'm sure they'd be way faster. Excellent idea.
- elcritch 5y ago
  They're pretty fast, with a few hexacores. Quite a bit more than a RPi4 but still under 25W each. There's a few NUCs for $350 or so.
  - vetinari 5y ago
    They have a small fan, which is quite annoying when the NUC is under load. Not a problem in colo, but annoying in homelab. Not a deal breaker, but something to think about.
jpdb 5y ago
NUCs are a bit difficult to source right now, but I'd be super interested in coloing ~4 of them + a switch + a nas of some sort in a 1/2U chassis running k3s.
vetinari 5y ago
Yes, the slim -DNKE NUCs do fit into 1U, three of them next to each other; you can find mounts like these to do that: https://www.myelectronics.nl/us/nuc-minipc-19-rackmount-kit-1-3-nucs.html https://www.myelectronics.nl/us/nuc-minipc-19-rackmount-kit-.... If you have 3U to work with, you can fit 12 of them. For the thick (-DNHE) NUCs, you either need 1.5U for three of them, or you can fit 8 of them into 3U.
- postpawl 5y ago

techsupporter 5y ago

I remember when there were several providers offering "Pi Colo." I had a Pi at PCextreme in Amsterdam for years before they ended the service. This is pretty clever, I like it.

jbverschoor 5y ago

I dunno.. a macmini M1 idles 7watt. This cluster does 36watt idle, 43watt normal use, or 48watt idle with the fan, and probably 62watt peak load with high fan. The M1 uses 26 under load and 39watt peak load (16GB with the 2TB ssd version according to apple). Measured with kill-a-watt, I've read 4watt idle and 20watt full load. So I think the 26watt is probably a more realistic value. 16GB version: $899 1TB usb-c ssd: $109 1U rackmount: $199 This will give you 16GB ram and 1.2TB ssd for $1207. It can fit two mac minis. As the purpose was 1U with max 1A/120V, you'd be able to run 3 mac minis in 1U (or 4 if you make sure they don't exceed 30watt). In terms of performance the pi does 295single and 830multi. 5 rpis is 295single (does not increase with 5 rpis!) and 4150multi. The M1 does 1748single and 7660multi. So you get about twice the cpu performance (ignoring the super fast 256gb ssd, and other hardware features the M1 has, and the overhead of running a 5 node cluster). A rackshelf for 4 mac-minis is $359. If you'd actually put in 3 (because of the 1A limit), you'd have 6x the cpu performance for somewhere between $3000 and $3300 (depending on the storage you want). All still in 1U. If power usage can be higher, you could actually fit up to 8 mac minis (36" depth) for 2.6A @120V. The comparison table in the article compares this setup with a Dell using 2.08A ("not at peak?"). But if you consider a max of about 20watt per M1, you can host 4-6 M1s with 1A. Also, you won't have to deal with breaking sd cards, the heat, noise, the wiring mess, and it has many hardware features that an rpi doesn't have. You won't have to deal with the storage, memory and cpu overhead of running something distributed. Still it's fun to tinker around with rpis. Although I stopped using them, as storage is really flakey because of the SSDs. Other SBCs offer built-on ssd storage.

hutrdvnj 5y ago
That's why I hope that the pi 5 has built-on ssd storage + m.2 slot and gets a significant performance boost to come more close to M1 speeds.
- the_biot 5y ago
  The point is that the Raspberry Pi -- any version -- has always been underpowered and overhyped. Wishing for the next version to be better is nonsense. Pick any recent SBC, and it'll be better bang for the buck. This has been true every since the first RPi.

spoonjim 5y ago

Raspberry Pis are not the right hardware for this. At a minimum you would want an SBC with onboard storage instead of paying this much to run your storage on thumb drives or SD cards.

andyjohnson0 5y ago

This is really neat, and I appreciate how much work the author did to describe the build process in so much detail. A couple of general questions: 1. Wouldn't compute modules allow a higher density of pi's? I appreciate that the power management would be more tricky though. I don't know whether there are backplanes available that would address this. 2. Do colos commonly accept self-built kit like this? With single strand wire and electrical tape on the gpios, etc. (No disrespect intended to the author. As I said, I think this is a great piece of work.)

gnfargbl 5y ago
> 2. Do colos commonly accept self-built kit like this? My (UK) colo agreement says: In exercising the rights granted in this Agreement, the Customer agrees to maintain the Apparatus to a standard which ensures that at all times it is safe and complies with all applicable health and safety standards. So I guess it depends on how you interpret those kinds of provisions. Personally I wouldn't choose to risk installing a homebrew design, because the colo provider would likely attempt to hold me liable for consequential losses if it started a fire.
- andyjohnson0 5y ago
  That's kind of what I thought. > Personally I wouldn't choose to risk installing a homebrew design, because the colo provider would likely attempt to hold me liable for consequential losses if it started a fire. Even if a fire occurred and your kit didn't start it, it might be difficult to prove that. And thinking about what happened at ovh, the other colo customers might pile in too.
  - genewitch 5y ago
    we put a dishrack with GPUs ziptied to it in our colo and they let it go for several months, then decided it was "a fire hazard" - probably the power usage was the issue more than anything else, if you have a cage/rack(s), what's the difference?
BackBlast 5y ago

nippoo 5y ago

It's worth figuring out what your colo provider's power usage policy is. Mine were pretty lax (nominally 0.5A at 240V, but I had a PowerEdge 1950 there which would definitely draw m ore at startup!) I think they were mostly concerned about average rather than maximum current draw, and they certainly wouldn't power anything off if I drew a bit more current...

whalesalad 5y ago

I hope your cage is fireproof. Shoving all that consumer gear in a 1U seems… risky.

mikecoles 5y ago
Why would a 1U be different than a 2U or 6U?

Saris 5y ago

I have some passively cooled i3-7100u boxes I recently got, each with an M.2 slot and 2 RAM modules, and they idle at around 2W each when booted into Debian. I've never felt like Pis are that great for power vs performance for most applications, and make more sense in places where you can make use of all that GPIO and compact size.

johnklos 5y ago

I've built several 1U servers to colocate. One nice thing when you only have one feed and one static IPv4 address is that you can use IPv6 for management. For example, here's a server I build several years ago (pre Raspberry Pi 4): https://www.klos.com/~john/anath.jpg https://www.klos.com/~john/anath.jpg It has an NVIDIA Jetson TK2 as the main system plus a Raspberry Pi 3 that provides serial console and has a relay connected to a GPIO for remote reset for the TK2. The Pi can be reached directly via a static IPv6, so no additional IPv4 required. I also have a 1U server with an 8 gig Raspberry Pi 4 and two 8 TB drives set up as a mirror. This system is also significantly below 100 watts, both peak and average: https://twitter.com/AnachronistJohn/status/1287628712981049345 https://twitter.com/AnachronistJohn/status/12876287129810493...

neerlent 5y ago
Can you explain your combined use of IPv4 and IPv6? Is the Pi only reachable directly via IPv6, or can IPv4 requests be tunneled through the Jetson to reach the Pi?

exabrial 5y ago

Last time I checked, the RPI to really be unlocked needs an additional PCIe line. IIRC, right now, I think there's just one, and the consumer board uses it for USB. The blade edition makes it available on the connector, but then you have to choose between network or storage. It'd be ideal if there were three PCIe lines, one for usb, one for network, one for storage.

wwarren 5y ago
I think the Pi 4 and above don't have that issue any more? I could be wrong
myself248 5y ago
Or you need a PCIe switch on the base-board, and those are practically impossible to get without a giant NDA. My kingdom for a PCIe switch chip that's as easy to use as the TUSB series of hub chips.

bserge 5y ago

Dear God, those USB controllers are ticking time bombs! I believe it should be possible to make a sub-120W machine using an ITX or laptop board with more reliable PCIe and SATA connections for the drives.

BackBlast 5y ago
So much this. I've used rPi 4 as a home server and it croaks under load. Burns through SD cards if you go that route. The USB, while better, similarly does not operate well under load and I've similarly killed USB controllers when I subject them to too many IOPS. This thing is a maintenance nightmare. If you're just doing a one off server systems you can pick up older HP thin clients with better specs than a pi 4 and real IO for a similar price point. T630s can be had for around $60 shipped off eBay with 4GB ram and 8-32GB M.2 SATA upgradable to 64GB RAM (spec sheet says 32GB but some have made 64 work) CPU supports ECC but I haven't tried that yet and as much storage as you want to stuff into the SATA slots (1 22x80 and 1 22x47). You're not going to be able to cram as many in a U1. But you will have an actually stable system. I'd put the CPU at the near equivalent of 2 pi 4s - better if you're doing lots of AES.

ehutch79 5y ago

What is this actually useful for?

pacman2 5y ago

You better have a good liability insurance if someone hosts this for you.

alexellisuk 5y ago

A super cool project, but it's so easy to self-host a Raspberry Pi at home now with a public IP - even behind NAT and firewalls. I'm the creator of inlets (self-hosted tunnels - L4/L7) https://docs.inlets.dev/#/ https://docs.inlets.dev/#/ But Argo Tunnels from Cloudflare, and reverse SSH (for those with too much time on their hands) also work. For more on a teeny K3s cluster: https://blog.alexellis.io/state-of-netbooting-raspberry-pi-in-2021/ https://blog.alexellis.io/state-of-netbooting-raspberry-pi-i...