6 ms·

Hello, original author here. I saw a bunch of traffic from Hacker News and decided to drop by. Thanks for sharing it here! I'd be happy to answer any questions

Hello, original author here. I saw a bunch of traffic from Hacker News and decided to drop by. Thanks for sharing it here! I'd be happy to answer any questions.

gchamonlive 6y ago
Hey there. Thanks for the article! I wanted to ask you if it is worth it implementing something similar in the cloud to learn kubernetes. The hardware is prohibitively expensive on Brazil, so maybe a cluster of t2.micro on AWS running for the total amount of 750 hours could serve the learning purpose for free
- ikaruswill 6y ago
  Thanks for reading! It's probably worth implementing it in the cloud using the free credits if hardware is prohibitively expensive, at least for the purposes of learning. If you do intend to self-host some services in the longer term, the cost of running it on the cloud will catch up with the hardware cost in no time at all.
- nl 6y ago
  If you have a PC, then just run it on VM(s) on that. That's what AWS is giving you anyway. Here's an example: https://kubernetes.io/blog/2019/03/28/running-kubernetes-locally-on-linux-with-minikube-now-with-kubernetes-1.14-support/ https://kubernetes.io/blog/2019/03/28/running-kubernetes-loc...
- me551ah 6y ago
  Or you could also checkout lowendbox.com (not affiliated with them). They provide really cheap vps servers.
- tinco 6y ago
  I don't know how it is in Brazil but in Europe and the US offices frequently go out of business that have dozens if not hundreds tiny desktop computers in their bankruptcy auctions. They've got i5 cpu's 4-8gb of ram and 128gb SSD's and go for (much) less than a hundred bucks. Stack 3 of them and you have a kubernetes cluster that you could carry in one hand and I think outperforms the 7 raspberry Pi 3's. Servethehome has a series of articles about the machines, though they're focusing on the slightly higher end ones.
  - numpad0 6y ago
    I think it’s lease companies replacing paid off business PCs. They rent boxes to offices on multi year contracts, collect and recycle them when the term is up.
justinclift 6y ago
> An alternative was to use commercial-grade network switches with 16 ports but that was obviously was out of the question (and budget). Ebay?
- ikaruswill 6y ago
  Thanks for the tip. I did consider Ebay, but at that point, I figured it was more trouble than it's worth if I bought them used. Ebay doesn't exactly have the best reputation.
  - justinclift 6y ago
    Ahhh. I guess it depends on what you're looking for. Mellanox cards and related cabling is generally still pretty good (in UK and Australia anyway). A few years ago I obtained a few 24 port HP Procurve switches (3400cl-24G), with 10GbE (CX4 connectors) on the back. Those worked pretty well, though turned out to be slightly too noisy for use as desktop switches. ;) Taking a quick look on US Ebay now, something like this (US$39) might have been workable: https://www.ebay.com/itm/HP-J9145A-ProCurve-2910al-24G-Ethernet-Switch/114290990219 https://www.ebay.com/itm/HP-J9145A-ProCurve-2910al-24G-Ether... I don't know that seller at all, but ex-enterprise HP Procurve gear should generally be pretty safe. From reputable vendors anyway. :) For Singapore, I'm not sure. Might have needed to get something from the US shipped over. This one seems like it'd be ~US$80 including shipping, which was probably too much: https://www.ebay.com/itm/HP-ProCurve-Switch-1800-24G-switch-24-ports/233622141076 https://www.ebay.com/itm/HP-ProCurve-Switch-1800-24G-switch-...
    - ryall 6y ago
      To help with the noise try replacing the fans with high quality ones. I threw some nocturnas into an old managed dell switch and it quieted down immensely. Just be aware that sometimes they change the pins around from the standard pinout, I blew out one of the fans before discovering this. Fan still works, just lost the rpm monitoring :(
      - 6y ago
hevelvarik 6y ago
Fascinating, pardon my ignorance but I’m new to this home brew clustering thing. How is the coordination between the nodes handled for a use case like serving a wordpress blog? Or are we talking about n-1 nodes behind a load balancing master node?
- TobTobXX 6y ago
  Kubernetes would be my guess... Especially k3s, which has a low footprint is suited for rbpis.
jp0d 6y ago
Hey mate. The rig looks awesome. How is the performance output compared to a regular desktop with Ryzen-5/7 or Core-i7 processor in it? Can it run Spark? Apologies if my questions are irrelevant. I'm not sure if Spark/Java works well on the ARM platform yet. Although it'd be foolish on my part to expect too much performance from something like this that costs around $600! It'd be an interesting setup to test parallelism and learn how things work.
- redis_mlc 6y ago
  > How is the performance output compared to a regular desktop Sorry, but often budget (junk) CPUs are only 10% of the performance of current desktop-class processors. You can use "openssl speed" to test and compare performance. When I was experimenting with Intel's Atom netbook chip, it was exactly 10% of a Xeon thread. So it could be used for httpd static content, but not java. > It'd be an interesting setup to test parallelism and learn how things work. Long-term, they can be used for DNS, PXE, etc. but most people will not get their money back. If you plan to use docker or k8s, verify they have the right CPU options first. https://www.openssl.org/docs/man1.1.1/man1/openssl-speed.html https://www.openssl.org/docs/man1.1.1/man1/openssl-speed.htm... https://en.wikipedia.org/wiki/Intel_Atom https://en.wikipedia.org/wiki/Intel_Atom
  - jp0d 6y ago
    Thanks man.. I know. I've an old Asus laptop/netbook with an Atom processor. Quad-core I might add. The travesty! It's slow as hell. I bought it because it had an 11 inch screen and I wanted something small. I bought it for AUD $300 back in 2015. I haven't used it more than a month. It's just part of my electronics junk collection. Now a days I'm a lot more careful before buying these things. That's why I've been holding on to the idea of buying a bunch of RPi 4s or Jetson Nanos. It's probably a better idea to get a proper GPU for my Ryzen desktop.
    - justinclift 6y ago
      As a general concept, an old low-end laptop can sometimes be usefully repurposed as a low end NAS. It just needs a sufficient amount of ram (depends on the NAS software), and the ability to plug in disks via some means (USB, etc). It wouldn't be fast, but the battery in it (unless completely dead) even provides some power-outage protection. :)
Fiahil 6y ago
I have a similar cluster at home that I mostly use as a media center! One change I made over your setup is using an external NAS for storage. I'm also using k3s for making the whole setup completely portable. When I need to access my movie library within a deployment, I just mount the folder as NFS volume. It's working quite well and performances are still decent! Maybe something to consider for Leviathan?
himynameistimli 6y ago
hey nice one will
hestefisk 6y ago
Very nice work. Super geeky. I would love running a Plan9 cluster on this setup. Any thoughts on trying it out?
hhtoyou 6y ago
The site is down

Hello, original author here. I saw a bunch of traffic from Hacker News and decided to drop by. Thanks for sharing it here! I'd be happy to answer any questions.

gchamonlive 6y ago

Hey there. Thanks for the article! I wanted to ask you if it is worth it implementing something similar in the cloud to learn kubernetes. The hardware is prohibitively expensive on Brazil, so maybe a cluster of t2.micro on AWS running for the total amount of 750 hours could serve the learning purpose for free

ikaruswill 6y ago
Thanks for reading! It's probably worth implementing it in the cloud using the free credits if hardware is prohibitively expensive, at least for the purposes of learning. If you do intend to self-host some services in the longer term, the cost of running it on the cloud will catch up with the hardware cost in no time at all.
nl 6y ago
If you have a PC, then just run it on VM(s) on that. That's what AWS is giving you anyway. Here's an example: https://kubernetes.io/blog/2019/03/28/running-kubernetes-locally-on-linux-with-minikube-now-with-kubernetes-1.14-support/ https://kubernetes.io/blog/2019/03/28/running-kubernetes-loc...
me551ah 6y ago
Or you could also checkout lowendbox.com (not affiliated with them). They provide really cheap vps servers.
tinco 6y ago
I don't know how it is in Brazil but in Europe and the US offices frequently go out of business that have dozens if not hundreds tiny desktop computers in their bankruptcy auctions. They've got i5 cpu's 4-8gb of ram and 128gb SSD's and go for (much) less than a hundred bucks. Stack 3 of them and you have a kubernetes cluster that you could carry in one hand and I think outperforms the 7 raspberry Pi 3's. Servethehome has a series of articles about the machines, though they're focusing on the slightly higher end ones.
- numpad0 6y ago
  I think it’s lease companies replacing paid off business PCs. They rent boxes to offices on multi year contracts, collect and recycle them when the term is up.

justinclift 6y ago

> An alternative was to use commercial-grade network switches with 16 ports but that was obviously was out of the question (and budget). Ebay?

ikaruswill 6y ago
Thanks for the tip. I did consider Ebay, but at that point, I figured it was more trouble than it's worth if I bought them used. Ebay doesn't exactly have the best reputation.
- justinclift 6y ago
  Ahhh. I guess it depends on what you're looking for. Mellanox cards and related cabling is generally still pretty good (in UK and Australia anyway). A few years ago I obtained a few 24 port HP Procurve switches (3400cl-24G), with 10GbE (CX4 connectors) on the back. Those worked pretty well, though turned out to be slightly too noisy for use as desktop switches. ;) Taking a quick look on US Ebay now, something like this (US$39) might have been workable: https://www.ebay.com/itm/HP-J9145A-ProCurve-2910al-24G-Ethernet-Switch/114290990219 https://www.ebay.com/itm/HP-J9145A-ProCurve-2910al-24G-Ether... I don't know that seller at all, but ex-enterprise HP Procurve gear should generally be pretty safe. From reputable vendors anyway. :) For Singapore, I'm not sure. Might have needed to get something from the US shipped over. This one seems like it'd be ~US$80 including shipping, which was probably too much: https://www.ebay.com/itm/HP-ProCurve-Switch-1800-24G-switch-24-ports/233622141076 https://www.ebay.com/itm/HP-ProCurve-Switch-1800-24G-switch-...
  - ryall 6y ago
    To help with the noise try replacing the fans with high quality ones. I threw some nocturnas into an old managed dell switch and it quieted down immensely. Just be aware that sometimes they change the pins around from the standard pinout, I blew out one of the fans before discovering this. Fan still works, just lost the rpm monitoring :(
    - 6y ago

hevelvarik 6y ago

Fascinating, pardon my ignorance but I’m new to this home brew clustering thing. How is the coordination between the nodes handled for a use case like serving a wordpress blog? Or are we talking about n-1 nodes behind a load balancing master node?

TobTobXX 6y ago
Kubernetes would be my guess... Especially k3s, which has a low footprint is suited for rbpis.

jp0d 6y ago

Hey mate. The rig looks awesome. How is the performance output compared to a regular desktop with Ryzen-5/7 or Core-i7 processor in it? Can it run Spark? Apologies if my questions are irrelevant. I'm not sure if Spark/Java works well on the ARM platform yet. Although it'd be foolish on my part to expect too much performance from something like this that costs around $600! It'd be an interesting setup to test parallelism and learn how things work.

redis_mlc 6y ago
> How is the performance output compared to a regular desktop Sorry, but often budget (junk) CPUs are only 10% of the performance of current desktop-class processors. You can use "openssl speed" to test and compare performance. When I was experimenting with Intel's Atom netbook chip, it was exactly 10% of a Xeon thread. So it could be used for httpd static content, but not java. > It'd be an interesting setup to test parallelism and learn how things work. Long-term, they can be used for DNS, PXE, etc. but most people will not get their money back. If you plan to use docker or k8s, verify they have the right CPU options first. https://www.openssl.org/docs/man1.1.1/man1/openssl-speed.html https://www.openssl.org/docs/man1.1.1/man1/openssl-speed.htm... https://en.wikipedia.org/wiki/Intel_Atom https://en.wikipedia.org/wiki/Intel_Atom
- jp0d 6y ago
  Thanks man.. I know. I've an old Asus laptop/netbook with an Atom processor. Quad-core I might add. The travesty! It's slow as hell. I bought it because it had an 11 inch screen and I wanted something small. I bought it for AUD $300 back in 2015. I haven't used it more than a month. It's just part of my electronics junk collection. Now a days I'm a lot more careful before buying these things. That's why I've been holding on to the idea of buying a bunch of RPi 4s or Jetson Nanos. It's probably a better idea to get a proper GPU for my Ryzen desktop.
  - justinclift 6y ago
    As a general concept, an old low-end laptop can sometimes be usefully repurposed as a low end NAS. It just needs a sufficient amount of ram (depends on the NAS software), and the ability to plug in disks via some means (USB, etc). It wouldn't be fast, but the battery in it (unless completely dead) even provides some power-outage protection. :)

Fiahil 6y ago

I have a similar cluster at home that I mostly use as a media center! One change I made over your setup is using an external NAS for storage. I'm also using k3s for making the whole setup completely portable. When I need to access my movie library within a deployment, I just mount the folder as NFS volume. It's working quite well and performances are still decent! Maybe something to consider for Leviathan?

himynameistimli 6y ago

hey nice one will

hestefisk 6y ago

Very nice work. Super geeky. I would love running a Plan9 cluster on this setup. Any thoughts on trying it out?

hhtoyou 6y ago

The site is down