5 ms·

I'm currently in the process of setting up a HA Hashicorp stack (Vault/Consul/Nomad) + GlusterFS cluster in my free time at home with Raspberrys and some other

I'm currently in the process of setting up a HA Hashicorp stack (Vault/Consul/Nomad) + GlusterFS cluster in my free time at home with Raspberrys and some other small-board computers. Completely overkill but it's great fun. My goal is to see just how self-reliant I can be in terms of digital services. Right now it feels completely reasonable to get to a point where the only external services I really rely on is a CA and domain registrar. Possibly DNS, and external endpoints to front traffic. If the HA part works well and I can make a failover region somewhere, self-hosted e-mail doesn't seem that unreasonable anymore. There are several attractive alternatives to RasPi (Odroid already mentioned, beware though as most of the small boards are 32bit only. Also Khadas VIM3, FriendlyELEC NanoPi, Rock Pi. Many people also seem to like Orange Pi). The earlier generation Raspis are honestly quite disappointing from a performance perspective, mostly because of the shared bus between Ethernet/Wifi/USB/storage. The 4B is actually the first to hold its ground, and still does price/performance-wise compared to the above. Honestly it feels like the market's stagnated a bit around the RK3399 and Allwinner H5/H6, hoping there's going to be a new wave of interesting stuff during 2020. A really nice feature of some of these ARM boards is that you can go so much more free (as in libre) than with x86 chips. Raspberry Pi excluded, unfortunately. If you're open to x86 and want a bit more power, Intel NUCs have been around for a good time and AMD is pushing out Ryzen NUCs now. First out is ASROCK. I've also been very happy with the PC Engines APU2 router boards - they are great as small-form-factor servers or NAS builds as well. Note that even if Raspbian is 32-bit only, you can totally run 64-bit OS's on the 3B+ and 4 series. For 3B+ there's even a pretty stable UEFI bootloader: https://www.raspberrypi.org/forums/viewtopic.php?f=50&t=249449 https://www.raspberrypi.org/forums/viewtopic.php?f=50&t=2494... For Raspi 4 you can just rebuild the kernel for aarch64 and change the config a bit. Or if you're lazy, sakaki- is providing weekly builds here: https://github.com/sakaki-/bcm2711-kernel https://github.com/sakaki-/bcm2711-kernel I have both 3B+ and 4B running vanilla 64-bit Debian Buster.

vedses 7y ago
Opensuse also provides 64bit images for RaspberryPi https://en.opensuse.org/HCL:Raspberry_Pi4 https://en.opensuse.org/HCL:Raspberry_Pi4
alias_neo 7y ago
It'd be nice to read about your setup if you post it somewhere. I just built a new server with a 12c/24t 3900x and bought 3x RPi4-4GB, the goal was to have the Pis and a couple of light-weight VMs run a Kubernetes cluster using K3s (I'm currently running 3 Alpine VMs as a cluster on the server, and planning to add the Alpine aarch64 on the 3 Pis. I was thinking of deploying OpenFAAS to the Pis for serverless functions and have more "heavy" services on the server. I'm a developer so it's more of an academic task than running production services (I have another home server running FreeNAS with my production services in jails). The alternative would be the Hashi stack, and I want to work with Vault anyway as I'm looking at it for secret management in our infrastructure at work.
- Legogris 7y ago
  I'm thinking I might do a write-up when I've had some more progress! And yeah, my requirements in terms of performance are pretty light but my main candidate for whenever I need more power on a single node (though I guess I'd have to get at least two to keep my HA approach, so I want to see how far I can get with horizontally scaling SBCs first) is a mini-ITX with Ryzen 3900.
  - alias_neo 7y ago
    I made the mistake of putting the 3900X in the CS381, which, sadly has zero cooler clearance, and so the NH-L9a I had to use so it'd fit, has horrid cooling performance. I'd just suggest being careful with case choice to make sure you can put a big cooler or an AIO on the CPU.
zek 7y ago
I have this exact setup (vault/consul/nomad + glusterfs and zerotier for networking) and its pretty awesome. Still dependent on letsencrypt for SSL certificates. It runs plex amazingly well (though I am using some machines which are probably overkill for this purpose), even with the data coming from a glusterfs drive. Most of my nomad tasks can just launch anywhere because of gluster. For internal DNS at least, you can just use consul. I set up dnsmasq to forward to consul on all of my machines which is super convenient (esp when that DNS just points to a docker container ipv6 address on the zerotier network, not port remapping on networking insanity needed)
- Legogris 7y ago
  Cheers, haven't checked out zerotier before. I'm curious, do you run GlusterFS on nomad or standalone? Seems like it could work fine as raw_exec, but I'm not sure if that's a good idea or not.
  - zek 7y ago
    kinda a late response (sorry) but I run it standalone, setup via ansible. I have ansible scripts that bootstrap everything "below" nomad so it sets up zerotier, then vault, then consul, then dnsmasq (pointing to consul) and then nomad. You could probably run gluster in nomad but given that I give most nomad tasks a gluster directory that feels odd. The only thing I still need to figure out about this setup is that I currently use a single glusterfs volume for most of my nomad tasks, I would love to have a nomad integration that could provision and mount the gluster volume when I specify a volume for a docker task in nomad.
- ckrailo 7y ago
  Any additional build details from you or Legogris? Haven't considered using glusterfs but the rest of those pieces, absolutely. Only piece I could find similar to the topic is https://www.mockingbirdconsulting.co.uk/blog/2019-01-05-hashicorp-at-home/ https://www.mockingbirdconsulting.co.uk/blog/2019-01-05-hash....
  - Legogris 7y ago

vedses 7y ago

Opensuse also provides 64bit images for RaspberryPi https://en.opensuse.org/HCL:Raspberry_Pi4 https://en.opensuse.org/HCL:Raspberry_Pi4

alias_neo 7y ago

It'd be nice to read about your setup if you post it somewhere. I just built a new server with a 12c/24t 3900x and bought 3x RPi4-4GB, the goal was to have the Pis and a couple of light-weight VMs run a Kubernetes cluster using K3s (I'm currently running 3 Alpine VMs as a cluster on the server, and planning to add the Alpine aarch64 on the 3 Pis. I was thinking of deploying OpenFAAS to the Pis for serverless functions and have more "heavy" services on the server. I'm a developer so it's more of an academic task than running production services (I have another home server running FreeNAS with my production services in jails). The alternative would be the Hashi stack, and I want to work with Vault anyway as I'm looking at it for secret management in our infrastructure at work.

Legogris 7y ago
I'm thinking I might do a write-up when I've had some more progress! And yeah, my requirements in terms of performance are pretty light but my main candidate for whenever I need more power on a single node (though I guess I'd have to get at least two to keep my HA approach, so I want to see how far I can get with horizontally scaling SBCs first) is a mini-ITX with Ryzen 3900.
- alias_neo 7y ago
  I made the mistake of putting the 3900X in the CS381, which, sadly has zero cooler clearance, and so the NH-L9a I had to use so it'd fit, has horrid cooling performance. I'd just suggest being careful with case choice to make sure you can put a big cooler or an AIO on the CPU.

zek 7y ago

I have this exact setup (vault/consul/nomad + glusterfs and zerotier for networking) and its pretty awesome. Still dependent on letsencrypt for SSL certificates. It runs plex amazingly well (though I am using some machines which are probably overkill for this purpose), even with the data coming from a glusterfs drive. Most of my nomad tasks can just launch anywhere because of gluster. For internal DNS at least, you can just use consul. I set up dnsmasq to forward to consul on all of my machines which is super convenient (esp when that DNS just points to a docker container ipv6 address on the zerotier network, not port remapping on networking insanity needed)

Legogris 7y ago
Cheers, haven't checked out zerotier before. I'm curious, do you run GlusterFS on nomad or standalone? Seems like it could work fine as raw_exec, but I'm not sure if that's a good idea or not.
- zek 7y ago
  kinda a late response (sorry) but I run it standalone, setup via ansible. I have ansible scripts that bootstrap everything "below" nomad so it sets up zerotier, then vault, then consul, then dnsmasq (pointing to consul) and then nomad. You could probably run gluster in nomad but given that I give most nomad tasks a gluster directory that feels odd. The only thing I still need to figure out about this setup is that I currently use a single glusterfs volume for most of my nomad tasks, I would love to have a nomad integration that could provision and mount the gluster volume when I specify a volume for a docker task in nomad.
ckrailo 7y ago
Any additional build details from you or Legogris? Haven't considered using glusterfs but the rest of those pieces, absolutely. Only piece I could find similar to the topic is https://www.mockingbirdconsulting.co.uk/blog/2019-01-05-hashicorp-at-home/ https://www.mockingbirdconsulting.co.uk/blog/2019-01-05-hash....
- Legogris 7y ago