4 ms·

Colocation providers seem to charge an additional $5-10 per Ethernet port. That’s part of why I ended up cramming all the Pis into the 1U and using the network

Colocation providers seem to charge an additional $5-10 per Ethernet port. That’s part of why I ended up cramming all the Pis into the 1U and using the network switch. There’s a similar rack mount for Pis, but I would have needed to pay for the additional Ethernet ports monthly.

elcritch 5y ago
That makes sense. BTW, fun project write up! I'm torn between doing the colo or just upgrading to 1G fiber at home. To run 3-4 NUC's I'd use 2-3 120V-to-14V DC module's in parallel. Then if one power supply failed, you'd have redundancy! They run off 12V-19V nicely and you can source switches that run off 12V-19V as well. It's even possible to put in a LiFePO4 battery for backup. I'll probably do that at home to skip the UPS. In, all you can have a decent server cluster with top of the line redundancy. Well, stick a pi4 in for remote management. I use Nerves for my flashing remote devices then using docker or containers to run the software. One project I wanted to toy around with (and it'd work with Pi4's) would be a secure cluster setup without k8s. There's apparently support in KVM to run a VM with hot-swap if you can setup a shared storage. It seems KVM supports Gluster (or Cepher?) for shared storage. I'm a bit surprised it's not already "a thing" for small businesses.
- vetinari 5y ago
  If you want to play with live migrations, try a cluster running Proxmox. It does support both Gluster and Ceph, and for smaller clusters ZFS (yes, you can live migrate VMs with local disks backed by ZFS, once you set up a replication). Also note that distributed filesystems (Ceph at least) want to own entire block device; thus you need something else to boot & run the system from (which is going to be a problem for slim NUCs above, that have only a single M.2 slot). Ceph and Gluster are not a thing for small businesses, because for them to be practical, they want cluster sizes that are beyond needs of small businesses. It is a gun too big for them.
  - elcritch 5y ago
    Thanks I’ll have to check out Proxmox. For the NUC’s I’ve been booting from USB drives using Nerves. It works well! That leaves the entire m.2 for the application. Nerves targets embedded buts works perfectly for setting up a cluster of small devices. Using ZFS could be a good method, hmmm. Ceph and gluster aren’t meant for small offices, sure. But small businesses still spend a lot to setup IT servers. A system with reliable replication costs even more. So if one of those file systems could provide redundancy it’d be worth the extra disks. Of course many IT staff won’t know anything about those file systems, much less running them. I’d like one to just have a storage cluster for resilient backups for personal data.
vetinari 5y ago
Yeah, running in colo has another constrains than running a homelab. In my case, a switch with appropriate amount of slots was just few Us above, for free ;). What I was wondering, why unmanaged switch without PoE (OK, I then it would need additional IP address), no PoE hats and then having to play around with relays (their control doesn't require IP?). With PoE managed switch, you would get control of PoE and could turn off/on the Pis.

elcritch 5y ago

That makes sense. BTW, fun project write up! I'm torn between doing the colo or just upgrading to 1G fiber at home. To run 3-4 NUC's I'd use 2-3 120V-to-14V DC module's in parallel. Then if one power supply failed, you'd have redundancy! They run off 12V-19V nicely and you can source switches that run off 12V-19V as well. It's even possible to put in a LiFePO4 battery for backup. I'll probably do that at home to skip the UPS. In, all you can have a decent server cluster with top of the line redundancy. Well, stick a pi4 in for remote management. I use Nerves for my flashing remote devices then using docker or containers to run the software. One project I wanted to toy around with (and it'd work with Pi4's) would be a secure cluster setup without k8s. There's apparently support in KVM to run a VM with hot-swap if you can setup a shared storage. It seems KVM supports Gluster (or Cepher?) for shared storage. I'm a bit surprised it's not already "a thing" for small businesses.

vetinari 5y ago
If you want to play with live migrations, try a cluster running Proxmox. It does support both Gluster and Ceph, and for smaller clusters ZFS (yes, you can live migrate VMs with local disks backed by ZFS, once you set up a replication). Also note that distributed filesystems (Ceph at least) want to own entire block device; thus you need something else to boot & run the system from (which is going to be a problem for slim NUCs above, that have only a single M.2 slot). Ceph and Gluster are not a thing for small businesses, because for them to be practical, they want cluster sizes that are beyond needs of small businesses. It is a gun too big for them.
- elcritch 5y ago
  Thanks I’ll have to check out Proxmox. For the NUC’s I’ve been booting from USB drives using Nerves. It works well! That leaves the entire m.2 for the application. Nerves targets embedded buts works perfectly for setting up a cluster of small devices. Using ZFS could be a good method, hmmm. Ceph and gluster aren’t meant for small offices, sure. But small businesses still spend a lot to setup IT servers. A system with reliable replication costs even more. So if one of those file systems could provide redundancy it’d be worth the extra disks. Of course many IT staff won’t know anything about those file systems, much less running them. I’d like one to just have a storage cluster for resilient backups for personal data.

vetinari 5y ago

Yeah, running in colo has another constrains than running a homelab. In my case, a switch with appropriate amount of slots was just few Us above, for free ;). What I was wondering, why unmanaged switch without PoE (OK, I then it would need additional IP address), no PoE hats and then having to play around with relays (their control doesn't require IP?). With PoE managed switch, you would get control of PoE and could turn off/on the Pis.