3 ms·

Hey Jonah, Perhaps you already covered this in the video and I missed it, but I was wondering how the team went about hardware upgrades/disposal? I maintain a

by lprd 6y ago

Hey Jonah, Perhaps you already covered this in the video and I missed it, but I was wondering how the team went about hardware upgrades/disposal? I maintain a just few servers in my own homelab so upgrading is pretty trivial for me, but I can't imagine what its like on that scale. Also, which hypervisor are you using to manage all of those VMs? Thanks again for the talk, it was very insightful and fun to watch!

jonah-archive 6y ago
Thank you, so glad you enjoyed it! We try to keep a regular upgrade cycle (to hold to our tight budget), typically with a tick-tock of adding new hardware (expanding within our existing footprint) and cycling out old hardware and drives as they reach the far end of the works/doesn't work spectrum. We have a local partner who takes care of some disposal for us, but we also have no shortage of physical storage space, so we will also accumulate (sometime intentionally -- nearly our entire "red box" deployment -- see "previous version": https://archive.org/web/petabox.php https://archive.org/web/petabox.php -- is packed into a shipping container. We don't like to throw things away!). For a hypervisor, we use Ganeti (running over KVM). Because our fleet is so heterogenous we need to be able to control a lot of VM parameters in order to efficiently pack our computational resources, and Ganeti is kind of in a sweet spot for us in terms of providing a lot more tooling than a bunch of virsh scripts, and being much smaller than systems like OpenStack geared towards large, homogenous deployments).