Something we did early on was to just get a rack of a dozen ci/test servers. These machines were really cheap since they can use consumer grade stuff. No UPS, n

Something we did early on was to just get a rack of a dozen ci/test servers. These machines were really cheap since they can use consumer grade stuff. No UPS, no redundant PSUs, no raid, etc. Most importantly they are always there and always “hot”: most runs they can just git pull rather than git clone. They already have tons of intermediate results like node/nuget modules cached. There is a bit of maintenance but unlike maintaining production servers, maintaining these is pretty easy. If you screw up nothing bad happens. When they act up, it’s not extremely urgent.