3 ms·

Yup -- whole heartedly agree. It's so easy that at this point, most of my worry is wondering how I would deal with a disaster scenario, whether the server getti

Yup -- whole heartedly agree. It's so easy that at this point, most of my worry is wondering how I would deal with a disaster scenario, whether the server getting wiped or any other kind of catastrophic failure -- is my automation good enough that I can go from bare-machine to all my projects fully running? One thing I was a little disappointed with was the ignition provisioning system (https://coreos.com/ignition/docs/latest/ https://coreos.com/ignition/docs/latest/) -- it's revolutionary (I mean not really, but it is to me) in the amount of the setting-up-the-server routine that you can skip, but I was disappointed that you can only run it once (by design, that's exactly what it's supposed to do, set up the machine ONCE). I currently manage one larger dedicated host that is more pet than cattle, but if I was truly in someone's cloud spinning up instances, then I could take time to build my perfect ignition config, but I only got one shot (and it worked the first time so), and didn't want to wipe the harddrive.

kuschku 9y ago
> Yup -- whole heartedly agree. It's so easy that at this point, most of my worry is wondering how I would deal with a disaster scenario, whether the server getting wiped or any other kind of catastrophic failure -- is my automation good enough that I can go from bare-machine to all my projects fully running? I’m running my system currently on Ubuntu (I’d have to pay extra for additional iSCSI drives), and I had to recreate the system hundreds of times due to mistakes. So I built an automated system to do that, and tested it hundreds of times. So I’m at least sure that mine works.
- hardwaresofton 9y ago
  Would you mind sharing the system or what it looks like? I haven't tested the re-build process, which is why I'm almost certain it would fail me/require a lot of terrible manual steps.
  - kuschku 9y ago
    It’s thousands of shell scripts that test the status, set package sources, install kubernetes, and install the actual services. Very hackily implemented.
robszumski 9y ago
I do this today with Terraform. If you have any existing cluster, disaster recovery is as easy as swapping the name in your tfvars, then running a new apply. [1] All new infra + kubernetes is set up. I then have a very simple bash script that is just ~30 `kubectl create` lines. I executed this when our automatic clean up script nuked my cluster due to incorrect tagging on my part. Time from monitoring critical alert to back up was under an hour. [1]: https://coreos.com/tectonic/docs/latest/install/aws/manual-boot.html#deploy-the-cluster https://coreos.com/tectonic/docs/latest/install/aws/manual-b...
- hardwaresofton 9y ago
  The link you posted is to AWS -- have you ever used terraform on a bare-metal machine? I feel like I've said this at least 5 times so far, but every time I read the Terraform documentation, I get the distinct feeling it's not really for "bare metal" (that's what everyone seems to be calling a machine you actually own in someone's data center) I really want to use Terraform, but it just doesn't look like it really wants to do the single server use case.

kuschku 9y ago

> Yup -- whole heartedly agree. It's so easy that at this point, most of my worry is wondering how I would deal with a disaster scenario, whether the server getting wiped or any other kind of catastrophic failure -- is my automation good enough that I can go from bare-machine to all my projects fully running? I’m running my system currently on Ubuntu (I’d have to pay extra for additional iSCSI drives), and I had to recreate the system hundreds of times due to mistakes. So I built an automated system to do that, and tested it hundreds of times. So I’m at least sure that mine works.

hardwaresofton 9y ago
Would you mind sharing the system or what it looks like? I haven't tested the re-build process, which is why I'm almost certain it would fail me/require a lot of terrible manual steps.
- kuschku 9y ago
  It’s thousands of shell scripts that test the status, set package sources, install kubernetes, and install the actual services. Very hackily implemented.

robszumski 9y ago

I do this today with Terraform. If you have any existing cluster, disaster recovery is as easy as swapping the name in your tfvars, then running a new apply. [1] All new infra + kubernetes is set up. I then have a very simple bash script that is just ~30 `kubectl create` lines. I executed this when our automatic clean up script nuked my cluster due to incorrect tagging on my part. Time from monitoring critical alert to back up was under an hour. [1]: https://coreos.com/tectonic/docs/latest/install/aws/manual-boot.html#deploy-the-cluster https://coreos.com/tectonic/docs/latest/install/aws/manual-b...

hardwaresofton 9y ago
The link you posted is to AWS -- have you ever used terraform on a bare-metal machine? I feel like I've said this at least 5 times so far, but every time I read the Terraform documentation, I get the distinct feeling it's not really for "bare metal" (that's what everyone seems to be calling a machine you actually own in someone's data center) I really want to use Terraform, but it just doesn't look like it really wants to do the single server use case.