6 ms·

> Personal Anecdote: I switched all my personal project infrastructure to use Kubernetes (granted, it's a single node cluster), and went through the process of

> Personal Anecdote: I switched all my personal project infrastructure to use Kubernetes (granted, it's a single node cluster), and went through the process of setting it up myself (no tectonic). The process was difficult but immensely rewarding (I have a super long blog post series incoming about it), and various bits of documentation helped me every step of the way. For the curious, my latest ops iteration was docker-containers-managed-by-systemd-provisioned-by-ansible setup to docker-containers-managed-by-kubernetes-provisioned-by-ansible-on-coreos. I’ve done something very similar myself, and you’re exactly right. The effort to set up a new project, or tear it down, is basically zero. I can properly version my entire system via git, by just exporting the configs as json, putting them in a git repo, and importing them back if I want to. I honestly recommend everyone to at least try it out, it’s immensely useful (and a good skill to have)

hardwaresofton 9y ago
Yup -- whole heartedly agree. It's so easy that at this point, most of my worry is wondering how I would deal with a disaster scenario, whether the server getting wiped or any other kind of catastrophic failure -- is my automation good enough that I can go from bare-machine to all my projects fully running? One thing I was a little disappointed with was the ignition provisioning system (https://coreos.com/ignition/docs/latest/ https://coreos.com/ignition/docs/latest/) -- it's revolutionary (I mean not really, but it is to me) in the amount of the setting-up-the-server routine that you can skip, but I was disappointed that you can only run it once (by design, that's exactly what it's supposed to do, set up the machine ONCE). I currently manage one larger dedicated host that is more pet than cattle, but if I was truly in someone's cloud spinning up instances, then I could take time to build my perfect ignition config, but I only got one shot (and it worked the first time so), and didn't want to wipe the harddrive.
- kuschku 9y ago
  > Yup -- whole heartedly agree. It's so easy that at this point, most of my worry is wondering how I would deal with a disaster scenario, whether the server getting wiped or any other kind of catastrophic failure -- is my automation good enough that I can go from bare-machine to all my projects fully running? I’m running my system currently on Ubuntu (I’d have to pay extra for additional iSCSI drives), and I had to recreate the system hundreds of times due to mistakes. So I built an automated system to do that, and tested it hundreds of times. So I’m at least sure that mine works.
  - hardwaresofton 9y ago
    Would you mind sharing the system or what it looks like? I haven't tested the re-build process, which is why I'm almost certain it would fail me/require a lot of terrible manual steps.
    - kuschku 9y ago
      It’s thousands of shell scripts that test the status, set package sources, install kubernetes, and install the actual services. Very hackily implemented.
- robszumski 9y ago
  I do this today with Terraform. If you have any existing cluster, disaster recovery is as easy as swapping the name in your tfvars, then running a new apply. [1] All new infra + kubernetes is set up. I then have a very simple bash script that is just ~30 `kubectl create` lines. I executed this when our automatic clean up script nuked my cluster due to incorrect tagging on my part. Time from monitoring critical alert to back up was under an hour. [1]: https://coreos.com/tectonic/docs/latest/install/aws/manual-boot.html#deploy-the-cluster https://coreos.com/tectonic/docs/latest/install/aws/manual-b...
  - hardwaresofton 9y ago
    The link you posted is to AWS -- have you ever used terraform on a bare-metal machine? I feel like I've said this at least 5 times so far, but every time I read the Terraform documentation, I get the distinct feeling it's not really for "bare metal" (that's what everyone seems to be calling a machine you actually own in someone's data center) I really want to use Terraform, but it just doesn't look like it really wants to do the single server use case.

hardwaresofton 9y ago

Yup -- whole heartedly agree. It's so easy that at this point, most of my worry is wondering how I would deal with a disaster scenario, whether the server getting wiped or any other kind of catastrophic failure -- is my automation good enough that I can go from bare-machine to all my projects fully running? One thing I was a little disappointed with was the ignition provisioning system (https://coreos.com/ignition/docs/latest/ https://coreos.com/ignition/docs/latest/) -- it's revolutionary (I mean not really, but it is to me) in the amount of the setting-up-the-server routine that you can skip, but I was disappointed that you can only run it once (by design, that's exactly what it's supposed to do, set up the machine ONCE). I currently manage one larger dedicated host that is more pet than cattle, but if I was truly in someone's cloud spinning up instances, then I could take time to build my perfect ignition config, but I only got one shot (and it worked the first time so), and didn't want to wipe the harddrive.

kuschku 9y ago
> Yup -- whole heartedly agree. It's so easy that at this point, most of my worry is wondering how I would deal with a disaster scenario, whether the server getting wiped or any other kind of catastrophic failure -- is my automation good enough that I can go from bare-machine to all my projects fully running? I’m running my system currently on Ubuntu (I’d have to pay extra for additional iSCSI drives), and I had to recreate the system hundreds of times due to mistakes. So I built an automated system to do that, and tested it hundreds of times. So I’m at least sure that mine works.
- hardwaresofton 9y ago
  Would you mind sharing the system or what it looks like? I haven't tested the re-build process, which is why I'm almost certain it would fail me/require a lot of terrible manual steps.
  - kuschku 9y ago
    It’s thousands of shell scripts that test the status, set package sources, install kubernetes, and install the actual services. Very hackily implemented.
robszumski 9y ago
I do this today with Terraform. If you have any existing cluster, disaster recovery is as easy as swapping the name in your tfvars, then running a new apply. [1] All new infra + kubernetes is set up. I then have a very simple bash script that is just ~30 `kubectl create` lines. I executed this when our automatic clean up script nuked my cluster due to incorrect tagging on my part. Time from monitoring critical alert to back up was under an hour. [1]: https://coreos.com/tectonic/docs/latest/install/aws/manual-boot.html#deploy-the-cluster https://coreos.com/tectonic/docs/latest/install/aws/manual-b...
- hardwaresofton 9y ago
  The link you posted is to AWS -- have you ever used terraform on a bare-metal machine? I feel like I've said this at least 5 times so far, but every time I read the Terraform documentation, I get the distinct feeling it's not really for "bare metal" (that's what everyone seems to be calling a machine you actually own in someone's data center) I really want to use Terraform, but it just doesn't look like it really wants to do the single server use case.