10 ms·

Creating my personal cloud with HashiCorp

205 points by CGamesPlay 5y ago

yosito 5y ago
I'd be curious to know more about the average cost of this setup over a few months, and how scaling up/down affects the cost.
vgb2k18 5y ago
> they still feel usable after 6 months of use. By this time, most of the context that I learned to develop the project has faded away and I have to rely on the documentation that I left myself. This was a much-needed reminder that the target audience for my documentation is my future-self.
brunooliv 5y ago
Great stuff!!
kosolam 5y ago
Thanks for sharing. I was looking for such examples
reidjs 5y ago
What does it mean to host a personal cloud vs hosting a server? I thought the point of using cloud services was so you didn’t have to host it yourself.
- shell0x 5y ago
  Aren't a lot of people just using cloud instead of server now? It's similar to people saying I run on EC2, which is a server, and 95% of the time it's basically a VM.
- CGamesPlay 5y ago
  I am using "personal cloud" to refer to the set of "cloud services" that I am hosting myself on my own server. The advantage of having the server is that my devices have a central syncing hub, and the advantage of hosting myself is no need to trust third parties. There are obviously disadvantages, like having to host it myself.
  - deleted 5y ago
    [deleted]
shell0x 5y ago
As a DevOps guy, I'm not a huge fan of Terraform. Often I hear from enterprises that Terraform is cloud agnostic, but that's often very wrong. Terraform modules are still specific to the cloud platform and a rewrite is required to port an app running on AWS to GCP. If you use AWS, you're probably better off to use AWS Cloudformation and for GCP Google Cloud Deployment manager. A business reason is often that the engineers are more familiar with Terraform already, but learning Cloudformation is really not that hard...And if you can't work out how to use a basic tool, you should not be running infrastructure anyway, because IT is a constant cycle of change. I'm not saying Terraform is not good, but I just think a native solution of the platform is preferable over a 3rd party tool.
- brightball 5y ago
  Terraform also had the advantage of being able to pull together multiple services outside of a single provider. Want to use Cloudflare with AWS and another 3rd party provider that offers services in an AWS region? Simple with Terraform.
- 404mm 5y ago
  Also a DevOps guy, I like Terraform from the “user” experience. It’s much more comfortable eco system to be in than AWS CF. Managing your resources is also better experience. That is until you hit an issue with resource or situation not being correctly supported by TF. CF has the vast advantage of being native and fully supporting AWS resources. Unfortunately it gets complicated (not complex) so quickly and has a strong feel of rushed MVP.
- c7DJTLrn 5y ago
  I seem to recall that years ago, Terraform was touted as an abstraction layer for cloud providers meaning that you could write some HCL and move between them seamlessly. A solution for vendor lock-in. Maybe that was nonsense but that's certainly not the case today.
- kosolam 5y ago
  If you are saying that it’s better to run native than Terraform then to me it means that Terraform is not good enough - which is also my experience from poking it. It covers all use cases on all clouds, buy only ~80% of each use case. The rest you have to figure out by yourself. Which often means diving into CloudFormation and such - so basically losing the advantage of Terraform.
zoidb 5y ago
I was looking into doing something similar with Nomad recently since lately I have been using systemd to launch containers and managing that config with some janky shell scripts. How are you configuring everything to run on a single server, including consul? Isn't Nomad designed to to run on multiple servers or are you running nomad in multiple containers on your VPS? When I looked into this previously I got to https://stackoverflow.com/questions/56112422/nomad-configuration-for-single-node-to-act-as-production-server-and-client https://stackoverflow.com/questions/56112422/nomad-configura... which has some suggestions for running Nomad on a single node but generally recommends against it.
- Jedd 5y ago
  For development purposes you very much can run Nomad and Consul on a single host. They recommend you don't, as you lose any HA, of course, but for those of us not seeking 5 nines of availability, that's quite acceptable. In my lab I'm actually running a two-node cluster, but that's 'even worse' and engenders the occasional mildly surprising failure states. Anyway, I can highly recommend setting up Nomad (and some friends) on a single host. It's going to be much more robust & interesting than 'some janky shell scripts'. : )
  - CGamesPlay 5y ago
    What's the configuration here? 1 server-client and 1 client? Or are you actually running 2 server-clients?
    - Jedd 5y ago
      It's not my ideal situation, but pandemic relocation etc means I'm not next to my normal lab (two ESX servers + NAS). So, that said, yes, server (whitebox) plus desktop (xeon, 32gb) - both running Debian, with Consul, Nomad, Traefik, promtail etc running on both, but no shared NFS between the two, so I've got constraints on most of the important stuff to run on the server (prometheus, cortex, loki, nodered). In practice, desktop is running 24/7, just it occasionally gets a reboot. Having consul/nomad running as server-client on two machines is undeniably weird, and requires some careful consideration around bootstrap_expect= settings. Almost all of this is around having a useful facsimile of my work environment, rather than (say) running an SSG public site.
stavros 5y ago
I had many of the same needs, so I wrote Harbormaster: https://gitlab.com/stavros/harbormaster https://gitlab.com/stavros/harbormaster All it does is manage Compose applications, with a sane directory structure. It's been working great, both for my personal use and for a few companies running production workloads on it. I love that it's super simple and the workflow it has is fantastic, I just push to a repo and everything else happens automatically.
- CGamesPlay 5y ago
  Nice! I may even have looked at that when I was getting started on this project. The two things that jump out at me as "not as easy" are private docker repos and cron jobs. I know that you can run cron images for Docker, but I didn't want the extra layer of indirection.
  - stavros 5y ago
    Hmm, what do you mean? You don't have to use either of those with Harbormaster.
    - CGamesPlay 5y ago
      I wanted to use both of them :)
      - stavros 5y ago
        Oh, well, you can :P Can you go into a bit more detail about your setup?
        CGamesPlay 5y ago
        Nomad has native support for periodic jobs that does not require running a cron docker image, and that was a big draw. As far as the private registry goes, I don't see why that would be particularly hard with a set of compose scripts, but I really liked the fact that Nomad has everything about them documented. Honestly, if I had stuck with Compose files then Harbormaster looks like it's a reasonable way to manage them. But I felt like everything I wanted to do was just a little bit more difficult when I was using them, especially after comparing to Nomad.
moondev 5y ago
the qemu driver for nomad seems pretty bare bones compared to kubevirt https://www.nomadproject.io/docs/drivers/qemu https://www.nomadproject.io/docs/drivers/qemu https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#cloudinitnocloud https://kubevirt.io/user-guide/virtual_machines/disks_and_vo... Is terraform generally used to deploy workloads to nomad instead of writing tasks directly?
- CGamesPlay 5y ago
  I believe that it does have a provider that talks to Nomad, however I just directly write the Nomad jobs (or use levant, which is a lightweight template for Nomad jobs). The only use of Terraform is to set up the barebones VM and other infrastructure (S3 buckets, etc.)
- chromatin 5y ago
  With the Nomad qemu driver, any unsupported options (like bridge networking) can be passed as args.
- glynnforrest 5y ago
  > Is terraform generally used to deploy workloads to nomad instead of writing tasks directly? It can be, although it has some weird shortcomings. For example, if the job is already present in Nomad but not running ("dead"), I don't think you can use the terraform provider to start it again. HashiCorp themselves suggest using terraform to provision the base nomad system (ACL, quotas, base system jobs), but perhaps not your actual applications: > This can be used to initialize your cluster with system jobs, common services, and more. In day to day Nomad use it is common for developers to submit jobs to Nomad directly, such as for general app deployment. In addition to these apps, a Nomad cluster often runs core system services that are ideally setup during infrastructure creation. This resource is ideal for the latter type of job, but can be used to manage any job within Nomad. https://registry.terraform.io/providers/hashicorp/nomad/latest/docs/resources/job https://registry.terraform.io/providers/hashicorp/nomad/late... In my work helping companies with nomad, I've seen jobs run a few different ways: * Write and submit HCL jobs directly to nomad * Terraform templating and the nomad provider (as above) * https://github.com/hashicorp/levant https://github.com/hashicorp/levant * A DIY templating thing (e.g. python and jinja2 templates) * A webapp that submits jobs as JSON directly to the nomad API, perhaps modifying it to match certain policies (kind of like k8s validating / mutating admission webhooks) There's also a new thing similar to Helm: https://github.com/hashicorp/nomad-pack https://github.com/hashicorp/nomad-pack
protoman3000 5y ago
I am curious, why is cloud-init not enough?
- capableweb 5y ago
  cloud-init is for instance initialization, commonly called "provisioning". Nomad is a application (container/non-container) orchestration platform. The two couldn't be more different.
chromatin 5y ago
Great article. I’ve also just started using nomad to manage personal services on my home loan. What I particularly like is that I have a mixture of Docker containers, VMs, and LXD containers all centrally managed. Overall I found nomad to be fairly intuitive, and the single binary/single job paradigm of the Hashicorp stack is very appealing.
- lupire 5y ago
  home loan?
jokethrowaway 5y ago
I was in a similar situation a few months ago. I had previous positive experience with k8s and some negative experience with nomad, so I was torn between k3s/k0s and just docker stack+swarm. I didn't want to install anything as I started with a minuscole node (5$ digital ocean node). In the end I went with docker stack + machine + swarm + docker flow proxy and it's been pretty smooth sailing. I don't need anything else installed on the machine, I can develop stacks locally with docker compose, deploy them to swarm and I can even add replication on multiple machines for services that need it. I manage secrets with stackoverflow's blackbox and the secrets live in the repository encrypted. I can easily run commands on the local or remote stack via docker machine. I cheated with cron jobs and I just synchronise /etc/cron on all hosts with a directory in my repo. The commands are mostly running commands inside containers. I have one stack for redis, postgres, docker registry and multiple stacks for other applications. The flow to deploying stuff is basically: generate id from git hash, build docker image, push to self hosted registry, stack deploy with the git hash as version. A few caveats I found: - Dependencies with docker health checks will take the duration of the health checks to become available to services needing them. Either you skip health checks or setup things so that infra is running before your services. - In order to use docker-machine on multiple machines / with different contributors you need to export the key used to setup the machine and export it (there is a npm package that works very well, but it would be nice to have it natively). - You have to cleanup the docker registry manually every once and then (I had to write a cron job for that) - I'm unsure about the future of docker, albeit things work pretty well as they are.
humblepie 5y ago
I've been using nomad for smaller setups in AWS and it's been great. The biggest issue I've encountered on it is when you move out of AWS (and you can't apply EC2-based IAM) but still have S3-hosted artifacts. Specifically it cannot receive Vault secrets for the artifact credentials because the nomad templates get applied at a much later stage.
- glynnforrest 5y ago
  Yeah, this pains me too. Here's a relevant issue to keep an eye on: https://github.com/hashicorp/nomad/issues/3854 https://github.com/hashicorp/nomad/issues/3854 I've used an nginx-based S3 proxy in the past to get around this. Not ideal but it works.
- CGamesPlay 5y ago
  Have you looked into levant? Seems like it would allow you to do this. Now, with levant the developer machine would be the thing retrieving the vault secrets, but it may be a useful stopgap. https://github.com/hashicorp/levant https://github.com/hashicorp/levant
  - humblepie 5y ago
    Hmm, would the stored job data still include the AWS credentials? That is, if I change the artifact S3 credentials and I run "nomad job plan" it will show the diff of the AWS keys. That means somewhere in the nomad raft logs the keys are exposed.
    - CGamesPlay 5y ago
      Yes, exactly. Definitely not ideal, but potentially a workaround depending on the security requirements.
la6471 5y ago
Good article!

205 points by CGamesPlay 5y ago

I'd be curious to know more about the average cost of this setup over a few months, and how scaling up/down affects the cost.

vgb2k18 5y ago

> they still feel usable after 6 months of use. By this time, most of the context that I learned to develop the project has faded away and I have to rely on the documentation that I left myself. This was a much-needed reminder that the target audience for my documentation is my future-self.

brunooliv 5y ago

Great stuff!!

kosolam 5y ago

Thanks for sharing. I was looking for such examples

reidjs 5y ago

What does it mean to host a personal cloud vs hosting a server? I thought the point of using cloud services was so you didn’t have to host it yourself.

shell0x 5y ago
Aren't a lot of people just using cloud instead of server now? It's similar to people saying I run on EC2, which is a server, and 95% of the time it's basically a VM.
CGamesPlay 5y ago
I am using "personal cloud" to refer to the set of "cloud services" that I am hosting myself on my own server. The advantage of having the server is that my devices have a central syncing hub, and the advantage of hosting myself is no need to trust third parties. There are obviously disadvantages, like having to host it myself.
- deleted 5y ago
  [deleted]

shell0x 5y ago

As a DevOps guy, I'm not a huge fan of Terraform. Often I hear from enterprises that Terraform is cloud agnostic, but that's often very wrong. Terraform modules are still specific to the cloud platform and a rewrite is required to port an app running on AWS to GCP. If you use AWS, you're probably better off to use AWS Cloudformation and for GCP Google Cloud Deployment manager. A business reason is often that the engineers are more familiar with Terraform already, but learning Cloudformation is really not that hard...And if you can't work out how to use a basic tool, you should not be running infrastructure anyway, because IT is a constant cycle of change. I'm not saying Terraform is not good, but I just think a native solution of the platform is preferable over a 3rd party tool.

brightball 5y ago
Terraform also had the advantage of being able to pull together multiple services outside of a single provider. Want to use Cloudflare with AWS and another 3rd party provider that offers services in an AWS region? Simple with Terraform.
404mm 5y ago
Also a DevOps guy, I like Terraform from the “user” experience. It’s much more comfortable eco system to be in than AWS CF. Managing your resources is also better experience. That is until you hit an issue with resource or situation not being correctly supported by TF. CF has the vast advantage of being native and fully supporting AWS resources. Unfortunately it gets complicated (not complex) so quickly and has a strong feel of rushed MVP.
c7DJTLrn 5y ago
I seem to recall that years ago, Terraform was touted as an abstraction layer for cloud providers meaning that you could write some HCL and move between them seamlessly. A solution for vendor lock-in. Maybe that was nonsense but that's certainly not the case today.
kosolam 5y ago
If you are saying that it’s better to run native than Terraform then to me it means that Terraform is not good enough - which is also my experience from poking it. It covers all use cases on all clouds, buy only ~80% of each use case. The rest you have to figure out by yourself. Which often means diving into CloudFormation and such - so basically losing the advantage of Terraform.

zoidb 5y ago

I was looking into doing something similar with Nomad recently since lately I have been using systemd to launch containers and managing that config with some janky shell scripts. How are you configuring everything to run on a single server, including consul? Isn't Nomad designed to to run on multiple servers or are you running nomad in multiple containers on your VPS? When I looked into this previously I got to https://stackoverflow.com/questions/56112422/nomad-configuration-for-single-node-to-act-as-production-server-and-client https://stackoverflow.com/questions/56112422/nomad-configura... which has some suggestions for running Nomad on a single node but generally recommends against it.

Jedd 5y ago
For development purposes you very much can run Nomad and Consul on a single host. They recommend you don't, as you lose any HA, of course, but for those of us not seeking 5 nines of availability, that's quite acceptable. In my lab I'm actually running a two-node cluster, but that's 'even worse' and engenders the occasional mildly surprising failure states. Anyway, I can highly recommend setting up Nomad (and some friends) on a single host. It's going to be much more robust & interesting than 'some janky shell scripts'. : )
- CGamesPlay 5y ago
  What's the configuration here? 1 server-client and 1 client? Or are you actually running 2 server-clients?
  - Jedd 5y ago
    It's not my ideal situation, but pandemic relocation etc means I'm not next to my normal lab (two ESX servers + NAS). So, that said, yes, server (whitebox) plus desktop (xeon, 32gb) - both running Debian, with Consul, Nomad, Traefik, promtail etc running on both, but no shared NFS between the two, so I've got constraints on most of the important stuff to run on the server (prometheus, cortex, loki, nodered). In practice, desktop is running 24/7, just it occasionally gets a reboot. Having consul/nomad running as server-client on two machines is undeniably weird, and requires some careful consideration around bootstrap_expect= settings. Almost all of this is around having a useful facsimile of my work environment, rather than (say) running an SSG public site.

stavros 5y ago

I had many of the same needs, so I wrote Harbormaster: https://gitlab.com/stavros/harbormaster https://gitlab.com/stavros/harbormaster All it does is manage Compose applications, with a sane directory structure. It's been working great, both for my personal use and for a few companies running production workloads on it. I love that it's super simple and the workflow it has is fantastic, I just push to a repo and everything else happens automatically.

CGamesPlay 5y ago
Nice! I may even have looked at that when I was getting started on this project. The two things that jump out at me as "not as easy" are private docker repos and cron jobs. I know that you can run cron images for Docker, but I didn't want the extra layer of indirection.
- stavros 5y ago
  Hmm, what do you mean? You don't have to use either of those with Harbormaster.
  - CGamesPlay 5y ago
    I wanted to use both of them :)
    - stavros 5y ago
      Oh, well, you can :P Can you go into a bit more detail about your setup?
      - CGamesPlay 5y ago
        Nomad has native support for periodic jobs that does not require running a cron docker image, and that was a big draw. As far as the private registry goes, I don't see why that would be particularly hard with a set of compose scripts, but I really liked the fact that Nomad has everything about them documented. Honestly, if I had stuck with Compose files then Harbormaster looks like it's a reasonable way to manage them. But I felt like everything I wanted to do was just a little bit more difficult when I was using them, especially after comparing to Nomad.

moondev 5y ago

the qemu driver for nomad seems pretty bare bones compared to kubevirt https://www.nomadproject.io/docs/drivers/qemu https://www.nomadproject.io/docs/drivers/qemu https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#cloudinitnocloud https://kubevirt.io/user-guide/virtual_machines/disks_and_vo... Is terraform generally used to deploy workloads to nomad instead of writing tasks directly?

CGamesPlay 5y ago
I believe that it does have a provider that talks to Nomad, however I just directly write the Nomad jobs (or use levant, which is a lightweight template for Nomad jobs). The only use of Terraform is to set up the barebones VM and other infrastructure (S3 buckets, etc.)
chromatin 5y ago
With the Nomad qemu driver, any unsupported options (like bridge networking) can be passed as args.
glynnforrest 5y ago
> Is terraform generally used to deploy workloads to nomad instead of writing tasks directly? It can be, although it has some weird shortcomings. For example, if the job is already present in Nomad but not running ("dead"), I don't think you can use the terraform provider to start it again. HashiCorp themselves suggest using terraform to provision the base nomad system (ACL, quotas, base system jobs), but perhaps not your actual applications: > This can be used to initialize your cluster with system jobs, common services, and more. In day to day Nomad use it is common for developers to submit jobs to Nomad directly, such as for general app deployment. In addition to these apps, a Nomad cluster often runs core system services that are ideally setup during infrastructure creation. This resource is ideal for the latter type of job, but can be used to manage any job within Nomad. https://registry.terraform.io/providers/hashicorp/nomad/latest/docs/resources/job https://registry.terraform.io/providers/hashicorp/nomad/late... In my work helping companies with nomad, I've seen jobs run a few different ways: * Write and submit HCL jobs directly to nomad * Terraform templating and the nomad provider (as above) * https://github.com/hashicorp/levant https://github.com/hashicorp/levant * A DIY templating thing (e.g. python and jinja2 templates) * A webapp that submits jobs as JSON directly to the nomad API, perhaps modifying it to match certain policies (kind of like k8s validating / mutating admission webhooks) There's also a new thing similar to Helm: https://github.com/hashicorp/nomad-pack https://github.com/hashicorp/nomad-pack

protoman3000 5y ago

I am curious, why is cloud-init not enough?

capableweb 5y ago
cloud-init is for instance initialization, commonly called "provisioning". Nomad is a application (container/non-container) orchestration platform. The two couldn't be more different.

chromatin 5y ago

Great article. I’ve also just started using nomad to manage personal services on my home loan. What I particularly like is that I have a mixture of Docker containers, VMs, and LXD containers all centrally managed. Overall I found nomad to be fairly intuitive, and the single binary/single job paradigm of the Hashicorp stack is very appealing.

lupire 5y ago
home loan?

jokethrowaway 5y ago

I was in a similar situation a few months ago. I had previous positive experience with k8s and some negative experience with nomad, so I was torn between k3s/k0s and just docker stack+swarm. I didn't want to install anything as I started with a minuscole node (5$ digital ocean node). In the end I went with docker stack + machine + swarm + docker flow proxy and it's been pretty smooth sailing. I don't need anything else installed on the machine, I can develop stacks locally with docker compose, deploy them to swarm and I can even add replication on multiple machines for services that need it. I manage secrets with stackoverflow's blackbox and the secrets live in the repository encrypted. I can easily run commands on the local or remote stack via docker machine. I cheated with cron jobs and I just synchronise /etc/cron on all hosts with a directory in my repo. The commands are mostly running commands inside containers. I have one stack for redis, postgres, docker registry and multiple stacks for other applications. The flow to deploying stuff is basically: generate id from git hash, build docker image, push to self hosted registry, stack deploy with the git hash as version. A few caveats I found: - Dependencies with docker health checks will take the duration of the health checks to become available to services needing them. Either you skip health checks or setup things so that infra is running before your services. - In order to use docker-machine on multiple machines / with different contributors you need to export the key used to setup the machine and export it (there is a npm package that works very well, but it would be nice to have it natively). - You have to cleanup the docker registry manually every once and then (I had to write a cron job for that) - I'm unsure about the future of docker, albeit things work pretty well as they are.

humblepie 5y ago

I've been using nomad for smaller setups in AWS and it's been great. The biggest issue I've encountered on it is when you move out of AWS (and you can't apply EC2-based IAM) but still have S3-hosted artifacts. Specifically it cannot receive Vault secrets for the artifact credentials because the nomad templates get applied at a much later stage.

glynnforrest 5y ago
Yeah, this pains me too. Here's a relevant issue to keep an eye on: https://github.com/hashicorp/nomad/issues/3854 https://github.com/hashicorp/nomad/issues/3854 I've used an nginx-based S3 proxy in the past to get around this. Not ideal but it works.
CGamesPlay 5y ago
Have you looked into levant? Seems like it would allow you to do this. Now, with levant the developer machine would be the thing retrieving the vault secrets, but it may be a useful stopgap. https://github.com/hashicorp/levant https://github.com/hashicorp/levant
- humblepie 5y ago
  Hmm, would the stored job data still include the AWS credentials? That is, if I change the artifact S3 credentials and I run "nomad job plan" it will show the diff of the AWS keys. That means somewhere in the nomad raft logs the keys are exposed.
  - CGamesPlay 5y ago
    Yes, exactly. Definitely not ideal, but potentially a workaround depending on the security requirements.

la6471 5y ago

Good article!