6 ms·

Cheap Kubernetes Cluster on AWS with Kubeadm

180 points by unkoman 8y ago

crb 8y ago
> If anyone has any suggestions on a better way of doing this without shelling out $20 a month for an ELB, please open an Issue! kubehost was designed for this purpose on GKE: https://github.com/GoogleContainerTools/kubehost https://github.com/GoogleContainerTools/kubehost
- acd 8y ago
  You could use route53 health checks and do dns failover https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover-configuring.html https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dn...
zegl 8y ago
Can it be considered cheap if the cluster is also small? The overhead of the control plane+networking can be quite big, and can easily use more resources than what a m1.small has to offer.
pstadler 8y ago
I'm running a three node cluster on Hetzner Cloud for less than $10 a month. Comprehensive guide and automated provisioning available here: https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide
- hardwaresofton 8y ago
  +1 for Hetzner, it's amazing -- I'm particularly fond of their dedicated server offerings. One of the things about Kubernetes I like most is the likelihood that it reduces the barrier to entry for "PaaS land" (and resultingly "cloud land") for providers like Hetzner (see OVH's recently announced k8s offering[0]). Once these "baremetal" providers get in the managed (by k8s) game, I'm certain companies will start to spring up and offer/chip away at cloud provider offerings like S3/RDS/etc -- they'll just connect to your k8s infrastructure, and bring their kubernetes-compatible know-how. This is going to cause prices to plummet, as it will make the tiers of value added services more distinct -- i.e. "a platform to run things" on vs. "really good managed databases". [0]: https://www.ovh.co.uk/kubernetes/ https://www.ovh.co.uk/kubernetes/
  - stingraycharles 8y ago
    What’s the network of Hetzner like? I like their offering, especially the AMD Epyc servers seem to give a great bang for the buck, but I am a bit put off by their low 99.9% network uptime guarantee. https://www.hetzner.com/rechtliches/agb https://www.hetzner.com/rechtliches/agb
    - dx034 8y ago
      I'm using both their dedicated servers and the cloud offering and never had any issues. Speed and latency are consistent both within Europe as well as to the US (not monitoring much traffic to Asia). Latency from their DCs in Germany to Frankfurt is around 3ms, so comparable to the central Europe offerings of most cloud providers.
      - stingraycharles 8y ago
        So their network hasn’t had any downtime? Any idea why they would only guarantee 99.9% uptime on their network?
        simplyinfinity 8y ago
        I've been a hetzner customer for the past 5.5 years now, so far I've received 21 emails from their status reports as follows: 10 from failures (2 or 3 of which directly affected me), 10 emails stating when planned maintenance of certain resources will occur 1 email to let me know of the spectre and meltdown vulnerabilities. So far i'm pretty happy customer, and sending business their way and migrating client's sites to their cloud offerings.
        Demiurge 8y ago
        In the past two years, I remember they have done scheduled infrastructure maintenance twice and unscheduled interruption for a few hours. So, no, it's not 100%. But, AWS has also had some interruptions. From personal experience, this Hetzner downtime has simply been negligible enough. Bigger issue have been replacing commodity memory a few times, which could have been avoided by paying a few more dollars for more quality.
- jimmy_ruska 8y ago
  Hetzner lists VMs that are ultra cheap but then has separate listing for more expensive VMs with dedicated cores. Alibaba does the same. With these shared core machines, I wonder what the performance reliability guarantees available and how it compares to the dedicated core machines.
  - grogenaut 8y ago
    Depending on your data I'd also wonder about the security restrictions between tenants. It's a thing I consider on AWS as well depending on what I'm doing.
    - dx034 8y ago
      Do shared cores add any attack vectors other shared machines (with dedicated cores) don't have?
      - grogenaut 8y ago
        There are always risks with shared harware, known and unknown veunerabilities in the hypervisor or hardware. All the recent intel stuff, row hammer, etc. The interesting question comes when people start implementing hardware hypervisors and what is the risk profile there. Security, at the end of the day, isn't about what is secure and what isn't. If you want to be secure, don't get on the internet. Everything else is a exercise in risk tradeoffs and mitigation. If I was doing anything with PII, cc#'s or any other data I never want to touch I wouldn't use shared hardware without hard thought on it.
        londons_explore 8y ago
        Shared hardware from the big cloud players adds attack vectors, but it also comes with some of the best security minds trying to keep the entire platform secure. For example, they'll typically be on secret mailing lists and aware of security vulnerabilities weeks before you know about them.
      - 8y ago
- kaivi 8y ago
  Indeed, cloud+dedicated infrastructure is really cheap after certain load/volume, compared to pure GCE/AWS. I have been using k8s with Hetzner for 6 months now, with dirt-cheap SSD/NVMe storage and 1080 GPUs. Can't recommend them enough, and I do not really see any competition here.
- forgot-my-pw 8y ago
  I like Hetzner's offerings. Wish they're also in North America and Asia.
moondev 8y ago
You can use haproxy as a tcp load balancer for the apiserver instead of an elb
- manishsharan 8y ago
  could you please comment on how you use Haproxy servers across multiple AWS availability zones ? how do you configre and manage dns for your haproxy etc ? My reason for asking is that I am super cheap and I want to avoid AWS ELB charges.
  - yebyen 8y ago
    Put your Haproxy (or nginx-ingress, or whatever) into HostPort mode, or enable hostNetwork. I have a comment somewhere in my history that explains in a bit more detail: https://news.ycombinator.com/item?id=18660503 https://news.ycombinator.com/item?id=18660503 that's it... hostPort mode, hostNetwork, and enable in DaemonSet mode. Now all of your nodes are load balancers for ingress, and you don't need any ELBs. This is not a recommended configuration because something has to point DNS at your nodes, and the nodes are really not designed to be permanent. If you autoscale, or scale your cluster manually, your DNS needs to be updated to keep up with that. You may be able to find a way to automate that, but DNS has limitations related to TTL, such that if you are doing this too frequently, visitors to your cluster are likely to experience issues. But if your nodes never come and go, this is a pretty good way to run a cluster and keep it on the cheap. If the traffic you want to balance is not HTTP then ingress won't help you (for now?), but the configuration for HAproxy will be similar.
lifeisstillgood 8y ago
I love the attitude here - because in my privileged western world worrying if my monthly hosting costs are one latte and crossiant or two seems quaint. But even in the "rich" EU Avergae monthly wages can be only 1,000 USD, making this a tenth of a day's work, and we don't have to go much further afield to see 6 bucks becoming a significant chunk of a workers day. So thank you, some kids somewhere will be able to afford to develop skills because you started penny pinching. cheers
- raehik 8y ago
  It certainly makes getting into this field open to students like me. Very glad for that
- skybrian 8y ago
  Even for people who can afford it, this makes a difference for preserving low-traffic hobby sites you only mildly care about. I have a couple of silly sites that I've run on App Engine for many years, using its generous free tier. If they were on Digital Ocean I'd have shut them down by now, as I eventually did for a Go search engine that I wasn't using much.
fridgamarator 8y ago
Google Cloud and Digital Ocean both offer managed kubernetes clusters, you only pay for the workers.
- bryanlarsen 8y ago
  Not only that, Digital Ocean's fully managed cluster starts at $10.
  - k__ 8y ago
    AWS Fargate starts at $0 If it's only a hobby project with a small cluster, the Fargate costs could very well be under $10.
    - yebyen 8y ago
      That's not really an apples-to-apples comparison though, is it? I mean yeah, a cluster that you never power up costs nothing, that's what I would have hoped. https://aws.amazon.com/fargate/pricing/ https://aws.amazon.com/fargate/pricing/ Assume that you are actually _doing something_ with the cluster, then there will be charges for vCPU and memory. The pricing on this page indicates that if you only need your containers for 10 minutes a day, every day, then you will pay $1.23 for the month at 2GB/1vCPU. That's pretty modest usage at a pretty modest cost. Sure, you can do it, but... you may need to re-architect your product to take advantage of transient workers. (Now I think we all should do that, but that's another conversation...) Compared to DigitalOcean's $10/mo (single-worker, managed) cluster, which provides one full-time node with the same specs, that you can slice-and-dice to run as many tasks as you can fit in 2GB of memory and a single vCPU. Now it becomes clearer that Fargate is priced at a premium. If I'm doing the math right, you'll pay $177.12 for that same task space with your Fargate cluster. If your hobby project needs to run for more than a couple of hours a month, you will usually pay a lot more with Fargate.
      - k__ 8y ago
        I had the impression that the $1.23 are for 10 workers?
triplewipeass 8y ago
How's this better than kops? https://github.com/kubernetes/kops https://github.com/kubernetes/kops
- jordanbeiber 8y ago
  Kubeadm gives you a ”core” or bare bones setup which makes it much more flexible in terms of addons and versions etc. It’s a bit more work though, which is the trade-off, but since the last few versions kubeadm makes it really easy to spin up clusters. Personally I opt for complete flexibility.
  - raesene9 8y ago
    Also Kubeadm has a pretty sane set of default security settings, which some other k8s distributions do not. Specifically kops (by default) does not enable authentication on the kubelet, meaning any attacker who gets access to one container in your cluster is very likely to be able to compromise the whole thing.
013a 8y ago
You can totally install Minikube on AWS [1], which removes the need for a dedicated master and drop the price to that of a single instance. Not sure why you'd ever want to do this, given that GKE or DO will always be cheaper and AWS's core services aren't all that special, but as a thought experiment it's interesting. [1] https://www.radishlogic.com/kubernetes/running-minikube-in-aws-ec2-ubuntu/ https://www.radishlogic.com/kubernetes/running-minikube-in-a...
- zwerdlds 8y ago
  New to the k8s scene, but does Minikube support multi-node systems?
  - crb 8y ago
    No. We discuss this exact topic with Minikube author Dan Lorenc here: https://kubernetespodcast.com/episode/039-minikube/ https://kubernetespodcast.com/episode/039-minikube/
    - tripue 8y ago
      I enjoy listening to your podcast every week ! Thank for your work What a small world here on hn
    - shitloadofbooks 8y ago
      Your podcast is great! You two make a fairly dry topic engaging and I look forward to every episode.
  - tripue 8y ago
    An alternative that will soon support multi master is rancher/k3s if you want a lightweight k8
joseph 8y ago
My project keights[1] can build a cheap two node cluster in AWS, but is not limited to small clusters. Though, it does spin up an ELB unlike this one. 1. https://github.com/cloudboss/keights https://github.com/cloudboss/keights

180 points by unkoman 8y ago

crb 8y ago

> If anyone has any suggestions on a better way of doing this without shelling out $20 a month for an ELB, please open an Issue! kubehost was designed for this purpose on GKE: https://github.com/GoogleContainerTools/kubehost https://github.com/GoogleContainerTools/kubehost

acd 8y ago
You could use route53 health checks and do dns failover https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover-configuring.html https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dn...

zegl 8y ago

Can it be considered cheap if the cluster is also small? The overhead of the control plane+networking can be quite big, and can easily use more resources than what a m1.small has to offer.

pstadler 8y ago

I'm running a three node cluster on Hetzner Cloud for less than $10 a month. Comprehensive guide and automated provisioning available here: https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide

hardwaresofton 8y ago
+1 for Hetzner, it's amazing -- I'm particularly fond of their dedicated server offerings. One of the things about Kubernetes I like most is the likelihood that it reduces the barrier to entry for "PaaS land" (and resultingly "cloud land") for providers like Hetzner (see OVH's recently announced k8s offering[0]). Once these "baremetal" providers get in the managed (by k8s) game, I'm certain companies will start to spring up and offer/chip away at cloud provider offerings like S3/RDS/etc -- they'll just connect to your k8s infrastructure, and bring their kubernetes-compatible know-how. This is going to cause prices to plummet, as it will make the tiers of value added services more distinct -- i.e. "a platform to run things" on vs. "really good managed databases". [0]: https://www.ovh.co.uk/kubernetes/ https://www.ovh.co.uk/kubernetes/
- stingraycharles 8y ago
  What’s the network of Hetzner like? I like their offering, especially the AMD Epyc servers seem to give a great bang for the buck, but I am a bit put off by their low 99.9% network uptime guarantee. https://www.hetzner.com/rechtliches/agb https://www.hetzner.com/rechtliches/agb
  - dx034 8y ago
    I'm using both their dedicated servers and the cloud offering and never had any issues. Speed and latency are consistent both within Europe as well as to the US (not monitoring much traffic to Asia). Latency from their DCs in Germany to Frankfurt is around 3ms, so comparable to the central Europe offerings of most cloud providers.
    - stingraycharles 8y ago
      So their network hasn’t had any downtime? Any idea why they would only guarantee 99.9% uptime on their network?
      - simplyinfinity 8y ago
        I've been a hetzner customer for the past 5.5 years now, so far I've received 21 emails from their status reports as follows: 10 from failures (2 or 3 of which directly affected me), 10 emails stating when planned maintenance of certain resources will occur 1 email to let me know of the spectre and meltdown vulnerabilities. So far i'm pretty happy customer, and sending business their way and migrating client's sites to their cloud offerings.
      - Demiurge 8y ago
        In the past two years, I remember they have done scheduled infrastructure maintenance twice and unscheduled interruption for a few hours. So, no, it's not 100%. But, AWS has also had some interruptions. From personal experience, this Hetzner downtime has simply been negligible enough. Bigger issue have been replacing commodity memory a few times, which could have been avoided by paying a few more dollars for more quality.
jimmy_ruska 8y ago
Hetzner lists VMs that are ultra cheap but then has separate listing for more expensive VMs with dedicated cores. Alibaba does the same. With these shared core machines, I wonder what the performance reliability guarantees available and how it compares to the dedicated core machines.
- grogenaut 8y ago
  Depending on your data I'd also wonder about the security restrictions between tenants. It's a thing I consider on AWS as well depending on what I'm doing.
  - dx034 8y ago
    Do shared cores add any attack vectors other shared machines (with dedicated cores) don't have?
    - grogenaut 8y ago
      There are always risks with shared harware, known and unknown veunerabilities in the hypervisor or hardware. All the recent intel stuff, row hammer, etc. The interesting question comes when people start implementing hardware hypervisors and what is the risk profile there. Security, at the end of the day, isn't about what is secure and what isn't. If you want to be secure, don't get on the internet. Everything else is a exercise in risk tradeoffs and mitigation. If I was doing anything with PII, cc#'s or any other data I never want to touch I wouldn't use shared hardware without hard thought on it.
      - londons_explore 8y ago
        Shared hardware from the big cloud players adds attack vectors, but it also comes with some of the best security minds trying to keep the entire platform secure. For example, they'll typically be on secret mailing lists and aware of security vulnerabilities weeks before you know about them.
    - 8y ago
kaivi 8y ago
Indeed, cloud+dedicated infrastructure is really cheap after certain load/volume, compared to pure GCE/AWS. I have been using k8s with Hetzner for 6 months now, with dirt-cheap SSD/NVMe storage and 1080 GPUs. Can't recommend them enough, and I do not really see any competition here.
forgot-my-pw 8y ago
I like Hetzner's offerings. Wish they're also in North America and Asia.

moondev 8y ago

You can use haproxy as a tcp load balancer for the apiserver instead of an elb

manishsharan 8y ago
could you please comment on how you use Haproxy servers across multiple AWS availability zones ? how do you configre and manage dns for your haproxy etc ? My reason for asking is that I am super cheap and I want to avoid AWS ELB charges.
- yebyen 8y ago
  Put your Haproxy (or nginx-ingress, or whatever) into HostPort mode, or enable hostNetwork. I have a comment somewhere in my history that explains in a bit more detail: https://news.ycombinator.com/item?id=18660503 https://news.ycombinator.com/item?id=18660503 that's it... hostPort mode, hostNetwork, and enable in DaemonSet mode. Now all of your nodes are load balancers for ingress, and you don't need any ELBs. This is not a recommended configuration because something has to point DNS at your nodes, and the nodes are really not designed to be permanent. If you autoscale, or scale your cluster manually, your DNS needs to be updated to keep up with that. You may be able to find a way to automate that, but DNS has limitations related to TTL, such that if you are doing this too frequently, visitors to your cluster are likely to experience issues. But if your nodes never come and go, this is a pretty good way to run a cluster and keep it on the cheap. If the traffic you want to balance is not HTTP then ingress won't help you (for now?), but the configuration for HAproxy will be similar.

lifeisstillgood 8y ago

I love the attitude here - because in my privileged western world worrying if my monthly hosting costs are one latte and crossiant or two seems quaint. But even in the "rich" EU Avergae monthly wages can be only 1,000 USD, making this a tenth of a day's work, and we don't have to go much further afield to see 6 bucks becoming a significant chunk of a workers day. So thank you, some kids somewhere will be able to afford to develop skills because you started penny pinching. cheers

raehik 8y ago
It certainly makes getting into this field open to students like me. Very glad for that
skybrian 8y ago
Even for people who can afford it, this makes a difference for preserving low-traffic hobby sites you only mildly care about. I have a couple of silly sites that I've run on App Engine for many years, using its generous free tier. If they were on Digital Ocean I'd have shut them down by now, as I eventually did for a Go search engine that I wasn't using much.

fridgamarator 8y ago

Google Cloud and Digital Ocean both offer managed kubernetes clusters, you only pay for the workers.

bryanlarsen 8y ago
Not only that, Digital Ocean's fully managed cluster starts at $10.
- k__ 8y ago
  AWS Fargate starts at $0 If it's only a hobby project with a small cluster, the Fargate costs could very well be under $10.
  - yebyen 8y ago
    That's not really an apples-to-apples comparison though, is it? I mean yeah, a cluster that you never power up costs nothing, that's what I would have hoped. https://aws.amazon.com/fargate/pricing/ https://aws.amazon.com/fargate/pricing/ Assume that you are actually _doing something_ with the cluster, then there will be charges for vCPU and memory. The pricing on this page indicates that if you only need your containers for 10 minutes a day, every day, then you will pay $1.23 for the month at 2GB/1vCPU. That's pretty modest usage at a pretty modest cost. Sure, you can do it, but... you may need to re-architect your product to take advantage of transient workers. (Now I think we all should do that, but that's another conversation...) Compared to DigitalOcean's $10/mo (single-worker, managed) cluster, which provides one full-time node with the same specs, that you can slice-and-dice to run as many tasks as you can fit in 2GB of memory and a single vCPU. Now it becomes clearer that Fargate is priced at a premium. If I'm doing the math right, you'll pay $177.12 for that same task space with your Fargate cluster. If your hobby project needs to run for more than a couple of hours a month, you will usually pay a lot more with Fargate.
    - k__ 8y ago
      I had the impression that the $1.23 are for 10 workers?

triplewipeass 8y ago

How's this better than kops? https://github.com/kubernetes/kops https://github.com/kubernetes/kops

jordanbeiber 8y ago
Kubeadm gives you a ”core” or bare bones setup which makes it much more flexible in terms of addons and versions etc. It’s a bit more work though, which is the trade-off, but since the last few versions kubeadm makes it really easy to spin up clusters. Personally I opt for complete flexibility.
- raesene9 8y ago
  Also Kubeadm has a pretty sane set of default security settings, which some other k8s distributions do not. Specifically kops (by default) does not enable authentication on the kubelet, meaning any attacker who gets access to one container in your cluster is very likely to be able to compromise the whole thing.

013a 8y ago

You can totally install Minikube on AWS [1], which removes the need for a dedicated master and drop the price to that of a single instance. Not sure why you'd ever want to do this, given that GKE or DO will always be cheaper and AWS's core services aren't all that special, but as a thought experiment it's interesting. [1] https://www.radishlogic.com/kubernetes/running-minikube-in-aws-ec2-ubuntu/ https://www.radishlogic.com/kubernetes/running-minikube-in-a...

zwerdlds 8y ago
New to the k8s scene, but does Minikube support multi-node systems?
- crb 8y ago
  No. We discuss this exact topic with Minikube author Dan Lorenc here: https://kubernetespodcast.com/episode/039-minikube/ https://kubernetespodcast.com/episode/039-minikube/
  - tripue 8y ago
    I enjoy listening to your podcast every week ! Thank for your work What a small world here on hn
  - shitloadofbooks 8y ago
    Your podcast is great! You two make a fairly dry topic engaging and I look forward to every episode.
- tripue 8y ago
  An alternative that will soon support multi master is rancher/k3s if you want a lightweight k8

joseph 8y ago

My project keights[1] can build a cheap two node cluster in AWS, but is not limited to small clusters. Though, it does spin up an ELB unlike this one. 1. https://github.com/cloudboss/keights https://github.com/cloudboss/keights