6 ms·

Sucks that there isn’t anything simpler than k8s that’s production grade. Maybe it’s time to yolo with a regular container that just restarts on failures, ha…

datadeft 3y ago
There is AWS Autoscaling, ECS, EKS, Hashicorp Nomad, Fly.io and the few more. However I would ask the question (same question I have been asking since the first appearance of k8s) what do you need so much that it justifies this complexity? Implementing a simple autoscaling system or using the provided solutions like AWS ASG was and is results in more reliable infra. If you want to use cutting edge there Lamda function that most cloud vendors offer. That is also less complex option.
bsdnoob 3y ago
Hashicorp nomad comes to mind.
- jarym 3y ago
  It does and most people that used Nomad were very positive about it. But with the turn that Hashicorp have taken I'm not sure that it now stands much of a chance.
- jdoss 3y ago
  Seconded. Hashicorp Nomad has been a breath of fresh air for doing HA deployments for my workloads. Getting a small cluster setup to self host Nomad is so easier than Kubernetes and defining workloads is much easier to understand too IMO. The only negatives about Nomad is the Hashicorp license drama that has happened recently and persistent storage can be a pain in the ass.
  - hughw 3y ago
    I hope somebody can elaborate on the licensing problem? I see a community edition...
    - gabeio 3y ago
      https://news.ycombinator.com/item?id=37081306 https://news.ycombinator.com/item?id=37081306 They pulled the rug out on a lot of people and people tend to hold grudges. Community edition is peanuts compared to using an open source tool you _could have_ fixed your own bugs with by creating PRs or adding features with PRs. I used to be a huge hashicorp fanboy... _used to_.
      - sofixa 3y ago
        Just to clarify, you can still access the code, submit PRs to add features, etc. The only difference is that during a fixed time period (4 years) you cannot use Nomad to compete with HashiCorp. If you want to do that, you can ask for a license.
    - jdoss 3y ago
      They changed to the BDL from the MPL https://www.hashicorp.com/license-faq https://www.hashicorp.com/license-faq which in FOSS culture is considered a dick move.
- dwroberts 3y ago
  I love Nomad but having used it in two different roles and now invested time to understand k8s properly, I would absolutely not recommend it. Nomad is simple on the surface and could have been a great tool - but it is basically unusable on its own without tighter integration with Hashi’s own tools (eg Vault). Configuring and maintaining all those things is nontrivial and ends up being more annoying (for a lesser end result) than just using a fully managed kubernetes cluster like GKE
  - robertlagrant 3y ago
    You'd imagine that any scheduler that's well integrated with Vault would have a huge advantage over other ones. Surprising that it's not like that for another product from the same company.
    - throitallaway 3y ago
      Vault is highly integrated with K8s via the vault-agent.
dewey 3y ago
Maybe https://kamal-deploy.org https://kamal-deploy.org could be worth a try there, as 37Signals almost moved all their services from Kubernetes I'd say it can be called production grade.
speedgoose 3y ago
"Don't throw the baby out with the bathwater". It’s just a bug in something most people don’t use.
acedTrex 3y ago
HA k3s?
sofixa 3y ago
Disclaimer: I work at HashiCorp, but have been of this opinion since before joining (source: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...) HashiCorp Nomad is a perfectly easy and flexible orchestrator (not only containers, also VMs, random binaries, Firecracker, WASM, etc.) that is production grade. It has a number of advantages over Kubernetes (being drastically easier to deploy, use and maintain; having integrated native templating with logic; being much nimbler but also being able to scale much more, etc. etc.) and it's a very good choice in many scenarios. Of course there's also disadvantages, most notably ecosystem size, but often that doesn't matter. Kubernetes is great and extremely powerful, but also very complex and straight up overkill for many organisations.
- wg0 3y ago
  I've run both. First k8s is free and nothing beats free but that's not about it. Nomad's mental model is hard to grasp. Or let's say is not as elegant. It's is nowhere near k8s. And k8s at least from a usage standpoint can be explained in under ten minutes. Pod, Deployment, Service, Ingress and you're pretty much done.
- stuff4ben 3y ago
  yeah, but ya'lls licensing tho...
  - orangepurple 3y ago
    https://www.hashicorp.com/products/nomad/pricing https://www.hashicorp.com/products/nomad/pricing There is an unsupported community edition
    - dharmab 3y ago
      In case you haven't heard, the next version of Nomad is moving to a non Open Source license.
  - deleted 3y ago
    [deleted]
nailer 3y ago
AWS ECS
ctvo 3y ago
I use AWS ECS (sometimes with ECS Anywhere to use my own servers + ECS's control plane) for orchestration where I can. It's a little surprising how many folks are unaware of the non-Kubernetes orchestration options. Going full Kubernetes is rarely what organizations need. Getting CI/CD -> containers on servers and being able to configure the resources containers need, having the placement handled, etc. gets most of these folks 90% of what they want with much much less complexity. - AWS ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... - ECS Anywhere: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
- willthames 3y ago
  Does ECS support mounting configuration files (without needing the configuration file to be on the host)? Being able to mount secrets and configmaps into the container file system (without having to modify the container image to provide an entrypoint) definitely seemed to be one major advantage of kubernetes over ECS a few years back.
  - ctvo 3y ago
    In the case of secrets, it can populate secrets automatically from SecretsManager into environment variables. See a CDK example: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs-readme.html#environment-variables https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_... You can setup something to mount configuration files stored in say S3 -> into an EFS volume that you attach to all ECS tasks, for example. The problem with ECS is that everything, including service discovery, is an integration with another AWS service. Making it even more difficult to ever migrate, but it does support much of what folks use Kubernetes for.
- ldjkfkdsjnv 3y ago
  ECS Fargate is a bit priecy, but is also a no brainer. It just works
- tebbers 3y ago
whoknowsidont 3y ago
Erlang/Elixir screaming in the background It's a wonderful, observable, manageable orchestration layer that's easily and safely customizable if you're wise enough to treat it as such.
Spivak 3y ago
There is, you create a load balancer, an autoscaling group, put an ubuntu image in it, set the userdata to install docker and docker-compose up your desired containers. * Instance refresh to deploy new images. * Every so often update your AMI to patch. * Create an RDS database and hook the instances up to that if you need such things. It's really hard to get more boring but works than this. It can be set up in a few hundred lines of terraform. I promise the code for your entire AWS account will be less than just the k8s YAML.
- secondcoming 3y ago
  VM images aren't cool enough! We do this but create custom images that have all their requirements installed via ansible. Unfortunately there's now a push to move to GKE instead of GCE, for no real reason.
- turtlebits 3y ago
  What happens when all your services won't fit on a single box, or you need X more of a specific service?
  - Draiken 3y ago
    You create a new instance and run the same setup. If your workload is extremely variable, something like k8s with auto-scaling makes sense. But a lot of applications (maybe the majority?) have very predictable loads and you don't always need new services. When you're at the size of large corps with hundreds/thousands of services, then k8s is amazing. If you're not even close to that number, it's very likely premature optimization.

datadeft 3y ago

There is AWS Autoscaling, ECS, EKS, Hashicorp Nomad, Fly.io and the few more. However I would ask the question (same question I have been asking since the first appearance of k8s) what do you need so much that it justifies this complexity? Implementing a simple autoscaling system or using the provided solutions like AWS ASG was and is results in more reliable infra. If you want to use cutting edge there Lamda function that most cloud vendors offer. That is also less complex option.

bsdnoob 3y ago

Hashicorp nomad comes to mind.

jarym 3y ago
It does and most people that used Nomad were very positive about it. But with the turn that Hashicorp have taken I'm not sure that it now stands much of a chance.
jdoss 3y ago
Seconded. Hashicorp Nomad has been a breath of fresh air for doing HA deployments for my workloads. Getting a small cluster setup to self host Nomad is so easier than Kubernetes and defining workloads is much easier to understand too IMO. The only negatives about Nomad is the Hashicorp license drama that has happened recently and persistent storage can be a pain in the ass.
- hughw 3y ago
  I hope somebody can elaborate on the licensing problem? I see a community edition...
  - gabeio 3y ago
    https://news.ycombinator.com/item?id=37081306 https://news.ycombinator.com/item?id=37081306 They pulled the rug out on a lot of people and people tend to hold grudges. Community edition is peanuts compared to using an open source tool you _could have_ fixed your own bugs with by creating PRs or adding features with PRs. I used to be a huge hashicorp fanboy... _used to_.
    - sofixa 3y ago
      Just to clarify, you can still access the code, submit PRs to add features, etc. The only difference is that during a fixed time period (4 years) you cannot use Nomad to compete with HashiCorp. If you want to do that, you can ask for a license.
  - jdoss 3y ago
    They changed to the BDL from the MPL https://www.hashicorp.com/license-faq https://www.hashicorp.com/license-faq which in FOSS culture is considered a dick move.
dwroberts 3y ago
I love Nomad but having used it in two different roles and now invested time to understand k8s properly, I would absolutely not recommend it. Nomad is simple on the surface and could have been a great tool - but it is basically unusable on its own without tighter integration with Hashi’s own tools (eg Vault). Configuring and maintaining all those things is nontrivial and ends up being more annoying (for a lesser end result) than just using a fully managed kubernetes cluster like GKE
- robertlagrant 3y ago
  You'd imagine that any scheduler that's well integrated with Vault would have a huge advantage over other ones. Surprising that it's not like that for another product from the same company.
  - throitallaway 3y ago
    Vault is highly integrated with K8s via the vault-agent.

dewey 3y ago

Maybe https://kamal-deploy.org https://kamal-deploy.org could be worth a try there, as 37Signals almost moved all their services from Kubernetes I'd say it can be called production grade.

speedgoose 3y ago

"Don't throw the baby out with the bathwater". It’s just a bug in something most people don’t use.

acedTrex 3y ago

HA k3s?

sofixa 3y ago

Disclaimer: I work at HashiCorp, but have been of this opinion since before joining (source: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...) HashiCorp Nomad is a perfectly easy and flexible orchestrator (not only containers, also VMs, random binaries, Firecracker, WASM, etc.) that is production grade. It has a number of advantages over Kubernetes (being drastically easier to deploy, use and maintain; having integrated native templating with logic; being much nimbler but also being able to scale much more, etc. etc.) and it's a very good choice in many scenarios. Of course there's also disadvantages, most notably ecosystem size, but often that doesn't matter. Kubernetes is great and extremely powerful, but also very complex and straight up overkill for many organisations.

wg0 3y ago
I've run both. First k8s is free and nothing beats free but that's not about it. Nomad's mental model is hard to grasp. Or let's say is not as elegant. It's is nowhere near k8s. And k8s at least from a usage standpoint can be explained in under ten minutes. Pod, Deployment, Service, Ingress and you're pretty much done.
stuff4ben 3y ago
yeah, but ya'lls licensing tho...
- orangepurple 3y ago
  https://www.hashicorp.com/products/nomad/pricing https://www.hashicorp.com/products/nomad/pricing There is an unsupported community edition
  - dharmab 3y ago
    In case you haven't heard, the next version of Nomad is moving to a non Open Source license.
- deleted 3y ago
  [deleted]

nailer 3y ago

AWS ECS

ctvo 3y ago

I use AWS ECS (sometimes with ECS Anywhere to use my own servers + ECS's control plane) for orchestration where I can. It's a little surprising how many folks are unaware of the non-Kubernetes orchestration options. Going full Kubernetes is rarely what organizations need. Getting CI/CD -> containers on servers and being able to configure the resources containers need, having the placement handled, etc. gets most of these folks 90% of what they want with much much less complexity. - AWS ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... - ECS Anywhere: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

willthames 3y ago
Does ECS support mounting configuration files (without needing the configuration file to be on the host)? Being able to mount secrets and configmaps into the container file system (without having to modify the container image to provide an entrypoint) definitely seemed to be one major advantage of kubernetes over ECS a few years back.
- ctvo 3y ago
  In the case of secrets, it can populate secrets automatically from SecretsManager into environment variables. See a CDK example: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs-readme.html#environment-variables https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_... You can setup something to mount configuration files stored in say S3 -> into an EFS volume that you attach to all ECS tasks, for example. The problem with ECS is that everything, including service discovery, is an integration with another AWS service. Making it even more difficult to ever migrate, but it does support much of what folks use Kubernetes for.
ldjkfkdsjnv 3y ago
ECS Fargate is a bit priecy, but is also a no brainer. It just works
tebbers 3y ago

whoknowsidont 3y ago

Erlang/Elixir screaming in the background It's a wonderful, observable, manageable orchestration layer that's easily and safely customizable if you're wise enough to treat it as such.

Spivak 3y ago

There is, you create a load balancer, an autoscaling group, put an ubuntu image in it, set the userdata to install docker and docker-compose up your desired containers. * Instance refresh to deploy new images. * Every so often update your AMI to patch. * Create an RDS database and hook the instances up to that if you need such things. It's really hard to get more boring but works than this. It can be set up in a few hundred lines of terraform. I promise the code for your entire AWS account will be less than just the k8s YAML.

secondcoming 3y ago
VM images aren't cool enough! We do this but create custom images that have all their requirements installed via ansible. Unfortunately there's now a push to move to GKE instead of GCE, for no real reason.
turtlebits 3y ago
What happens when all your services won't fit on a single box, or you need X more of a specific service?
- Draiken 3y ago
  You create a new instance and run the same setup. If your workload is extremely variable, something like k8s with auto-scaling makes sense. But a lot of applications (maybe the majority?) have very predictable loads and you don't always need new services. When you're at the size of large corps with hundreds/thousands of services, then k8s is amazing. If you're not even close to that number, it's very likely premature optimization.