8 ms·

Cuber: Deploy your apps on Kubernetes easily

89 points by mariuz 3y ago

harpratap 3y ago
> We had 100% uptime and we saved 80% on cloud costs. Is there a source on such bold claims?
- Gys 3y ago
  > We have built Cuber for scaling Pushpad. Cuber has been used in production for over a year and it is stable and reliable. We had 100% uptime and we saved 80% on cloud costs. Pretty sure the 100% and 80% refer to using Cuber for Pushpad. So its probably more about using Kubernetes than about Cuber.
  - Terretta 3y ago
    That uptime is dropping: URL to pushpad.xyz in README doesn't work (as of this comment).
- dijit 3y ago
  to be fair cloud is (on the whole) about 5x the price compared to bare metal. Depending on several factors which make it really hard to compare (apples to oranges and all that). For my use-case, which does not require HA, is relatively B/W intensive etc; K8S on bare metal can be 11x cheaper (with the same amount of hands-off, click to deploy, no messing around) as GCP. So, fundamentally, there can be enormous gains w.r.t price vs cloud, 80% cost reduction leaves an infinitesimally small margin though; if they really are providing all the same benefits as a public cloud.
  - harpratap 3y ago
    This is not a bare-metal vs Cloud comparison, the Cuber PaaS is supposed to run on anything including cloud based on their docs. So when they say "save 80% on cloud costs" I am assuming they mean for eg. save 80% on using Cuber on GCP compared to deploying same workload on GKE. It would be very absurd to claim something like - I was using i9-13900KS but realized I could run the same workload on my raspberry pi, but hey I also used this packaging tool in the process therefore I saved 80% costs because of the packaging tool.
thelastknowngod 3y ago
Probably ok for very small startups but it should be a priority to remove this for the first SRE hire.
- mst 3y ago
  Why? If the generated configurations are suitable and the SRE hire is (as they should be) able to trace through how they're generated and audit the results, I don't see why said SRE would want to remove it unless/until it actively gets in the way. When I first start working on pre-existing infrastructure, finding out what my users' current pain points are and figuring out how to work on them is always the priority, and removing a tool that's currently working, probably creating additional hopefully-transitory pain points in the process, seems like the opposite of a good place to start.
- deleted 3y ago
  [deleted]
robertlagrant 3y ago
> Kubernetes is up to 80% cheaper compared to PaaS like Heroku and you can choose between different cloud providers (no lock-in). I'm not a fan of these business case buzzword sentences. There is lock-in, although less, and "up to" is meaningless. It could be 81% cheaper, or it could be more expensive.
- emanuele232 3y ago
  "up to 80%". isn't it to be read as "from the same price to a 80% discount" ?
  - quickthrower2 3y ago
    Which is meaningless, like a $20,000 - $100,000 salary range.
    - nebster 3y ago
      Pretty sure he is disputing that it could be 81% cheaper that the original comment said. "Up to 80% cheaper" means 80% or less, not 80% or more.
- Chris2048 3y ago
  > It could be 81% cheaper I think "up to 80% cheaper" implies "No more than 80% cheaper".
  - robertlagrant 3y ago
    Yes I know - I'm saying it could be even cheaper than that, or it could get more expensive to move to Kubernetes. A percentage like this isn't useful.
- deleted 3y ago
  [deleted]
TekMol 3y ago
What does "Deploy on Kubernetes" mean? I can deploy an application on a machine running a specific Debian version. Go to some provider of bare metal or VMs, they usually have a "Get a Debian version soandso machine" button. Then I copy some files to a specific directory, then I run a setup.sh script that turns this specific version of Debian into a server running my application. But how do I deploy "on Kubernetes"? Where do I get a button "Rent a Kubernetes"?
- cassianoleal 3y ago
  > What does "Deploy on Kubernetes" mean? What kind of a question is this? It means to deploy "on Kubernetes". It doesn't mean to run a script on a server. It means you need a Kubernetes cluster where you have deploy permissions. Then you can deploy on Kubernetes. > Where do I get a button "Rent a Kubernetes"? https://cloud.google.com/kubernetes-engine/ https://cloud.google.com/kubernetes-engine/ https://www.digitalocean.com/products/kubernetes https://www.digitalocean.com/products/kubernetes https://azure.microsoft.com/en-us/products/kubernetes-service/ https://azure.microsoft.com/en-us/products/kubernetes-servic... https://aws.amazon.com/eks/ https://aws.amazon.com/eks/ (In no particular order)
- dividedbyzero 3y ago
  > What does "Deploy on Kubernetes" mean? Kubernetes is an orchestrator that you control via an API, there is no file system to put things on and you don't run anything directly. You tell Kubernetes where to get your software from (as a container image) and how it should run it, and Kubernetes will then make sure the specified number of instances of your software is running with the settings you specify, on a suitable machine, with any storage volumes and other resources you request, or tell you it can't. > But how do I deploy "on Kubernetes"? Depends on the way it's set up. With vanilla Kubernetes you put container images in a registry that Kubernetes can access, write a manifest YAML and send that to the Kubernetes API (e.g. using the kubectl utility). > Where do I get a button "Rent a Kubernetes"? GCP, AWS, Azure, DigitalOcean, ... though they're labelled slightly differently.
- crabbone 3y ago
  Well, you could think about "deploying on Debian" as a combination of these things (but it's not limited to these things): * Comply with general Linux rules for creating applications, i.e. either provide a file containing ELF executable with the appropriate metadata s.a. execution bit set etc. or similarly, a script file with a properly formatted shebang line. * Comply with Debian package manager rules, i.e. create a properly formatted DEB file that can, in principle, be installed by "deb" package manager (i.e. doesn't create dependency problems, pre/post-install scripts execute etc.) It's similar for Kubernetes deployment: except on top of creating a DEB package, you now also need to provide a "chart", that is a document (or a collection of documents) that describe in a formal way how the application is to be installed. In a way very similar to how you can side-step creation of DEB package by, say, writing a Makefile and using that to install, you could also side-step creating a chart by, eg. writing a script that creates all the necessary Kubernetes entities. It's also not necessary that you create a chart in addition to DEB: it's possible that you use a Makefile-like approach for the part that would be better managed by DEB package, but still provide a chart for Kubernetes (it also has a package manager, however poorly designed and implemented: Helm). --- In a way similar to how you could rent a bare-metal box or a VM, you can rent a Kubernetes cluster. Prices very a lot, and similar to the VM prices will greatly depend on the components you choose to put in your cluster. This being said, running applications on Kubernetes is anything but cheap: Kubernetes adds a significant management layer overhead, but that pails on comparison to the typical way to deploy applications where there's rarely any form of cohesion between components (i.e. Linux container images that support individual components rarely have much of an overlap that can be automatically deduplicated, which means the application deployment ends up having tens or hundreds of copies of the same popular libraries, like libc or pthreads etc.) But it doesn't end there: Etcd, the Kuberenetes' database backend is hugely overloaded by all kinds of traffic sent to it. Most of that traffic is of no interest to the operator. So, your network also takes a noticeable performance hit from running Kubernetes in it (a typical solution is to dedicate an entirely separate physical network for Kubernetes' "control plane"). Of course, it's possible to come up with an equally bad if not worse solution in-house, if you want the same features provided by Kubernetes, but it's also possible and relatively easy to come up with a solution that will be less resource-hungry, while performing the same function. Bottom line: claims about 80% cheaper are pulled out of... well, we'll not discuss where they pulled them out of.
benterix 3y ago
> Kubernetes is up to 80% cheaper compared to PaaS like Heroku What do the authors ever mean? How so? The smallest k8s cluster that makes sense has 3 nodes. This means you need to run at least 3 VMs and pay for everything else you need (storage, secrets, egress). And on some cloud providers like AWS, you pay an extra hourly fee just for the privilege of using k8s (that amounts to around $70 last time I checked). So the above statement makes no any sense.
- danpalmer 3y ago
  I would guess "up to" means once these costs you mention are amortized. Sure k8s isn't going to beat a $10 dyno or however much they cost now, but I bet a cluster of 10 large VMs each running 50 containers blows the equivalent Heroku setup out of the water on costs. K8s basically gets cheaper the more you run (as cloud costs, setup costs, engineering, bin-packing, volume discounts stack up), whereas Heroku basically scales linearly.
- jiggawatts 3y ago
  For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Monoliths get too huge and unwieldy if that many developers need to push changes on different release cadences. The key thing is that a container needs about 10-20% of the memory of a virtual machine. Generally, a large cluster allows for very efficient "bin packing" so that workloads with different requirements all get distributed so that every VM is loaded to about 80-90%, whereas you'd be lucky to have 10% across your fleet of VMs. (Don't forget to include all the non-prod instances!) At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. However, it does scale down to a single node. E.g.: Azure Kubernetes Service (AKS) will cheerfully run with just one Linux box and has no extra hidden costs.
  - dividedbyzero 3y ago
    > For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more Is there a source for this statement?
sandGorgon 3y ago
genuine question - why this over ArgoCD ? which is a huge ecosystem and all the declarative setups. Plus has clean github integration for incremental deploys, etc
- SOLAR_FIELDS 3y ago
  Argo is kind of antithetical to what this project is trying to solve. This project wants to be a Dockerfile like approach to Kubernetes. It focuses on simplicity. ArgoCD is decidedly not that, even if longer term something like that is the more correct way to manage production deployments. I could see this project being picked up by some dev who just wants it on the cluster then an SRE ports it to something like Argo. Ideally the dev should learn Argo in the first place but a world where every dev understands what Argo is doing feels pretty unrealistic. Side note (kind of off topic but I’ve been thinking about Argo a lot lately) I’ve been a bit more bearish on Argo for non production environments because it encourages pets and not cattle. I think nowadays the correct way to do environments is a gitops production cluster and dynamic pushed based ephemeral environments that are spawned from that production configuration and templates accordingly. Actually maybe it isn’t off topic: that’s what I’d like to see to offer to developers: some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
  - sandGorgon 3y ago
    that's a fair point, but Argocd + Packer is just that right ? >some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo Totally with you there, but you can build this on top of Argo by having your opinionated manifest language (like Packer or Cuberfile or something else). But there is no joy to maintaining the Kubernetes api interfaces.
    - SOLAR_FIELDS 3y ago
      you could achieve this with packer and some custom tooling yeah. But just saying “and some custom tooling” is like saying “you could do what Dropbox does with a custom rsync script”. There’s no off the shelf tooling to do it. One does not simply “build this on top of Argo by having your opinionated manifest language”. There’s probably 10 tools I could customize to do this, packer being one of them, but no real patterns I’m aware of where people are actually doing it with argocd or fluxcd config as a base. Usually you’re either all in or all out on gitops, so I would be interested in seeing blog posts or libraries of people who have done hybrid approaches (push and pull deployments) and the trade offs they had to make. There is a decent amount of decisions that have to be made to build an abstraction like that, so anyone who has done it before I’m happy to stand on the shoulders of. It’s been on my list of unsolved devops/infra problems to unwrap for a couple of years now In the end there are really three steps that need to happen for true “copy prod” ephemeral environments. You need to provision some stack of infra and then deploy some stuff on top of it and then pull some arbitrary, extremely customizable state into that deployment. I’ve been quite interested in Layerform recently for the first part of that, I think they are getting close to the right abstraction on the infra side. But then you still have to deploy (Argo, helm, flux, something) into that infra stack. Then you have to get the state into that deployment somehow. there is no answer for the latter two from Layerform (terraform is not the right tool for deploys or state backups once you do anything beyond a hello world app). Getting the state in has its own whole Pandora’s box (can I copy customer data from prod into another environment? How do I do so on a way that doesn’t violate SOC2/hipaa/whatever? How will my system’s data and configurations affect the third party systems it connects to?) that I think is simply just not generalizable. But the first two steps should be able to be generalized, and the last step is something that tools can be made available to make it easier with the knowledge that it’s basically business logic at that point
lakomen 3y ago
k8s is, and I wrote it countless times, a huge waste of resources. The whole event driven distributed design is. There aren't many use cases that require more than 5k concurrent clients and the number is even higher without the HTTP overhead (a single bare metal server can handle with the proper language). For most companies it's a waste of money. k8s is hype tech designed to drive business to public cloud providers, because of its complexity and high hardware demands. 9 bare metal servers minimum. And then you have those pods, which are collections of services (e.g. postgres + api server + some cache). But in the case of postgres, why wouldn't you have a dedicated cluster instead of multiple, isolated instances as it is the case with k8s? I know, now the EDA faction chimes in with their dogma. Again most cases don't require even 5k concurrent clients. And most projects never will grow to the point of that many concurrent clients. Addendum: And then I read the who is hiring thread and it's all k8s k8s k8s... frustrating
- achempion 3y ago
  k8s is a complex solution for complex problems. You don't need k8s to host landing or personal blog type of service. Once you move to a type of project with complex demands around deployments, log management, failover management, secrets, environments, resource isolation, etc., then building something to solve this set of problems over and over again is very expensive when off-the-shelf battle-tested solution is available.
  - sureglymop 3y ago
    What about using docker swarm or even a single node docker swarm cluster? I think that could be a viable alternative for small stuff.
    - SirensOfTitan 3y ago
      The thing is, k8s has a lot of mindshare. It's entirely easier to go and find some pertinent issue with your k8s set-up through a simple search, whereas e.g. nomad and swarm have much less mindshare. You don't have to go full complexity with k8s. You can just use a managed k8s provider and ship a simple deployment with an anti-affinity policy to distribute across VMs on creation. Sure, there are other solutions [and this one is likely to not be the cheapest], but they're all vendor locked--k8s is about as close to a cloud agnostic compute as we're likely to get.
anilakar 3y ago
This kind of defeats the purpose of k8s which is to run arbitrary software on a flexible stack. If you need a separate packaging tool you might as well compile a native package.
moondev 3y ago
> First you need to install the prerequisites: ruby, git, docker, pack, then gem install cuber, then create a Cuberfile Seems more complex than a yaml manifest - Is there a Kubernetes operator available for these Cuberfiles
- cacois 3y ago
  Just seems like another abstraction layer to me. With its own DSL, which will have bugs and limitations. And of course buildpacks - yet another abstraction thats simple until you need to build and maintain your own because none fit quite right. I don't know, not sure this solves it.
- tveita 3y ago
  It looks like a fairly thin wrapper around Docker, Heroku buildpacks, and this template file: https://github.com/cuber-cloud/cuber-gem/blob/master/lib/cuber/templates/deployment.yml.erb https://github.com/cuber-cloud/cuber-gem/blob/master/lib/cub...
- deleted 3y ago
  [deleted]
skinnyarms 3y ago
I really like skaffold for stuff like this, and it's a single binary.
- intelVISA 3y ago
  A single bin? Doesn't that spoil the fun of this random amalgam of langs, packages and CVEs? It's like a lucky dip :)
Scene_Cast2 3y ago
How's k3s doing these days? I'm not familiar with the area, but it was marketing itself as a simpler k8s.
- theossuary 3y ago
  K3s is fine, but it was started on an older version of kubernetes, so there's a few places where K8s has solved an issue (like bootstrapping) that k3s had to solve already in a different way. K0s is a new project I like that is similar to k3s in spirit. Microk8s is a snap version from canonical with some neat out of the box stuff (like the simplest GPU support); but again, only available as a snap. Honestly though just use kubespray for production, it's fine
  - gchamonlive 3y ago
    Kubespray is fine if you can spare hours of Dev time. Where I used to work I wasn't involved directly with maintaining the k8s cluster using kubespray, but from the daily meetings it was clear that it is no easy task. Sure it helps with deploying a serviceable production cluster, but when you start throwing at it things like disaster recovery, the automation scripts can grow quite large and complex, even leveraging kubespray. If you can, use a managed service like gks.
    - quickthrower2 3y ago
      I was going to say. Using a managed service is specialized enough. I can't imagine building one up with everything from scratch. Feels like a full time role "Kubernetes Sys Admin"
  - cyberpunk 3y ago
    kubespray is not good advice unless it's changed massively since I last used it in 2020 or so. Imagine a 3 hour Ansible-playbook execution, which randomly failed on different tasks, most of which got fixed by hitting <up><enter> until it applied... We switched to RKE, which runs even on our bigger clusters (nothing mad, maybe 40 nodes) in a few minutes and works perfectly..
WildGreenLeave 3y ago
If I understand correctly, Cuber is meant to fit between "I run a simple Digital Ocean droplet with my code" and "I have people managing my kubernetes cluster and deployments". I am a freelance software developer running sizeable client applications (low millions requests per month, 300 ~ 500 domains, fully custom made) in my own kubernetes cluster along with my own (side)projects so I feel that I fit that bill somewhat. My first impression is that it looks quite awesome! If this is able to remove a big build process of containers then that would be awesome. I don't want to build my docker containers locally, I don't want to 'manage' a docker registry, I don't want to build my containers in my cluster (e.g. Kaniko). However, everything says it will push to the Docker registry and I haven't been able to find any private registry or non-registry option. That would be quite a dealbreaker because I don't want to 'store' my images long term, I will only use them for the few moments where the pods are deployed with a new image and the few days/weeks where they might restart. But retention is not a huge requirement. Just my 2c.
woopwoop24 3y ago
if it works for them, let them have it. i stopped reading the spec, after i have seen that the only label they attach is the lb. anyone who really uses k8s will not at this product longer than it takes, reading the readme or the first bit of code.

89 points by mariuz 3y ago

harpratap 3y ago

> We had 100% uptime and we saved 80% on cloud costs. Is there a source on such bold claims?

Gys 3y ago
> We have built Cuber for scaling Pushpad. Cuber has been used in production for over a year and it is stable and reliable. We had 100% uptime and we saved 80% on cloud costs. Pretty sure the 100% and 80% refer to using Cuber for Pushpad. So its probably more about using Kubernetes than about Cuber.
- Terretta 3y ago
  That uptime is dropping: URL to pushpad.xyz in README doesn't work (as of this comment).
dijit 3y ago
to be fair cloud is (on the whole) about 5x the price compared to bare metal. Depending on several factors which make it really hard to compare (apples to oranges and all that). For my use-case, which does not require HA, is relatively B/W intensive etc; K8S on bare metal can be 11x cheaper (with the same amount of hands-off, click to deploy, no messing around) as GCP. So, fundamentally, there can be enormous gains w.r.t price vs cloud, 80% cost reduction leaves an infinitesimally small margin though; if they really are providing all the same benefits as a public cloud.
- harpratap 3y ago
  This is not a bare-metal vs Cloud comparison, the Cuber PaaS is supposed to run on anything including cloud based on their docs. So when they say "save 80% on cloud costs" I am assuming they mean for eg. save 80% on using Cuber on GCP compared to deploying same workload on GKE. It would be very absurd to claim something like - I was using i9-13900KS but realized I could run the same workload on my raspberry pi, but hey I also used this packaging tool in the process therefore I saved 80% costs because of the packaging tool.

thelastknowngod 3y ago

Probably ok for very small startups but it should be a priority to remove this for the first SRE hire.

mst 3y ago
Why? If the generated configurations are suitable and the SRE hire is (as they should be) able to trace through how they're generated and audit the results, I don't see why said SRE would want to remove it unless/until it actively gets in the way. When I first start working on pre-existing infrastructure, finding out what my users' current pain points are and figuring out how to work on them is always the priority, and removing a tool that's currently working, probably creating additional hopefully-transitory pain points in the process, seems like the opposite of a good place to start.
deleted 3y ago
[deleted]

robertlagrant 3y ago

> Kubernetes is up to 80% cheaper compared to PaaS like Heroku and you can choose between different cloud providers (no lock-in). I'm not a fan of these business case buzzword sentences. There is lock-in, although less, and "up to" is meaningless. It could be 81% cheaper, or it could be more expensive.

emanuele232 3y ago
"up to 80%". isn't it to be read as "from the same price to a 80% discount" ?
- quickthrower2 3y ago
  Which is meaningless, like a $20,000 - $100,000 salary range.
  - nebster 3y ago
    Pretty sure he is disputing that it could be 81% cheaper that the original comment said. "Up to 80% cheaper" means 80% or less, not 80% or more.
Chris2048 3y ago
> It could be 81% cheaper I think "up to 80% cheaper" implies "No more than 80% cheaper".
- robertlagrant 3y ago
  Yes I know - I'm saying it could be even cheaper than that, or it could get more expensive to move to Kubernetes. A percentage like this isn't useful.
deleted 3y ago
[deleted]

TekMol 3y ago

What does "Deploy on Kubernetes" mean? I can deploy an application on a machine running a specific Debian version. Go to some provider of bare metal or VMs, they usually have a "Get a Debian version soandso machine" button. Then I copy some files to a specific directory, then I run a setup.sh script that turns this specific version of Debian into a server running my application. But how do I deploy "on Kubernetes"? Where do I get a button "Rent a Kubernetes"?

cassianoleal 3y ago
> What does "Deploy on Kubernetes" mean? What kind of a question is this? It means to deploy "on Kubernetes". It doesn't mean to run a script on a server. It means you need a Kubernetes cluster where you have deploy permissions. Then you can deploy on Kubernetes. > Where do I get a button "Rent a Kubernetes"? https://cloud.google.com/kubernetes-engine/ https://cloud.google.com/kubernetes-engine/ https://www.digitalocean.com/products/kubernetes https://www.digitalocean.com/products/kubernetes https://azure.microsoft.com/en-us/products/kubernetes-service/ https://azure.microsoft.com/en-us/products/kubernetes-servic... https://aws.amazon.com/eks/ https://aws.amazon.com/eks/ (In no particular order)
dividedbyzero 3y ago
> What does "Deploy on Kubernetes" mean? Kubernetes is an orchestrator that you control via an API, there is no file system to put things on and you don't run anything directly. You tell Kubernetes where to get your software from (as a container image) and how it should run it, and Kubernetes will then make sure the specified number of instances of your software is running with the settings you specify, on a suitable machine, with any storage volumes and other resources you request, or tell you it can't. > But how do I deploy "on Kubernetes"? Depends on the way it's set up. With vanilla Kubernetes you put container images in a registry that Kubernetes can access, write a manifest YAML and send that to the Kubernetes API (e.g. using the kubectl utility). > Where do I get a button "Rent a Kubernetes"? GCP, AWS, Azure, DigitalOcean, ... though they're labelled slightly differently.
crabbone 3y ago
Well, you could think about "deploying on Debian" as a combination of these things (but it's not limited to these things): * Comply with general Linux rules for creating applications, i.e. either provide a file containing ELF executable with the appropriate metadata s.a. execution bit set etc. or similarly, a script file with a properly formatted shebang line. * Comply with Debian package manager rules, i.e. create a properly formatted DEB file that can, in principle, be installed by "deb" package manager (i.e. doesn't create dependency problems, pre/post-install scripts execute etc.) It's similar for Kubernetes deployment: except on top of creating a DEB package, you now also need to provide a "chart", that is a document (or a collection of documents) that describe in a formal way how the application is to be installed. In a way very similar to how you can side-step creation of DEB package by, say, writing a Makefile and using that to install, you could also side-step creating a chart by, eg. writing a script that creates all the necessary Kubernetes entities. It's also not necessary that you create a chart in addition to DEB: it's possible that you use a Makefile-like approach for the part that would be better managed by DEB package, but still provide a chart for Kubernetes (it also has a package manager, however poorly designed and implemented: Helm). --- In a way similar to how you could rent a bare-metal box or a VM, you can rent a Kubernetes cluster. Prices very a lot, and similar to the VM prices will greatly depend on the components you choose to put in your cluster. This being said, running applications on Kubernetes is anything but cheap: Kubernetes adds a significant management layer overhead, but that pails on comparison to the typical way to deploy applications where there's rarely any form of cohesion between components (i.e. Linux container images that support individual components rarely have much of an overlap that can be automatically deduplicated, which means the application deployment ends up having tens or hundreds of copies of the same popular libraries, like libc or pthreads etc.) But it doesn't end there: Etcd, the Kuberenetes' database backend is hugely overloaded by all kinds of traffic sent to it. Most of that traffic is of no interest to the operator. So, your network also takes a noticeable performance hit from running Kubernetes in it (a typical solution is to dedicate an entirely separate physical network for Kubernetes' "control plane"). Of course, it's possible to come up with an equally bad if not worse solution in-house, if you want the same features provided by Kubernetes, but it's also possible and relatively easy to come up with a solution that will be less resource-hungry, while performing the same function. Bottom line: claims about 80% cheaper are pulled out of... well, we'll not discuss where they pulled them out of.

benterix 3y ago

> Kubernetes is up to 80% cheaper compared to PaaS like Heroku What do the authors ever mean? How so? The smallest k8s cluster that makes sense has 3 nodes. This means you need to run at least 3 VMs and pay for everything else you need (storage, secrets, egress). And on some cloud providers like AWS, you pay an extra hourly fee just for the privilege of using k8s (that amounts to around $70 last time I checked). So the above statement makes no any sense.

danpalmer 3y ago
I would guess "up to" means once these costs you mention are amortized. Sure k8s isn't going to beat a $10 dyno or however much they cost now, but I bet a cluster of 10 large VMs each running 50 containers blows the equivalent Heroku setup out of the water on costs. K8s basically gets cheaper the more you run (as cloud costs, setup costs, engineering, bin-packing, volume discounts stack up), whereas Heroku basically scales linearly.
jiggawatts 3y ago
For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Monoliths get too huge and unwieldy if that many developers need to push changes on different release cadences. The key thing is that a container needs about 10-20% of the memory of a virtual machine. Generally, a large cluster allows for very efficient "bin packing" so that workloads with different requirements all get distributed so that every VM is loaded to about 80-90%, whereas you'd be lucky to have 10% across your fleet of VMs. (Don't forget to include all the non-prod instances!) At the tiniest scales, Kubernetes stops making sense because it's too complex to manage. However, it does scale down to a single node. E.g.: Azure Kubernetes Service (AKS) will cheerfully run with just one Linux box and has no extra hidden costs.
- dividedbyzero 3y ago
  > For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more Is there a source for this statement?

sandGorgon 3y ago

genuine question - why this over ArgoCD ? which is a huge ecosystem and all the declarative setups. Plus has clean github integration for incremental deploys, etc

SOLAR_FIELDS 3y ago
Argo is kind of antithetical to what this project is trying to solve. This project wants to be a Dockerfile like approach to Kubernetes. It focuses on simplicity. ArgoCD is decidedly not that, even if longer term something like that is the more correct way to manage production deployments. I could see this project being picked up by some dev who just wants it on the cluster then an SRE ports it to something like Argo. Ideally the dev should learn Argo in the first place but a world where every dev understands what Argo is doing feels pretty unrealistic. Side note (kind of off topic but I’ve been thinking about Argo a lot lately) I’ve been a bit more bearish on Argo for non production environments because it encourages pets and not cattle. I think nowadays the correct way to do environments is a gitops production cluster and dynamic pushed based ephemeral environments that are spawned from that production configuration and templates accordingly. Actually maybe it isn’t off topic: that’s what I’d like to see to offer to developers: some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
- sandGorgon 3y ago
  that's a fair point, but Argocd + Packer is just that right ? >some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo Totally with you there, but you can build this on top of Argo by having your opinionated manifest language (like Packer or Cuberfile or something else). But there is no joy to maintaining the Kubernetes api interfaces.
  - SOLAR_FIELDS 3y ago
    you could achieve this with packer and some custom tooling yeah. But just saying “and some custom tooling” is like saying “you could do what Dropbox does with a custom rsync script”. There’s no off the shelf tooling to do it. One does not simply “build this on top of Argo by having your opinionated manifest language”. There’s probably 10 tools I could customize to do this, packer being one of them, but no real patterns I’m aware of where people are actually doing it with argocd or fluxcd config as a base. Usually you’re either all in or all out on gitops, so I would be interested in seeing blog posts or libraries of people who have done hybrid approaches (push and pull deployments) and the trade offs they had to make. There is a decent amount of decisions that have to be made to build an abstraction like that, so anyone who has done it before I’m happy to stand on the shoulders of. It’s been on my list of unsolved devops/infra problems to unwrap for a couple of years now In the end there are really three steps that need to happen for true “copy prod” ephemeral environments. You need to provision some stack of infra and then deploy some stuff on top of it and then pull some arbitrary, extremely customizable state into that deployment. I’ve been quite interested in Layerform recently for the first part of that, I think they are getting close to the right abstraction on the infra side. But then you still have to deploy (Argo, helm, flux, something) into that infra stack. Then you have to get the state into that deployment somehow. there is no answer for the latter two from Layerform (terraform is not the right tool for deploys or state backups once you do anything beyond a hello world app). Getting the state in has its own whole Pandora’s box (can I copy customer data from prod into another environment? How do I do so on a way that doesn’t violate SOC2/hipaa/whatever? How will my system’s data and configurations affect the third party systems it connects to?) that I think is simply just not generalizable. But the first two steps should be able to be generalized, and the last step is something that tools can be made available to make it easier with the knowledge that it’s basically business logic at that point

lakomen 3y ago

k8s is, and I wrote it countless times, a huge waste of resources. The whole event driven distributed design is. There aren't many use cases that require more than 5k concurrent clients and the number is even higher without the HTTP overhead (a single bare metal server can handle with the proper language). For most companies it's a waste of money. k8s is hype tech designed to drive business to public cloud providers, because of its complexity and high hardware demands. 9 bare metal servers minimum. And then you have those pods, which are collections of services (e.g. postgres + api server + some cache). But in the case of postgres, why wouldn't you have a dedicated cluster instead of multiple, isolated instances as it is the case with k8s? I know, now the EDA faction chimes in with their dogma. Again most cases don't require even 5k concurrent clients. And most projects never will grow to the point of that many concurrent clients. Addendum: And then I read the who is hiring thread and it's all k8s k8s k8s... frustrating

achempion 3y ago
k8s is a complex solution for complex problems. You don't need k8s to host landing or personal blog type of service. Once you move to a type of project with complex demands around deployments, log management, failover management, secrets, environments, resource isolation, etc., then building something to solve this set of problems over and over again is very expensive when off-the-shelf battle-tested solution is available.
- sureglymop 3y ago
  What about using docker swarm or even a single node docker swarm cluster? I think that could be a viable alternative for small stuff.
  - SirensOfTitan 3y ago
    The thing is, k8s has a lot of mindshare. It's entirely easier to go and find some pertinent issue with your k8s set-up through a simple search, whereas e.g. nomad and swarm have much less mindshare. You don't have to go full complexity with k8s. You can just use a managed k8s provider and ship a simple deployment with an anti-affinity policy to distribute across VMs on creation. Sure, there are other solutions [and this one is likely to not be the cheapest], but they're all vendor locked--k8s is about as close to a cloud agnostic compute as we're likely to get.

anilakar 3y ago

This kind of defeats the purpose of k8s which is to run arbitrary software on a flexible stack. If you need a separate packaging tool you might as well compile a native package.

moondev 3y ago

> First you need to install the prerequisites: ruby, git, docker, pack, then gem install cuber, then create a Cuberfile Seems more complex than a yaml manifest - Is there a Kubernetes operator available for these Cuberfiles

cacois 3y ago
Just seems like another abstraction layer to me. With its own DSL, which will have bugs and limitations. And of course buildpacks - yet another abstraction thats simple until you need to build and maintain your own because none fit quite right. I don't know, not sure this solves it.
tveita 3y ago
It looks like a fairly thin wrapper around Docker, Heroku buildpacks, and this template file: https://github.com/cuber-cloud/cuber-gem/blob/master/lib/cuber/templates/deployment.yml.erb https://github.com/cuber-cloud/cuber-gem/blob/master/lib/cub...
deleted 3y ago
[deleted]

skinnyarms 3y ago

I really like skaffold for stuff like this, and it's a single binary.

intelVISA 3y ago
A single bin? Doesn't that spoil the fun of this random amalgam of langs, packages and CVEs? It's like a lucky dip :)

Scene_Cast2 3y ago

How's k3s doing these days? I'm not familiar with the area, but it was marketing itself as a simpler k8s.

theossuary 3y ago
K3s is fine, but it was started on an older version of kubernetes, so there's a few places where K8s has solved an issue (like bootstrapping) that k3s had to solve already in a different way. K0s is a new project I like that is similar to k3s in spirit. Microk8s is a snap version from canonical with some neat out of the box stuff (like the simplest GPU support); but again, only available as a snap. Honestly though just use kubespray for production, it's fine
- gchamonlive 3y ago
  Kubespray is fine if you can spare hours of Dev time. Where I used to work I wasn't involved directly with maintaining the k8s cluster using kubespray, but from the daily meetings it was clear that it is no easy task. Sure it helps with deploying a serviceable production cluster, but when you start throwing at it things like disaster recovery, the automation scripts can grow quite large and complex, even leveraging kubespray. If you can, use a managed service like gks.
  - quickthrower2 3y ago
    I was going to say. Using a managed service is specialized enough. I can't imagine building one up with everything from scratch. Feels like a full time role "Kubernetes Sys Admin"
- cyberpunk 3y ago
  kubespray is not good advice unless it's changed massively since I last used it in 2020 or so. Imagine a 3 hour Ansible-playbook execution, which randomly failed on different tasks, most of which got fixed by hitting <up><enter> until it applied... We switched to RKE, which runs even on our bigger clusters (nothing mad, maybe 40 nodes) in a few minutes and works perfectly..

WildGreenLeave 3y ago

If I understand correctly, Cuber is meant to fit between "I run a simple Digital Ocean droplet with my code" and "I have people managing my kubernetes cluster and deployments". I am a freelance software developer running sizeable client applications (low millions requests per month, 300 ~ 500 domains, fully custom made) in my own kubernetes cluster along with my own (side)projects so I feel that I fit that bill somewhat. My first impression is that it looks quite awesome! If this is able to remove a big build process of containers then that would be awesome. I don't want to build my docker containers locally, I don't want to 'manage' a docker registry, I don't want to build my containers in my cluster (e.g. Kaniko). However, everything says it will push to the Docker registry and I haven't been able to find any private registry or non-registry option. That would be quite a dealbreaker because I don't want to 'store' my images long term, I will only use them for the few moments where the pods are deployed with a new image and the few days/weeks where they might restart. But retention is not a huge requirement. Just my 2c.

woopwoop24 3y ago

if it works for them, let them have it. i stopped reading the spec, after i have seen that the only label they attach is the lb. anyone who really uses k8s will not at this product longer than it takes, reading the readme or the first bit of code.