28 ms·

Kubernetes is a red flag signalling premature optimisation

• tenfourty 4y ago
  [OP here] It feels bizarre saying this, having spent so much of my life advocating for and selling a distribution of Kubernetes and consulting services to help folks get the most of out it, but here goes! YOU probably shouldn't use Kubernetes and a bunch of other "cool" things for your product. Most folks building software at startups and scale-ups should avoid Kubernetes and other premature optimisations. If your company uses Kubernetes, you are likely expending energy on something that doesn't take you towards your mission. You have probably fallen into the trap of premature optimisation. Please don't take this post to be only aimed against Kubernetes. It is not. I am directing this post at every possible bit of premature optimisation engineers make in the course of building software.
  • mschuster91 4y ago
    Running a self-hosted Kubernetes is indeed ... questionable, but a managed Kubernetes? That's a pretty sane thing to do IMO. The alternatives are running your Docker containers either manually on some EC2 or other bare-metal server which is a nightmare to do deployments, or using something like Elastic Beanstalk which is even worse. For me at least, Kubernetes has become something like an universal standard: if you're already running Docker or other containerization, "using Kubernetes" is nothing more than a clear, "single source of truth" documentation on what infrastructure your application expects. Certainly beats that Confluence document from 2018, last updated 2019, on how the production server is set up that doesn't even match up with what was reality in 2020 much less today.
    • k8sToGo 4y ago
      A managed cluster is not as much managed as you would think. Still have to configure and install a lot of things yourself.
      • dj_mc_merlin 4y ago
        Of course, managed means you don't take care of say, creating your own X.509 CA and issuing certificates, tending to their expiry, installing Tiller, setting up pod networking etc. etc. All of these much more annoying and harder than `helm update --install`ing some charts to your own cluster.
        • k8sToGo 4y ago
          That is true. But if you compare it to, for example, managed web hosting, then it is a very different managed experience.
        • grumpy-de-sre 4y ago
          To be fair installing tiller hasn't been a thing for years. And pretty much everyone using cert-manager and lets encrypt which makes the whole X.509 story pretty much a no brainer.
  • xyzzy123 4y ago
    Doesn't it depend what staff you have? I would have agreed with this in 2018 but the world has moved on. If you have a bunch of people who know it, you can deploy a cluster into gcloud or aws with a few clicks or lines of IaC. Would I recommend a startup team learn kube while trying to ship a product? No. Would I think it's a red flag if a team who already know it choose it as their preferred platform? Also no.
    • ramraj07 4y ago
      Even if they know it, unless they absolutely need to implement it, why would you waste such valuable resources on of all things infra? (Unless your product IS infra). No engineer I know who’s smart enough to effortlessly deploy k8s on their own would want to do that as the job. There’s a million other interesting things (hopefully?) that they can be doing.
      • xyzzy123 4y ago
        Often to get access to things that integrate well with kubernetes. Personally I don't see too much difference between kube yamls and systemd units and cloudformation or whatever and the "cluster maintenance" itself is not the burden it used to be if you stay on the "paved road" provided by your cloud.
      • dinkleberg 4y ago
        Not at all. If you know Kubernetes well and your needs are fairly simple it takes no time at all. On a project a couple of years ago my cofounder and I opted to use Kubernetes after running into constant frustration after following the advice to keep it as simple as possible and just use VMs. Kubernetes makes many complicated things very easy and lets you stop worrying about your infra (provided you’re using a managed provider). On our next project we used a PaaS offering because we had a bunch of credits for it and it was painful in comparison to what we had with Kubernetes. It was way more expensive (if we didn’t have credits), the deployments were slow, and we had less flexibility. Kubernetes isn’t perfect, far from it. But for those who know it well it is usually a good option.
    • zelphirkalt 4y ago
  • hadlock 4y ago
    You can get away without an orchestrator right up until about when your ARR hits $10mm Hiring people with k8s experience in 2022 is not a difficult task, it's not an obscure technology anymore, and when your initial crop of devops decides to leave, the new guys coming in can scan your deployments and namespaces and pretty much hit the ground running within 48-72 hours. That's a big, important part of running a business. Being able to hire people with the right skill set, and being able to keep the lights on. Bespoke systems built on top of EC2 require time, extra documentation and a steep learning curve, not to mention the fact that the bespoke system probably isn't getting security audits or being built to modern standards or best practices, tooling isn't being kept up to date. I can build a mechanical wristwatch in my garage, but if I had full access to the Rolex factory floor for free, I'd probably take that option.
    • pjmlp 4y ago
      So much for theory. I just came out of a project where Kubernetes performance issues involved wild guess and blind tuning until the so called experts actually found out why the cloud cluster was behaving strangely, including support from Cloud vendor. And good luck making sense of all the YAML spaghetti available for bootstrapping the whole cluster from scratch. 72 hours? They better be 10x DevOps team.
      • secondcoming 4y ago
        This is our world right now. A kubernetes cronjob sometimes fails and we have no idea why.
      • hadlock 4y ago
        If your devops guys are struggling, you are hiring the wrong devops folks, or at the wrong end of the pay band. Most devops guys I know are paid in the same band as a senior developer.
        • pjmlp 4y ago
          Sure, it is like bug free C code. It is only a matter of having the top of the cream. Pity there aren't enough of them in the world, including on cloud vendor support team.
  • subhajeet2107 4y ago
    What about standardisation that comes with using a framework like kubernetes , while not using k8s you end up with adhoc deployment methods, clunky work arounds of handling networking, policies, secrets etc, with Kubernetes or even ECS it signals that the team or the developer is looking to use fixed set of rules for infrastructure, also k8s scales well even for smaller apps
    • adastra22 4y ago
      Seriously, I use k8s for the same reason I use docker: it's a standard language for deployment. Yeah I could do the same stuff manually for a small project.. but why?
      • p_l 4y ago
        Last year I tried to "simplify" by doing things mostly the old way (though we used containers for a bunch of things later on). The experience pushed me to decide "never again" and to plonk k3s or at least something like podman from the start on the server.
• mattbillenstein 4y ago
  In the limit, there are some startups that could run production on a single Linux host - I recently helped one get off Heroku and their spend went from ~$1k/mo to ~$50/mo and it made debugging and figuring out performance issues so much easier than what they were doing previously...
  • pantulis 4y ago
    The amount of traffic you can serve on a bunch of Linode boxes is pretty high. I know this sounds like a boomer yelling at the clouds --see what I did here? Kubernetes is the right solution to a difficult problem you may or may not have in your future.
  • moreira 4y ago
    I'm very much with you on this, but I do understand that it's one of those things that is just not feasible when your team has no sysadmin/devops experience. You were able to do it, but what happens to them when you're not around? Does their team have the required experience to handle it? That's the difference in cost. It's like DYI - yes, if I have all the skills and experience I can do everything myself incredibly cheaply, but... I don't. So I gotta pay. But if your team does have the skills and experience, it's definitely worth looking into. I do think people deeply underestimate what can be achieved with a single (or two, if you need a standby) dedicated Linux server these days. A single server can easily go up to 2TB of RAM and 128+ cores. Long before you ever get to a scale where that's a limitation, you'll have more than enough resources to figure things out.
    • robertlagrant 4y ago
      > I'm very much with you on this, but I do understand that it's one of those things that is just not feasible when your team has no sysadmin/devops experience. Exactly - $950/mo is nowhere near enough to pay for those skills if you don't have them. It's good value for money.
    • p_l 4y ago
      Funnily enough, small amount of servers that you want to utilize as much as possible is pretty much one of the original use cases for kubernetes. My first deployment involved virtual machines, but we were essentially packing as much as possible into lowest amount of VMs, then doubled it up so that we had failover. This way we had clear visibility of how many resources we were using and could allocate them according to how much we had.
• cube2222 4y ago
  Having worked with Kubernetes, it's great - I think even smaller setups can benefit from it and its approach (especially when using a managed provider). But for startups and/or simpler setups, ECS/Lambda is so much less work, while usually being powerful enough.
  • carom 4y ago
    I've looked at it a few times but haven't been able to get my head around the concept. Reading the intro documentation I haven't been able to map the concept of "nodes" to a server, database, and GPU server.
    • robertlagrant 4y ago
      A node is a server, virtual or physical, which may or may not have a hardware capability such as a GPU.
    • FridgeSeal 4y ago
      The nodes are just the individual machines in your K8s cluster. Which one(s) run your API/server, your database, etc is basically arbitrary. You’re free to let the K8s scheduler do as it see fits, or you can get increasing degrees of control by adjusting things like node and pod affinity/anti-affinity, etc.
    • angio 4y ago
      Kubernetes is fairly simple: * You describe what type of state you need (Deployment = one or more Pods running your container, Service = expose the deployment) * Kubernetes runs a control loop that reconciles the current cluster's state with your desired state Nodes are the virtual machines on which everything runs, you don't need to interact with them unless you have special requirements (e.g. GPU instances). In that case, you annotate GPU instances with some type of label (like `has-gpu=true`), then in your deployment you add a node affinity saying it needs to run on nodes that have that label. Kubernetes will schedule it for you if there's any node matching.
    • mstipetic 4y ago
      I've made a k8s 101 webinar a while ago that I think explains the basic concepts pretty well https://www.youtube.com/watch?v=qQw91VeJMGY https://www.youtube.com/watch?v=qQw91VeJMGY
• ocius 4y ago
  Our startup has a webapp that acts as a UI for a machine learning application. We have two different types of heavy workloads (ML and something else). The workers for these run on Kubernetes, which makes them easy to scale (which we do a lot, automatically). The app itself doesn't run on Kubernetes yet (simple VM). It would be better if it did, though! We keep building a lot of functionality ourselves (e.g. deployment without downtime) that would be easier with K8s. I think what you really want to avoid is Microservices, not K8s. K8s gives you many advantages, such as the entire config in Git, containerized workloads, deploy management, etc., which sooner or later, you'll find yourself wanting. Depending on how you get started, the learning curve can be steep, and some of the tools are not mature but supposedly used by everyone (ArgoCD for deployment?).
  • WatchDog 4y ago
    What is it giving you over an ec2 auto scaling group or ECS/Fargate? Both can scale as much as you like, your config can live as cdk/cloudformation/terraform code?
    • k8sToGo 4y ago
      Being independent of AWS stuff. Kubernetes community is much larger than ECS/Fargate, so many things already have been solved by others. Also something like karpenter.sh is much nicer than ec2 autoscaling group.
    • ocius 4y ago
      I'm not an expert on auto scaling groups, but we have used the Google Cloud equivalent for a time. The biggest issue for us was that deployment is not as easy. Can you update the software on the ec2 instance without turning it on, for example? With K8s, we can leave the deployment scaled to 0 and just patch the image of the deployment to perform a release while the workers are all shut down. Similarly, we don't have to write code to wait for the workers to finish their current job before we shut them down in order to be replaced by a newer version; this is all managed by K8s, and the configuration for it lives in Git. As others have already pointed out, it is also important for us to remain independent from Google Cloud / AWS.
• te_chris 4y ago
  k8s can be easy to set up, and if you know how to use it then you should. You shouldn't go to the trouble of learning it, but if you do know it already, then there is no reason not to go with services like gke autopilot.
• sorry_outta_gas 4y ago
  I don't use it much right now but I've found kubernetes (at least the managed service variety) fairly staright forward compared to some of the altearntives it beats the pants off the the random rubricks of ansible and puppet projects I've run into the past anyway
  • iostream24 4y ago
    How does it beat the pants off Ansible?
• moomin 4y ago
  I feel like this is probably good advice for startups and AWS-using companies. But there remains a lot of us who have, for many reasons, some good, some bad, a lot of investment in physical infrastructure already. For those, the Hashicorp or Kubernetes stack makes a lot of sense, if only to help standardise the insanity of what you’re running in-house.
• Dave3of5 4y ago
  Sort of agree. There are caveats of course but I tend to think if you can just start with a bunch of manually setup instances and roll from there. If you find yourself spending hours per day starting up and shutting down instances manually then yes, automate that part. Nothing I've ever done has taken off at all so automating DevOps would have been a total waste of time. Then again the projects themselves have been a complete waste of time.
  • p_l 4y ago
    I think it depends on whether you automate it so that it goes out of your way so you can spend more time on things crucial to your work - or you're just pontificating because you hate your job anyway.
• ale42 4y ago
  Why should using different languages for front-end and back-end be a problem? I rather think that it is better to use languages that are appropriate for the given problem. It is not premature optimization to have parts of a back-end implemented in C/C++/Go/whatever else if high performance is needed. It would rather be a waste of resources, money and energy not to use an high-performance language for high-performance applications. Of course using the same language for the front-end might make no sense at all.
  • pjmlp 4y ago
    Its seems to be a new thing with younger generations. We never had an issue with multiple languages across tier-n architectures. Suddenly with the uptake of HTML 5, it became an issue not being able to use JavaScript everywhere.
    • maverwa 4y ago
      I think the argument was/is not "its a problem we cannot have javascript" but "if we can have javascript everywhere, we only need to hire javascript devs and only need to care about javascript tooling", which is a fair point. That does ignore the fact that an experienced frontend JS dev is not necessarily also a good productive backend JS dev, but at least they know the language basics, can use the same IDE, etc. If thats worth it is something that depends on what you try to achive, I guess. I personally would not pick JS (nor TS) for the backend.
      • biztos 4y ago
        And isn’t Wasm supposed to (some day) free us from the need to pick JS for the front end?
        • ggktk 4y ago
          Unfortunately, that doesn't seem the way WASM is going. It's been 5 years and we still can't access the DOM without going through JS.
• mstipetic 4y ago
  It depends how experienced you are with it. If your team has no experience with it, don't do it. I'm pretty experienced, having set up a bunch of infras, so I'd run a blog on it now
  • p_l 4y ago
    Indeed - having used it for few years, I'd start the initial infra (for testing/playground/whatever) as small k8s on a local VM or similar solution so that we could prototype against the target abstractions from day one. Much, much less work to then move the deployments elsewhere, including building and teardown of integration testing environments.
    • mstipetic 4y ago
      When I get a contract to move a startups infra to k8s the majority of the work is actually untangling the services and making them accept configuration and communication in a sane way, the actual yamls are easy. Starting with k8s actually enforces some structure and standards and makes for a much more friendly environment to work in, usually you can run the whole infra on your laptop
• jpgvm 4y ago
  I disagree. K8s signals the team values infra and sees it as an important part of delivering their product. It's an investment like any other and for some teams it makes sense to set infra on a solid path sooner rather than later. You can move fast on things like serverless but the limitations of those systems can force architecture decisions that are sub-optimal whilst control of your own infrastructure can allow broader interpretation of problems and enablement of more wholistic solutions. i.e it opens up building things "the right way" sooner in many cases and that can save a ton of time and frustration. Also in the case where it's obvious you will outgrow serverless it also saves on a sometimes painful migration effort.
  • CRConrad 4y ago
    > K8s signals the team values infra and sees it as an important part of delivering their product. But isn't "infra[structure]" what you run your product on, and not part of the product itself? If your "product" is so big it needs stuff like that built into it, then it feels to me that by defintion that stuff isn't "infra". All in all, your sentence feels like "The Excel-Dell Humongo-XPS bundle signals that Microsoft values infra and sees it as an important part of delivering their product" to me. No, that only signals that A) Dell and Microsoft are using the dominant marketing position of Excel to milk money from naive customers, and B) If Excel can't run on more modest hardware, it's an unnecessarily bloated and inefficient product; look around for another spreadsheet.
    • jpgvm 4y ago
      Not at all. First of all k8s isn't bloated, it's actually surprisingly compact if you think about what you are getting. Second it doesn't imply the product -needs- it. Anything you can run on k8s can be run on bare boxes. Just doing so probably means you either lack the scale to need k8s (i.e my spreadsheet doesn't have many users) or you lack the competence to understand you have reached the scale that it's required (i.e your spreadsheet is run by n00bs). So it's much more of a positive than negative signal if you are seeking a vendor. I think the product relationship thing is a poor analogy from that side though, it -mostly- matters for people that want to work there on/in/around the product itself. As a primarily infra guy I have a preference for k8s because it allows me to have some reasonable set of services I can rely on being there if a place is "using k8s". If not I just have to hope either the existing infra team is competent enough to have things like service discovery, secret management, process monitoring, etc under control or that they are amenable to efforts to get it under control. If I am approaching it from a backend developer perspective where most of that isn't my problem then it gives me some confidence the infra team has some idea what they are doing and that I won't need to deal with some sort of bespoke deployment system that has endless quirks and company-specific bugs that I shouldn't need to concern myself with. Generally k8s just eliminates uncertainties by increasing standardisation so I don't need to watch people solve the same problem poorly over and over again for no reason at all.
• shele 4y ago
  > Imagine spending a lot of time and money picking out the best possible gear for a hobby before actually starting the hobby. Haha, this is exactly what “hobby” means to a lot of people. Less judgmental: thinking and dreaming about the right tools in disproportion to the need is something people do a lot, presumably because it is a source of joy.
  • mrweasel 4y ago
    That bit was pretty interesting to me as well, because that's exactly what many people do. I don't agree that this is what "hobby" means, it's more that we tend to try to buy our way into a life style. I feel that Kubernetes is exactly that for a large number of developers, operations people less so, they want to be part of a professional, world class, trendy environment/community. So they try leverage Kubernetes to elevate them to this Silicon Valley type tech company. There's absolutely value to be had from Kubernetes, but I agree with the article, it's not for the majority. Except I think there is one valid reasoning: If you purely see Kubernetes as tooling for deployment, it's hard to present any good alternatives for deploying to VMs or physical hardware.
    • ozim 4y ago
      You know these types of people who like to wear army clothes to look bad-ass even though they are wimps? I see it the same type of folks as ones using FAANG tooling :) just so they can feel better about their job or more important when they work on business line CRUD. I am working on business line CRUD and I like it with boring servers :)
• rkangel 4y ago
  The flipside of this philosophy is that you have to be able to revisit your decisions when your requirements change. This is a superpower that you want to have individually as an engineer but you need organisationally as a culture too. If you have that power then you when you need to write some similar code you can copy and paste, because you can trust that when you have a third use you will actually refactor properly. It means that you can get the service up and running quickly on a box you've got, because you can trust that if the service actually gets used your org will allow you the time to put it on the cloud properly. It's the most important part of agile - make the right decision for what you've got now, and then make a different one later.
• supermatt 4y ago
  I don’t get these complaints AT ALL. I don’t use kubernetes, simply because I am running apps in managed environments, and have been using docker-compose with vscode remote to emulate those environments. But being able to define your resources and how they are linked via a schema, makes sense even from a dev perspective. Isn’t that all that kubernetes is doing at it’s most basic? Sounds like that saves time to me over manually setting everything up for every project you work on.
  • gjulianm 4y ago
    From what I've seen with Kubernetes, the problem is that in order to define the resources and links with a schema, it needs several abstractions and extra tooling. So you not only need to define the schema (which isn't that trivial) but also understand what Kubernetes does, how to work with it, and how to solve problems. It's a tool that adds complexity and difficulty, if you're not using the advantages it provides (mainly multi-node management and scale capabilities) you're just handicapping yourself.
    • jpgvm 4y ago
      This is a common misconception. k8s isn't about scale, multi-node or even reliability/resiliency. We had solutions for all of that before it came along. It's about having a standard API for deployment artifacts. The k8s manifests are trivial (if verbose) the complexity comes from running the underlying layer which when you are small you simply outsource to AWS or GCP. There is some k8s know-how that is table stakes for a good experience, namely knowing what extra stuff you need on top of a base k8s cluster. i.e certmanager, external-dns. Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes. Before anyone starts with "but serverless!" you need to consider the spagetti you end up with if you go down that route, API Gateway, 100x Lambdas and thousands upon thousands of lines of Terraform boilterplate does not a happy infra team make.
      • ramraj07 4y ago
        It would be great if otherwise smart people Also learn to know when what’s trivial to them is not trivial to the masses. K8s is not trivial, not to me at the least. I’m not some super duper engineer but I do alright? That’s all I can say at the least.
• ianpurton 4y ago
  If you have one docker container to deploy then yes you can get away with heroku or fly.io etc. More than one container then cloud hosted kubernetes makes sense. It's a transferable skill a kind of learn once apply to any cloud technology.
• imjonse 4y ago
  When I see k8s I automatically think o13g due witnessing it being tried a couple of times for the coolness factor, not because it was actually required.
• iostream24 4y ago
  “So you want to run a bunch of stuff on one computer, why?” In a quest to get closer to the metal, Kubernetes keeps you far away, which is the opposite of what any production service should want. What is the purpose of adding layers when uni-kernels and eco-kernels give you better isolation and better performance? Your cloud provider already runs your virtual machines OS on a hardware hypervisor. Then running Kubernetes on top of the OS and then a zillion containers on it is a recipe for poor performance. What is the logic here that I am clearly missing? Cloud providers native Kubernetes stacks don’t improve performance or pricing, compared to their cloud compute instances virtual machines and a dedicated virtual machine per would-be-container that thankfully doesn’t need to now share processing resources with others. What gives? Why on earth would anyone run production processes in Kubernetes?
  • iostream24 4y ago
    Merely making scriptable infrastructure doesn’t require kubernetes or containers… Ansible or it’s ilk will do just fine. Deploy cloud instances like you currently deploy containers. Save money, get better performance, have real networking. *i have used kubernetes and understand how to use it as intended. I feel like I am missing the motivation for it’s current widespread usage in web apps
  • jiggawatts 4y ago
    It deduplicates the kernel memory and system image base disk. The minimum virtual machine size for a Windows server that is at all useful for anything is 4 GB of memory. Okay, okay, so you can technically boot it up on 2 GB and some roles will work fine, this will last only until some dingbat remotes to it with RDP with a 4K monitor and it starts swapping to disk. Even if you use Server Core and block port 3389, it still needs a ton of memory just to start. Running in a container it uses a few hundred megabytes. Similarly, the minimum system disk size you can get away with is 32 GB if it is a discardable / ephemeral instance. You need 64 GB minimum if you ever intend to run Windows Update on it. With containers, the unique parts of the image might be just a few hundred megabytes, even for complex apps. My experience is with Windows, but from what I hear Linux VMs vs Linux containers have vaguely similar ratios. So with containers, a single host can run dozens of applications, all sharing the same base disk, and all sharing the same OS kernel. The savings can be staggering. At $dayjob, the admins are very much stuck in the dedicated VMs for every role mentality, and they're burning through enormous piles of taxpayer money to run them at literally 0.1% load. Having said that, Kubernetes has its own problems. As you said, layering it on top of cloud VMs is a bit silly, and can easily result in the container running in a nested hypervisor at molasses speeds. Similarly, every single typical process changes dramatically: Deployment, updates, monitoring, auditing, etc... Combine the above with the incompatible underlying cloud layer and things get really messy really quickly. In my experience 90% of the world just isn't ready for the learning curve. Windows as an operating system isn't ready, certainly. Microsoft Azure isn't really ready either. Their AKS managed offering is still undergoing massive churn and seems to have more preview features than stable features. Even in the Linux world I hear more horror stories than success stories. It seems that everyone who says they love Kubernetes is using it on like... one machine. Come back and tell me how you feel after troubleshooting a failed upgrade on a cluster managing $100M of finance transactions. What I would like to see is "native Kubernetes clouds" where the hosts are bare metal and there is no impedance mismatch between K8s and the cloud provider APIs because K8s is the API. Instead of the Azure portal or the AWS console you literally log into a Kubernetes console. IMHO that would allow a true commoditisation of the public cloud and start to erode the near-duopoly of AWS and Azure.
• glintik 4y ago
  I can understand developers and devops - overengineering allows them to play with technologies for company money. Tech managers should control level of overengineering, but in many cases they just miss..
• cutler 4y ago
  Same for Docker.
• amirteshome 4y ago
  Docker Swarm Rocks " .... I would recommend for teams of less than 200 developers, or clusters of less than 1000 machines. This includes small / medium size organizations (like when you are not Google or Amazon), startups, one-man projects, and "hobby" projects. " https://dockerswarm.rocks/ https://dockerswarm.rocks/
• Stormwalker 4y ago
  I have read that this article isn't about k8s, but in my experience, companies that try to avoid k8s in favor of other specialized solutions are leaning more into premature optimisations. The company that used Nomad spent weeks in analisys of simple auth service, but every company that used k8s, went with whatever and moved on.
  • ramraj07 4y ago
    There are other options. You can use elastic beanstalk, and RDS. You get scaling, reliability and backups. And it’s not complicated and you can set up easy CICD with just GitHub actions. If you don’t need scaling, just use lightsail and litestream. There are many many options to do things simply.
    • Stormwalker 4y ago
      You just showcased my point! People would spend weeks in search for solutions that implement part of required functionality, instead of using whatever they're most familiar with and move on.
• deleted 4y ago
  [deleted]
• jhoelzel 4y ago
  Guys Kubernetes is a container platform for multiple nodes. I know it seems hard to understand from the outside, but its really not. You would naturally come up with ALL the same componets if you were to take your container strategy onto multiple computers. What if you dont need multiple servers? Well go the single node approach and have a flexible, true and tested way to spin up containers, which can and should be able to crash whenever they have to. Using Containers is pre mature optimization too? maybe I should get my typewriter.
  • mtrower 4y ago
    > Using Containers is pre mature optimization too? maybe I should get my typewriter More like grab a damn pen and scrawl out what you actually need. It's flexible and fast, which in many cases suits the situation best.
  • CRConrad 4y ago
    > Using Containers is pre mature optimization too? maybe I should get my typewriter. You seem to have missed the alternative "get a computer" in between there.
    • jhoelzel 4y ago
      well I grew up provisioning and using dedicated servers and nowadays you get much tighter security from the container ecosystem that I would on a dedicated server. I have just provisioned another bare metal k8s cluster, taking software of dedicated servers and.... well if I dont have to, im not running bare metal anymore in 2022. If you want to just "get a computer in there" have a look at harvester, its the best of the two worlds
• pritambarhate 4y ago
  Doesn't look like the author knows what he is talking about. His point about early stage startup should not use K8S is fine. But the next advice about not using a different language for frontend and backend is wrong. I think the most appropriate advice is to choose a stack which the founding team is most familiar with. If that means RoR then RoR is fine. If it means PHP then PHP is fine too. Another option is to use a technology which is best suited for the product your are trying to build. For example, if you are building a managed cloud service, then building on top of K8S, or FireCracker or Nomad can be a good choice. But then it means you need to learn the tech being used inside out. Also he talks about all of this and then gives example of WhatsApp at the end. WhatsApp chose Erlang for the backend and their front end was written in Java and Objective-C. They could have chosen Java for backend to keep frontend language same but they didn't. They used Erlang because they based their architecture on Ejabberd which was open source and was built with Erlang. Also WhatsApp managed all their servers by themselves and didn't even move to managed cloud services when they became available. They were self hosting till FB acquired them and moved them to FB data centres later on (Source: http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...).
  • DrewADesign 4y ago
    Yep. In fact, the front/back language bit is the most egregious premature optimization I can think of.
  • RapperWhoMadeIt 4y ago
    I also thought WhatsApp is a bad example. They not only hosted themselves, but they used solely FreeBSD (as far as I know) in their servers. (which don't get me wrong, I find great as a FreeBSD sysadmin myself).
    • brendamn 4y ago
      Using WhatsApp as an example of a lean engineering org should almost be banned at this point. WhatsApp had a high performing engineering team that used basically the perfect set of tools to build their application (which also had a narrow feature scope; plaintext messaging). Even with hindsight there is very little you could do to improve on how they executed. Just because WhatsApp scaled to almost half a billion users with a small engineering team doesn't mean that's the standard, or even achievable, for almost all teams.
• waffles2021 4y ago
  The most starry of starry eyed startups will want a team that is already fit for scale, so they build for scale before they need it. Possibly convinced by VC behaviour and getting as much from each funding round as possible. If you have to scrap your 1st round team and re-employ for 2nd at scale its going to be a hard sell.
• grumpy-de-sre 4y ago
  At this point I'm pretty much convinced these repeated "Kubernetes is bad for startups" rants are some kind of FUD campaign (probably grass roots from people who've missed the containerisation and declarative infra train). Kubernetes is actually pretty great for the vast majority of use-cases. Sure if you don't have experience with kubernetes leave learning it until after you've hit product-market fit. But rejecting the advances made in this ecosystem is a red flag upon itself. My ultimate MVP starter pack is probably a big fat VM running a single node K3s instance and a cloud provider managed database. Software wise nginx-ingress / cert-manager / argo-cd is a pretty good default set.
  • pas 4y ago
    could you elaborate on the "ideal" argocd workflow? let's say devs push to gitlab, which starts a pipeline, tests run, container images get built, then in the last step in the gitops repo a version bump happens and then argocd will pick it up automatically? (and if one wants to be able to rollback then that's the same workflow just instead of version "bump" the pipeline tags images with the commit hash and at the last step sets the version to the right tag?) are there any gotchas, best practices? :) thanks!
    • Chilinot 4y ago
      For deployments using Argo, yea that's pretty much it. You can set it to automatically deploy to your target environment as soon as it detects changes to your k8s mainfests, or you can require a manual "approve" step which is a push of a button for configs that are out of sync with what is applied to k8s. For rollbacks, you have a button in argo that lists previous deployments and you can just choose which previous deployment you want to go back to. It works pretty well those (thankfully few) times i have had to rollback any deployments.
      • isbvhodnvemrwvn 4y ago
        Can it run smoke and acceptance tests? Or does it require calling external services?
• htormey 4y ago
  The big missing piece of this article is a sense of at what scale and why should a startup decide to invest in a piece of infrastructure like Kubernetes. The author mentions other things he considers red flags such as using a different language for backend and frontend development with no additional context. Is the author talking about a startup in the context of one person who just knows JavaScript working on their own building a prototype? Is he talking about a series B company with 500k MAUs? Some additional context would improve the article a lot. I think the author should have had a few people read over the article and given feedback before publication.
• halotrope 4y ago
  I don't know. I default to GKE for all new deployments. It really reduces the mental overhead of infrastructure for me. I can add services, cronjobs and whatnot as I see fit in a standardized manner and don't worry about. A couple YML's get you started and you can scale as you see fit. Anything you would like to deploy can share 1-2 vCPU. The whole thing is also relatively portable across clouds or even on premise if you want it. Since everything is in containers, I have an escape hatch to just deploy on fly.io, Vercel or whatnot. I get the criticism, that people over complicate projects that maybe don't even have traction yet. Bashing k8s is the wrong conclusion here IMHO. It feels a bit like people bashing React and SPAs just because what they have built so far never needed it. Just stick to your guns and stop evangelizing your way of doing things.
• kgeist 4y ago
  I am curious - there are markets where usage of managed platforms like AWS is restricted due to data protection laws (no data center on a country's soil) or what if a large client wants an on-premises installation (we have plenty of such clients) - what's the alternative to k8s if you want your solution to be "portable" across providers? From what I understand, our DevOps team leveraged k8s to streamline it using a standardized way to define resources and to scale them, but I'm not an ops guy so maybe I'm missing something.
• darkstar_16 4y ago
  How does everyone think about Kubernetes on-prem ? Where the requirement is to run an application/services hands-off without any platform team to maintain it etc ? Is Kubernetes also the right platform for that sort of environment ?
• CSDude 4y ago
  My rule to use Kubernetes is only if my containers exceed the extra containers I need to make Kubernetes work, including but not limited to: - kube-state-metrics - Coredns - Prometheus - Alert manager - Node Exporter - Spot instance termination handler - Karpenter/cluster-autoscaler - Event Exporter - Kubernetes Dashboard - Ingress - Cert-Manager - Custom CNI? - Istio (not always) - ArgoCD (not always) Even if they are configured with fractional CPU (which mostly causes stupid throttling issues) they still are headachaes to maintain. I used to run 4000 pods in a single cluster, using Kubernetes was great. Otherwise, nowadays I'd just go with Fargate/Cloudrun/Fly.io etc.
• avereveard 4y ago
  having repeatable infrastructure from day 1 is great. kubernetes is the simplest way to have that. it's not the only way, but it's provider agnostic, has a lot of well maintained and understood tooling around, and splits clearly artifacts from deployment (i.e. no scripts configuring stuff after startup, everything is packaged and frozen when you build and upload the container image) > Solve problems as they arise, not in advance. while this does make sense for supporting varying requirements the lean way, it fails to address the increased costs that rearchitecting a solution mid-lifecycle incurs. > Do more with less. goddamn slogan driven blogging. what is the proposed solution? doesn't say. are we supposed to log in every prod/test/dev machine and maintain the deps by hand with yum/apt? write our own chef/puppet scripts? how is that better than docker images running kubernetes? The comparison between solutions is the interesting part. op never say. guess "works on his pc" is enough for him, we can only assume he envision a battery of mac laptops serving pages from a basement, with devs cautiously tipotoing around network cables to deliver patches via external usb drives
• bamboozled 4y ago
  Man these opinion pieces get boring fast and very rarely add value. I honestly couldn't give a toss if some guy thinks using some < language / tech / library > is <blah>, it's a technology, a tool, try it, use it, if it works for you good, if not use something else. Move on. The only time it's a allowed is if it's bashing JIRA :)
• tsujamin 4y ago
  as someone who is currently tearing down a managed k8s cluster and migrating the services into SaaS I sympathise with this blog heavily
  • tr33house 4y ago
    What does migrating into Saas mean? Those are two different concepts
• CipherThrowaway 4y ago
  I lean conservative in my tech choices but I just don't see the big issue with Kubernetes. If you use a managed service like GKE it is really a breeze. I have seen teams with no prior experience set up simple deployments in a day or two and operate them without issues. Sure, it is often better to avoid the "inner platform" of K8s and run your application using a container service + managed SQL offering. But the difference isn't huge and the IaC ends up being about as complex as the K8s YAML. For setting up things like background jobs, cron jobs, managed certificates and so on I haven't found K8s less convenient than using whatever infrastructure alternatives are provided by cloud vendors. The main issue I have seen in startups is premature architecture complexity. Lambda soup, multiple databases, self-managed message brokers, unnecessary caching, microservices etc. Whether you use K8s or not, architectural complexity will bite your head off at small scales. K8s is an enabler for overly complicated architectures but it is not problematic with simple ones. >Did users ask for this? Not an argument. Users don't ask for implementation details. They don't ask us to use Git or build automation or React. But if you always opt for less sophisticated workflows and technologies in the name of "just getting stuff done right now" you will end up bogged down really quickly. As in, weeks or months. I've worked with teams who wanted to email source archives around because Git was "too complicated." At some point you have to make the call of what is and isn't worth it. And that depends on the product, the team, projected future decisions and so on.
• discordianfish 4y ago
  I feel like most of these rants come from people who never built the alternative to kubernetes to support a modern workflow (CI/CD with branch deploys, monitoring, access control etc). I love kubernetes because I don't need to build bespoken platforms at every company I join. I probably would have switched careers by now if I still had to deal with site specific tooling that all essentially implement a worst version of what Kubernetes has to offer.
  • regularfry 4y ago
    Except that he's not arguing to build the alternative to k8s. He's arguing to buy a higher-level abstraction.
• lebowen 4y ago
  I'm a developer that uses Kubernetes in production purely because I want to be able to use the same Docker images that I use in development. I am not a Kubernetes advocate but what else is there that handles all of the issues faced when deploying containers? Such as scaling, deployment, configuration etc? There are alternatives such as Hashicorps Nomad, but I don't see how this is any better/worse that K8s.
  • clubdorothe 4y ago
    Check into Amazon ECS or Google Cloud Run. It's basically a higher level of managed Kubernetes ( EKS or GKE ). You give your containers, and they take care of running it. Tim Hockin (one of kubernetes creator) support the idea to use something as much managed and automatic as possible : https://twitter.com/thockin/status/1539987108521054208 https://twitter.com/thockin/status/1539987108521054208
    • lebowen 4y ago
      I'm aware of these products, but do not fit my use case. I need to run some services on premises and have set up a self hosted Kubernetes instance on a physical server in a rack. It could be overkill and maybe I could use something like Docker Swarm. Apart from this I am unsure what I can use that isn't K8s to orchestrate my containers on site.
      • MaKey 4y ago
        Which features do you need that docker compose can't meet?
        • lebowen 4y ago
          Rolling restart for zero downtime deployments.
      • everfrustrated 4y ago
        AWS ECS can be run on-prem.
• almostdigital 4y ago
  I can't recommend docker swarm enough, it does 95% of what you need in k8s without requiring someone full-time just to manage it.
  • politelemon 4y ago
    Agreed, it's a shame that k8s won the mindshare, and other solutions don't get a second glance. Swarm will do most of what you need, and can help avoid the premature optimisation trap. Going from Swarm to Fargate or K8S is a small step on a journey.
• AtNightWeCode 4y ago
  The same language argument is plain wrong. It is easier to change between languages within the same domain than to swap domains using the same language. If you done web services programming with NodeJS you can easily switch to GO. Starting with React will be much harder.
• jhugo 4y ago
  Anything can be a red flag signalling premature optimisation, depending on the scope of the problem you're solving. Like any dogma, "you don't need k8s" will sometimes help you, sometimes harm you.
• revskill 4y ago
  One stable Kafka cluster is worth more than 1000 kubernetes clusters (for free) to me. Kubernetes to me is kind of useless. I don't use it for the sake of learning it. What's the point ? I write code for human, to serve real benefit/profit from business standpoint. I build value and trust based on what i delivered, not from what i "want to learn". Currently, i go all in for unidirectional architecture. Think redux all the way, not just for frontend. The point here is, it's the architecture that's drive business profit, not Kubernetes. Kubernetes is not your architecture.
• wodenokoto 4y ago
  It’s kinda curious that the HN sentiment in the comments has lately switched from “kubernetes is hell” to “kubernetes is quite useful and well worth it” To me, kubernetes is the kind of giant, complicated, hard to understand software that is ripe for HN disdain.
  • jhugo 4y ago
    If using k8s doesn't reduce the overall complexity of your system, then you probably don't need it. But not all systems are the same as yours.
  • regularfry 4y ago
    What it is, is a low-level abstraction that probably wants something over the top of it to hide the messy details. I don't think anyone's really cracked that yet.
  • grumpy-de-sre 4y ago
    I'm pretty sure the "kubernetes is hell" proponents were always just a vocal minority. But as always technology choices have to be made in the context of each business (available experience etc).
  • cjalmeida 4y ago
    Modern CPUs are giant, complicated pieces of hardware. But they’re useful, and have a well defined interface that you can interact with it at a higher level. Same with k8s. People conflated running k8s from scratch, with administrating. The former is very hard, you should buy it from a cloud provider. The latter is IMO not much harder than VMs at small scale and much simpler at you grow bigger.
    • mirceal 4y ago
      Yes, let's compare CPUs that took literally decades to mature and there are a handful of companies that can design and manufacture them to K8s. We don't want to run k8s from scratch? Let's buy the managed version in the cloud? It sounds like an argument for containers. I mean, at that point why not use a simpler abstraction, like a VM. Or just use functions.
• drzel 4y ago
  I maintain a free, open source, fast paced shooter that really only works with <100ms pings. To this end I have a dozen servers around the globe. Each server runs a docker-compose with four instances of the game server (each for 24 players), along with some auxiliary services that do things like download updates, upload match demos and statistics. I frequently need to re-provision new servers and close existing ones. I've been doing all this with the help of docker-machine (now deprecated), and a collection of handy bash scripts. While this has worked, it's become more and more fragile and I need something better. For years I've resisted Kubernetes because I keep hearing "premature optimisation", but at this point I'm not sure I have any other options. So before I dive head-long into Kuburnetes by what feels like necessity, how else could I possibly: - Push button provisioning of new hosts across a number of cloud providers - Monitoring so I can restart services under certain conditions. - Notification of failures etc. - Automatic / continuous deployment across all servers and regions.
  • punkrex 4y ago
    This is gonna be hard, I’d honestly look at fly.io. But argocd hooked up to each provider along with provider specific overrides should work with enough tweaking.
  • kasey_junk 4y ago
    What you are describing is fairly trivially accomplished using a combination of vm image provisioning software (e.g packer) and a traditional configuration management system (e.g. ansible). It will be simpler, more straightforward and most likely use less computing resources. It’s also a decades old technique at this point that is extremely battle hardened. You don’t say it outright but you imply that you are doing this as a single operator. That removes giant swathes of the advantages K8s brings to the table. Further your implication that there are multiple edge clusters is a case that is fairly new to K8s and the best practice would be to run n K8s clusters. If you are looking for a more managed/turn key option I’m a huge fly.io fan boy.
• sweaver 4y ago
  For sure, there is power at the cost of agility! I've seen cases where we started off as simply as possible with no k8's. We built the initial product really quickly using a ton of managed services. Whilst it was great to get us going, once we hit "growth" things just didn't scale. (1) The cost of cloud was getting astronomical for us (and growing with each new deployment) and (2) it was totally inflexible (whether that be wanting to deploy in a different cloud, or extend the platform's featureset) because we didn't own the underlying service. We ended up porting everything to k8's. That was a long & arduous process but it gave us total flexibility at significant cost savings. The benefits were great, but not everyone has access to the engineers/skillset needed to be successful with k8's. That's why we built Plural.sh – it takes the hard work out of k8's deployments. I've seen people go from zero to a full production deployment of a datastack on k8's in just 2 weeks. It deploys in your cloud, and you own the underlying infra and conf so you have total control of it. And because we believe in being open, you can eject your stack out of plural if you don't like it and keep everything running. Great post, and hope all is well with you!
  • tr33house 4y ago
    Had the same exact experience! Just posted a comment. I think could companies have done a good job marketing how cheap it is to get started on cloud. Once things scale, the bills change and it's no longer as cheap to use managed infra. The night and day difference is hard to explain when people haven't had to deal with all these issues on top of scaling, uptime, performance issues etc. I just can't recommend k8s enough
  • candiddevmike 4y ago
    I think it's funny that everyone uses/loves k8s ends up building a product to make it easier to use. There are a lot of examples in the comments here. To me, that's enough of a red flag that k8s doesn't understand their users or can't meet them where they're at.
    • nouveaux 4y ago
      Heroku and friends is literally built to simplify the complexities of VMs. Are you also arguing that deploying to VMs is too complex?
• KronisLV 4y ago
  I have previously talked both about the advantages and disadvantages of Kubernetes, but some of the points in this article seem interesting to say the least. I know that a lot of folks are already disagreeing with some of them, but I wouldn't suggest that the author is plain wrong, merely that there is a lot more nuance to it. > Companies using Kubernetes for an application that is a web app. There is probably a whole spectrum, with manually deploying code to shared hosting through SFTP being on one end and having full Infrastructure as Code running on Kubernetes with CRDs and other functionality on the other. Personally, I think that you need to find the sweet spot: - use containers, otherwise your app environments will rot over time, or at the least become inconsistent - consider something like Ansible or another GitOps approach for managing the app config, consider the same for server seetup - use some sort of container orchestration, whatever is easier and whatever you're more familiar with (Docker Swarm, Hashicorp Nomad, K3s) - if you do go for Kubernetes (K3s or otherwise) make it as simple as possible, don't go for fancy service meshes or bunches of operators etc. - document what you can: with Markdown or whatever for describing things that need to be known and done manually, code for the rest This approach has allowed me to be able to add any server to a cluster easily, to distribute the load of all the app services, or even be able to treat many projects with different tech stacks the same way: just launch the container, configure resource limits, parameters, storage, networking/certificates and it's done. Things crash? Automatic restarts and liveness probes/health checks. Want the logs? Just look at the output or easily configure shipping them somewhere. Need to deliver new version to clients? Just push to Nexus/Artifactory/Harbor. And should your org have separate Ops people? Well, they won't really have to care about someone wanting to bump the JDK version or whatever, let the devs care about what's inside of the app themselves. Security concerns? Scan the whole container with Trivy? Clients have a different environment? They probably can run OCI containers somehow. Many of those advantages also apply to monolithic apps, like some garbage that may only run on JDK $OLD_VERSION or $OLD_DISTRO, but should still have its surface area for attacks limited as much as possible. If done right, developing apps like this will be a breeze without getting too overcomplicated. If done wrong, you will get nothing done and will have to suffer with figuring out how to get your Kubernetes cluster up and running properly, then will have problems with resource usage etc. > More than one language for your application. For example, a backend in Golang, Ruby, PHP etc. and a Frontend Web App in React, Vue etc. This is probably the most controversial argument: I will admit that using something like Ruby on Rails or Laravel, or any other server side rendering option will probably be a pretty decent and fast way to get up and running. Yet, nowadays people like developing APIs so that other apps can be connected to those relatively easily, which is somewhat handled by most of these options as well, though developing a separate front end and dogfooding the API yourselves will perhaps be the better way to ensure that everything works as expected. More so, in my personal experience trying to develop both the front end and back end as a single bundle can sometimes be a major pain in the behind, which gets infinitely worse when you do have some sort of a full web app that you try to embed and serve from the back end container/process/deployment as static files, because of issues with permissions, threads/performance etc. If anyone is curious, doing that with Spring (not Boot) was really annoying and kind of brittle, having to always deploy them together and having unreasonably long startup times aside. So I can't help but to lean towards a split setup (in most cases): - RESTful API for the back end, can be used by the front end or other integrations (I don't think that GraphQL is all that good, but it's also an option) - some sort of a separate web app front end (Vue, React, Angular) that is served by Nginx/Caddy/Apache and may later be replaced/migrated over to another solution etc. > Not using a cloud service to host your app. Examples are Heroku, Vercel, Netlify and Fly.io. Most product teams will have over-architected their solution if they have to have an ops or infra team. This probably depends on corporate policies or whatever people are comfortable with. Using those cloud services will result in vendor lock which, as Heroku showed, isn't always cost effective or even that good of an idea long term. Currently I pay around 260 Euros/year for my servers (5 VPSes) on which I run my containers, some people pay more than that per month on certain platforms. Because I settled on OCI containers as the common runtime format and the aforementioned approaches to orchestration, I can switch between whatever hosts that I want. Previously, I've used DigitalOcean, Vultr, Scaleway, Hetzner, Contabo and Time4VPS (my current one) and can easily move, whenever a better offer comes along. Of course, for many startups with SV funding that doesn't matter much, but there is definitely a lot of merit to avoiding vendor lock and focusing on open source standards. Or, you know, you decide to work in an industry where the law book gets thrown at you and you have to do on-prem.
• pelasaco 4y ago
  I'm not sure if I can agree with the author. We moved from plain AWS EC2 to K8 to improve our continuous delivery pipeline and it worked. Before we had dozen of custom terraform scripts, custom runners on gitlab, and it became much cleaner now with K8. Other aspect that improved was monitoring and introspection. I think mainly because K8 offers a reach set of tools and patterns helping exactly in that matters. It comes with its own costs, but I can definitely say that we went from a tailor made solution to a standard set of techniques and tools, which is a huge improvement, IMO.
• ploppyploppy 4y ago
  There are two things that guarantee a click on HN due to the community's ignorance and arrogance: Crypto and Kubernetes.
  • CRConrad 4y ago
    Both of them seem to have about equally many proponents as contrarians, so in each case it's half due to the community's ignorance and arrogance, half to its wisdom and humility. The only question is which is which.
• wiredone 4y ago
  I'll be honest - i've worked places where kubernetes was a thing, and places where it wasn't. Both within the last 5 years. Kubernetes is a layer of complexity that just isn't warranted for most companies. Hell even Amazon still sticks with VMs. Autoscaling and firecracker solve most things. Its nice to have cluster (pod?) management as a first-class concept, but you get that with tagging of instances just as well for most use cases. In short - i think the author has a point.
  • p_l 4y ago
    Amazon is IMHO an anti-example - they have huge installed base of tooling to handle VMs, and most importantly, they have huge scale of capital available. Throwing single-use VMs at the wall is cheap for them. Meanwhile, I was chugging around with k8s because it was balance between "easy to deploy our workloads with minimal time spent" vs "increasing the cost by few pizzas will cause noticeable loss so pack those VMs tight".
• tr33house 4y ago
  As a startup founder that's not VC funded, I would totally recommend you look into building with kubernetes from the get go. The biggest learning curve is for the person setting up the initial deployments, services, ingress etc Most other team members may just need to maybe change the image name and kubectl apply to roll things out. Knowing that rollouts won't bring down prod and that they can be tested in different environments consistently is really valuable. I started out with Iaas namely Google App Engine and we suffered a ton with huge bills especially from our managed db instance. Once the costs were too high we moved to VMs. Doing deployments was fine but complicated enough that only seasoned team members could do it safely. We needed to build a lot of checks, monitoring etc to do this safely. A bunch of random scripts existed to set things up and migrating base operating system etc required a ton of time. Moving to kubernetes was a breath of fresh air and I wish we'd done it earlier. We now have an easy repeatable process . Infra is easier to understand. Rollouts are safer and honestly, the system is safer too. We know exactly what ports can allow ingress, what service boundaries exist. What cronjobs are configured, their state etc with simple kubectl commands. Using kubernetes forces you to write configurable code and is very similar to testing: it sounds like it'll slow you down and shouldn't be invested in until the codebase is at a certain size but we've all learned from experience how is actually speeds everything up, makes larger changes faster, cheaper customer support and saves you from explaining why a certain feature has been broken for 10 without anyone's knowledge
  • lbriner 4y ago
    Totally agree! Kubernetes isn't just about global scale that most people will never need, which would agree with the article. It is about deploying new apps to an existing production system really quickly and easily. We can deploy a new app alongside an old app and proxy between them. Setting up a new application on IIS or a new web server to scale is a mare, doing the same on AKS (managed!) is a breeze. It is also really good value for money, because we can scale relatively quickly compared to dedicated servers. It is also harder to break something existing with a new deployment because of the container isolation. We might not need 1000 email services now but we could very quickly need that kind of scale and I don't want to be running up 100s of VMs at short notice as the business starts taking off when I can simply scale out the K8S deployment and add a few nodes. There is relatively little extra work (a dockerfile?) compared to hosting the same services on a web server.
• clubdorothe 4y ago
  Tim Hockin (one of kubernetes creator) supports the idea to use something as much managed and automatic as possible if you don't have time for it [1]. He probably refers to use Cloud Run [2], over a (managed?) Kubernetes cluster (such as EKS or GKE) [1] https://twitter.com/thockin/status/1539987108521054208 https://twitter.com/thockin/status/1539987108521054208 [2] https://twitter.com/the_thagomizer/status/1539757049071800320 https://twitter.com/the_thagomizer/status/153975704907180032...
• twawaaay 4y ago
  The startup I am working for has seen entire engineering fired after they hired a lot of people and spun an enormous amount of infrastructure, spent tons of cash and produced absolutely nothing useful to the company (presumably because they were engrossed in spinning the infrastructure). The company isn't very large and does not provide software products to the clients. The systems are mostly business intelligence, billing, ERP-like, etc. Now we are restarting lean (but wiser for the experience). 10x less people and technology accomplishes more and faster. Most companies with that kind of problem DO NOT need Kubernetes because they don't have problems for which Kubernetes is the cheapest solution. Most companies do need focus on keeping things simple, minimise amount of stuff to make it possible to hire a development team that has a shot at understanding what is happening. For example, we made decision to restrict ourselves to only one cloud provider (AWS), to only one backend language, one frontend language, etc. Our prod environment is simple, too. No effing microservices. One application on one application server easily serves the needs of a company of over 200 employees and our clients. Simplifying the environment made wonders to productivity in many different ways, the most important being that the discussion now focuses on what the company needs and the functionality that would solve the problem rather than on technical trivia. All this makes possible following things: * having one development team where everybody can contribute to everything (no need to Zoom with local K8s guru to accomplish anything or spend time in endless meetings with "frontend people".) * hiring a relatively simple developer profile. This does not mean hiring bad developers. But it means I am looking for people who know X and Y very well rather than X, Y, Z, A, B, C each a little bit... It is stupid to assume a person will know well every one of 50 different technologies listed on their LinkedIn profile -- most likely they only really know couple of things and everything else only superficially. * hiring from talent pool that has a lot of experience but not necessarily in the stuff that other companies require. Other companies may hive overlooked them because they don't know a bunch of new tech, but we don't care. We only care that they are great people that get shit done, fit our company, and know how to design and implement functionality in their core programming language. * engineers being able to accomplish their individual tasks from start to end, right after joining the company. No need for a long learning period. * company needs main center of attention rather than spending large part of the effort on technical trivia. My take? If you need to add something, first ask yourself what are the costs and better understand what you get in return for that cost.
• deleted 4y ago
  [deleted]
• choeger 4y ago
  Funny thing that I read this line: > Solve problems as they arise, not in advance. It strikes me that this particular philosophy, that I associate for no other reason than a gut feeling with US-American culture, is one of the biggest reasons for our shitty software development quality. Every successful team that I have seen so far had solved the worst problems in advance and thanks to this investment ended up with a much higher capacity for times of crisis. The assumption that problems can be solved (really, solved, not somehow worked around until things will explode in your face Tuesday next week) when they show up in practice looks like fake optimisim. Of course it takes good judgement to distinguish actual future problems from mere inconveniences and phantom issues. But that's a skill that people should try to cultivate not bullshitting their way out of responsible management with terms like "Pareto principle" and "agile".
• Noughmad 4y ago
  I really don't understand all these complaints about how Kubernetes is so complex, how it's an investment etc. I am a single developer that uses Kubernetes for two separate projects, and in both cases it has been a breeze. Each service gets a YAML file (with the Deployment and Service together), then add an Ingress and a ConfigMap. That's all. It's good practice so still have a managed DB, so the choice of Kubernetes vs something running on EC2 doesn't change anything here. Setting up a managed kubernetes cluster for a single containerized application is currently no more complicated than setting up AWS Lambda. What you get out of it for free is amazing though. The main one for me is simplicity - each deployment is a single command, which can be (but doesn't have to be) triggered by CI. I can compare this to the previous situation of running "docker-compose up" on multiple hosts. Then, if what you're just deploying is broken, Kubernetes will tell you and will not route traffic to the new pods. Nothing else comes close to this. Zero-downtime deployments is a nice bonus. Simple scaling, just add or remove a node, and you're set. Oh, and finally, you can take your setup to a different provider, and only need some tweaks on the Ingress.
  • iasay 4y ago
    It’s really complicated when something goes wrong. That is my only criticism. Particularly in the various CNI layers out there. You really have to know exactly how everything works to get through those days and that is beyond the average person who can create a docker container and push it into the cluster which is the usual success metric.
    • jpgvm 4y ago
      99% of people aren't going to use a different CNI plugin to what their managed distribution ships with. Same goes for peeking under the covers of storage plugins, kubelet config, etc. You pay AWS/GCP for that these days and just use the API.
      • iasay 4y ago
        It was AWS who had trouble fixing our CNI issues…
• Linda703 4y ago
  [dead]
• mt42or 4y ago
  Kubernetes is the standard for infrastructure, everyone should use it instead of VM or all others singular infrastructure. All others options are costly on the mid and long term.
  • iostream24 4y ago
    I don’t think so. I run 2 dedicated-cpu cloud instances that will thrash any load you throw at it high availability, and I can spin up an entire 3-node production cluster in a few minutes with good old Ansible and some cloud provider module. Kibernetes brings me overhead and wastes system resources. I already have infrastructure as code. Why would I want to containerize systems I already have YAML to describe? People forget they real hardware has to run this stuff, and layers os schedulers are not helpful. Look at the disaster of fibers threads and processes already!
• deleted 4y ago
  [deleted]
• SassyGrapefruit 4y ago
  Why build it when you can buy it? Right? The decision is more involved that this. The real question is... What is the shortest route to a sustainable, defensible revenue stream? In the 80's IBM built a super quick computer out of pieces you could buy at Radio Shack. The ultimate solution! Why incur all that upfront manufacturing cost when you can assemble it out of existing parts? They learned the answer really quickly. The lower barriers of entry meant anyone could recreate the product. And so a million clones popped up using the same parts. For a decade Apple ate their lunch by investing in a proprietary platform(which a lot people said was madness using the exact reasoning on display in this article) SaaS is great but you have to have some sort of moat for whatever you are building. Sometimes that moat can be as simple the idiosyncrasies of your over-engineered, prematurely optimized product. I had a hand in the development of one of the major VPS players during the 2010's. There were things we could do just because of the weird decisions people had made in the past. That translated directly to competitive advantage.
• williamcotton 4y ago
  I have the strange urge to create a sort of web app Olympics. Let’s see just how long it takes for participants using their preferred tools to build and deploy a Todo app with user accounts! And then we see how long it takes to go from the initial application to one that supports 1000 requests a second.
• Annatar 4y ago
  It takes a month to set up a Kickstart infrastructure (DHCP, TFTP, NFS, Kickstart profile(s)), a day to three days for an experienced system engineer to package the components, and a week to three weeks to test and package the configuration (by automating in %pre and %post of the RPM's). That's far less than one year, and far less than it takes to set up Kubernetes, Terraform, Docker and a complete CI / CD pipeline.
• sudarshnachakra 4y ago
  While I tend to agree to the conclusion on premature optimization - I disagree with the assumption that it is premature for most startups. In fact it's a reasonable insurance for startups (that is - if at all they succeed) it'll solve the problem of scale at that point without needing to make huge changes. BTW Google open-sourced Kubernetes not for charity (like all businesses they also want to make money), they knew they had lost the cloud war with Amazon/Azure gulping up almost 80-90% market share. So they wanted a platform on which they can lure users back to Google Cloud when they start providing kick-ass services (to avoid the much famed vendor lock-in). And since docker was able to solve the dependency management in a reasonable way (not in a very esoteric way like nix) and were dipping their toes into distributed orchestration, they took it as a open fundamental unit to solve orchestration of distributed systems. But yes Google succeeded in convincing dev/ops to use k8s by frightening them with vendor lock-in. But the most ironic thing that I see about k8s is that all these HPA, Restart on crash all those things are being touted as great new features. These features have existed in Erlang for decades (supervisors and spawning actors). I'm not sure why Google did not try to leverage the Erlang ecosystem - it might have been much faster to market (may be NIH).
  • p_l 4y ago
    It's because OTP does not integrate with anything not running on Erlang VM, and k8s instead derives from different family tree of general language-independent schedulers/environments.
    • sudarshnachakra 4y ago
      Language independence is not a trait of k8s it's an artifact of docker packaging java/c++/perl/python/go/rust etc. as an arch dependent image. TBH I find k8s support for languages other than Golang pretty poor (there have been attempts to get java into k8s by redhat with native-image, but it seems to have not made it big).
      • p_l 4y ago
        Language independence is a trait of k8s in the sense that none of its interfaces are in any way specific to a language - the most restrictive in that are the few APIs that are based on gRPC, because state of gRPC libraries is still poor in some places. Unless you want to embed another language in-process of some k8s components, but the need to do that is disappearing as things are moved out of process.
• CRConrad 4y ago
  Reacting only to the headline, before reading the article: > Kubernetes is a red flag signalling premature optimisation And even that presupposes that it's an optimisation at all in the first place.
• m0llusk 4y ago
  Comparison to internationalization might be interesting. Supporting multiple languages almost always expands the potential customer base, sometimes by a large amount. But even with the potential business impact it is extremely rare for applications to be localized early on. Doing the work early makes it much easier and pays off in organizing communication with users. Is this because the tools that are used are not impressive or powerful enough? Maybe needing to know multiple languages and team up with others is the blocker? Working on localization it is often remarkable how great effort often gets put into pretty much everything else and then translations are added hastily and with as little sophistication as possible.
• ki_ 4y ago
  personally i believe you should try to avoid as many services as possible. Why do you want to rely on a service? What if the service says "bye bye", you'd be like.. ok i guess our software doesnt work anymore.. RIP. I also feel like many developers use services to hide the fact they just suck at solving certain problems. E.g. we need to deploy.. ok we need this 3rd party service that will deploy for us, we will give them access to all our servers and therefore all data and that's absolutely not weird, + you'll have to read their docs for a few hours to know how it works... ok.. how about i write a simple deploy script in 5 minutes? "NO WTF! Are u insane??? Everything will crash and burn unless we use this service". :: So that's basically how my conversations go with these other devs. Really annoying, i just gave up on them, i just let them do what they want, you cant change their mind.
• movedx 4y ago
  I agree entirely. I like to call what the author is referring to as, "What-If Engineering". It's the science of thinking you'll be Google next week, so you build for that level of scale today. It involves picking extremely complicated, expensive (both in compute and skilled labour) technologies to deploy a Rails app that has two features. And it all boils down to, "But what if..." pre-optimising. It happens at all levels. At the individual unit level: "I'll make these four lines of code a function in case I need to call it more than once later on - you know, what if that's needed?" It also happens at the database level: "What if we need to change the schema later on? Do we really want to be restricted to a hard schema in MySQL? Let's use MongoDB". What's even worse, is Helm and the likes make it possible to spin up these kinds of solutions in a heart beat. And, as witnessed and evidenced by several comments below, developers think that's that... all done. It's a perfect solution. It won't fail because K8s will manage it. Oh boy. Start with a monolith on two VMs and a load balancer. Chips and networks are cheaper than labour, and right now, anyone with K8s experience is demanding $150k + 10% Superannuation here in Australia... minimum! https://martinfowler.com/bliki/MonolithFirst.html https://martinfowler.com/bliki/MonolithFirst.html
  • iso1631 4y ago
    You're forgetting that many people will want to use K8s for a project because they want it on their CV to get the high paying jobs. I saw the term on HN a couple of weeks ago -- CVOps
    • movedx 4y ago
      I'm not forgetting that fact, I'm simply choosing to ignore such people. They're not really what the industry is about. That's not in the spirit of a healthy society. That's just leeching. Good luck to them, but they're not going to occupy time and space in my mind.
      • iso1631 4y ago
        Maybe it's just my organization, but I see the behavior across the corporation far more than I'd like, and inevitably these people move on leaving a complex mess in their wake that long term support staff have to deal with. We seem to mostly manage to avoid that in my department, but we have a very low turnover.
• j_m_b 4y ago
  Kubernetes seems like subversion before git came out. Overly complex but got the job done. When git came out, all other VCSs were quickly abandoned. Just waiting for the day when the git of container management comes out.
• d23 4y ago
  I wanted to agree with this, but this line (mentioned as a premature optimization) is absolutely baffling to me: “More than one language for your application. For example, a backend in Golang, Ruby, PHP etc. and a Frontend Web App in React, Vue etc.” When would your front end and backend ever be in the same language? I guess JavaScript would be your only option?
• spoiler 4y ago
  I disagree. There are a lot of valid reasons why a start-up might prefer Kubernetes to other solutions. A simple and good enough reason is the need for variable compute: You might need more computing sporadically (say for example you're doing software in the sports industry; your demand changes with the number and size of events, as well as on weekends). Another reason might be a start up that allows their customers to execute some type of arbitrary code (eg a no-code product). This can vary from customer to customer, and it can also vary within the customers use cases. Imagine having to manage all these storage, networking and compute resources manually... Or going full circle and managing it with a bunch of puppet/ansible/shell scripts. Now you slap some automated scale triggers on top that fires off these scripts. Congratulations! We've built something that looks like Kubernetes; if we squint. There's some smoke coming out of it. Documentation? Lol, we don't have time for that. we are a company that gets shit done, we don't toy around with Shcubernetes and documentation! Error handling/monitoring/reporting? Eh, just read the code if something fails. Need to add cert issuance? Yeah let's implement our own ACME integration. Network ingress? Let's just have a 500line haproxy infront of an 2000line nginx config; no big deal. DNS? just append to named; who cares? The name for the "we get shit done" crowd should probably be "we don't give a shit about anything and have others solve problems we created because of our lack of thinking and foresight", but it doesn't sound quite as memorable. It's just people who are comfortable cutting corners at other people's expenses. When they have to own the shit they made, they start blaming others and leave the company. Sorry for the rant.
• justinsb 4y ago
  The problem with this sort of argument is that it sets up a straw-man, without describing what you should do instead. Kubernetes could be better - of course - but what better approach is the author recommending to use instead? It would be better if we could use one language on frontend and backend - of course - but what is this one language that works everywhere? It would be better - of course - if there was a SaaS that could do everything for us, is cheap and is open source - but what is that SaaS? It's very easy to say something could be improved, but much harder to propose something to replace it that doesn't have its own shortcomings. It's a pity because this article actually makes a good point, that we should think about what the actual user/business goal is before making technical decisions, and be careful lest we spend all our time on technology. I've heard this idea described as "innovation tokens". It's disappointing that the article chose to wrap this idea in clickbait, but I guess we wouldn't be talking about it otherwise! Disclaimer: I have no end of personal biases in favor of kubernetes!
  • ryanbrunner 4y ago
    > The problem with this sort of argument is that it sets up a straw-man, without describing what you should do instead. Kubernetes could be better - of course - but what better approach is the author recommending to use instead? The article explicitly advocates for high-level PaaS tools as an alternative to Kubernetes.
    • justinsb 4y ago
      Maybe it could be read that way, but I think it's a good indication of the problem that a strict reading doesn't include that. The article advocates that you should be using Heroku or Vercel or Netlify or Fly as not doing so is an antipattern, but it then says that instead of k8s "Most organisations should consider some of the higher-level building blocks available via cloud providers", which seems contradictory. I quite liked the actual message of the article (which I take as "be mindful of your technology choices") - but I think that it is really devalued by the less coherent but more click-baity arguments (e.g. "javascript is the only rational language choice") Edit: Actually, re-reading the article yet again, I think I can see your reading (though I wish you had written the article, as your writing is much clearer). As I understand it, the article says you should use a PaaS-style solution until you outgrow it, when you should move to something from the cloud providers. For me, that would be CloudRun -> GKE-Autopilot -> GKE (skipping CloudRun if you don't want to learn two systems), but there may be some google-bias there!
• ImPleadThe5th 4y ago
  I think it depends on where your "target" architecture is no? If you're doing monolith-first but know you eventually need to move to microservices, it might make sense to put in some effort to learning k8s in the simplest case first and expanding it out. I think the issue with people that use k8s first is that they go for super trendy architectures before really nailing down the business logic first. This leads to technical debt and a lot of confusion around the tool.
• muskmusk 4y ago
  I guess you wanted attention, congratulations you got it. As far as the advice goes. No not really, lots of people are familiar enough with Kubernetes that it makes it the fastest choice possible for them. Engineers like to pretend that tech choices are oh so important and can be discussed in a vacuum, but really most tech choices are made once you know the people who are working on the project. Pick stuff that is familiar to you unless you have a good reason to do otherwise.
• dcolkitt 4y ago
  Disagree if you’re already familiar with K8s. It literally takes 15 minutes to setup a cluster and manifest on GKE
• blown_gasket 4y ago
  Speaking from an infrastructure focused perspective who writes to make the provisioning faster, I understand why software-writing focused individuals like Kubernetes both on-prem and in the cloud. It's super easy to deploy to once the platform is provided to you. On the other hand which is a REQUIREMENT, is to get the platform available no matter the method, it can be a laborious chore that doesn't need to exist unless you need it. on-prem: procuring hardware, installing hardware, installing the host OS/deploy the host virtual machines, networking, deploying K8s, CNI, CSI, etc if cloud: increasing quotas, verifying that CPU sizes are available for more nodes or new node pools, performing virtual networking and subnetting for the K8s clusters, etc. both: security everything from RBAC to networking to image validation, upgrades - you made sure to have a scalable app right? if not fingers crossed on availability. Would the chore exist outside of Kubernetes, of course there would be a chore. I'm not sure so great though. While deploying to Kubernetes is simple. Setting up an on-prem platform is not what-so-ever and if you with public cloud you now have to be aware of all the caveats and gotchas that aren't handcuffs with on-prem.
• gxt 4y ago
  Thank you, I've been struggling with the decision to move away from k8s as I am spending much time making it work, and much more again every time it needs to change, or be updated, when everything is already fine on metal. I just need to find a way to keep or convert the dockerfiles so that it just runs on metal and without docker.
• dmarlow 4y ago
  I often hear a lot about you should vs you shouldn't. How about just highlighting your experience with? Not all companies, people or circumstances are the same. To me, it's akin to comparing athletes and trying to debate who is the goat. Just talk about what you did and let people figure it out. If their startup failed because of it, then they got what they were after, to learn and grow for the next one. Or, maybe to not do one again. I can guarantee you that for every claim you make, there's someone who was successful in doing the opposite.
• mixedCase 4y ago
  I'm getting tired of the "You don't actually need Kubernetes while starting out!" crowd, despite being part of it. Of course you don't. Of course if you don't know Kubernetes, learning it as you try to get a company going on a minimum headcount is not the most efficient approach. But for pete's sakes man, if you have used K8s before, know what you're doing and you're running on cloud, just shove a couple off-the-shelf Terraform modules at the problem and there's your execution environment needs solved, with easy and reliable automations available for handling certs, external access, load balancing, etc, all of it reasonably easy to get going once you have done it before. Stop pretending Kubernetes is this humongous infrastructure investment that mandates a full time job to keep up at low scale. Of course if you have done this multiple times before you don't need to be told this, but people new to it shouldn't be fed a barrage of exaggerations.
  • mirceal 4y ago
    > Stop pretending Kubernetes is this humongous infrastructure investment that mandates a full time job to keep up at low scale. It is a full time job. The cost of using something is not just the setup cost. The same way that software development is not just the cost of writing the software.
    • mixedCase 4y ago
      How? What is keeling over so often on that is not on a simpler setup? I've set up EKS multiple times across different companies with few nodes and everything runs smoothly all of the time. Can you go into detail?
      • thaneross 4y ago
        Indeed. Hosted k8s has been low maintenance in my experience, much lower than managing a bunch of VMs. I think the common thread between people who have horror stories using k8s is either pains from self-hosting or teams who adopted it without experience and used the ample rope it gives to hang themselves.
    • steveBK123 4y ago
• vbezhenar 4y ago
  May be I don't see something. I'm learning about Kube right now and I like it. It's like docker-compose done right. Managing Kube cluster might not be that easy, but that's why managed offerings are there. And you can easily scale with Kube. And by scale I don't necessary mean scale up, because you can scale down as well and that's important to save your precious money. Few other Kube goodies: 1. Plenty of apps are available as helm charts. It's like pre-build docker image, but even better and easier to install and manage (!). 2. There are some marvelous operator like Postgres operators which will do the hard work of managing Postgres database for you, including replication, backups, upgrading. On the down side is the fact that when it broke, you need an expertise... So may be managed database still is better for small scale. I'm coming to kube as someone who spend some time dealing with docker-compose servers. And I feel like kube will be better. For new projects, even on small scale, it's likely that I'll choose kube.
• parkingrift 4y ago
  I just don’t understand this type of article. If you understand how to use and deploy kubernetes and you have confidence in it… you should use it. What you spend in extra infra costs is trivial. …and if you don’t understand how to use and deploy kubernetes just what in the fresh hell are you doing? Stick with technology you know and then move to kubernetes later if or when you need it. We’re a relatively small shop, and we’re using k8s. All of our senior stuff know and understand it. The pipeline is fully automated. If we prematurely optimizing anything it’s developer output over saving money.
• decidertm 4y ago
  There is a lot to agree on in this article. Though teams should be able to choose whatever stack (languages, frameworks, databases, queues) they want. What’s important is there is a coherent way to deploy, build, operate, scale, observe these services, without relying on one or two team members. Getting a cluster setup and a few deployments is pretty simple, however scaling that cluster, setting up secure runtime (network policies, mTLS, Kata, KVM, gVisor), prometheus, CI/CD, preview, staging and production environments is a huge investment (personnel, cloud costs, delay to production delivery). Teams should be able to benefit from Kubernetes and the surrounding cloud-native ecosystem without directly consuming or modifying it. You don’t need to reinvent the wheel, and that’s what you could say a lot of platform teams are doing today. That’s what we’re working on at https://northflank.com https://northflank.com a next-generation deployment platform using Kubernetes either in a secure multi-tenant environment, traditional PaaS or deployed directly into your GKE, EKS and AKS clusters. (disclaimer: Northflank co-founder)
• bane 4y ago
  I remember working with a client in the last 5 years that demanded a Kubernetes cluster to run custom analytics on very fast incoming data streams (several GB per hour). By "custom analytics" I mean, python scripts that loaded a day's worth of data, computed something, wrote the results to disk, and quit. During development of the scripts, the developers/data scientists wrote and tested everything outside of the cluster since they were simple scripts. They had no problem grinding through a day's worth of data in their testing. But going into prod, had to shove it into the cluster. So now we had to maintain the scripts AND the fscking cluster. Why? "What if the data volume increases or we need to run lots of analytics at once?" "You'll still be dominated by I/O overhead and your expected rate of growth in data volume is <3% per year. You can just move to faster disks to more than keep up. Also there's some indexing techniques that would help..." Nope, had to have the cluster. So we had the cluster. At the expense of 10x the hardware, another rack of equipment, a networking guy, and a dedicated cluster admin (with associated service contracts from a support vendor). It literally all ran fine on a single system with lots of RAM and SSDs -- which we proved by replicating all of the tasks the cluster was doing. Argh...
• ezekiel11 4y ago
  With the advent of serverless product offerings, I agree 95% here. The other 5% are on premise, own their own server enterprises, which I know, are rare. It no longer makes sense to spend so much resources on orchestration or distributing across multiple hardware. Not when you can simply write code and have it running globally distributed on edge. You would be doing yourself a disservice to not follow this new paradigm shift.
• AtNightWeCode 4y ago
  The part of Kubernetes and startups is 100% correct I believe. It is very hard for some people to not own a thing. People can be very narrow-minded about buying a service. Even though they at the same time gladly buys work-hours to offload their personal workload. To set up complete envs for dev, test and prod with GIT, full CI/CD and everything you need like databases and storage is less than 20 hours work in a modern cloud. This is something that comes with low maintenance as well. If you think containerization and Kubernetes is a better option, you are basically incompetent when it comes startups.
  • orange-mentor 4y ago
    This person gets it.
• honkycat 4y ago
  Sometimes I wonder what planet these people are coding on. Hosted k8s makes it so easy. The other day I spun up a new eks cluster, created some Iam roles, installed the cert-manager, external dns, and aws alb ingress controller. Then I pushed a deployment, a service, and an ingress and BOOM. Log aggregation, automatic dns, secret handling, ...I can go on. It is simpler than normal aws services in my experience.
• bg24 4y ago
  The author has given 1 specific example: "Another time our Data Science team told us that they needed an orchestration tool for their data pipelines. I steered the selection process towards Argo Workflows (which runs inside Kubernetes) instead of Prefect (a SaaS offering), which they already used for a PoC. There were all sorts of justifications in my head for this decision. Unfortunately, they were all based on premature optimisations. In the end, our team needed to build a new set of Terraform and Helm charts to automate the deployment of Argo Workflows and integrate it into our SSO etc. I regret this decision. I think we lost weeks or even months shipping something to end-users because of this decision. Premature optimisation!" There is absolutely NO issue in adopting kubernetes, and then using SAAS for everything possible. In the above case, if you had a data pipeline proficient engineer in the team, you would not spend months to set up one. It comes down to making decisions based on your current situation (business urgency, team skills, future plans).
• bhauer 4y ago
  While I partially agree with the premise, I perceive the situation a bit differently. By not adequately prioritizing performance at the early technology-selection phase of a development project, teams often find themselves forced to prematurely adopt highly-complex deployment systems and other approaches to scale their low-performance system. In my experience, a well-functioning new team will include performance in their selection process. Doing so allows a project to defer optimizations such as higher-complexity deployment (e.g., cluster orchestration, highly sophisticated caching, and so on) for much longer than those who build on low-performance platforms and frameworks. Choosing a low-performance platform and framework sets an artificially low performance ceiling. Developers dealing with a low performance ceiling will either instinctively (through received or learned patterns) or reactively (through user complaints, issue reports, troubleshooting, and tuning) deal with performance challenges they should not need to deal with so early in a product's lifespan. Ironically, some would say that including performance in your technology selection criteria is "premature optimization," but I argue the opposite: considering performance a feature early allows you to defer costly optimizations (such as the matter at hand, Kubernetes deployment), perhaps even indefinitely.
• funstuff007 4y ago
  What's good back of envelope break-even for when I should move to K8S from a set of hand-rolled scripts and manual processes to manage a bunch of VMs? It definitely feels like more than 10, but how much more than 10? And do people manage database servers using K8S? Or is it better than hand manage those?
• antholeole 4y ago
  First, to respond directly to the post: having a (stable, one you don’t tinker with every other day) k8 setup isn’t bad, its only bad when you have to pay the setup overhead. I built a boilerplate app with: - authentication (Google auth) - cloud run backend -CDK - CICD (GitHub actions) - full-boilerplate frontend - full boilerplate backend - Signal error reporting - Postgres - graphQL - all my other preferences like git hooks, linter rules, vscode workspace, local environment etc. Now, whenever I need to build an app, I just git clone this repo and its plug in credentials and play. My backend cluster doesn’t ever run more than one instance, so I guess having a scalable backend is pointless, but who cares? It’s free, time wise (and almost money wise since cloud run is pretty efficient). I built it a year ago and don’t touch most of the dev ops for any app, except to maybe rip out authentication when I don’t need it. Imo, I think this is a highly effective method of engineering. Everyone has their favorite stack, so it might be worth it you to take a weekend and fully build out your boilerplate app so that if you ever need to, you can get straight to building the actual features.
• nojvek 4y ago
  I feel most places over complicate K8s. The foundational ideas are sane, especially with managed k8s in GCE, AKS, Digital Ocean e.t.c The basic idea is that of "be like this spec" via a yaml/json file - a controller is continuously monitoring an object and making adjustments until spec matches actual. I've seen many startups have one person who sets up and maintains their k8s cluster. If you know what you're doing, it has pretty solid ROI.
• oldsj 4y ago
  > Imagine spending a lot of time and money picking out the best possible gear for a hobby before actually starting the hobby. Are you kidding me that's my favorite part!
• _jezell_ 4y ago
  This post is a red flag signaling the author doesn't know what he's talking about.
• the_scoop 4y ago
  Build for today's needs but have a notional roadmap for how to scale your architecture as the company grows. The author is mostly correct that K8 is complete overkill for most early/mid stage companies and startups, however extensibility is a key architectural concern and one that cannot afford to ignored at the outset. If you need to re-write large amounts of code from scratch every time your business scales up that's a good sign that you aren't thinking far enough ahead. For instance here are some straight-forward things you can probably do today that require little additional effort but will make a migration to k8 much less painful down the road (should you ever get there): 1) Use domain driven design to properly abstract your code 2) Make services and auth stateless where-ever possible/practical 3) Containerize your codebase and leverage a container registry as part of your CI/CD process BLUF always have both a current state architecture as well as a future state architecture in mind and a plan for how to gradually evolve from one to the other over time.