40 ms·

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I wa

• flak48 6y ago
  > We've since migrated to cloud functions (on GCP; but AWS lambdas could also work) and it's just been a breeze. Did you run into cold start delays with GCP cloud functions? AFAIK it is one of the reasons many prefer AWS Lambda over GCP
  • dvt 6y ago
    We actually had a cold start tolerance of ±30 seconds and so far, very few jobs are out of that range. This was one of our main sticking points, and the Google guys gave us some great tips on how to reduce cold starts.
• thrwn_frthr_awy 6y ago
  > This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. I've experienced this as well. At the last large company I worked at we had a Heroku-like system to run our apps on. That was deprecated for a Docker-based solution. And then _that_ has now been deprecated for a Kubernetes offering. We just ran some Python web apps–we didn't want to have to learn and support an entire system. And here's the thing, most big tech companies I've worked with are all made up of these small, "internal" services, that just want a simple place to run their services.
  • croh 6y ago
    Been there. Simple python django app. Max 20 APIs. Expecting 100 requests per month. But POs and directors wanted shiny DevOps tools. Problems with typical MNCs. Every quarter manager/director comes with some new hype.
    • dvcrn 6y ago
      If it's a hosted k8s, all you'd do is containerize your application, then create a deployment that pulls that container and exposes it through a loadbalancer. The container is the same if you'd push it to herokus container registry or beanstalk
      • treeman79 6y ago
        Heroku I configure external items with some environments variables. Often there is a simple command to do it for me. Helm. I’m staring at 20 pages of Yaml. Most of it seems arbitrary. Crap ton of interconnections. Plus defining all iam roles. Doesn’t help that devops team keeps writing custom scripts to interact with it.
        • dvcrn 6y ago
          On k8s, you write those env variables into the deployment manifest that describes the container. If they are sensitive, pack them into a k8s secret and add that to the container instead. You can use the external services the same way if they are already setup with things like DATABASE_URL that holds a postgres connection string to amazon RDS, etc. Running k8s doesn't mean you have to move your entire stack into k8s, you can still use hosted services just fine. In all my time with k8s, I never used Helm. If you don't need it, don't use it and keep it simple. k8s can do a ton of stuff but in reality, you barely need more than the basic.
• outworlder 6y ago
  > so we kept running into dev-ops walls This reads like an oxymoron. 'DevOPS' was supposed to break down walls between teams. You probably have a traditional "Ops" organization, regardless of team naming conventions.
• deleted 6y ago
  [deleted]
• avita1 6y ago
  I agree with most of this, but was surprised by your comment: > Oh, also, the implementation of Kubernetes cron jobs is also complete garbage (spawning a new pod every job is insanely wasteful). How often/how many cron jobs are you running that spawning a new pod per job is a problem ?
  • dvt 6y ago
    The first iteration (I actually wasn't around for that) was trying to run a cron for every "data ingestion job" -- at some points, we were doing about 50k+ API requests daily (FB/Instagram/Twitter/etc.) and that was absolutely not tenable using k8s cronjobs.
    • deleted 6y ago
      [deleted]
    • snuxoll 6y ago
      Why use cronjobs at all for this? This is a classic work queue problem.
      • dvt 6y ago
        I wasn't there for this decision, but I assume cronjobs were being treated as "cloud functions" -- and to be fair, the k8s documentation kind of makes it seem that you could technically do that, but fails miserably if you try to do so in practice.
    • dilyevsky 6y ago
      50k/day is less than 1 qps. This is nothing. This is either not the full story or your cluster was setup completely wrong
      • regularfry 6y ago
        Depends how spiky the distribution was. 20k a second at 2pm? Gonna have a bad time.
• flowerlad 6y ago
  > If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from JavaScript/React to maintaining backend infra, and I couldn't have done it without k8s.
  • lifeisstillgood 6y ago
    Firstly tell me more about your 1-person full-stack venture, but second how comes you, with barely any time for sitting down can use k8s happily but it falls over for others. I am struggling to see truth amount the comments here :-(
    • dashwav 6y ago
      TBF with a completely greenfield project and managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly if you are willing to learn how to do so. I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to have a lot of problems. The big thing to note is that starting a project on Erlang very well might solve the problems that a Java EE project ran into, but that is because they were able to solve them at the ground floor, and just popping a Java EE project with all of it's architecture into an Erlang project will probably end up in a worse spot. I think that this is what often happens with k8s as well - if you or your company have a currently working implementation that isn't on k8s, of course you won't be able to just easily plop it into a k8s cluster and everything be all well and good, but I think the problem is that people are equating that issue with k8s itself, which is a completely different paradigm.
      • solidasparagus 6y ago
        > managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.
• deleted 6y ago
  [deleted]
• whatsmyusername 6y ago
  The 'needing a team' aspect of Kubernetes sounds remarkably similar to conversations I had like 8 years ago when Openstack was the new hotness. We went with ECS and have been happy with it. It plays well with all of AWS's other products and features. For the few things we have to run On-Prem we use Docker Swarm in single node mode and it works well (albeit missing a few features like crons from Kubernetes).
  • threeseed 6y ago
    AWS Kubernetes (EKS) with Fargate is just as simple as ECS. Far simpler if you can include the fact that with Kubernetes you can install new applications as simply as "helm install prometheus".
    • whatsmyusername 6y ago
      Are they still charging something ridiculous for control planes? When I looked at it they were like $200/month. I wasn't aware they were offering fargate on EKS now. They weren't when I looked at it last.
      • dtech 6y ago
        Since December 2019: https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-available/ https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-n...
  • StreamBright 6y ago
    Yep. I remember the Openstack consultants swarming to office trying to sell it as a solution and not a giant overhead / problem. Luckily they failed the POC so we did not need to waste our life on a non-issue trying to solve it with a non-solution. Now it is k8s' time to do the same. We will see how far this buzzword train gonna go.
    • whatsmyusername 6y ago
      TBF I like k8s as a service or platform. Once metallb came out to solve the "bare metal load balancer" problem it could deliver the entire product I was looking for. My big issue with it is the underlying complexity and house of cards nature of running your control plane as sidecar containers on your runtime infrastructure. You CAN set it up and run the control plane out of band, but last I looked there wasn't step by step documentation for doing so. I also couldn't find anyone doing it in prod which to me is a nonstarter. If I can't figure it out myself and can't hire for it I'm not doing it. I'm sure it's fine on google cloud, but ECS solves 90% of our problems AND integrates with everything else we're using already.
• theptip 6y ago
  > > We would need to build/staff a full-time Compute team I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so programmable. The idea that you'd need a completely new full-time team doesn't grok for me; that new team should replace another team that's no longer needed, or more likely, involve a combination of hiring some experts and retraining your existing engineers. I do take seriously the other issues raised RE: security (though one GKE cluster per security boundary is a perfectly reasonable approach and gets you further than you might think). > Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy I think this is a different issue than the OP was raising; in any case, in order for a technology to succeed, you need to have subject matter experts embedded in your dev teams, or a separate function that provides the service to the teams that use it. In the context of the OP, I think your case would be more like saying "hey data team, you need to build your data jobs into AMIs, go figure it out". Regardless of the technology chosen, it's not going to succeed if the teams doing the work don't know how to use the tools.
  • brown9-2 6y ago
    Note that they already have a team who has built their current compute platform, who built the pipeline to run containers/processes on VMs with auto scaling groups. It’s great if their own solution works well for them at less cost, but that system didn’t built itself and has non-zero maintenance costs.
    • rumanator 6y ago
      > Note that they already have a team who has built their current compute platform, who built the pipeline to run containers/processes on VMs with auto scaling groups. In that case I would expect Coinbase to write blog posts on how their setup is the absolute best solution to their problem, and not how they refrain from adopting the best solution to their problem because they claim they don't have anyone on the team that is able to pull that off.
• jupp0r 6y ago
  > I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls This seems like an organizational problem to me. "DevOps walls" sounds like there is a "DevOps team" (a famous DevOps anti-pattern) and there are knowledge silos between development and ops, which ironically is the exact opposite of what DevOps is about. What this also means is that developers need to be aware of the environment in which their services run and should be very familiar with how k8s works, why not take that as an opportunity to learn?
• manigandham 6y ago
  You can't make the complexity disappear. Kubernetes just offers a very standardized, stable and relatively polished way to handle it. The other alternatives, like running your own orchestration system, are just as complex if not more so, although you might be more familiar with it since you built it. That being said, many companies don't need any of it in the first place for their scale, and that's probably the biggest issue with K8S today.
  • yjftsjthsd-h 6y ago
    I'd say it the other way; k8s has to do all the things that another management system would have to do, but sufficiently tied together that you can't ease into it as needed.
    • manigandham 6y ago
      I guess that depends primarily on whether you're installing and operating K8S yourself. Use something like GKE and it's a very seamless experienced, with AKS getting pretty good and the rest being rather crummy. Once you have a managed cluster, the deploying apps is fairly easy. A single container/pod is a 1-liner and you can work your way up from there.
      • yjftsjthsd-h 6y ago
        That's fair; all of this is heavily influenced by your operating environment. If you can run GKE or if you have an ops team to deal with that stuff, then yeah k8s is great. Unfortunately, I'm part of the ops team, and our company is too small to have a dedicated k8s team and too low-budget to (likely) do well with a managed service (we get absurd value per money out of bare metal servers, which is very much a tradeoff that is sometimes painful). So to me, k8s looks like a very iffy tradeoff. Bigger company, bigger budget, different constraints? Yeah, k8s would be great.
        • mwcampbell 6y ago
          > we get absurd value per money out of bare metal servers, which is very much a tradeoff that is sometimes painful What do you find most painful about your use of bare-metal servers? The thing that I like most about a hyperscale cloud provider is the level of redundancy, including even multiple data centers per region, and their built-in health checks and recovery (e.g. through auto-scaling groups) based on that redundancy. With bare-metal servers, I'd have to cobble together my own failover system for the occasional time when one of those servers goes down or becomes unreachable due to a network issue. And of course, I'd probably find out that my home-made failover system doesn't actually work at the worst possible time.
• dvcrn 6y ago
  I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on digitalocean. If I need more power for my cluster, I increase the nodes through digitalocean and k8s automatically moves my containers around how it deems fit. I used normal blue/green deployments on self-managed VMs in the past, then worked with beanstalk, heroku, appengine and I much prefer k8s. Yes it's easier on heroku, but try to run 2-3 different containers on the same dyno for dev to keep cost down. On k8s I can run my entire stack on one single small digitalocean $10 VM if I wanted to. I wouldn't even know what I else could pick that gives me equal flexibility and power?
  • arcticfox 6y ago
    I'm also on a 2-person project on DigitalOcean k8s, also very happy. K8s is kind of messy compared to Heroku, which I don't love, but is also way more powerful and can be more secure. I don't know what I'd use instead of it, exactly as you said. Also, we run a VPC-only K3s node for some simple internal tools that works great as well.
    • dvcrn 6y ago
      > Also, we run a VPC-only K3s node for some simple internal tools that works great as well. We do exactly the same thing! We have a one-node k8s for all these dev things that just works. Everything is containerized for local dev anyway so moving it to k8s was just writing the deployment manifest. On heroku, all of these would be separate dynos (or one glued-together dyno that does everything). On a self-hosted VM we'd have to deal with managing that. I liked this approach so much that I now have a small 1-node personal cluster that hosts all of my private hobby projects that aren't ready for prime time yet, that were on heroku previously. Costs me only $10 + (persistent storage + IP address if needed)
      • meigwilym 6y ago
• 013a 6y ago
  > If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. I think this is roughly similar to saying "if you want linux, you need people that know how to maintain it." Which is to say, you can create an architecture where this is absolutely true, but it doesn't need to be true. The big issue with K8S right now isn't K8S; its that there aren't (big, well established) solutions like Heroku or Zeit or whatever, for K8S, where you don't need to worry about "the cluster", just like those solutions don't make you worry about Linux. K8S really is two parts; the API and the Cluster. The API is the more valuable of the two. And, you know, maybe it won't ever get there. Heroku and Zeit solve strikingly similar problems to K8S. Maybe K8S just is a platform like that, but for enterprises who want to home-grow, and maybe most companies shouldn't worry about it. But I think the platform, and thus the community, simply needs more time to figure out where it makes sense. Most companies shouldn't touch K8S. You'll probably regret it. But, to your second point: AWS literally has nothing beyond EKS/ECS + Fargate which approaches a Heroku-like service. Beanstalk is supposed to be that, but its really just a layer on-top of EC2 which doesn't touch the "ultra low maintenance" of a Heroku, or Zeit, or App Engine. So if you're on AWS, and you want to use their other excellent managed services, you either go outside AWS, or you'll go EKS, or you'll end up trying to in-house something even worse.
• harpratap 6y ago
  I have said this multiple times in the past (https://news.ycombinator.com/item?id=23361176 https://news.ycombinator.com/item?id=23361176 https://news.ycombinator.com/item?id=23243626 https://news.ycombinator.com/item?id=23243626) and will say it again - You (business logic developers) are not meant to use K8s directly, you are supposed to use a PaaS built on top of K8s (like Cloud Functions, Lambda etc)
  • Lx1oG-AWb6h_ZG0 6y ago
    But this makes no sense. Why would a cloud operator even bother with k8s if the customer is only interacting with functions? It’s much more efficient to bypass Kubernetes and run directly on the cloud’s native system (like borg). I think you’re right though - Kubernetes is a massive red herring, we should ideally be running containers/functions on as close to bare metal as possible. Fundamentally, VMs are the wrong abstraction if all your code is containerized. Joyent’s triton is the closest thing we have to this... I really don’t know why AWS/Azure/GCP haven’t cottened onto this, it would massively reduce their COGS and improve our developer experience.
    • harpratap 6y ago
      > Why would a cloud operator even bother with k8s if the customer is only interacting with functions? Because there are various "tiers" of users, some companies (like coinbase) could actually leverage K8s in their Codeflow/Odin project and prevent a lock-in. But a regular developer looking to just "get things shipped" isn't meant to waste his/her time with pure K8s. > Kubernetes is a massive red herring We agree but on a different note. The biggest selling point of K8s is it's API design. The entire industry needs to converge on one "defacto" standard of packing and deployment. Google's Cloud function is a perfect example of this. The API is based on K8s and Knative but under the hood it actually runs directly on GCE rather than GKE. What happens underneath is hidden from business developer, you only care about the data in your yaml and your docker image.
    • StreamBright 6y ago
      >> I really don’t know why AWS/Azure/GCP haven’t cottened onto this Conflict of interest. If k8s yields to the most revenue why would they try to decrease that? If some customers are so delusional that they go for an inefficient abstraction so be it. Btw. this is my experience with k8s too, people use it because it is a trend. Not a single company / developer could justify using it to me over leaner resources like EC2, ASG, cloud native resources, etc.
• rhizome 6y ago
  Your comment and the post and threads in general are making me chuckle, having just spent the last half hour reading this post, for no particular reason, from a few months ago about the transformation of racknut-and-perl roles like neteng and sysadmin into text editor-based devops: https://news.ycombinator.com/item?id=22508968 https://news.ycombinator.com/item?id=22508968 I have to say, certain comments -- which I'm sure are just as real-world as yours -- lie in...let's call it tension...with the threads and comments here in this post. That everything in the old post appears to be subsumed into a "sysadmin II: internet boogaloo" of orchestrator expertise is humorously ironic.
• brianwawok 6y ago
  Counterpoint: Run 11 person startup. Use hosted GKE. We spend less than 1 man-hour per week dealing with K8s or anything like that. K8S is a big reason we are able to out-execute our competition.
  • bvm 6y ago
    I agree with you completely, we use Kops and see a similar workload. The real boon for us is not in production, where the HA/error-tolerant/easy horizontal scaling certainly helps, but in development, where we can easily bring up ephemeral feature branch deploys as part of the CI/CD pipeline (which itself runs on k8s using Gitlab CI)
    • brianwawok 6y ago
      Yah. I mean, k8s has 1000s of features. I am not using every one of them. I carved out the subset I need, and it is FANTASTIC. I get * Great rolling deploys * Self healing clusters * Resource sharing / bin packing (rather than have 20 half used servers, I can have 6 much more heavily utilized servers) I could maybe frankenstein these features on top of something else.. but frankly it seems absurd. If these are the only 3 features I use to run docker pods, k8s is a HUGE win to me.
  • donedealomg 6y ago
    You could do the same with AWS ECS there is nothing special about K8s in your situation.
    • brianwawok 6y ago
      I do not use AWS ECS does not bin pack, does it?
      • notaharvardmba 6y ago
        https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-strategies.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
• Thev00d00 6y ago
  GKE is fantastic compared to EKS.
• omginternets 6y ago
  What did the typical "devops wall" look like? Can you recall any specific examples?
• tootie 6y ago
  Did you evaluate Cloud Run/Fargate for deploying the same Docker images without k8s?
• jillesvangurp 6y ago
  It's a pity that docker swarm did not make it. It wasn't perfect but it was a lot simpler to setup and manage than kubernetes. If you can get away with it, vanilla docker hosts running docker compose provide most of the same benefits with a fraction of the cost. For most startups, that's a great way to avoid getting sucked into a black hole of non value adding devops activity. You lose some flexibility but vanilla ubuntu hosts with vanilla docker installs are easy to setup and manage. We used packer and ansible to build amis for this a few years ago with some very minimal scripts for container deploys.
  • santamarias 6y ago
    > It's a pity that docker swarm did not make it. Sorry I do not understand that statement, in my naive opinion Docker Swarm seems to be a thing. Care to elaborate, please?
    • jillesvangurp 6y ago
      It exists but in terms of people using it or it being actively developed, it's dead as a doornail ever since Docker was more less forced to also support kubernetes and basically gave in to the reality that world + dog was opting for kubernetes instead of swarm. They never really retired it but at this point it's a footnote in Docker releases. I've not actually encountered it in the wild in four years or so and never in a production setup.
    • silvester23 6y ago
      It is, but at this point it is unclear for how long Docker Swarm will be supported, see e.g. https://boxboat.com/2019/12/10/migrate-docker-swarm-to-kubernetes/ https://boxboat.com/2019/12/10/migrate-docker-swarm-to-kuber... We are actually currently in the process of migrating from Docker Swarm to k8s and I am not 100% sure that's a good idea. We will see.
      • patricjansson 6y ago
        "conversations have led us to the conclusion that our customers want continued support of Swarm without an implied end date." https://www.mirantis.com/blog/mirantis-will-continue-to-support-and-develop-docker-swarm/ https://www.mirantis.com/blog/mirantis-will-continue-to-supp...
• bule0bird 6y ago
  I had a similar problem, setup a small cluster and it worked fine until next upgrade and nightmare repeated on each upgrade. I realize spending most of my time on fixing k8s issues rather than doing any valuable work
• maurys 6y ago
  I've found that ECS on AWS works great for similar use-cases. I guess it gets more love than EKS on AWS. Seems like EKS is great for people to start looking at AWS without vendor lock-in, but then people switch over to ECS and/or Lambdas for compute.
• tylermenezes 6y ago
  Nomad + Consul has been pretty great to us over the last year. We're a small nonprofit and chose it specifically because we can't afford to pay someone to keep watch over k8s
  • deleted 6y ago
    [deleted]
• ricksharp 6y ago
  I don’t understand the value of container tech (unless dealing with legacy code). Serverless seems better and cheaper (lambda, GC Functions, Azure Functions). Pay as you go with no overhead costs. In many cases usage may never exceed the free credits. For the database and persistent storage, you need a cloud native service like S3 and Postgres RDS. (Running a database in a container is asking for complete data loss from what I understand.) This is the primary cost for my tech stack and for lite loads $15 is about as minimal as possible. So I don’t see how using containers could be cheaper unless you are keeping the data in the container, which is a bad idea since containers are supposed to be stateless. So serverless cloud native tech seems better by far from everything I’ve experienced. Also when using a tool like serverless framework, deploying an entire stack of serverless resources is dead simple. I’ve tried a few times to see the value of containers, but I just don’t get it.