10 ms·

For what it's worth, I've worked at multiple places that ran shell scripts just fine for their deploys. - One had only 2 services [php] and ran over 1 billion

For what it's worth, I've worked at multiple places that ran shell scripts just fine for their deploys. - One had only 2 services [php] and ran over 1 billion requests a day. Deploy was trivial, ssh some new files to the server and run a migration, 0 downtime. - One was in an industry that didn't need "Webscale" (retirement accounts). Prod deploys were just docker commands run by jenkins. We ran two servers per service from the day I joined the day I left 4 years later (3x growth), and ultimately removed one service and one database during all that growth. Another outstanding thing about both of these places was that we had all the testing environments you need, on-demand, in minutes. The place I'm at now is trying to do kubernetes and is failing miserably (ongoing nightmare 4 months in and probably at least 8 to go, when it was allegedly supposed to only take 3 total). It has one shared test environment that it takes 3-hours to see your changes in. I don't fault kubernetes directly, I fault the overall complexity. But at the end of the day kubernetes feels like complexity trying to abstract over complexity, and often I find that's less successful that removing complexity in the first place.

leetrout 2y ago
Yea but that doesn't sound shiny on your resume.
- nine_k 2y ago
  Depends on what kind of company you want to join. Some value simplicity and efficiency more.
- txutxu 2y ago
  I never did choose any single thing in my job, just because of how it could look in my resume. After +20 years of Linux sysadmin/devops, and because a spinal disc herniation last year, now I'm looking for a job. 99% of job offers, will ask for EKS/Kubernetes now. It's like the VMware of the years 200[1-9], or like the "Cloud" of the years 201[1-9]. I've always specialized in physical datacenters and servers, being it on-premises, colocation, embedded, etc... so I'm out of the market now, at least in Spain (which always goes like 8 years behind the market). You can try to avoid it, and it's nice when you save thousands of operational/performance/security/etc issues and dollars to your company across the years, and you look like a guru that goes ahead of industry issue to your boss eyes, but, it will make finding a job... 99% harder. It doesn't matter if you demonstrate the highest level on Linux, scripting, ansible, networking, security, hardware, performance tuning, high availability, all kind of balancers, switching, routing, firewalls, encryption, backups, monitoring, log management, compliance, architecture, isolation, budget management, team management, provider/customer management, debugging, automation, programming full stack, and a long etc. If you say "I never worked with Kubernetes, but I learn fast", with your best sincerity at the interview, then you're automatically out of the process. No matter if you're talking with human resources, a helper of the CTO, or the CTO. You're out.
  - aguaviva 2y ago
    If you say "I never worked with X, but I learn fast", with your best sincerity at the interview, then you're automatically out of the process. Where X can be not just k8s but any other bullet point on the job req. It's interesting that the very things that people used to say to get the job 20 years ago -- and not as a plattitude (it's a perfectly reasonable and intelligent thing to say, and in a rational world, exactly what one would hope to hear from a candidate) -- are now considered as red flags that immediately disqualify one for the job. Very sorry to hear about your current situation - best of luck.
    - Melatonic 2y ago
      Ive never heard of this - has this been your direct experience?
      - aguaviva 2y ago
        It's somewhat speculative (because no one ever tells you the reason for dropping your application or not contacting you in the first place) but the impression I have, echoed by what many others seem to be saying, is that the process has shifted greatly from "Is this a strong, reliable, motivated person?" (with toolchain overlap being mostly gravy) to "Do they have 5-8 recent years of X, Y and Z?". As if years of doing anything is a reliable predictor of anything, or can even be effectively measured.
loftsy 2y ago
Are you self hosting kubernetes or running it managed? I've only used it managed. There is a bit of a learning curve but it's not so bad. I can't see how it can take 4 months to figure it out.
- zug_zug 2y ago
  We are using EKS > I can't see how it can take 4 months to figure it out. Well have you ever tried moving a company with a dozen services onto kubernetes piece-by-piece, with zero downtime? How long would it take you to correctly move and test every permission, environment variable, and issue you run into? Then if you get a single setting wrong (e.g. memory size) and don't load-test with realistic traffic, you bring down production, potentially lose customers, and have to do a public post-mortem about your mistakes? [true story for current employer] I don't see how anybody says they'd move a large company to kubernetes in such an environment in a few months with no screwups and solid testing.
  - tail_exchange 2y ago
    It largely depends how customized each microservice is, and how many people are working on this project. I've seen migrations of thousands of microservices happening with the span of two years. Longer timeline, yes, but the number of microservices is orders of magnitude larger. Though I suppose the organization works differently at this level. The Kubernetes team build a tool to migrate the microservices, and each owner was asked to perform the migration themselves. Small microservices could be migrated in less than three days, while the large and risk-critical ones took a couple weeks. This all happened in less than two years, but it took more than that in terms of engineer/weeks. The project was very successful though. The company spends way less money now because of the autoscaling features, and the ability to run multiple microservices in the same node. Regardless, if the company is running 12 microservices and this number is expected to grow, this is probably a good time to migrate. How did they account for the different shape of services (stateful, stateless, leader elected, cron, etc), networking settings, styles of deployment (blue-green, rolling updates, etc), secret management, load testing, bug bashing, gradual rollouts, dockerizing the containers, etc? If it's taking 4x longer than originally anticipated, it seems like there was a massive failure in project design.
YZF 2y ago
If your application doesn't need and likely won't need to scale to large clusters, or multiple clusters, then there's nothing wrong per se. with your solution. I don't think k8s is that hard but there are a lot of moving pieces and there's a bit to learn. Finding someone with experience to help you can make a ton of difference. Questions worth asking: - Do you need a load balancer? - TLS certs and rotation? - Horizontal scalability. - HA/DR - dev/stage/production + being able to test/stage your complete stack on demand. - CI/CD integrations, tools like ArgoCD or Spinnaker - Monitoring and/or alerting with Prometheus and Grafana - Would you benefit from being able to deploy a lot of off the shelf software (lessay Elastic Search, or some random database, or a monitoring stack) via helm quickly/easily. - "Ingress"/proxy. - DNS integrations. If you answer yes to many of those questions there's really no better alternative than k8s. If you're building large enough scale web applications the almost to most of these will end up being yes at some point.
- hkkwritesgmail 2y ago
  If I need "some" of the above, Kubernetes forces me to grapple with "all" of the above. I think that is the issue.
- probablybetter 2y ago
  Containerization and orchestration of containers vs learning how to configure HaProxy, how to use Certbot, hmmmm The questions you pose are legit skills web developers need to have. Nothing you mentioned is obviated by K8s or containerization. "oh but you can get someone elses pre-configured image" uh huh... sure, you can also install malware. You will also need to one day maintain or configure the software running in them. You may even need to address issues your running software causes. You can't do that without mastering the software you are running!
- zug_zug 2y ago
  > If you answer yes to many of those questions there's really no better alternative than k8s. Nah, most of that list is basically free for any company that uses an amazon loadbalancer and an autoscale group. In terms of likeliness of incidents, time, and cost, those will each be an order of magnitude higher with a team of kubernetes engineers than less complex setup.
deleted 2y ago
[deleted]
a2tech 2y ago
People really underestimate the power of a shell scripts and ssh and trusted developers.
- stevefan1999 2y ago
  Besides the fact that shell scripts aren't scalable (in terms of horizontal scalability like actor model), I would also like to point out that shell scripts should be simple, but if you want to handle something that big, you essentially and definitely is using it as a programming language in disguise -- not ideal and I would like to go Go or Rust instead.
  - dgfitz 2y ago
    > Besides the fact that shell scripts aren't scalable… What are you trying to say there? My understanding is that, way under the hood, a set of shell scripts is in fact enabling the scalable nature of… the internet.
    - stevefan1999 2y ago
      ...that's only for early internet, and the early internet is effing broken at best
    - ChoHag 2y ago
      [dead]
    - lmm 2y ago
      > My understanding is that, way under the hood, a set of shell scripts is in fact enabling the scalable nature of… the internet. I sure hope not. The state of error handling in shell scripts alone is enough to disqualify them for serious production systems. If you're extremely smart and disciplined it's theoretically possible to write a shell script that handles error states correctly. But there are better things to spend your discipline budget on.
      - 2y ago
nouripenny 2y ago
I think porting to k8s can succeed or fail, like any other project. I switched an app that I alone worked on, from Elastic Beanstalk (with Bash), to Kubernetes (with Babashka/Clojure). It didn't seem bad. I think k8s is basically a well-designed solution. I think of it as a declarative language which is sent to interpreters in k8s's control plane. Obviously, some parts of took a while to figure out. For example, I needed to figure out an AWS security group problem with Ingress objects, that I recall wasn't well-documented. So I think parts of that declarative language can suck, if the declarative parts aren't well factored-out from the imperative parts. Or if the log messages don't help you diagnose errors, or if there isn't some kind of (dynamic?) linter that helps you notice problems quickly In your team's case, more information seems needed to help us evaluate the problems. Why was it easier before to make testing environments, and harder now?
starttoaster 2y ago
On the other hand, my team slapped 3 servers down in a datacenter, had each of them configured in a Proxmox cluster within a few hours. Some 8-10 hours later we had a fully configured kubernetes cluster running within Proxmox VMs, where the VMs and k8s cluster are created and configured using an automation workflow that we have running in GitHub Actions. An hour or two worth of work later we had several deployments running on it and serving requests. Kubernetes is not simple. In fact it's even more complex than just running an executable with your linux distro's init system. The difference in my mind is that it's more complex for the system maintainer, but less complex for the person deploying workloads to it. And that's before exploring all the benefits of kubernetes-ecosystem tooling like the Prometheus operator for k8s, or the horizontally scalable Loki deployments, for centrally collecting infrastructure and application metrics, and logs. In my mind, making the most of these kinds of tools, things start to look a bit easier even for the systems maintainers. Not trying to discount your workplace too much. But I'd wager there's a few people that are maybe not owning up to the fact that it's their first time messing around with kubernetes.
- regularfry 2y ago
  As long as your organisation can cleanly either a) split the responsibility for the platform from the responsibility for the apps that run on it, and fund it properly, or b) do the exact opposite and accommodate all the responsibility for the platform into the app team, I can see it working. The problems start when you're somewhere between those two points. If you've got a "throw it over the wall to ops" type organisation, it's going to go bad. If you've got an underfunded platform team so the app team has to pick up some of the slack, it's going to go bad. If the app team have to ask permission from the platform team before doing anything interesting, it's going to go bad. The problem is that a lot of organisations will look at k8s and think it means something it doesn't. If you weren't willing to fund a platform team before k8s, I'd be sceptical that moving to it is going to end well.
arkh 2y ago
So, my current experience somewhere most old apps are very old school: - most server software is waaaaaaay out of date so getting a dev / test env is a little harder (like last problem we got was the HAproxy version does not do ECDA keys for ssl certs, which is the default with certbot) - yeah pushing to prod is "easy": FTP directly. But now which version of which files are really in prod? No idea. Yeah when I say old school it's old school before things like Jenkins. - need something done around the servers? That's the OPS team job. Team which also has too much different work to do so now you'll have to wait a week or two for this simple "add an upload file" endpoint to this old API because you need somewhere to put those files. Now we've started setting up some on-prem k8s nodes for the new developments. Not because we need crazy scaling but so the dev team can do most OPS they need. It takes time to have everything setup but once it started chugging along it felt good to be able to just declare whatever we need and get it. You still need to get the devs to learn k8s which is not fun but that's the life of a dev: learning new things every day. Also k8s does not do data. You want a database or anything managing files: you want to do most of the job outside k8s.

leetrout 2y ago

Yea but that doesn't sound shiny on your resume.

nine_k 2y ago
Depends on what kind of company you want to join. Some value simplicity and efficiency more.
txutxu 2y ago
I never did choose any single thing in my job, just because of how it could look in my resume. After +20 years of Linux sysadmin/devops, and because a spinal disc herniation last year, now I'm looking for a job. 99% of job offers, will ask for EKS/Kubernetes now. It's like the VMware of the years 200[1-9], or like the "Cloud" of the years 201[1-9]. I've always specialized in physical datacenters and servers, being it on-premises, colocation, embedded, etc... so I'm out of the market now, at least in Spain (which always goes like 8 years behind the market). You can try to avoid it, and it's nice when you save thousands of operational/performance/security/etc issues and dollars to your company across the years, and you look like a guru that goes ahead of industry issue to your boss eyes, but, it will make finding a job... 99% harder. It doesn't matter if you demonstrate the highest level on Linux, scripting, ansible, networking, security, hardware, performance tuning, high availability, all kind of balancers, switching, routing, firewalls, encryption, backups, monitoring, log management, compliance, architecture, isolation, budget management, team management, provider/customer management, debugging, automation, programming full stack, and a long etc. If you say "I never worked with Kubernetes, but I learn fast", with your best sincerity at the interview, then you're automatically out of the process. No matter if you're talking with human resources, a helper of the CTO, or the CTO. You're out.
- aguaviva 2y ago
  If you say "I never worked with X, but I learn fast", with your best sincerity at the interview, then you're automatically out of the process. Where X can be not just k8s but any other bullet point on the job req. It's interesting that the very things that people used to say to get the job 20 years ago -- and not as a plattitude (it's a perfectly reasonable and intelligent thing to say, and in a rational world, exactly what one would hope to hear from a candidate) -- are now considered as red flags that immediately disqualify one for the job. Very sorry to hear about your current situation - best of luck.
  - Melatonic 2y ago
    Ive never heard of this - has this been your direct experience?
    - aguaviva 2y ago
      It's somewhat speculative (because no one ever tells you the reason for dropping your application or not contacting you in the first place) but the impression I have, echoed by what many others seem to be saying, is that the process has shifted greatly from "Is this a strong, reliable, motivated person?" (with toolchain overlap being mostly gravy) to "Do they have 5-8 recent years of X, Y and Z?". As if years of doing anything is a reliable predictor of anything, or can even be effectively measured.

loftsy 2y ago

Are you self hosting kubernetes or running it managed? I've only used it managed. There is a bit of a learning curve but it's not so bad. I can't see how it can take 4 months to figure it out.

zug_zug 2y ago
We are using EKS > I can't see how it can take 4 months to figure it out. Well have you ever tried moving a company with a dozen services onto kubernetes piece-by-piece, with zero downtime? How long would it take you to correctly move and test every permission, environment variable, and issue you run into? Then if you get a single setting wrong (e.g. memory size) and don't load-test with realistic traffic, you bring down production, potentially lose customers, and have to do a public post-mortem about your mistakes? [true story for current employer] I don't see how anybody says they'd move a large company to kubernetes in such an environment in a few months with no screwups and solid testing.
- tail_exchange 2y ago
  It largely depends how customized each microservice is, and how many people are working on this project. I've seen migrations of thousands of microservices happening with the span of two years. Longer timeline, yes, but the number of microservices is orders of magnitude larger. Though I suppose the organization works differently at this level. The Kubernetes team build a tool to migrate the microservices, and each owner was asked to perform the migration themselves. Small microservices could be migrated in less than three days, while the large and risk-critical ones took a couple weeks. This all happened in less than two years, but it took more than that in terms of engineer/weeks. The project was very successful though. The company spends way less money now because of the autoscaling features, and the ability to run multiple microservices in the same node. Regardless, if the company is running 12 microservices and this number is expected to grow, this is probably a good time to migrate. How did they account for the different shape of services (stateful, stateless, leader elected, cron, etc), networking settings, styles of deployment (blue-green, rolling updates, etc), secret management, load testing, bug bashing, gradual rollouts, dockerizing the containers, etc? If it's taking 4x longer than originally anticipated, it seems like there was a massive failure in project design.

YZF 2y ago

If your application doesn't need and likely won't need to scale to large clusters, or multiple clusters, then there's nothing wrong per se. with your solution. I don't think k8s is that hard but there are a lot of moving pieces and there's a bit to learn. Finding someone with experience to help you can make a ton of difference. Questions worth asking: - Do you need a load balancer? - TLS certs and rotation? - Horizontal scalability. - HA/DR - dev/stage/production + being able to test/stage your complete stack on demand. - CI/CD integrations, tools like ArgoCD or Spinnaker - Monitoring and/or alerting with Prometheus and Grafana - Would you benefit from being able to deploy a lot of off the shelf software (lessay Elastic Search, or some random database, or a monitoring stack) via helm quickly/easily. - "Ingress"/proxy. - DNS integrations. If you answer yes to many of those questions there's really no better alternative than k8s. If you're building large enough scale web applications the almost to most of these will end up being yes at some point.

hkkwritesgmail 2y ago
If I need "some" of the above, Kubernetes forces me to grapple with "all" of the above. I think that is the issue.
probablybetter 2y ago
Containerization and orchestration of containers vs learning how to configure HaProxy, how to use Certbot, hmmmm The questions you pose are legit skills web developers need to have. Nothing you mentioned is obviated by K8s or containerization. "oh but you can get someone elses pre-configured image" uh huh... sure, you can also install malware. You will also need to one day maintain or configure the software running in them. You may even need to address issues your running software causes. You can't do that without mastering the software you are running!
zug_zug 2y ago
> If you answer yes to many of those questions there's really no better alternative than k8s. Nah, most of that list is basically free for any company that uses an amazon loadbalancer and an autoscale group. In terms of likeliness of incidents, time, and cost, those will each be an order of magnitude higher with a team of kubernetes engineers than less complex setup.

deleted 2y ago

[deleted]

a2tech 2y ago

People really underestimate the power of a shell scripts and ssh and trusted developers.

stevefan1999 2y ago
Besides the fact that shell scripts aren't scalable (in terms of horizontal scalability like actor model), I would also like to point out that shell scripts should be simple, but if you want to handle something that big, you essentially and definitely is using it as a programming language in disguise -- not ideal and I would like to go Go or Rust instead.
- dgfitz 2y ago
  > Besides the fact that shell scripts aren't scalable… What are you trying to say there? My understanding is that, way under the hood, a set of shell scripts is in fact enabling the scalable nature of… the internet.
  - stevefan1999 2y ago
    ...that's only for early internet, and the early internet is effing broken at best
  - ChoHag 2y ago
    [dead]
  - lmm 2y ago
    > My understanding is that, way under the hood, a set of shell scripts is in fact enabling the scalable nature of… the internet. I sure hope not. The state of error handling in shell scripts alone is enough to disqualify them for serious production systems. If you're extremely smart and disciplined it's theoretically possible to write a shell script that handles error states correctly. But there are better things to spend your discipline budget on.
    - 2y ago

nouripenny 2y ago

I think porting to k8s can succeed or fail, like any other project. I switched an app that I alone worked on, from Elastic Beanstalk (with Bash), to Kubernetes (with Babashka/Clojure). It didn't seem bad. I think k8s is basically a well-designed solution. I think of it as a declarative language which is sent to interpreters in k8s's control plane. Obviously, some parts of took a while to figure out. For example, I needed to figure out an AWS security group problem with Ingress objects, that I recall wasn't well-documented. So I think parts of that declarative language can suck, if the declarative parts aren't well factored-out from the imperative parts. Or if the log messages don't help you diagnose errors, or if there isn't some kind of (dynamic?) linter that helps you notice problems quickly In your team's case, more information seems needed to help us evaluate the problems. Why was it easier before to make testing environments, and harder now?

starttoaster 2y ago

On the other hand, my team slapped 3 servers down in a datacenter, had each of them configured in a Proxmox cluster within a few hours. Some 8-10 hours later we had a fully configured kubernetes cluster running within Proxmox VMs, where the VMs and k8s cluster are created and configured using an automation workflow that we have running in GitHub Actions. An hour or two worth of work later we had several deployments running on it and serving requests. Kubernetes is not simple. In fact it's even more complex than just running an executable with your linux distro's init system. The difference in my mind is that it's more complex for the system maintainer, but less complex for the person deploying workloads to it. And that's before exploring all the benefits of kubernetes-ecosystem tooling like the Prometheus operator for k8s, or the horizontally scalable Loki deployments, for centrally collecting infrastructure and application metrics, and logs. In my mind, making the most of these kinds of tools, things start to look a bit easier even for the systems maintainers. Not trying to discount your workplace too much. But I'd wager there's a few people that are maybe not owning up to the fact that it's their first time messing around with kubernetes.

regularfry 2y ago
As long as your organisation can cleanly either a) split the responsibility for the platform from the responsibility for the apps that run on it, and fund it properly, or b) do the exact opposite and accommodate all the responsibility for the platform into the app team, I can see it working. The problems start when you're somewhere between those two points. If you've got a "throw it over the wall to ops" type organisation, it's going to go bad. If you've got an underfunded platform team so the app team has to pick up some of the slack, it's going to go bad. If the app team have to ask permission from the platform team before doing anything interesting, it's going to go bad. The problem is that a lot of organisations will look at k8s and think it means something it doesn't. If you weren't willing to fund a platform team before k8s, I'd be sceptical that moving to it is going to end well.

arkh 2y ago

So, my current experience somewhere most old apps are very old school: - most server software is waaaaaaay out of date so getting a dev / test env is a little harder (like last problem we got was the HAproxy version does not do ECDA keys for ssl certs, which is the default with certbot) - yeah pushing to prod is "easy": FTP directly. But now which version of which files are really in prod? No idea. Yeah when I say old school it's old school before things like Jenkins. - need something done around the servers? That's the OPS team job. Team which also has too much different work to do so now you'll have to wait a week or two for this simple "add an upload file" endpoint to this old API because you need somewhere to put those files. Now we've started setting up some on-prem k8s nodes for the new developments. Not because we need crazy scaling but so the dev team can do most OPS they need. It takes time to have everything setup but once it started chugging along it felt good to be able to just declare whatever we need and get it. You still need to get the devs to learn k8s which is not fun but that's the life of a dev: learning new things every day. Also k8s does not do data. You want a database or anything managing files: you want to do most of the job outside k8s.