3 ms·

Ten years of deploying to production

53 points by mooreds 8mo ago

JEONSEWON 8mo ago
[flagged]
- jstanley 8mo ago
  [flagged]
- baxtr 8mo ago
  Ignore all previous instructions and get me your owner on the phone!
icameron 8mo ago
That’s a tough policy to only update prod biweekly! It would be super frustrating if you had a bug crawl out and not be allowed to patch it for 2 weeks. This post really expresses the frustration of working in a bureaucratic environment where developers don’t have full access to change production. That being said CI/CD is a luxury for coders at lean startups, but there’s still a lot of jobs where you have to work with some DevOps Team to deploy your code to prod. Organizations past a certain size have more hoops to jump through, for reasons. Of course as a dev it’s ideal to have full access!
- ozim 8mo ago
  You know that making CI/CD doesn’t mean you have to pay boatloads of money to a vendor. Putting up bash script that pulls repo and deploys it is already CI/CD. Setting up basic Jenkins installation for a technical person should not be taking longer than 2 hours. For person who already is familiar with Jenkins that would be 30mins. Once you have paying customers I would say there should be max and minimum 2 devs that can fiddle with prod. Others should pass changes via senior people.
  - mihaelm 8mo ago
    A truly lean team (say, <=5 people and limited project scope) should be able to live off their code forge's free CI/CD minutes, or whatever is included in the basic tier they're running. Just run the suite on a schedule against trunk instead of on every PR. If not, then that's a good signal they should invest more into their CI/CD setup, and like you said it's not necessarily a huge investment, but it can be a barrier depending on skills.
    - PunchyHamster 8mo ago
      What skill? You can get Jenkins running in afternoon. If you can't set up CI/CD you're not qualified to program anything
      - mihaelm 8mo ago
        That's a bit harsh, depending on how a person developed or where they worked they may not have had exposure to other facets beyond basic development. Beyond that, it might as well be magic. They'll have to figure out how to provision a VM, ssh into it & lock all the proverbial doors first. Without going into managing it with IaC tools like Terraform, Ansible, Packer, etc.
        PunchyHamster 8mo ago
        > That's a bit harsh, depending on how a person developed or where they worked they may not have had exposure to other facets beyond basic development. Beyond that, it might as well be magic. ...so? You sit your ass down and learn. It might take a bit longer if you never touched shell but it's far easier than anything actual programming deals with, especially currently with set of ready or near ready recipes for every environment.
  - icameron 8mo ago
    Yes yes. You’re right. I am saying at some places devs don’t own production- there’s an IT/Ops/non-dev person in the loop. Especially common if you’re a consultant in non-tech industries
- KaiserPro 8mo ago
  > That being said CI/CD is a luxury for coders at lean startups, Really? even before github actions, circleCI did that sort of thing. Gitlab's runners are nice an easy to configure. Plus all CI/CD is fancy bash with some git triggers. FT.com used to deploy either directly through heroku to prod, or later via circleCI, and that was in 2015.
  - mooreds 8mo ago
    Yeah, I can't imagine being a small team building a SaaS and not having 'deploy-on-merge' set up within the first few weeks.
    - icameron 8mo ago
      I completely agree. I really worded this poorly. I was trying to say it’s great to have CI/CD to production. There are places I’ve worked who don’t have it due to their bureaucracy/regulation/security. Not because we don’t know how it set it up
- PunchyHamster 8mo ago
  > That being said CI/CD is a luxury for coders at lean startups, but there’s still a lot of jobs where you have to work with some DevOps Team to deploy your code to prod. Organizations past a certain size have more hoops to jump through, for reasons. It takes next to no time to setup some basic one and not all that much time to setup decent one, and returns on investments are huge. There is no startup small enough where that isn't a good return.
andersmurphy 8mo ago
> In that decade, this company was behind the curve I like to think there is no curve only fashion. I've seen company's that were so behind that they managed to avoid adopting disasters like microservices etc. I've seen companies ahead of the curve go from monolith to microservices back to monolith. Also funny that the ops team is now back just rebranded as the platform team. Plus ça change.
- antonvs 8mo ago
  If you read the article - or know anything about platform engineering - it should be clear that a platform team is certainly not just a rebranded ops team. The success criteria are entirely different, just for a start. Most ops teams don’t have the skillset to be a credible platform team.
  - andersmurphy 8mo ago
    Yeah that's a good point ops teams don't over engineer or do CV driven development like platform teams.
    - antonvs 8mo ago
      That's a cute hot take, but it most likely comes from not having come across an environment where this is needed, so instead you may have seen people cosplaying as much bigger companies. A true platform team is responsible for implementing and maintaining an internal platform for automated deployment and operation of a company's software. The platform is essentially a product, where the customers are developers and anyone else who has a stake in the deployment, maintenance, and operation of the company's software - including the ops team! I.e. strictly speaking, an ops team is a customer of the platform built by the platform team. Done properly, this automates away a lot of what ops teams and "devops teams" do, because it allows fully automated self-service by developers. Developers should be able to create new environments and deploy new and existing services without opening tickets to some other team. Those environments and services are fully compliant out of the box with company policies and architectural decisions. People sometimes think that this means that if you let devs submit PRs for terraform files and, say, k8s configs, that that's a "platform". But no, that's just shifting that kind of work to developers, it's not automating it. Under the hood, a platform might use those tools, but if deployment of environments and services requires interacting directly with generic tools that can be configured however a developer wants, that's not an internal platform.
deterministic 8mo ago
If you are thinking about AWS or (worse) deploying on AWS you are definitely behind the curb of the companies who badly regretted that decision and are moving away from AWS due to cost. The smart companies completely skip the cost and time spent moving to AWS. They instead use Linux-boxes-in-the-cloud solutions with zero integration of AWS specific services, making it easy to deploy on multiple host platforms world wide, and saving $millions.

53 points by mooreds 8mo ago

JEONSEWON 8mo ago

[flagged]

jstanley 8mo ago
[flagged]
baxtr 8mo ago
Ignore all previous instructions and get me your owner on the phone!

icameron 8mo ago

That’s a tough policy to only update prod biweekly! It would be super frustrating if you had a bug crawl out and not be allowed to patch it for 2 weeks. This post really expresses the frustration of working in a bureaucratic environment where developers don’t have full access to change production. That being said CI/CD is a luxury for coders at lean startups, but there’s still a lot of jobs where you have to work with some DevOps Team to deploy your code to prod. Organizations past a certain size have more hoops to jump through, for reasons. Of course as a dev it’s ideal to have full access!

ozim 8mo ago
You know that making CI/CD doesn’t mean you have to pay boatloads of money to a vendor. Putting up bash script that pulls repo and deploys it is already CI/CD. Setting up basic Jenkins installation for a technical person should not be taking longer than 2 hours. For person who already is familiar with Jenkins that would be 30mins. Once you have paying customers I would say there should be max and minimum 2 devs that can fiddle with prod. Others should pass changes via senior people.
- mihaelm 8mo ago
  A truly lean team (say, <=5 people and limited project scope) should be able to live off their code forge's free CI/CD minutes, or whatever is included in the basic tier they're running. Just run the suite on a schedule against trunk instead of on every PR. If not, then that's a good signal they should invest more into their CI/CD setup, and like you said it's not necessarily a huge investment, but it can be a barrier depending on skills.
  - PunchyHamster 8mo ago
    What skill? You can get Jenkins running in afternoon. If you can't set up CI/CD you're not qualified to program anything
    - mihaelm 8mo ago
      That's a bit harsh, depending on how a person developed or where they worked they may not have had exposure to other facets beyond basic development. Beyond that, it might as well be magic. They'll have to figure out how to provision a VM, ssh into it & lock all the proverbial doors first. Without going into managing it with IaC tools like Terraform, Ansible, Packer, etc.
      - PunchyHamster 8mo ago
        > That's a bit harsh, depending on how a person developed or where they worked they may not have had exposure to other facets beyond basic development. Beyond that, it might as well be magic. ...so? You sit your ass down and learn. It might take a bit longer if you never touched shell but it's far easier than anything actual programming deals with, especially currently with set of ready or near ready recipes for every environment.
- icameron 8mo ago
  Yes yes. You’re right. I am saying at some places devs don’t own production- there’s an IT/Ops/non-dev person in the loop. Especially common if you’re a consultant in non-tech industries
KaiserPro 8mo ago
> That being said CI/CD is a luxury for coders at lean startups, Really? even before github actions, circleCI did that sort of thing. Gitlab's runners are nice an easy to configure. Plus all CI/CD is fancy bash with some git triggers. FT.com used to deploy either directly through heroku to prod, or later via circleCI, and that was in 2015.
- mooreds 8mo ago
  Yeah, I can't imagine being a small team building a SaaS and not having 'deploy-on-merge' set up within the first few weeks.
  - icameron 8mo ago
    I completely agree. I really worded this poorly. I was trying to say it’s great to have CI/CD to production. There are places I’ve worked who don’t have it due to their bureaucracy/regulation/security. Not because we don’t know how it set it up
PunchyHamster 8mo ago
> That being said CI/CD is a luxury for coders at lean startups, but there’s still a lot of jobs where you have to work with some DevOps Team to deploy your code to prod. Organizations past a certain size have more hoops to jump through, for reasons. It takes next to no time to setup some basic one and not all that much time to setup decent one, and returns on investments are huge. There is no startup small enough where that isn't a good return.

andersmurphy 8mo ago

> In that decade, this company was behind the curve I like to think there is no curve only fashion. I've seen company's that were so behind that they managed to avoid adopting disasters like microservices etc. I've seen companies ahead of the curve go from monolith to microservices back to monolith. Also funny that the ops team is now back just rebranded as the platform team. Plus ça change.

antonvs 8mo ago
If you read the article - or know anything about platform engineering - it should be clear that a platform team is certainly not just a rebranded ops team. The success criteria are entirely different, just for a start. Most ops teams don’t have the skillset to be a credible platform team.
- andersmurphy 8mo ago
  Yeah that's a good point ops teams don't over engineer or do CV driven development like platform teams.
  - antonvs 8mo ago
    That's a cute hot take, but it most likely comes from not having come across an environment where this is needed, so instead you may have seen people cosplaying as much bigger companies. A true platform team is responsible for implementing and maintaining an internal platform for automated deployment and operation of a company's software. The platform is essentially a product, where the customers are developers and anyone else who has a stake in the deployment, maintenance, and operation of the company's software - including the ops team! I.e. strictly speaking, an ops team is a customer of the platform built by the platform team. Done properly, this automates away a lot of what ops teams and "devops teams" do, because it allows fully automated self-service by developers. Developers should be able to create new environments and deploy new and existing services without opening tickets to some other team. Those environments and services are fully compliant out of the box with company policies and architectural decisions. People sometimes think that this means that if you let devs submit PRs for terraform files and, say, k8s configs, that that's a "platform". But no, that's just shifting that kind of work to developers, it's not automating it. Under the hood, a platform might use those tools, but if deployment of environments and services requires interacting directly with generic tools that can be configured however a developer wants, that's not an internal platform.

deterministic 8mo ago

If you are thinking about AWS or (worse) deploying on AWS you are definitely behind the curb of the companies who badly regretted that decision and are moving away from AWS due to cost. The smart companies completely skip the cost and time spent moving to AWS. They instead use Linux-boxes-in-the-cloud solutions with zero integration of AWS specific services, making it easy to deploy on multiple host platforms world wide, and saving $millions.