4 ms·

YAML is great in my opinion. What isn’t great is how hard we have made the deployment part of CD. I’ll admit that our setup within Azure DevOps isn’t amazing, b

by devjab 3y ago

YAML is great in my opinion. What isn’t great is how hard we have made the deployment part of CD. I’ll admit that our setup within Azure DevOps isn’t amazing, but the fact that you have organisations with multiple teams or 5-6 operators working on these tools is just mind blowing to me. Maybe not for Google, but for us in regular boring enterprise where we top out at 50.000 concurrent users or typically way, way, less. It was easier for me to deploy enterprise web-applications to our on-prem IIS that had all the load balancing and networking and whatever else done by less than a quarter of a full time employee back in the early 00’s than it is for me to deploy the same damn thing into our “modern” setup. Yes, there are advantages to our modern pipeline. We’ve moved far beyond the whole “it works on my computer” thing and we’ve increased the quality control immensely by putting up much better acceptance gates, but the actual deployment? That’s a damn nightmare in 2023. Again, this is likely not an issue for many HN programmers who work in actual tech companies, or in companies with good dedicated DevOps teams, but in the world of non-tech enterprise CI/CD has frankly never felt worse in my career. I guess you can chose to blame YAML, and the loads of it you need to do anything and how hard templating seems to be. But in my opinion it’s far more of an organisational issue than a technical one. We need CD tools that are far more automated, so that it doesn’t become the job of the developer to describe the infrastructure as code. It’s fine that you can do that, but the reality is also that we are asking millions of developers to deploy infrastructure that they might not have the slightest clue how works. I’ve never met a developer who didn’t just want to hand off their container and then have all the networking and “server stuff” just done for them. And when you don’t, you end up with a lot of VNETs and subnets that nobody really knows how works and that your organisation is losing a ton of money on because your developers didn’t know you could so /x and then start the subnet where it came to.

jiggawatts 3y ago
The cloud is the new mainframe. You work offline on a "job definition" of some sort, submit it to the proprietary shared system, wait in a queue, then you receive a log file generated by a system you don't control. You can't run the proprietary system code locally on a workstation, and hence the inner loop is often tens of minutes long at best, hours or days at worst. There's no preview, no "what if" mode or "dry run". You work directly on production even if it's called "test" because there's only one system. The real problem isn't YAML. It wouldn't matter if the pipelines were scripted in God's own programming language. Software development on workstations instead of central timeshare mainframes became wildly popular because it allowed a dramatically faster inner loop, it allowed isolation from production environments, and it gave control back into the hands of the developers. Current-generation CI/CD pipelines generally undo all of that. Single-box Kubernetes reintroduces most of what made workstation-based development good, but it is still a very new system and has many teething issues. PS: A related issue to yours is that there are great solutions for the solo dev doing click-ops for one app, there are great solutions for megacorops doing automation at scale for thousands of devs, but in the middle where you have a couple of enterprise devs managing a few dozen apps its just madness.
- bewo001 3y ago
  Got it, I'll use JCL for the next project. More seriously, this is a very good comparison. IBM always dreamed to be something like AWS is today.
  - jiggawatts 3y ago
    The cloud is just a disaggregated mainframe. You don't even have to squint very hard for IBM's Job Control Language to start looking an awful lot like some CI/CD pipeline YAML file.
- pramodbiligiri 3y ago
  What do you mean by single-box kubernetes?
  - diarrhea 3y ago
    Not the OP, but probably Minikube and similar.
globular-toast 3y ago
What's your problem with deployment? In everything I've set up I just tag a commit, push it, then that commit gets deployed. It seems really straightforward to set that up in any CI/CD system.
- devjab 3y ago
  It's in the resource allocation/generation process of your deployment things get tricky. The first thing you're going to have to do is the networking. You're not going to deploy things on the internet in an Enterprise Organisation, so you're going to have something that can make sure your application gets on the right VNET and the right subnet. Obtains the correct private-endpoints. Gets a DNS address so people can go to https://yourapp.yourcompany.com https://yourapp.yourcompany.com from the internal on-prem network that will need a DNS routing as well because it's not going to ask the internet. Once in a while, when you actually want your app to be accessed from the internet, you're going to need some sort of gateway resource that can transfer the traffic through whatever security your org has setup in such a way that it blocks, every, IP you're not intending on letting through. The second thing you're going to want is a place to persist your data. You can have a centralized shared database, and most places use this, but the issue with that is that it'll eventually end up a mess. Not by bad intention, but because it's just very easy to build stored procedures, views and whatever else on top of the data from all those 100+ applications that are sharing the same DB server because it's a quick way to deliver business value (until it isn't). It also doesn't scale, which will eventually be an issue for you even if you aren't netflix. The flip-side is that you can't actually do a bunch of managed DBs because that is ridicilously expensive, and you likely can't use the CosmosDB (and whatever the AWS equvilient is called) because of company red-tape and also because your 3rd party DB backup vendor doesn't support it. So you either run them in containers, or share servers, or run up the bill. Two of those solutions require all the steps in the networking step, because they too need to be on the right private endpoints on the right subnets on the right VNETs, and all of them will need you to grant some sort of access rights through your IDM. Which is often as simple as giving your apps some sort of managed ID and then adding it to some AD group, but that's still something that needs to be done. Deployment slots: now we're moving into the more ridicilous parts of it, but you're probably going to need to deploy your application at least twice. It'll probably be three or more, but it'll be at least twice. Each of these slots require that you go through the two previous steps. If you're wondering why, then it can be as silly as because some Enterprise Architect lied in a meeting and convinced everyone it was required by law, similar to how you're also forced to put 4-6 eyes on every pull request (though to be fair, this part makes sense). And that's if you're using serverless dockerized function apps in Azure in an enterprise organisation. If you're not doing serverless it gets way worse. Sure kubernetes, travis, openshift and so on can do it much smarter, but in non-tech enterprise you don't have anyone employed to manage it. @jiggawatts really put it well when he said this in one of the other comments: > there are great solutions for the solo dev doing click-ops for one app, there are great solutions for megacorops doing automation at scale for thousands of devs, but in the middle where you have a couple of enterprise devs managing a few dozen apps its just madness. In all seriousness I've written and deployed business critical serverless applications, which are less code than the bicep and yaml required to deploy them, and here is the biggest issue, I have no idea if my bicep is actually good because GPT wrote most of it.
marcosdumay 3y ago
> We’ve moved far beyond the whole “it works on my computer” thing I dunno. I've had things working on my docker image quite a few times while breaking at the deployed one. This is only passed if you have complete transparency of the entire build and deployment pipeline, total access to the images repository, actual control over the building instructions, etc. That's about as restrictive as control over a local OS, and I do expect it to fail on about as many organizations as the ones that had problems with code breaking when moved into another machine.
- devjab 3y ago
  Access to our container registry breaks all the time for no reason anyone has been able to figure out. Not even the third party people we pay a lot of money to help. Basically we now add both a managed ID to the AD group with pull rights, and we setup rights the old fashined way and now it works maybe 95% of the time. Sometimes it still doesn't, and usually the fix is to delete everything and deploy again. As far as actually running the build images, however, if they run in Docker locally then we never have issues with running them in the cloud.

by devjab 3y ago

jiggawatts 3y ago

The cloud is the new mainframe. You work offline on a "job definition" of some sort, submit it to the proprietary shared system, wait in a queue, then you receive a log file generated by a system you don't control. You can't run the proprietary system code locally on a workstation, and hence the inner loop is often tens of minutes long at best, hours or days at worst. There's no preview, no "what if" mode or "dry run". You work directly on production even if it's called "test" because there's only one system. The real problem isn't YAML. It wouldn't matter if the pipelines were scripted in God's own programming language. Software development on workstations instead of central timeshare mainframes became wildly popular because it allowed a dramatically faster inner loop, it allowed isolation from production environments, and it gave control back into the hands of the developers. Current-generation CI/CD pipelines generally undo all of that. Single-box Kubernetes reintroduces most of what made workstation-based development good, but it is still a very new system and has many teething issues. PS: A related issue to yours is that there are great solutions for the solo dev doing click-ops for one app, there are great solutions for megacorops doing automation at scale for thousands of devs, but in the middle where you have a couple of enterprise devs managing a few dozen apps its just madness.

bewo001 3y ago
Got it, I'll use JCL for the next project. More seriously, this is a very good comparison. IBM always dreamed to be something like AWS is today.
- jiggawatts 3y ago
  The cloud is just a disaggregated mainframe. You don't even have to squint very hard for IBM's Job Control Language to start looking an awful lot like some CI/CD pipeline YAML file.
pramodbiligiri 3y ago
What do you mean by single-box kubernetes?
- diarrhea 3y ago
  Not the OP, but probably Minikube and similar.

globular-toast 3y ago

What's your problem with deployment? In everything I've set up I just tag a commit, push it, then that commit gets deployed. It seems really straightforward to set that up in any CI/CD system.

devjab 3y ago
It's in the resource allocation/generation process of your deployment things get tricky. The first thing you're going to have to do is the networking. You're not going to deploy things on the internet in an Enterprise Organisation, so you're going to have something that can make sure your application gets on the right VNET and the right subnet. Obtains the correct private-endpoints. Gets a DNS address so people can go to https://yourapp.yourcompany.com https://yourapp.yourcompany.com from the internal on-prem network that will need a DNS routing as well because it's not going to ask the internet. Once in a while, when you actually want your app to be accessed from the internet, you're going to need some sort of gateway resource that can transfer the traffic through whatever security your org has setup in such a way that it blocks, every, IP you're not intending on letting through. The second thing you're going to want is a place to persist your data. You can have a centralized shared database, and most places use this, but the issue with that is that it'll eventually end up a mess. Not by bad intention, but because it's just very easy to build stored procedures, views and whatever else on top of the data from all those 100+ applications that are sharing the same DB server because it's a quick way to deliver business value (until it isn't). It also doesn't scale, which will eventually be an issue for you even if you aren't netflix. The flip-side is that you can't actually do a bunch of managed DBs because that is ridicilously expensive, and you likely can't use the CosmosDB (and whatever the AWS equvilient is called) because of company red-tape and also because your 3rd party DB backup vendor doesn't support it. So you either run them in containers, or share servers, or run up the bill. Two of those solutions require all the steps in the networking step, because they too need to be on the right private endpoints on the right subnets on the right VNETs, and all of them will need you to grant some sort of access rights through your IDM. Which is often as simple as giving your apps some sort of managed ID and then adding it to some AD group, but that's still something that needs to be done. Deployment slots: now we're moving into the more ridicilous parts of it, but you're probably going to need to deploy your application at least twice. It'll probably be three or more, but it'll be at least twice. Each of these slots require that you go through the two previous steps. If you're wondering why, then it can be as silly as because some Enterprise Architect lied in a meeting and convinced everyone it was required by law, similar to how you're also forced to put 4-6 eyes on every pull request (though to be fair, this part makes sense). And that's if you're using serverless dockerized function apps in Azure in an enterprise organisation. If you're not doing serverless it gets way worse. Sure kubernetes, travis, openshift and so on can do it much smarter, but in non-tech enterprise you don't have anyone employed to manage it. @jiggawatts really put it well when he said this in one of the other comments: > there are great solutions for the solo dev doing click-ops for one app, there are great solutions for megacorops doing automation at scale for thousands of devs, but in the middle where you have a couple of enterprise devs managing a few dozen apps its just madness. In all seriousness I've written and deployed business critical serverless applications, which are less code than the bicep and yaml required to deploy them, and here is the biggest issue, I have no idea if my bicep is actually good because GPT wrote most of it.

marcosdumay 3y ago

> We’ve moved far beyond the whole “it works on my computer” thing I dunno. I've had things working on my docker image quite a few times while breaking at the deployed one. This is only passed if you have complete transparency of the entire build and deployment pipeline, total access to the images repository, actual control over the building instructions, etc. That's about as restrictive as control over a local OS, and I do expect it to fail on about as many organizations as the ones that had problems with code breaking when moved into another machine.

devjab 3y ago
Access to our container registry breaks all the time for no reason anyone has been able to figure out. Not even the third party people we pay a lot of money to help. Basically we now add both a managed ID to the AD group with pull rights, and we setup rights the old fashined way and now it works maybe 95% of the time. Sometimes it still doesn't, and usually the fix is to delete everything and deploy again. As far as actually running the build images, however, if they run in Docker locally then we never have issues with running them in the cloud.