5 ms·

Just learn Ansible already, it will take you at most one day.

TameAntelope 5y ago
I'm not sure how learning Ansible (which I already do know) is going to help troubleshoot or actually maintain the deployed services. The problem is not deployment or orchestration (though that is one of them), the problem is building a team with the necessary expertise to support what could be a gigantic variety of tools and technologies at an expert level.
- nirvdrum 5y ago
  Isn't that true of cloud providers, too? I'll use AWS as an example because that's where I have the most experience. There's a dizzying number of services, some of which have a high degree of overlap. Someone needs to understand them to make an informed choice as to what to use. Every place I've been at using AWS has had to deal with undocumented API issues and responses that the docs say are impossible, but happen. There are performance pitfalls that require in-depth knowledge of the product to avoid, but usually you don't find out until it's too late. Adopting many of the cloud native services isn't just a technology swap (your RDBMS knowledge isn't going to help much with DynamoDB). There's a cottage industry helping people just understand their bills and how to reduce costs. And for non-trivial deployments, you often have to learn orchestration tools like CloudFormation or Terraform. I'm at a place that uses GCP now and very little of that AWS knowledge I've accumulated is applicable here. Cloud platforms/services have a lot going for them, but I think they've become complex monsters that don't really save people as much time as they think. If you're just pushing buttons on the EC2 console, then it's faster than provisioning an Ubuntu server with Ansible for sure. But, those toy deployments aren't indicative of the actual effort to manage a production system. At least in my (mostly start-up) experience with AWS going back to 2009.
  - acdha 5y ago
    > Isn't that true of cloud providers, too? I'll use AWS as an example because that's where I have the most experience. There's a dizzying number of services, some of which have a high degree of overlap. Someone needs to understand them to make an informed choice as to what to use. Every place I've been at using AWS has had to deal with undocumented API issues and responses that the docs say are impossible, but happen. There are performance pitfalls that require in-depth knowledge of the product to avoid, but usually you don't find out until it's too late. Adopting many of the cloud native services isn't just a technology swap (your RDBMS knowledge isn't going to help much with DynamoDB). There's a cottage industry helping people just understand their bills and how to reduce costs. And for non-trivial deployments, you often have to learn orchestration tools like CloudFormation or Terraform. It's true to some extent but I would typically characterize it as having DIY add additional layers you have to be concerned with. For example, I've seen more problems with SANs in some weeks than I've had total using EBS since the late 2000s (repeat for dodgy network cabling, server firmware causing issues, etc.). The other thing to consider is what the alternatives are: for example, the reason why some of your RDBMS knowledge isn't relevant to DynamoDB is because it's a NoSQL database – if you wanted a SQL database, RDS is going to be at least an order of magnitude less time than running your own, especially if you are concerned with performance or reliability. If you've determined that NoSQL is a good fit for your application, that means that the relevant comparison is how much it costs to run DynamoDB versus, say, MongoDB. > I'm at a place that uses GCP now and very little of that AWS knowledge I've accumulated is applicable here. My experience has been the opposite. There are some differences but an awful lot of the principles transfer well to the other major cloud platforms and a solid foundation will make it easier for you to transition between the major cloud providers more easily than on-premise. > If you're just pushing buttons on the EC2 console, then it's faster than provisioning an Ubuntu server with Ansible for sure. But, those toy deployments aren't indicative of the actual effort to manage a production system. It's definitely not trivial but I certainly have no desire to go back to running a VMware cluster, either. I still have memories of being on the phone with their support team explaining how the flaws in their the design of their cluster health-check system lead to a big outage.
- NorwegianDude 5y ago
  > the problem is building a team with the necessary expertise to support what could be a gigantic variety of tools and technologies at an expert level. That is just as valid for cloud solutions, though.
- raffraffraff 5y ago
  Yeah. I remember years ago working at a shop that had hundreds of baremetal hypervisors in a bunch of colos. We had a problem where a clocks would sometimes skew massively (maybe dodgy hardware)? The thing was, ntpd would be able to correct it sometimes, but not always. It has a cut-off point where it throws its little arms up and says "I dunno!", and the skew just gets larger until someone manually fixed it. So we added a remediation step to the Sensu time check it if exceeded a threshold. Then one time we got blacklisted by the public ntpd servers we used in one region because we had a shitload of servers hitting them directly with ntpd and sensu checks. So we had to set up our own servers (and monitor them). And we still had occasional skew even though the remediation action would (eventually) force a correction, and these would cause odd failures with authentication or database replication. We eventually ditched ntpd and moved to chrony (which will continually adjust the clock regardless of the drift). But that took research, testing, puppet code, scheduled deployment, documentation etc. The whole episode was boring and stupidly time-consuming and wasn't even some cool thing that "moved the needle forward" for the company. It's just the fucking time on your servers. Now, take any number of stupid little things like this and sprinkle them over every single sprint, and see how the infrastructure/sre/devops/dogsbody team's promotion cycle works. "Why was the database upgrade delayed this time?"
  - southerntofu 5y ago
    I've never witnessed that kind of hardware problem precisely, but now just think about it: what would happen if the same situation happened on an AWS instance? How would you go about debugging anything and/or fixing it? It's not even sure you could diagnose the problem in the first place, let alone deploy a workaround. You'd have to send dozens of email to tech support who of course would say there is no problem because their machines have 999999999999999999% uptime and nothing could be wrong on their side, but hey they can sell you advanced support/engineering if you've got way too much money to help you find the problem in your code. Some commenter mentioned the dark days of Oracle/Microsoft/SAP ruling over the server market. But at least these companies had the most basic decency to let you house/access your own hardware and diagnose things yourself if you had the skills. Now in the "cloud" you can just go to hell and suffer all the problems/inconsistencies, or rather your users can suffer since you as a sysadmin have zero control/monitoring over what's happening. Oh and bonus point: if users report some problems, they will be reproducible 0% of the time since there are great chances your users are connected to a different availability zone: yeah it's easy to reach more than 5 9's when "ping works from a specific location" counts as uptime for a global service. So in the end, is it better to have a silly answer to "Why was the database upgrade delayed this time?"? Or is it better for the answer to be "i don't know, but upgrading the database cost us 37k€ in compute, 31k€ for storage and 125k€ for egress for backup of the previous db" ? I much prefer silly answers, but maybe that's because i don't have dozens of thousands of euros to be shorted of even if i wanted to :-)

I'm not sure how learning Ansible (which I already do know) is going to help troubleshoot or actually maintain the deployed services. The problem is not deployment or orchestration (though that is one of them), the problem is building a team with the necessary expertise to support what could be a gigantic variety of tools and technologies at an expert level.

nirvdrum 5y ago
Isn't that true of cloud providers, too? I'll use AWS as an example because that's where I have the most experience. There's a dizzying number of services, some of which have a high degree of overlap. Someone needs to understand them to make an informed choice as to what to use. Every place I've been at using AWS has had to deal with undocumented API issues and responses that the docs say are impossible, but happen. There are performance pitfalls that require in-depth knowledge of the product to avoid, but usually you don't find out until it's too late. Adopting many of the cloud native services isn't just a technology swap (your RDBMS knowledge isn't going to help much with DynamoDB). There's a cottage industry helping people just understand their bills and how to reduce costs. And for non-trivial deployments, you often have to learn orchestration tools like CloudFormation or Terraform. I'm at a place that uses GCP now and very little of that AWS knowledge I've accumulated is applicable here. Cloud platforms/services have a lot going for them, but I think they've become complex monsters that don't really save people as much time as they think. If you're just pushing buttons on the EC2 console, then it's faster than provisioning an Ubuntu server with Ansible for sure. But, those toy deployments aren't indicative of the actual effort to manage a production system. At least in my (mostly start-up) experience with AWS going back to 2009.
- acdha 5y ago
  > Isn't that true of cloud providers, too? I'll use AWS as an example because that's where I have the most experience. There's a dizzying number of services, some of which have a high degree of overlap. Someone needs to understand them to make an informed choice as to what to use. Every place I've been at using AWS has had to deal with undocumented API issues and responses that the docs say are impossible, but happen. There are performance pitfalls that require in-depth knowledge of the product to avoid, but usually you don't find out until it's too late. Adopting many of the cloud native services isn't just a technology swap (your RDBMS knowledge isn't going to help much with DynamoDB). There's a cottage industry helping people just understand their bills and how to reduce costs. And for non-trivial deployments, you often have to learn orchestration tools like CloudFormation or Terraform. It's true to some extent but I would typically characterize it as having DIY add additional layers you have to be concerned with. For example, I've seen more problems with SANs in some weeks than I've had total using EBS since the late 2000s (repeat for dodgy network cabling, server firmware causing issues, etc.). The other thing to consider is what the alternatives are: for example, the reason why some of your RDBMS knowledge isn't relevant to DynamoDB is because it's a NoSQL database – if you wanted a SQL database, RDS is going to be at least an order of magnitude less time than running your own, especially if you are concerned with performance or reliability. If you've determined that NoSQL is a good fit for your application, that means that the relevant comparison is how much it costs to run DynamoDB versus, say, MongoDB. > I'm at a place that uses GCP now and very little of that AWS knowledge I've accumulated is applicable here. My experience has been the opposite. There are some differences but an awful lot of the principles transfer well to the other major cloud platforms and a solid foundation will make it easier for you to transition between the major cloud providers more easily than on-premise. > If you're just pushing buttons on the EC2 console, then it's faster than provisioning an Ubuntu server with Ansible for sure. But, those toy deployments aren't indicative of the actual effort to manage a production system. It's definitely not trivial but I certainly have no desire to go back to running a VMware cluster, either. I still have memories of being on the phone with their support team explaining how the flaws in their the design of their cluster health-check system lead to a big outage.
NorwegianDude 5y ago
> the problem is building a team with the necessary expertise to support what could be a gigantic variety of tools and technologies at an expert level. That is just as valid for cloud solutions, though.
raffraffraff 5y ago
Yeah. I remember years ago working at a shop that had hundreds of baremetal hypervisors in a bunch of colos. We had a problem where a clocks would sometimes skew massively (maybe dodgy hardware)? The thing was, ntpd would be able to correct it sometimes, but not always. It has a cut-off point where it throws its little arms up and says "I dunno!", and the skew just gets larger until someone manually fixed it. So we added a remediation step to the Sensu time check it if exceeded a threshold. Then one time we got blacklisted by the public ntpd servers we used in one region because we had a shitload of servers hitting them directly with ntpd and sensu checks. So we had to set up our own servers (and monitor them). And we still had occasional skew even though the remediation action would (eventually) force a correction, and these would cause odd failures with authentication or database replication. We eventually ditched ntpd and moved to chrony (which will continually adjust the clock regardless of the drift). But that took research, testing, puppet code, scheduled deployment, documentation etc. The whole episode was boring and stupidly time-consuming and wasn't even some cool thing that "moved the needle forward" for the company. It's just the fucking time on your servers. Now, take any number of stupid little things like this and sprinkle them over every single sprint, and see how the infrastructure/sre/devops/dogsbody team's promotion cycle works. "Why was the database upgrade delayed this time?"
- southerntofu 5y ago
  I've never witnessed that kind of hardware problem precisely, but now just think about it: what would happen if the same situation happened on an AWS instance? How would you go about debugging anything and/or fixing it? It's not even sure you could diagnose the problem in the first place, let alone deploy a workaround. You'd have to send dozens of email to tech support who of course would say there is no problem because their machines have 999999999999999999% uptime and nothing could be wrong on their side, but hey they can sell you advanced support/engineering if you've got way too much money to help you find the problem in your code. Some commenter mentioned the dark days of Oracle/Microsoft/SAP ruling over the server market. But at least these companies had the most basic decency to let you house/access your own hardware and diagnose things yourself if you had the skills. Now in the "cloud" you can just go to hell and suffer all the problems/inconsistencies, or rather your users can suffer since you as a sysadmin have zero control/monitoring over what's happening. Oh and bonus point: if users report some problems, they will be reproducible 0% of the time since there are great chances your users are connected to a different availability zone: yeah it's easy to reach more than 5 9's when "ping works from a specific location" counts as uptime for a global service. So in the end, is it better to have a silly answer to "Why was the database upgrade delayed this time?"? Or is it better for the answer to be "i don't know, but upgrading the database cost us 37k€ in compute, 31k€ for storage and 125k€ for egress for backup of the previous db" ? I much prefer silly answers, but maybe that's because i don't have dozens of thousands of euros to be shorted of even if i wanted to :-)