5 ms·

I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get l

I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get lost because nobody thought to talk to the right team. But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un production ready it was for the longest time. It _felt_ like progress because it required a lot of (to dev eyes) hidden magic to make it work. Now, we are back with sysadmins, but they are called SREs. They provide a platform, which devs talk to to deploy stuff. Basically we're back where we were in 2015, just with more yaml. The good things that have come out of devops has been the dashboarding and metrics tools. Its just a shame that everything else touched by it appears to be an infinite source of busy work (looking at you kubernetes) (before anyone asks, I'm an SRE at a FAANG.)

notanormalnerd 3y ago
So you mean, DevOps has made Sysadmins more service oriented and now instead of having their pets and being in overprotective silos, they actually provide best practices and proper platforms for the poeple developing the software?
- KaiserPro 3y ago
  thats how it _should_ be. The place it worked the best was where a team of sysadmins were split up and embedded with each dev team. each devop would then rotate onto another team every 6 months or so. This meant that _we_ had to document our shit for the next devop, but also helped eliminate key person dependencies. We would then have a weekly offsite where we'd bitch and moan about our teams and conspire to make things better.
- sisve 3y ago
  Im not sure how former sysadmins feel about it, but from a dev standpoint, DevOps worked great for us. Making a clean cut on a plattform. Was my app down, I got message/called. If the platform was down it was the Infrastructure team / PaaS provider that needed to fix it. In the old days we had so few deploys and a hotfix was something that was extraordinary. I actually had a board of something in another country approving that we where allowed to deploy. They knew nothing about the application..
- tryauuum 3y ago
  god do I hate the "pet" analogy yes this server is special to me. No, I cannot move workloads somewhere else and shoot the server in the head. Because customers are running GPU workloads there, I cannot transparently migrate them
  - fragmede 3y ago
    How does the analogy falling flat for you? GPUs are interchangable resources; an H100 is just like any other H100, even if you can't take a snapshot of an intermediate state from one and load it onto another and resume.
    - tryauuum 3y ago
      I thought about it and I don't hate the analogy. Although usually I'm very against comparing IT stuff with physical stuff because it tends to produce pointless conversations. What I hate is the attitude of "treating your server is a pet is wrong, you should be able to kill it and replace any time". I feel like people who say it never run anything more complex than an HTTP API.
      - fragmede 3y ago
        Ah yeah I totally agree. Having pets is fine until it isn't. A single-digit handful is fine. 1,000 bespoke pets, way above Dunbar's number, isn't. Where exactly you run out of mental headroom and should convert is up to you; the point is that there is one and converting over should be planned for in the North Star roadmap of future development. My (Raspberry Pi) home server is very much a pet, but I'm never going to productionize that thing, so I'm totally fine with it being a pet, despite the practice at work of having no pets, all cattle.
mattgreenrocks 3y ago
More a general question, but I’ve gotten pings in the past to be an SRE at a FAANG despite not doing much cloud stuff. Is it the type of role people take for a tour of duty and then transfer out of? I imagine there are SREs who enjoy it and stay there, but curious why they’d invite someone heavy on the dev side to interview.
- dijit 3y ago
  Depends on the FAANG, Google famously throws devs at ops problems. Ironically they realised this wasn't a good idea in exclusivity, so now they hire two disciplines: SRE-SWE and SRE-SYS. SRE-SYS is as you would expect, pretty standard sysadmin knowledge from 2005 (IE; not helpdesk sysadmin, some scripting knowledge, heavy OS knowledge, heavy debugging knowledge). SRE-SWE is much more programmer heavy with a light touch on OS fundamentals (based on what I consider important for OS knowledge FWIW).
- KaiserPro 3y ago
  depends on the company and department. the downsides of SRE is that you tend to be super hyper specialised in something thats only really of any sense to that department inside a company. Moreover, you're dealing with real steel machines, and amounts of data that are prohibitorily expensive to just start again with. (the joy of AWS is that most of the times you can delete it all and spin it up again, often side by side) also, some SREs are really specialised software devs, others like me are sysadmins who'd been swept by the current into SRE roles. My company SREs are not like google's, where they are rock stars swanning about doing one line fixes that make everything .48% more reliable. We are computer nurses making sure that the wards of systems we look after aren't wiped out because a doctor forgot to wash their hands after taking a poo. However, they both require the same skills: murder mystery and the ability to dive super deep into the stack, go from python->c->assembly. or service->VFS->cache->network->time weirdness Also it could just be copy pasting service defs and making dashboards, then pinging devs saying "shits on fire yo"
tester756 3y ago
<conspiracy theory> Admins wanted to increase their $$ and started calling themselves DevOps And developed some automation to justify that </conspiracy theory>
- fragmede 3y ago
  I know a few old school sysadmins who got left behind because they couldn't upskill to DevOps and consequently lost out on $$, others who were fired after moving everything off prem to AWS because the company no longer needed them. You understand that your theory is a "conspiracy theory", so why even say it? how does that add to the discussion?
- KaiserPro 3y ago
  Shhhh, the union will get you. Although sadly in most cases the automation was harder than the original service to look after.
llama052 3y ago
Maybe in FAANG it's "back like we are in 2015" but the tools that have been brought into the mainstream like Kubernetes, automation tools and such have had a great impact across many small to large sized organizations. It's done a great job to enforce consistency and immutable infrastructure in a way that is more approachable for most when you get past the scale of a few resources. I've seen start ups write their on bash tooling, watch it devolve into some opaque thing that no one knows how it works or what it does. I've seen platform teams devolve into managing VM updates by hand, babysitting operating systems of every application deployed because it's treated as a long living pet, and it gets worse when team leads move on and then someone else comes in and half bakes their own flavor of change. Even better when networking gets involved, load balancing with HA where people roll their own solution. On top of that is the certificate story and how that gets managed. Don't even start with RBAC from a platform perspective, it's a nightmare. The toil filled platform days should be behind us. Plenty of companies are slow to evolve and you can find a lot of pain. Perhaps I've had bad luck in my history of being a network engineer, infrastructure engineer, and an SRE. The one big benefit to these tools that are coming out is it can give some sort of standardization and it's giving you a ton of power in abstraction so that platform teams aren't micromanaging things that are just sorted for you. Something like Kubernetes gives you load balancing, high availability, RBAC, networking and routing, resource management, node draining and rotations, auto scaling. Sure it's managed in yaml manifests, but it's doing a lot of lifting. I won't say that the abstraction can't bite you, you still need to understand the core concepts of things. However people should be understanding those concepts regardless. The amount of work that takes for a small team at a startup that needs to scale services is immense without it. Right now I'm working with a 3 man platform team and we are running over 200 services in multiple regions, multiple database technologies and analytics stacks, we auto scale aggressively and we serve over a billion requests and terabytes of traffic a month with over 99.99% of uptime. We also deal with operations, security and every other hat that someone wants to throw our way. The metrics solutions to come out of this are also such a huge win, I remember the days of Nagios, Zabbix, SolarWinds etc. They were mostly static graphs showing you snapshots of what happens at a very high level, now we have metrics for everything and you can feed that into anything to self-remediate or escalate. I absolutely love it and can't wait to see it evolve more! The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them. Some older sysadmins were legitimately afraid of rebooting servers because they were afraid they wouldn't come back. Some machines had over 900 days of uptime because they just couldn't keep up with the toil. Also this creates a wall between the platform and developers because it turns into a hand-off process where developers throw code over and the platform team has to implement some server and load balance logic, let alone meeting SLO/SLI criteria. Sadly, that's not uncommon to any industry. This team had 15+ people and had trouble keeping up let alone deploy and innovate. Outages in this mode are unavoidable, and pulling away from that old school mindset and starting over is VERY hard. I'm sure a lot of that can be contributed to bad management and bad hires, however from my perspective using tooling and standardization will only help you build a platform that can actually grow. Making that predictable and easy to digest is important, and so is understanding the details underneath. I just know I will never want to go back to 2015 or anytime before. I'll babysit kubernetes node issues or upgrade woes all day if that's what it takes (not that I need to). Granted this isn't the case at FAANG, they've got the brain power to build a consistent platform regardless. Everyone below that though benefits greatly.
lamontcg 3y ago
DevOps was originally about moving ops into dev teams. At Amazon they just gave devs pagers. Devs hated that. The SRE role seems to have taken over the app-facing side of what system engineers did being an interface between an app and the hardware. A lot of what I see SREs doing is things like app metrics and SLAs which were not what system engineers really did. And the SRE role has been pushed around from being embedded in dev teams, to being their own teams that managed apps to being a centralized DevOps team that looks an awful lot like the old silo'd split between Dev and Ops. I don't know where the actual system engineers went, they probably run Kubernetes clusters at FAANGs and have SRE titles these days.
nunez 3y ago
...and all of that is before looking at the tools in the CNCF diagram. Maybe this is "old man yells at cloud" me but I miss 2015 DevOps. If you knew Terraform, Ansible/Chef, a monitoring tool, a cloud (usually AWS) and a programming language (usually Python), you were all set and could still be idiomatically/culturally DevOps. And I say that as someone that knows Kubernetes very well

notanormalnerd 3y ago

So you mean, DevOps has made Sysadmins more service oriented and now instead of having their pets and being in overprotective silos, they actually provide best practices and proper platforms for the poeple developing the software?

KaiserPro 3y ago
thats how it _should_ be. The place it worked the best was where a team of sysadmins were split up and embedded with each dev team. each devop would then rotate onto another team every 6 months or so. This meant that _we_ had to document our shit for the next devop, but also helped eliminate key person dependencies. We would then have a weekly offsite where we'd bitch and moan about our teams and conspire to make things better.
sisve 3y ago
Im not sure how former sysadmins feel about it, but from a dev standpoint, DevOps worked great for us. Making a clean cut on a plattform. Was my app down, I got message/called. If the platform was down it was the Infrastructure team / PaaS provider that needed to fix it. In the old days we had so few deploys and a hotfix was something that was extraordinary. I actually had a board of something in another country approving that we where allowed to deploy. They knew nothing about the application..
tryauuum 3y ago
god do I hate the "pet" analogy yes this server is special to me. No, I cannot move workloads somewhere else and shoot the server in the head. Because customers are running GPU workloads there, I cannot transparently migrate them
- fragmede 3y ago
  How does the analogy falling flat for you? GPUs are interchangable resources; an H100 is just like any other H100, even if you can't take a snapshot of an intermediate state from one and load it onto another and resume.
  - tryauuum 3y ago
    I thought about it and I don't hate the analogy. Although usually I'm very against comparing IT stuff with physical stuff because it tends to produce pointless conversations. What I hate is the attitude of "treating your server is a pet is wrong, you should be able to kill it and replace any time". I feel like people who say it never run anything more complex than an HTTP API.
    - fragmede 3y ago
      Ah yeah I totally agree. Having pets is fine until it isn't. A single-digit handful is fine. 1,000 bespoke pets, way above Dunbar's number, isn't. Where exactly you run out of mental headroom and should convert is up to you; the point is that there is one and converting over should be planned for in the North Star roadmap of future development. My (Raspberry Pi) home server is very much a pet, but I'm never going to productionize that thing, so I'm totally fine with it being a pet, despite the practice at work of having no pets, all cattle.

mattgreenrocks 3y ago

More a general question, but I’ve gotten pings in the past to be an SRE at a FAANG despite not doing much cloud stuff. Is it the type of role people take for a tour of duty and then transfer out of? I imagine there are SREs who enjoy it and stay there, but curious why they’d invite someone heavy on the dev side to interview.

dijit 3y ago
Depends on the FAANG, Google famously throws devs at ops problems. Ironically they realised this wasn't a good idea in exclusivity, so now they hire two disciplines: SRE-SWE and SRE-SYS. SRE-SYS is as you would expect, pretty standard sysadmin knowledge from 2005 (IE; not helpdesk sysadmin, some scripting knowledge, heavy OS knowledge, heavy debugging knowledge). SRE-SWE is much more programmer heavy with a light touch on OS fundamentals (based on what I consider important for OS knowledge FWIW).
KaiserPro 3y ago
depends on the company and department. the downsides of SRE is that you tend to be super hyper specialised in something thats only really of any sense to that department inside a company. Moreover, you're dealing with real steel machines, and amounts of data that are prohibitorily expensive to just start again with. (the joy of AWS is that most of the times you can delete it all and spin it up again, often side by side) also, some SREs are really specialised software devs, others like me are sysadmins who'd been swept by the current into SRE roles. My company SREs are not like google's, where they are rock stars swanning about doing one line fixes that make everything .48% more reliable. We are computer nurses making sure that the wards of systems we look after aren't wiped out because a doctor forgot to wash their hands after taking a poo. However, they both require the same skills: murder mystery and the ability to dive super deep into the stack, go from python->c->assembly. or service->VFS->cache->network->time weirdness Also it could just be copy pasting service defs and making dashboards, then pinging devs saying "shits on fire yo"

tester756 3y ago

<conspiracy theory> Admins wanted to increase their $$ and started calling themselves DevOps And developed some automation to justify that </conspiracy theory>

fragmede 3y ago
I know a few old school sysadmins who got left behind because they couldn't upskill to DevOps and consequently lost out on $$, others who were fired after moving everything off prem to AWS because the company no longer needed them. You understand that your theory is a "conspiracy theory", so why even say it? how does that add to the discussion?
KaiserPro 3y ago
Shhhh, the union will get you. Although sadly in most cases the automation was harder than the original service to look after.

llama052 3y ago

Maybe in FAANG it's "back like we are in 2015" but the tools that have been brought into the mainstream like Kubernetes, automation tools and such have had a great impact across many small to large sized organizations. It's done a great job to enforce consistency and immutable infrastructure in a way that is more approachable for most when you get past the scale of a few resources. I've seen start ups write their on bash tooling, watch it devolve into some opaque thing that no one knows how it works or what it does. I've seen platform teams devolve into managing VM updates by hand, babysitting operating systems of every application deployed because it's treated as a long living pet, and it gets worse when team leads move on and then someone else comes in and half bakes their own flavor of change. Even better when networking gets involved, load balancing with HA where people roll their own solution. On top of that is the certificate story and how that gets managed. Don't even start with RBAC from a platform perspective, it's a nightmare. The toil filled platform days should be behind us. Plenty of companies are slow to evolve and you can find a lot of pain. Perhaps I've had bad luck in my history of being a network engineer, infrastructure engineer, and an SRE. The one big benefit to these tools that are coming out is it can give some sort of standardization and it's giving you a ton of power in abstraction so that platform teams aren't micromanaging things that are just sorted for you. Something like Kubernetes gives you load balancing, high availability, RBAC, networking and routing, resource management, node draining and rotations, auto scaling. Sure it's managed in yaml manifests, but it's doing a lot of lifting. I won't say that the abstraction can't bite you, you still need to understand the core concepts of things. However people should be understanding those concepts regardless. The amount of work that takes for a small team at a startup that needs to scale services is immense without it. Right now I'm working with a 3 man platform team and we are running over 200 services in multiple regions, multiple database technologies and analytics stacks, we auto scale aggressively and we serve over a billion requests and terabytes of traffic a month with over 99.99% of uptime. We also deal with operations, security and every other hat that someone wants to throw our way. The metrics solutions to come out of this are also such a huge win, I remember the days of Nagios, Zabbix, SolarWinds etc. They were mostly static graphs showing you snapshots of what happens at a very high level, now we have metrics for everything and you can feed that into anything to self-remediate or escalate. I absolutely love it and can't wait to see it evolve more! The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them. Some older sysadmins were legitimately afraid of rebooting servers because they were afraid they wouldn't come back. Some machines had over 900 days of uptime because they just couldn't keep up with the toil. Also this creates a wall between the platform and developers because it turns into a hand-off process where developers throw code over and the platform team has to implement some server and load balance logic, let alone meeting SLO/SLI criteria. Sadly, that's not uncommon to any industry. This team had 15+ people and had trouble keeping up let alone deploy and innovate. Outages in this mode are unavoidable, and pulling away from that old school mindset and starting over is VERY hard. I'm sure a lot of that can be contributed to bad management and bad hires, however from my perspective using tooling and standardization will only help you build a platform that can actually grow. Making that predictable and easy to digest is important, and so is understanding the details underneath. I just know I will never want to go back to 2015 or anytime before. I'll babysit kubernetes node issues or upgrade woes all day if that's what it takes (not that I need to). Granted this isn't the case at FAANG, they've got the brain power to build a consistent platform regardless. Everyone below that though benefits greatly.

lamontcg 3y ago

DevOps was originally about moving ops into dev teams. At Amazon they just gave devs pagers. Devs hated that. The SRE role seems to have taken over the app-facing side of what system engineers did being an interface between an app and the hardware. A lot of what I see SREs doing is things like app metrics and SLAs which were not what system engineers really did. And the SRE role has been pushed around from being embedded in dev teams, to being their own teams that managed apps to being a centralized DevOps team that looks an awful lot like the old silo'd split between Dev and Ops. I don't know where the actual system engineers went, they probably run Kubernetes clusters at FAANGs and have SRE titles these days.

nunez 3y ago

...and all of that is before looking at the tools in the CNCF diagram. Maybe this is "old man yells at cloud" me but I miss 2015 DevOps. If you knew Terraform, Ansible/Chef, a monitoring tool, a cloud (usually AWS) and a programming language (usually Python), you were all set and could still be idiomatically/culturally DevOps. And I say that as someone that knows Kubernetes very well