3 ms·

Maybe in FAANG it's "back like we are in 2015" but the tools that have been brought into the mainstream like Kubernetes, automation tools and such have had a gr

Maybe in FAANG it's "back like we are in 2015" but the tools that have been brought into the mainstream like Kubernetes, automation tools and such have had a great impact across many small to large sized organizations. It's done a great job to enforce consistency and immutable infrastructure in a way that is more approachable for most when you get past the scale of a few resources. I've seen start ups write their on bash tooling, watch it devolve into some opaque thing that no one knows how it works or what it does. I've seen platform teams devolve into managing VM updates by hand, babysitting operating systems of every application deployed because it's treated as a long living pet, and it gets worse when team leads move on and then someone else comes in and half bakes their own flavor of change. Even better when networking gets involved, load balancing with HA where people roll their own solution. On top of that is the certificate story and how that gets managed. Don't even start with RBAC from a platform perspective, it's a nightmare. The toil filled platform days should be behind us. Plenty of companies are slow to evolve and you can find a lot of pain. Perhaps I've had bad luck in my history of being a network engineer, infrastructure engineer, and an SRE. The one big benefit to these tools that are coming out is it can give some sort of standardization and it's giving you a ton of power in abstraction so that platform teams aren't micromanaging things that are just sorted for you. Something like Kubernetes gives you load balancing, high availability, RBAC, networking and routing, resource management, node draining and rotations, auto scaling. Sure it's managed in yaml manifests, but it's doing a lot of lifting. I won't say that the abstraction can't bite you, you still need to understand the core concepts of things. However people should be understanding those concepts regardless. The amount of work that takes for a small team at a startup that needs to scale services is immense without it. Right now I'm working with a 3 man platform team and we are running over 200 services in multiple regions, multiple database technologies and analytics stacks, we auto scale aggressively and we serve over a billion requests and terabytes of traffic a month with over 99.99% of uptime. We also deal with operations, security and every other hat that someone wants to throw our way. The metrics solutions to come out of this are also such a huge win, I remember the days of Nagios, Zabbix, SolarWinds etc. They were mostly static graphs showing you snapshots of what happens at a very high level, now we have metrics for everything and you can feed that into anything to self-remediate or escalate. I absolutely love it and can't wait to see it evolve more! The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them. Some older sysadmins were legitimately afraid of rebooting servers because they were afraid they wouldn't come back. Some machines had over 900 days of uptime because they just couldn't keep up with the toil. Also this creates a wall between the platform and developers because it turns into a hand-off process where developers throw code over and the platform team has to implement some server and load balance logic, let alone meeting SLO/SLI criteria. Sadly, that's not uncommon to any industry. This team had 15+ people and had trouble keeping up let alone deploy and innovate. Outages in this mode are unavoidable, and pulling away from that old school mindset and starting over is VERY hard. I'm sure a lot of that can be contributed to bad management and bad hires, however from my perspective using tooling and standardization will only help you build a platform that can actually grow. Making that predictable and easy to digest is important, and so is understanding the details underneath. I just know I will never want to go back to 2015 or anytime before. I'll babysit kubernetes node issues or upgrade woes all day if that's what it takes (not that I need to). Granted this isn't the case at FAANG, they've got the brain power to build a consistent platform regardless. Everyone below that though benefits greatly.

KaiserPro 3y ago
> but the tools that have been brought into the mainstream like Kubernetes unless you are running real steel, kubernetes is at best a complication layer over the underlying cloud tools. In 2018 me and my team built a real time, ML-based processing API. It was hugely scalable, optimised for speed, price and uptime. At no point did we use need to use kubernetes. Not only would that have eaten most of our innovation budget, all it would have done is add latency and driven up the money we spent on inter VPC transfers. > I remember the days of Nagios, Zabbix, SolarWinds etc So do I, they were recognised as shit even in 2015. $work allowed me to lead the migration away from them and by then we were half way through replacing most of splunk with graphite/grafana based metrics instead. The rest of the alerting was mostly handled by a custom inhouse /__status end point parser. > The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them. That, again, is just bad sysadmin. Unless you have complete control over your stack with ansible, then its not ansible, its just sparkling bash script. I worked at a VFX company that deployed to all real machines for virtually everything. They used a GNU configuration system that was older than SSH (I can't remember what its called, it used to use rsh, that's how old it was.) All the machines were fungible, We could replace a file server at the drop of a hat (the only limit was how fast we could copy 160tbs over 2x10gig nics) Most of the place ran on perl, in one git repo for just configuration. OS updates were applied weekly, and we pushed system changes daily. This was in 2013. salt made all of that easier, especially as we had way too many hosts to sensibly run ansible at the time(I think we worked out it would take >1day to run a basic role). They had automated it to the point that you could plug in an unconfigured blade host, turn it on, and within 30 minutes, the bios was updated, networking configured and each blade was starting to be installed. Automation is an attitude. You can either invent new tools to solve hypothetical problems badly (k8s supposedly is high scale, but cant practically get above 1k hosts, its also balls deep in ipv4 ), or actually use/create tools to solve the problems you have now.

KaiserPro 3y ago

> but the tools that have been brought into the mainstream like Kubernetes unless you are running real steel, kubernetes is at best a complication layer over the underlying cloud tools. In 2018 me and my team built a real time, ML-based processing API. It was hugely scalable, optimised for speed, price and uptime. At no point did we use need to use kubernetes. Not only would that have eaten most of our innovation budget, all it would have done is add latency and driven up the money we spent on inter VPC transfers. > I remember the days of Nagios, Zabbix, SolarWinds etc So do I, they were recognised as shit even in 2015. $work allowed me to lead the migration away from them and by then we were half way through replacing most of splunk with graphite/grafana based metrics instead. The rest of the alerting was mostly handled by a custom inhouse /__status end point parser. > The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them. That, again, is just bad sysadmin. Unless you have complete control over your stack with ansible, then its not ansible, its just sparkling bash script. I worked at a VFX company that deployed to all real machines for virtually everything. They used a GNU configuration system that was older than SSH (I can't remember what its called, it used to use rsh, that's how old it was.) All the machines were fungible, We could replace a file server at the drop of a hat (the only limit was how fast we could copy 160tbs over 2x10gig nics) Most of the place ran on perl, in one git repo for just configuration. OS updates were applied weekly, and we pushed system changes daily. This was in 2013. salt made all of that easier, especially as we had way too many hosts to sensibly run ansible at the time(I think we worked out it would take >1day to run a basic role). They had automated it to the point that you could plug in an unconfigured blade host, turn it on, and within 30 minutes, the bios was updated, networking configured and each blade was starting to be installed. Automation is an attitude. You can either invent new tools to solve hypothetical problems badly (k8s supposedly is high scale, but cant practically get above 1k hosts, its also balls deep in ipv4 ), or actually use/create tools to solve the problems you have now.