4 ms·

You touch on a point that has been bothering me for a while. In the 90s I remember what it was like to be in an industry beholden to MS and other proprietary s

by antod 5y ago

You touch on a point that has been bothering me for a while. In the 90s I remember what it was like to be in an industry beholden to MS and other proprietary software. I worked for an MS partner and did the cert thing etc but it felt uncomfortable to me. There were lots of people happy to soak in everything MS and ignore everything else. They were basically tiny pawns in a global Microsoft sales strategy. Most of their knowledge was product details, and how to glue multiple MS products together. I switched to the FOSS world to get away from that product focus and lockin. I enjoyed the freedom of evaluating and choosing tools and building things from interchangable layers. I wasn't beholden to any vendor and was part of a community instead. As Linux later grew into a mainstream OS it worked out great for my career too. Now with everything in my circles moving to AWS and moving into higher level managed services with massive lockin, the creeping notion of everything (incl the certified engineers) turning back into proprietary products is reminding me of how I felt in the late 90s about MS. It's all in the name of efficiency, simplicity and value for money. But I'm not seeing it play out like that. Our Infra/Devops/SRE/etc teams seem to be getting ever larger and ever more swamped in ever complexifying CI/CD pipelines and endless freaking IAM policy tweaking and hand holding of self service development teams. Meanwhile the annual AWS bills dwarf our previous on prem colo and hardware replacement budgets (eg 10x), even though we haven't saved anything on staff costs as that has gone way up too. We're back to just gluing vendor products together again, with general purpose tech knowledge being replaced by proprietary product knowledge. And the new generation of engineers and managers that don't remember the 90s see all this as progress and not a problem. I am now an "old man despairs at cloud" meme....

Dave3of5 5y ago
Yeah it's history repeating. I've said it before on here and I'll say it again I don't think running on AWS will ever be cheaper than DIY. It's kind of a fallacy and a sales tactic by AWS to tell you how the elasticity of their offering allows you to "save money during the quite hours". I don't think I've ever seen a system that actually does that.
- acdha 5y ago
  The savings usually come from the corners being cut on DIY solutions: not accounting for the time spent on infrastructure services, skipping various security and monitoring features, sacrificing reliability, etc. You can definitely beat it with certain workloads (e.g. high-volume egress) but I’ve seen more cases where someone thought they had beaten it but was really comparing unlike things (e.g. comparing S3 to unreliable storage in a single location) or leaving out operator time.
  - treis 5y ago
    The counter to this is that companies rarely cut ops staff when moving to the cloud. The problem is that you still have security and monitoring knobs to fiddle with in a cloud. And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems.
    - acdha 5y ago
      > And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems. More typically, it makes those problems visible — you get a bill from AWS every month in a way which you don't get a report that 70% of the hardware in your data center is either under-utilized or being used inefficiently. I think cloud migrations often suffer from that as a source of attribution error: “OMG, we don't have perfect security in the cloud!” “Aren't you running a production app on RHEL 6 under the account of someone who left 8 years ago?” “yeah, but we know about that!”
      - treis 5y ago
        I'm talking about things like exorbitant egress fees changing how you have to architect your systems. Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. It also caused much more operational complexity. Being in 10 times as many regions meant that we got hit with data center issues 10 times as often. Similarly, we had to run 10x the instances instead of going with fewer bigger ones. Ultimately that means we got 10x the issues caused by random things like disk failures. Our alerting got much harder because the volume was smaller since it was split into 10 parts. The smaller the baseline is the harder it is to detect problematic variations. We frequently got paged because some customer in a small region did something unusual in the middle of the night or someone tried an outgoing spam attack. That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.
stavros 5y ago
I share your frustration, but you have to factor in what we get for all the complexity. IAM roles are much more secure than "pwn one, pwn them all", containers are much easier to develop and deploy with, etc. I'm not saying it's worth it, but we should evaluate it on those merits. I, too, have a suspicion it's not, but maybe the larger ops team on AWS can run circles around the smaller ops team on on-prem? There has to be a reason, after all. > I am now an "old man despairs at cloud" meme.... "At cloud" indeed :P
- antod 5y ago
  It's not the cost or complexity that really bothers me (that was just a side rant hehe), more the vendor "productisation" of all the tools. After all, in many ways open source stuff is often lower level with the extra complexity that involves. With open source tools (hell, even they are often turning into products now) there was always a feeling you could dig in and understand them, then join the community and contribute, and even shape their future. Now tools are just things to consume on the vendors terms. Maybe the open pendulum will swing back again in 10yrs - probably at the hands of something like the Cloud Native Foundation, rather than the last peak 10yrs ago driven by Linux distros and some key apps/languages as well as dissatisfaction with MS etc.
  - stavros 5y ago
    Ah yeah, that's frustrating to me as well. The network is now the OS, and it's proprietary.
scarface74 5y ago
I also did the 6 developer MS cert thing in 2010. Not because I thought certifications were valuable. Just as a guided learning path. I never told anyone or put them on my resume. I was transitioning from a decade of C bit twiddling. I did the same thing 8 years later to learn the AWS ecosystem. Where I now work 2 years later in ProServe (just disclaiming my biases). There is a chance I could retire from AWS or I could move on to the next hype cycle. Managing a career in technology is about staying relevant. It’s not like my vast knowledge of 65C02 assembly that I learned in the 80s in middle school comes in handy in 2021. However “my general tech knowledge” has served me well as well as my ability to know what level of the stack provides a business an “unfair advantage”. Managing infrastructure is seldom it unless you’re a DropBox. Every company is dependent on dozens of vendors for their operations.

by antod 5y ago

Dave3of5 5y ago

Yeah it's history repeating. I've said it before on here and I'll say it again I don't think running on AWS will ever be cheaper than DIY. It's kind of a fallacy and a sales tactic by AWS to tell you how the elasticity of their offering allows you to "save money during the quite hours". I don't think I've ever seen a system that actually does that.

acdha 5y ago
The savings usually come from the corners being cut on DIY solutions: not accounting for the time spent on infrastructure services, skipping various security and monitoring features, sacrificing reliability, etc. You can definitely beat it with certain workloads (e.g. high-volume egress) but I’ve seen more cases where someone thought they had beaten it but was really comparing unlike things (e.g. comparing S3 to unreliable storage in a single location) or leaving out operator time.
- treis 5y ago
  The counter to this is that companies rarely cut ops staff when moving to the cloud. The problem is that you still have security and monitoring knobs to fiddle with in a cloud. And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems.
  - acdha 5y ago
    > And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems. More typically, it makes those problems visible — you get a bill from AWS every month in a way which you don't get a report that 70% of the hardware in your data center is either under-utilized or being used inefficiently. I think cloud migrations often suffer from that as a source of attribution error: “OMG, we don't have perfect security in the cloud!” “Aren't you running a production app on RHEL 6 under the account of someone who left 8 years ago?” “yeah, but we know about that!”
    - treis 5y ago
      I'm talking about things like exorbitant egress fees changing how you have to architect your systems. Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. It also caused much more operational complexity. Being in 10 times as many regions meant that we got hit with data center issues 10 times as often. Similarly, we had to run 10x the instances instead of going with fewer bigger ones. Ultimately that means we got 10x the issues caused by random things like disk failures. Our alerting got much harder because the volume was smaller since it was split into 10 parts. The smaller the baseline is the harder it is to detect problematic variations. We frequently got paged because some customer in a small region did something unusual in the middle of the night or someone tried an outgoing spam attack. That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.

stavros 5y ago

I share your frustration, but you have to factor in what we get for all the complexity. IAM roles are much more secure than "pwn one, pwn them all", containers are much easier to develop and deploy with, etc. I'm not saying it's worth it, but we should evaluate it on those merits. I, too, have a suspicion it's not, but maybe the larger ops team on AWS can run circles around the smaller ops team on on-prem? There has to be a reason, after all. > I am now an "old man despairs at cloud" meme.... "At cloud" indeed :P

antod 5y ago
It's not the cost or complexity that really bothers me (that was just a side rant hehe), more the vendor "productisation" of all the tools. After all, in many ways open source stuff is often lower level with the extra complexity that involves. With open source tools (hell, even they are often turning into products now) there was always a feeling you could dig in and understand them, then join the community and contribute, and even shape their future. Now tools are just things to consume on the vendors terms. Maybe the open pendulum will swing back again in 10yrs - probably at the hands of something like the Cloud Native Foundation, rather than the last peak 10yrs ago driven by Linux distros and some key apps/languages as well as dissatisfaction with MS etc.
- stavros 5y ago
  Ah yeah, that's frustrating to me as well. The network is now the OS, and it's proprietary.

scarface74 5y ago

I also did the 6 developer MS cert thing in 2010. Not because I thought certifications were valuable. Just as a guided learning path. I never told anyone or put them on my resume. I was transitioning from a decade of C bit twiddling. I did the same thing 8 years later to learn the AWS ecosystem. Where I now work 2 years later in ProServe (just disclaiming my biases). There is a chance I could retire from AWS or I could move on to the next hype cycle. Managing a career in technology is about staying relevant. It’s not like my vast knowledge of 65C02 assembly that I learned in the 80s in middle school comes in handy in 2021. However “my general tech knowledge” has served me well as well as my ability to know what level of the stack provides a business an “unfair advantage”. Managing infrastructure is seldom it unless you’re a DropBox. Every company is dependent on dozens of vendors for their operations.