10 ms·

Surprised to see cloud computing will be affected, but then again some analysts reported that AWS growth slowed to only 20% at the end of Q4. But based on what

Surprised to see cloud computing will be affected, but then again some analysts reported that AWS growth slowed to only 20% at the end of Q4. But based on what we are seeing they are still having a lot of success with on-prem to cloud migration deals (Southwest Airlines being the most recently announced one). We've also been tracking the shift in cloud spending patterns that Amazon is describing on earnings calls and how that may be compressing margins. In Q4 this was particularly noticeable, https://www.vantage.sh/cloud-cost-report/2022-q4 https://www.vantage.sh/cloud-cost-report/2022-q4

nimbius 4y ago
cloud computings darker secret is armies of overworked sysadmins and SRE's toiling in obscurity through exhausting oncall hours behind the scenes to keep everything running despite these sorts of cuts and mountains of tech debt.. anyone whos worked for a hosting provider can attest.
- spenczar5 4y ago
  Is that really a secret? I have worked both on the inside at AWS, and on the outside at groups which used AWS and Google Cloud. “They will do the huge amount of sysadmin toil” is, like, the number one selling point versus using on-prem systems, and its what justifies the prices.
  - throwboatyface 4y ago
    The cloud providers like to pretend their secret sauce is massive scale automation. They have tooling, alerting, etc. to make this all work with no people at all. Of course it's all BS, and the people I know who came from AWS were some of the worst for throwing bodies at problems and manually tweaking shit without trying to understand or automate solutions.
    - wnolens 4y ago
      It is their secret sauce and like a startup who licks and sticks envelopes initially while proving their business model, eventually it is automated. But the thirst for growth is never quenched, so there's always new features, new services. And maintenance of that automation. The work never ends. There's a 25-50% operations tax every AWS employee pays. But they're handling the operations load that several employees were 10y ago. I used to wake up in the middle of the night to restart individual machines. Now I'm alerted for much much more complex problems. But i'm still woken up :(
    - markus_zhang 4y ago
      I admin a small part of our Google cloud part time and I feel I absolutely have no time and no idea how to drill deeper than clickOps. Knowing where is everything and the general risks of toggling a switch are already nerve breaking. Sometimes I wish I could learn everything from scratch. I wish I could join an on prem team to learn all those small details slowly and painfully.
      - yourapostasy 4y ago
        > I wish I could join an on prem team to learn all those small details slowly and painfully. Learning "from the middle" of a tech stack outwards is the default case for everyone, so don't be so hard on yourself. There is no "from scratch", never was even in the Bombe days, just a question of how many abstraction layers you want to tackle. Give yourself time to learn and plenty of room to fail and try again, and "all those small details" will ingrain themselves into you as second nature.
    - tempnow987 4y ago
      They throw bodies because they have them and they are looking to solve a problem. And they do automate, claims that a service like S3 does not have automation are totally ridiculous, and few other general hosting providers have AWS scale here. That said, their development pace is crazy, with plenty of partially built solutions -> so yes, until something scales enough to need it I'm not surprised they have sysadmin toil. Reality is even the VMware worlds and other onprem stuff, even when "automated" have plenty of turmoil just staying fresh / current / backed up / secured etc. So that's what AWS is selling.
    - lightbendover 4y ago
      All of big tech is throwing bodies at problems at this point since decades of data is all rotted and programmatic solutions simply don't work. IF you stay at Amazon after all of this attrition, your life will be hell.
    - Aperocky 4y ago
      But it's not BS, automation is essential. I'd look for other reasons..
    - twblalock 4y ago
      And yet AWS is pretty reliable overall, and certainly better than most in-house systems even at large companies. I know multiple people who have worked on AWS. There is definitely some manual stuff going on, but there is also considerable infrastructure and automation in other areas.
  - pclmulqdq 4y ago
    The amount of sysadmin/SRE toil in some parts of the stack (all the distributed computing coordination layers) is super-linear in the growth of the system, but the nuanced version of the claim is that they can make up for it in the other parts of the stack (hardware cost and better global availability). A lot of people who have never seen these systems from the inside would be surprised at (1) how much engineering work is put into just getting things like "Postgres but it scales" and "Kubernetes but it works on our proprietary VM manager" to work and (2) how many internal incidents these cloud providers have that are quietly managed before they rise to the level where a customer would notice or care.
- hinkley 4y ago
  The “best” thing about layoffs/attrition is that onerous task you used to split with several people you now have to do twice as often.
- zarathustreal 4y ago
  (opinions my own) - I actually do work for $FAANG and can confirm this statement. Mountains of tech debt, 20+ year old tech running most of the stack, most tooling written in dynamic interpreted languages, and above all else every single engineer is expected to be part of on-call rotations, even if your role is not related to infrastructure or operations whatsoever. Even if you don't touch the service(s) your team maintains because you're in a FEE role. The SDE role at $FAANG I work at is essentially glorified tech support, which makes the leetcode interviewing process so much more ridiculous than it would be by default if you were actually writing code and solving meaningful problems most of the time.
  - shric 4y ago
    > The SDE role at (FAANG) I don't think you're generalising to all FAANG, just whichever one you work at. That said, this has not been my experience at all at another FAANG which doesn't use the TLA "SDE".
    - zarathustreal 4y ago
      Using (FAANG) for anonymity, but you're absolutely correct
    - influx 4y ago
      I’ve worked at Twitter, Uber, Amazon, and Facebook. They all had mountains of tech debt and oncall burden. Which one do you work at so I can go there?
      - ryandrake 4y ago
        I think you're going to find this at nearly all software companies, FAANG or smaller. The sad state of our industry is that everything is so poorly written that it's falling apart daily, and each company has an army of people trying to toss water on the last fire that popped up. This is the natural steady state of an industry that prioritizes schedule and feature cram over quality and craftsmanship, and prioritizes human heroics/toil over process and automation. Move fast and break things.
- walrus01 4y ago
  It's well known in the ISP industry that being an entry to mid level sysadmin/network eng at a hosting provider that has large numbers of very-low-dollar-amount per month customers is a new circle of personal hell. Working endless tickets from people who pay $20 a month for some VPS. The only way to get out of it is to become senior enough to not care anymore or work for a hosting operator that specializes in only higher dollar value colo/dedicated server customers.
  - deleted 4y ago
    [deleted]
- paxys 4y ago
  What's the "dark secret" exactly? The entire point of AWS is companies being able to outsource this army of sysadmins so they don't have to hire them themselves. That's also why they pay 3-5x the cost of buying an equivalent server to Amazon.
- vishnugupta 4y ago
  From multiple reliable sources I've heard that AWS's DynanamoDB team/ops is a meat grinder. The open tickets run into thousands with no hope of anyone getting time to work on permanent fixes.
j45 4y ago
Cloud computing can cost 3-5x self-managing with the open source versions of the standardized cloud tools that are sold to us. One big difference working against the cloud is the same thing that worked in it's favour 15 years ago. We don't always remember why we just adopted the cloud. Linux networking at times had scaling issues, and virtualization and linux was not where it is now. It's astounding how much more a $5 VPS can do. Imagine the medium end. Self-managed hosting tools have come so far to handle production quality, it doesn't even feel fair that it is not more well known. Whether it's Proxmox, or others, it is pretty close to most things out there. Making a tool cloud compatible, but cloud agnostic is one way to avoid a lot of technical debt (assuming it's not created another way). It doesn't have to be for everyone, but demand is probably slowing in part because of this. In a recession cost efficiency is everything.
- varispeed 4y ago
  I wish Rancher had never stopped the development of 1.6. It was such a simple and beautiful tool to run services. Now they moved to Kubernetes, it has added a ton of unnecessary complexity. It takes magnitude more effort to run anything on it and they call it progress...
  - j45 4y ago
    It’s probably making a developers life easier somewhere instead of the users. Sometimes it can be both but not always.
markus_zhang 4y ago
We are a medium large company (close to 10k employees globally) and the main focus of this year of the data teams is to figure out every possibility to reduce Google Cloud costs. The savings could be in 7 digits TBH annually. There are so many low hanging fruits no one cares when the day is good. Tables that have no partition expiration that grow into hundreds or even thousands of TBs, buckets with same amount of data that no one looks at, VMs that no service uses, large clusters well above their load (e.g. only 20% used maximum -- and a full replica too), very bad ETL pipelines that cost hundreds of bucks EVERY RUN, very costly queries generated by Looker dashboards EVERY view...there are just too many.
- lazystar 4y ago
  just messy bread making machines, as joel spoelsky would likely put it https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...
- ghaff 4y ago
  FinOps is definitely becoming a pretty hot topic. And, yeah, a lot of companies are realizing that they have some pretty low hanging fruit where they can cut spend significantly.
  - sneak 4y ago
    Feels like there is a startup in there? Give it your account number, and read only access keys, and it tells you what and where and how to change things.
    - deleted 4y ago
      [deleted]
    - ghaff 4y ago
      There are companies out there. I expect to see a number of them when I'm at Kubecon in a month or so.
    - 4y ago
varispeed 4y ago
Cloud computing is a massive rip off. From my experience with start ups, they were using things like AWS, because otherwise the investors would be reluctant to put their money in. As in they were afraid of dedicated hosting and something that small business wouldn't be able to handle. So for instance company would have their hosting costs upped from £500 a month (dedicated servers) to something like £5000, so that they could "scale up", while their dedicated servers could have easily served almost any traffic thrown at them and be more responsive. They have actually never reached the level of traffic that this old set up wouldn't be able to handle, but investors were always saying "what if we get sudden surge of users" which has never materialised. But then it was their money so nobody objected. I can see that now that hosting costs in the cloud have become ridiculous, cash strapped smaller businesses will be looking for savings and the cloud is quite an obvious area to look for cuts.
- unoti 4y ago
  > while their dedicated servers could have easily served almost any traffic thrown at them and be more responsive Sounds great, for some use cases. Particularly if you don't need multiple 9's of uptime, and just need a few 8's. Most companies don't have: * Guaranteed sla's for replacing components when the RAM needs to be reseated, a drive fails, or the fans get too dusty and things overheat * Redundancy set up in case the network connection fails * Redundancy set up in case the power fails. (Many places will have a UPS for under an hour, but generators, along with a regular maintenance schedule and processes for ensuring fuel is another thing) * On site security, to make sure nobody breaks in and does some corporate espionage * Ultra-high-speed links to the internet And if the little startup takes on a client with important certification requirements, it can be very helpful to have a cloud provider. A lot of contracts I've taken on over the years ask tough questions during the sales process about our practices regarding the above things and much more.
  - genmud 4y ago
    Have you never used colos? They literally tick all the boxes and it would be one of the worst colos I have ever seen to not have 3x9’s.
  - 4y ago
ptero 4y ago
This is a "you can write bad code in any language" paradigm. Doing computing efficiently takes skill, whether one does it in the cloud or on prem. Cloud solved a few common pain points and, at times of startup / tech boom, had a meteoric rise. But now many of the companies that moved to the cloud are hitting different challenges and are stuck in a setup that is pretty expensive to maintain. I am not expecting the death of the cloud computing, but that pendulum will likely swing back a little as the companies experiment with other options. I think a company that can help medium sized businesses optimize their compute setup with a mix of on prem and multi-provider cloud will do well. My 2c.
- KptMarchewa 4y ago
  >This is a "you can write bad code in any language" paradigm. It's extremely true! The fun is that bad code looks totally different from each other. The "bad" Scala type astronaut code would be unrecognizable from Python raw dictionary enthusiast code.

nimbius 4y ago

cloud computings darker secret is armies of overworked sysadmins and SRE's toiling in obscurity through exhausting oncall hours behind the scenes to keep everything running despite these sorts of cuts and mountains of tech debt.. anyone whos worked for a hosting provider can attest.

spenczar5 4y ago
Is that really a secret? I have worked both on the inside at AWS, and on the outside at groups which used AWS and Google Cloud. “They will do the huge amount of sysadmin toil” is, like, the number one selling point versus using on-prem systems, and its what justifies the prices.
- throwboatyface 4y ago
  The cloud providers like to pretend their secret sauce is massive scale automation. They have tooling, alerting, etc. to make this all work with no people at all. Of course it's all BS, and the people I know who came from AWS were some of the worst for throwing bodies at problems and manually tweaking shit without trying to understand or automate solutions.
  - wnolens 4y ago
    It is their secret sauce and like a startup who licks and sticks envelopes initially while proving their business model, eventually it is automated. But the thirst for growth is never quenched, so there's always new features, new services. And maintenance of that automation. The work never ends. There's a 25-50% operations tax every AWS employee pays. But they're handling the operations load that several employees were 10y ago. I used to wake up in the middle of the night to restart individual machines. Now I'm alerted for much much more complex problems. But i'm still woken up :(
  - markus_zhang 4y ago
    I admin a small part of our Google cloud part time and I feel I absolutely have no time and no idea how to drill deeper than clickOps. Knowing where is everything and the general risks of toggling a switch are already nerve breaking. Sometimes I wish I could learn everything from scratch. I wish I could join an on prem team to learn all those small details slowly and painfully.
    - yourapostasy 4y ago
      > I wish I could join an on prem team to learn all those small details slowly and painfully. Learning "from the middle" of a tech stack outwards is the default case for everyone, so don't be so hard on yourself. There is no "from scratch", never was even in the Bombe days, just a question of how many abstraction layers you want to tackle. Give yourself time to learn and plenty of room to fail and try again, and "all those small details" will ingrain themselves into you as second nature.
  - tempnow987 4y ago
    They throw bodies because they have them and they are looking to solve a problem. And they do automate, claims that a service like S3 does not have automation are totally ridiculous, and few other general hosting providers have AWS scale here. That said, their development pace is crazy, with plenty of partially built solutions -> so yes, until something scales enough to need it I'm not surprised they have sysadmin toil. Reality is even the VMware worlds and other onprem stuff, even when "automated" have plenty of turmoil just staying fresh / current / backed up / secured etc. So that's what AWS is selling.
  - lightbendover 4y ago
    All of big tech is throwing bodies at problems at this point since decades of data is all rotted and programmatic solutions simply don't work. IF you stay at Amazon after all of this attrition, your life will be hell.
  - Aperocky 4y ago
    But it's not BS, automation is essential. I'd look for other reasons..
  - twblalock 4y ago
    And yet AWS is pretty reliable overall, and certainly better than most in-house systems even at large companies. I know multiple people who have worked on AWS. There is definitely some manual stuff going on, but there is also considerable infrastructure and automation in other areas.
- pclmulqdq 4y ago
  The amount of sysadmin/SRE toil in some parts of the stack (all the distributed computing coordination layers) is super-linear in the growth of the system, but the nuanced version of the claim is that they can make up for it in the other parts of the stack (hardware cost and better global availability). A lot of people who have never seen these systems from the inside would be surprised at (1) how much engineering work is put into just getting things like "Postgres but it scales" and "Kubernetes but it works on our proprietary VM manager" to work and (2) how many internal incidents these cloud providers have that are quietly managed before they rise to the level where a customer would notice or care.
hinkley 4y ago
The “best” thing about layoffs/attrition is that onerous task you used to split with several people you now have to do twice as often.
zarathustreal 4y ago
(opinions my own) - I actually do work for $FAANG and can confirm this statement. Mountains of tech debt, 20+ year old tech running most of the stack, most tooling written in dynamic interpreted languages, and above all else every single engineer is expected to be part of on-call rotations, even if your role is not related to infrastructure or operations whatsoever. Even if you don't touch the service(s) your team maintains because you're in a FEE role. The SDE role at $FAANG I work at is essentially glorified tech support, which makes the leetcode interviewing process so much more ridiculous than it would be by default if you were actually writing code and solving meaningful problems most of the time.
- shric 4y ago
  > The SDE role at (FAANG) I don't think you're generalising to all FAANG, just whichever one you work at. That said, this has not been my experience at all at another FAANG which doesn't use the TLA "SDE".
  - zarathustreal 4y ago
    Using (FAANG) for anonymity, but you're absolutely correct
  - influx 4y ago
    I’ve worked at Twitter, Uber, Amazon, and Facebook. They all had mountains of tech debt and oncall burden. Which one do you work at so I can go there?
    - ryandrake 4y ago
      I think you're going to find this at nearly all software companies, FAANG or smaller. The sad state of our industry is that everything is so poorly written that it's falling apart daily, and each company has an army of people trying to toss water on the last fire that popped up. This is the natural steady state of an industry that prioritizes schedule and feature cram over quality and craftsmanship, and prioritizes human heroics/toil over process and automation. Move fast and break things.
walrus01 4y ago
It's well known in the ISP industry that being an entry to mid level sysadmin/network eng at a hosting provider that has large numbers of very-low-dollar-amount per month customers is a new circle of personal hell. Working endless tickets from people who pay $20 a month for some VPS. The only way to get out of it is to become senior enough to not care anymore or work for a hosting operator that specializes in only higher dollar value colo/dedicated server customers.
- deleted 4y ago
  [deleted]
paxys 4y ago
What's the "dark secret" exactly? The entire point of AWS is companies being able to outsource this army of sysadmins so they don't have to hire them themselves. That's also why they pay 3-5x the cost of buying an equivalent server to Amazon.
vishnugupta 4y ago
From multiple reliable sources I've heard that AWS's DynanamoDB team/ops is a meat grinder. The open tickets run into thousands with no hope of anyone getting time to work on permanent fixes.

j45 4y ago

Cloud computing can cost 3-5x self-managing with the open source versions of the standardized cloud tools that are sold to us. One big difference working against the cloud is the same thing that worked in it's favour 15 years ago. We don't always remember why we just adopted the cloud. Linux networking at times had scaling issues, and virtualization and linux was not where it is now. It's astounding how much more a $5 VPS can do. Imagine the medium end. Self-managed hosting tools have come so far to handle production quality, it doesn't even feel fair that it is not more well known. Whether it's Proxmox, or others, it is pretty close to most things out there. Making a tool cloud compatible, but cloud agnostic is one way to avoid a lot of technical debt (assuming it's not created another way). It doesn't have to be for everyone, but demand is probably slowing in part because of this. In a recession cost efficiency is everything.

varispeed 4y ago
I wish Rancher had never stopped the development of 1.6. It was such a simple and beautiful tool to run services. Now they moved to Kubernetes, it has added a ton of unnecessary complexity. It takes magnitude more effort to run anything on it and they call it progress...
- j45 4y ago
  It’s probably making a developers life easier somewhere instead of the users. Sometimes it can be both but not always.

markus_zhang 4y ago

We are a medium large company (close to 10k employees globally) and the main focus of this year of the data teams is to figure out every possibility to reduce Google Cloud costs. The savings could be in 7 digits TBH annually. There are so many low hanging fruits no one cares when the day is good. Tables that have no partition expiration that grow into hundreds or even thousands of TBs, buckets with same amount of data that no one looks at, VMs that no service uses, large clusters well above their load (e.g. only 20% used maximum -- and a full replica too), very bad ETL pipelines that cost hundreds of bucks EVERY RUN, very costly queries generated by Looker dashboards EVERY view...there are just too many.

lazystar 4y ago
just messy bread making machines, as joel spoelsky would likely put it https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...
ghaff 4y ago
FinOps is definitely becoming a pretty hot topic. And, yeah, a lot of companies are realizing that they have some pretty low hanging fruit where they can cut spend significantly.
- sneak 4y ago
  Feels like there is a startup in there? Give it your account number, and read only access keys, and it tells you what and where and how to change things.
  - deleted 4y ago
    [deleted]
  - ghaff 4y ago
    There are companies out there. I expect to see a number of them when I'm at Kubecon in a month or so.
  - 4y ago

varispeed 4y ago

Cloud computing is a massive rip off. From my experience with start ups, they were using things like AWS, because otherwise the investors would be reluctant to put their money in. As in they were afraid of dedicated hosting and something that small business wouldn't be able to handle. So for instance company would have their hosting costs upped from £500 a month (dedicated servers) to something like £5000, so that they could "scale up", while their dedicated servers could have easily served almost any traffic thrown at them and be more responsive. They have actually never reached the level of traffic that this old set up wouldn't be able to handle, but investors were always saying "what if we get sudden surge of users" which has never materialised. But then it was their money so nobody objected. I can see that now that hosting costs in the cloud have become ridiculous, cash strapped smaller businesses will be looking for savings and the cloud is quite an obvious area to look for cuts.

unoti 4y ago
> while their dedicated servers could have easily served almost any traffic thrown at them and be more responsive Sounds great, for some use cases. Particularly if you don't need multiple 9's of uptime, and just need a few 8's. Most companies don't have: * Guaranteed sla's for replacing components when the RAM needs to be reseated, a drive fails, or the fans get too dusty and things overheat * Redundancy set up in case the network connection fails * Redundancy set up in case the power fails. (Many places will have a UPS for under an hour, but generators, along with a regular maintenance schedule and processes for ensuring fuel is another thing) * On site security, to make sure nobody breaks in and does some corporate espionage * Ultra-high-speed links to the internet And if the little startup takes on a client with important certification requirements, it can be very helpful to have a cloud provider. A lot of contracts I've taken on over the years ask tough questions during the sales process about our practices regarding the above things and much more.
- genmud 4y ago
  Have you never used colos? They literally tick all the boxes and it would be one of the worst colos I have ever seen to not have 3x9’s.
- 4y ago

ptero 4y ago

This is a "you can write bad code in any language" paradigm. Doing computing efficiently takes skill, whether one does it in the cloud or on prem. Cloud solved a few common pain points and, at times of startup / tech boom, had a meteoric rise. But now many of the companies that moved to the cloud are hitting different challenges and are stuck in a setup that is pretty expensive to maintain. I am not expecting the death of the cloud computing, but that pendulum will likely swing back a little as the companies experiment with other options. I think a company that can help medium sized businesses optimize their compute setup with a mix of on prem and multi-provider cloud will do well. My 2c.

KptMarchewa 4y ago
>This is a "you can write bad code in any language" paradigm. It's extremely true! The fun is that bad code looks totally different from each other. The "bad" Scala type astronaut code would be unrecognizable from Python raw dictionary enthusiast code.