4 ms·

Yup, I hope to god we are moving past the age of 'everything's fast if you have enough machines' and 'money is not real' era of software development. I remembe

Yup, I hope to god we are moving past the age of 'everything's fast if you have enough machines' and 'money is not real' era of software development. I remember the point in my career when I moved from a cranky old .NET company, where we handled millions of users from a single cabinent's worth of beefy servers, to a cloud based shop where we used every cloud buzzword tech under the sun (but mainly everything was containerized node microservices). I shudder thinking back to the eldritch horrors I saw on the cloud billing side, and the funny thing is, we were constantly fighting performance problems.

bombcar 1y ago
My conspiracy theory is that "cloud scaling" was entirely driven by people who grew up watching sites get slash dotted and thought it was the absolute most important thing in the world that you can quickly scale up to infinity billion requests/second.
- colechristensen 1y ago
  No, cloud adoption was driven by teams having to wait 2 years for capex for their hardware purchase and then getting a quarter of what they asked for. You couldn't get things, people hoarded servers they pretended to be using because when they did need something they couldn't get it. Management just wouldn't approve budgets so you were stuck using too little hardware. On the cloud it takes five seconds to get a new machine I can ssh into and I don't have to ask anyone for the budget. You can save a lot of money with scaling, you have to actually do that though and very few places do.
  - dinvlad 1y ago
    And now, on cloud it’s the same but much more expensive and worse performance. We’ve been struggling for over a month to get a single (1) non-beefy non-GPU VM allocated on Azure, since they’ve been having insane capacity issues, to the extent that even “provisioned” capacity cannot be fulfilled ;-(
    - lazide 1y ago
      Sure, but that’s because Azure. I’m sorry someone made the decision to go there. AWS & GCP, stock outs at least used to be nearly unheard of.
      - dijit 1y ago
        Until you hit a certain scale. I totally agree about Azure being the worst of the three, they wanted us to commit to certain use before even buying hardware themselves. Crazy… But I also had capacity issues with Google at large scales in many zones.
        lazide 1y ago
        What sort of scale, if you don’t mind me asking?
        dijit 1y ago
        Hey, sure! That’s important context. One gameserver was 40vCPU and 256G of RAM, we had about 30-50 before we’d see some issues in some regions. (this is from memory now unfortunately). Sao Paulo and Tokyo being the worst, but Singapore, Australia and Mumbai also had issues at various times. The other places where we hit hard limits was Los Angeles, but we had more than a hundred instances then. The issue with the hard limits is that it’ll be one zone thats exceeded and the API will fail, so you have to retry with another zone in the same region, but you don’t get to practice building your autoscaler before you actually need it.
        lazide 1y ago
        Oh interesting. Yeah, and I’m guessing you don’t get much prior visibility into available stock, since that would also expose info to their competitors. That is a lot lower than I expected, but I also imagine that’s a sizable order that they like getting.
      - solatic 1y ago
  - everforward 1y ago
    I think this is partially that for the past decade or two, on-prem was partially preferred by very frugal companies. One of the places I worked that was on-prem enforced a "standard hardware profile" where the servers were all nearly the same except things that could be changed in house (like RAM sticks). When they ordered hardware, they'd order like 5% or 10% more than we thought we'd need to keep an "emergency pool". If you ended up needing more hardware than you thought and could justify why you needed it right now, they'd dip into that pool to give you more hardware on a pretty rapid schedule. It cost slightly more, but was invaluable. Need double the hardware for a week to do a migration? No problem. Product more popular than you thought? No problem.
    - colechristensen 1y ago
      >I think this is partially that for the past decade or two, on-prem was partially preferred by very frugal companies. Sure this is made worse by frugality, but I experienced this problem when virtualization was in its infancy, much less cloud anything even existing much less being popular.
  - bombcar 1y ago
    There's "cloud" as "server in the cloud I can use" which is what the majority of smaller players are using - it's just someone else's server. There's also "cloud" as the API-driven world of managed services that drain your wallet faster than you can blink.
  - drob518 1y ago
    This. Instant availability of compute and storage, purchasable with a corporate credit card, was the cloud killer app.
    - Foobar8568 1y ago
      I can't have instant ressources, azure Switzerland is overbooked and can't create much ressources, at two different clients. And I read something similar for AWS. And in most "large" companies, you still need to go to different teams and processes to have those cloud ressources. Oh and if your company made the mistake to outsource their IT, odds that you'll have a 4 digit bills to change a teraform. Welcome to Western Europe.
  - kaliszad 1y ago
    Seems like a few negotiation skills would be of better use than doing extreme amounts of work so somebody can take months on approving new hardware. Well guess what, the people that did slow down hardware procurement are now slowing down deployment of cloud resources as well because the fundamental problem wasn't addressed and that's organizational misalignment and disfunction.
    - colechristensen 1y ago
      You're going to negotiate with the structure of an entire corporation? Excuse me CEO your budgeting process is inconvenient to my work, please change it all to be more responsive. This is not how things work and not how changes are made. Unless you get into the C-suite or at least become a direct report of someone who is nobody cares and you're told to just deal with it. You can't negotiate because you're four levels of management away from being invited to the table. An organization that can make agile capital expenditures in response to the needs of an individual contributor is either barely out of the founder's garage or a magical fairyland unicorn.
      - kaliszad 1y ago
        You would be surprised. I have unclogged processes inside and between companies before where people would tell me it's not possible or that success is unlikely. And yes, if you are working with professionals most expenditures can be planned well in advance so you definitely can take months or even more than a year in the process. If there is a major issue affecting the business of the company you would be surprised what is possible. I've got things approved in weeks in an enterprise of 2000+ employees before and I was 5 levels down from the CEO then. I have improved my negotiation skills greatly since then (it helps when you co-found a consulting company).
    - bluedino 1y ago
      Even if you can get instant approval, Dell or HP won't Fedex you 50 servers next day like they will 50 laptops. And once you're a customer you get to deal with sales channels full of quotes and useless meetings and specialists. You can't just order from the website.
  - jjav 1y ago
    > On the cloud it takes five seconds to get a new machine I can ssh into and I don't have to ask anyone for the budget. This isn't exactly realistic, not for too long anyway. Once your cloud bill climbs into the millions, expect to see just as much scrutiny on what's costing so much and what can be cut and can you really justify the new thing you want to spin up (as there should be). Having been through many growing startups, I'd say the freewheeling days of spin up whatever you want only last to about 50K to 100K/month AWS billings.
neonsunset 1y ago
However old, .NET Framework was still using a decent JIT compiler with a statically typed language, powerful GC and a fully multi-threaded environment :) Of course Node could not compete, and the cost had to be paid for each thinly sliced microservice carrying heavy runtime alongside it.
- pdimitar 1y ago
  I don't think we need to bring languages and frameworks into this. Some of them make things worse, others -- much better. Furthermore, the microservices craze only made things worse regardless of PL / framework. IMO we have an entire generation (maybe two) of devs who never self-hosted. That's the main audience of the article and of many of the comments here. The rest of us who only scoffed at the cloud and microservices were just waiting the world to start coming back to its senses again.
dematz 1y ago
Tangential point but why is it that so often these leaving the cloud posts use the word "beefy" to describe the servers? It's always you don't need cloud because beefy servers handle pretty much any bla bla bla https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=beefy&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... If anyone from oxide computer or similar is reading, maybe you should rebrand to BEEFY server inc...
- fsckboy 1y ago
  >Tangential point... rebrand to BEEFY server idea for an ad campaign: "mmm, beefy!" https://www.youtube.com/watch?v=j6ImwKMRq98&t=21s https://www.youtube.com/watch?v=j6ImwKMRq98&t=21s i don't know how "worldwide" is the distribution of Chef "Boyardee" canned spaghetti (which today is not too good), but the founder's story is fairly interesting. He was a real Italian immigrant chef named Ettore Boiardi and he gets a lot of credit for originally evangelizing and popularizing "spaghetti" and "spaghetti sauce" in America when most people had never tried it. https://en.wikipedia.org/wiki/Ettore_Boiardi https://en.wikipedia.org/wiki/Ettore_Boiardi you know, "spaghetti code..."?
- gwking 1y ago
  The servers are always beefy and the software is always blazingly fast. Blazingly beefy is my new joke trademark.
- torginus 1y ago
  Because that is a casual word in the English language to describe an object with substantial power? If you would suggest a word that would make a better substitute in this case, that could move the conversation forward, and perhaps you could improve the aesthetic quality of posts about leaving the cloud.
  - saurik 1y ago
    Well, I wouldn't say I have a beefy car or a beefy sword: there's some historical bit of linguistic connection that seems to have caused people to describe a server with the adjective "beefy", rather than "powerful", "hefty", "stacked", or... "chonky" ;P. Coming up with some options, I think my favorite might be "cranked"?
jpfromlondon 1y ago
>to a cloud based shop where we used every cloud buzzword tech under the sun this is me currently, not so much "tech" as repackaged cloud services as SAAS and middleware, department is amortizing hours and effort against projects, the eldritch horrors you're referring to are starting to manifest and I want out.

bombcar 1y ago

My conspiracy theory is that "cloud scaling" was entirely driven by people who grew up watching sites get slash dotted and thought it was the absolute most important thing in the world that you can quickly scale up to infinity billion requests/second.

colechristensen 1y ago
No, cloud adoption was driven by teams having to wait 2 years for capex for their hardware purchase and then getting a quarter of what they asked for. You couldn't get things, people hoarded servers they pretended to be using because when they did need something they couldn't get it. Management just wouldn't approve budgets so you were stuck using too little hardware. On the cloud it takes five seconds to get a new machine I can ssh into and I don't have to ask anyone for the budget. You can save a lot of money with scaling, you have to actually do that though and very few places do.
- dinvlad 1y ago
  And now, on cloud it’s the same but much more expensive and worse performance. We’ve been struggling for over a month to get a single (1) non-beefy non-GPU VM allocated on Azure, since they’ve been having insane capacity issues, to the extent that even “provisioned” capacity cannot be fulfilled ;-(
  - lazide 1y ago
    Sure, but that’s because Azure. I’m sorry someone made the decision to go there. AWS & GCP, stock outs at least used to be nearly unheard of.
    - dijit 1y ago
      Until you hit a certain scale. I totally agree about Azure being the worst of the three, they wanted us to commit to certain use before even buying hardware themselves. Crazy… But I also had capacity issues with Google at large scales in many zones.
      - lazide 1y ago
        What sort of scale, if you don’t mind me asking?
        dijit 1y ago
        Hey, sure! That’s important context. One gameserver was 40vCPU and 256G of RAM, we had about 30-50 before we’d see some issues in some regions. (this is from memory now unfortunately). Sao Paulo and Tokyo being the worst, but Singapore, Australia and Mumbai also had issues at various times. The other places where we hit hard limits was Los Angeles, but we had more than a hundred instances then. The issue with the hard limits is that it’ll be one zone thats exceeded and the API will fail, so you have to retry with another zone in the same region, but you don’t get to practice building your autoscaler before you actually need it.
        lazide 1y ago
        Oh interesting. Yeah, and I’m guessing you don’t get much prior visibility into available stock, since that would also expose info to their competitors. That is a lot lower than I expected, but I also imagine that’s a sizable order that they like getting.
    - solatic 1y ago
- everforward 1y ago
  I think this is partially that for the past decade or two, on-prem was partially preferred by very frugal companies. One of the places I worked that was on-prem enforced a "standard hardware profile" where the servers were all nearly the same except things that could be changed in house (like RAM sticks). When they ordered hardware, they'd order like 5% or 10% more than we thought we'd need to keep an "emergency pool". If you ended up needing more hardware than you thought and could justify why you needed it right now, they'd dip into that pool to give you more hardware on a pretty rapid schedule. It cost slightly more, but was invaluable. Need double the hardware for a week to do a migration? No problem. Product more popular than you thought? No problem.
  - colechristensen 1y ago
    >I think this is partially that for the past decade or two, on-prem was partially preferred by very frugal companies. Sure this is made worse by frugality, but I experienced this problem when virtualization was in its infancy, much less cloud anything even existing much less being popular.
- bombcar 1y ago
  There's "cloud" as "server in the cloud I can use" which is what the majority of smaller players are using - it's just someone else's server. There's also "cloud" as the API-driven world of managed services that drain your wallet faster than you can blink.
- drob518 1y ago
  This. Instant availability of compute and storage, purchasable with a corporate credit card, was the cloud killer app.
  - Foobar8568 1y ago
    I can't have instant ressources, azure Switzerland is overbooked and can't create much ressources, at two different clients. And I read something similar for AWS. And in most "large" companies, you still need to go to different teams and processes to have those cloud ressources. Oh and if your company made the mistake to outsource their IT, odds that you'll have a 4 digit bills to change a teraform. Welcome to Western Europe.
- kaliszad 1y ago
  Seems like a few negotiation skills would be of better use than doing extreme amounts of work so somebody can take months on approving new hardware. Well guess what, the people that did slow down hardware procurement are now slowing down deployment of cloud resources as well because the fundamental problem wasn't addressed and that's organizational misalignment and disfunction.
  - colechristensen 1y ago
    You're going to negotiate with the structure of an entire corporation? Excuse me CEO your budgeting process is inconvenient to my work, please change it all to be more responsive. This is not how things work and not how changes are made. Unless you get into the C-suite or at least become a direct report of someone who is nobody cares and you're told to just deal with it. You can't negotiate because you're four levels of management away from being invited to the table. An organization that can make agile capital expenditures in response to the needs of an individual contributor is either barely out of the founder's garage or a magical fairyland unicorn.
    - kaliszad 1y ago
      You would be surprised. I have unclogged processes inside and between companies before where people would tell me it's not possible or that success is unlikely. And yes, if you are working with professionals most expenditures can be planned well in advance so you definitely can take months or even more than a year in the process. If there is a major issue affecting the business of the company you would be surprised what is possible. I've got things approved in weeks in an enterprise of 2000+ employees before and I was 5 levels down from the CEO then. I have improved my negotiation skills greatly since then (it helps when you co-found a consulting company).
  - bluedino 1y ago
    Even if you can get instant approval, Dell or HP won't Fedex you 50 servers next day like they will 50 laptops. And once you're a customer you get to deal with sales channels full of quotes and useless meetings and specialists. You can't just order from the website.
- jjav 1y ago
  > On the cloud it takes five seconds to get a new machine I can ssh into and I don't have to ask anyone for the budget. This isn't exactly realistic, not for too long anyway. Once your cloud bill climbs into the millions, expect to see just as much scrutiny on what's costing so much and what can be cut and can you really justify the new thing you want to spin up (as there should be). Having been through many growing startups, I'd say the freewheeling days of spin up whatever you want only last to about 50K to 100K/month AWS billings.

neonsunset 1y ago

However old, .NET Framework was still using a decent JIT compiler with a statically typed language, powerful GC and a fully multi-threaded environment :) Of course Node could not compete, and the cost had to be paid for each thinly sliced microservice carrying heavy runtime alongside it.

pdimitar 1y ago
I don't think we need to bring languages and frameworks into this. Some of them make things worse, others -- much better. Furthermore, the microservices craze only made things worse regardless of PL / framework. IMO we have an entire generation (maybe two) of devs who never self-hosted. That's the main audience of the article and of many of the comments here. The rest of us who only scoffed at the cloud and microservices were just waiting the world to start coming back to its senses again.

dematz 1y ago

Tangential point but why is it that so often these leaving the cloud posts use the word "beefy" to describe the servers? It's always you don't need cloud because beefy servers handle pretty much any bla bla bla https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=beefy&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... If anyone from oxide computer or similar is reading, maybe you should rebrand to BEEFY server inc...

fsckboy 1y ago
>Tangential point... rebrand to BEEFY server idea for an ad campaign: "mmm, beefy!" https://www.youtube.com/watch?v=j6ImwKMRq98&t=21s https://www.youtube.com/watch?v=j6ImwKMRq98&t=21s i don't know how "worldwide" is the distribution of Chef "Boyardee" canned spaghetti (which today is not too good), but the founder's story is fairly interesting. He was a real Italian immigrant chef named Ettore Boiardi and he gets a lot of credit for originally evangelizing and popularizing "spaghetti" and "spaghetti sauce" in America when most people had never tried it. https://en.wikipedia.org/wiki/Ettore_Boiardi https://en.wikipedia.org/wiki/Ettore_Boiardi you know, "spaghetti code..."?
gwking 1y ago
The servers are always beefy and the software is always blazingly fast. Blazingly beefy is my new joke trademark.
torginus 1y ago
Because that is a casual word in the English language to describe an object with substantial power? If you would suggest a word that would make a better substitute in this case, that could move the conversation forward, and perhaps you could improve the aesthetic quality of posts about leaving the cloud.
- saurik 1y ago
  Well, I wouldn't say I have a beefy car or a beefy sword: there's some historical bit of linguistic connection that seems to have caused people to describe a server with the adjective "beefy", rather than "powerful", "hefty", "stacked", or... "chonky" ;P. Coming up with some options, I think my favorite might be "cranked"?

jpfromlondon 1y ago

>to a cloud based shop where we used every cloud buzzword tech under the sun this is me currently, not so much "tech" as repackaged cloud services as SAAS and middleware, department is amortizing hours and effort against projects, the eldritch horrors you're referring to are starting to manifest and I want out.