5 ms·

When your cheap dedicated server goes down and your admin is on holiday and you have hundreds of angry customers calling you, you'll get it. Or you need to res

When your cheap dedicated server goes down and your admin is on holiday and you have hundreds of angry customers calling you, you'll get it. Or you need to restore your Postgres database and you find out that the backups didn't work. And finally you have a brilliant idea of hiring a second $150k/year dev ops admin so that at least one is always working and they can check each other's work. Suddenly, you're spending $300k on two dev ops admins alone and the cost savings of using cheaper dedicated servers are completely gone.

esseph 1y ago
Sure, but AWS has more downtime than I do :-)
- aurareturn 1y ago
  Yes but you go on holidays but AWS does not.
- fragmede 1y ago
  If we assume that you're a human being that sleeps, say 8 hrs/day, and not an LLM, that leaves you with 16 hours of consciousness a day, for an uptime of 66%. That's upsidedown nines. You don't even crack one nine of uptime. If we assume you've a diet of meth and cocaine, and only sleep 2 hours a day, that still puts you at only like 92% uptime.
  - esseph 1y ago
    > If we assume that you're a human being I'll have you know I am a cantaloupe, you insensitive clod!
grey-area 1y ago
It is statistically far more likely that your cloud service will go down for hours or days, and you will have no recourse and will just have to wait till AWS manage to resolve it.
- aurareturn 1y ago
  The difference is that if AWS goes down, I know for a fact that it'll be back up without me doing anything. If my own dedicated server goes down, I'm going to need to call my admin at 3am 10 times just to wake him up.
  - jrk 1y ago
    You know that AWS will come back up. You definitely don’t know whether your own instances will come back or if you’ll need to redeploy it all.
    - BrandoElFollito 11mo ago
      Why do you assume that the small dedicated server has a higher probability to come back?
  - KetoManx64 11mo ago
    If youe admin isn't competant enough to setup logging or notifications, how is it going to be better when your Cloud VM runs out of storage or doesn't reboot properly due to AWS swapping out hardware?
- n4r9 1y ago
  I suspect that this is really about liability. When AWS goes down you can just throw up your hands, everyone's in the same boat. If your own server goes down you worry that your customers doubt your competence. It's actually kinda frustrating - as an industry we're accepting worse outcomes due to misperceptions. That's how the free market goes sometimes.
  - deleted 1y ago
    [deleted]
  - PaulKeeble 1y ago
    Nobody gets fired for hiring IBM. This is the new version, when you go down because AWS did its someone else’s fault. Of course AWS will compare their downtime to industry standards for on premise and conclude they are down less often. On Premise engineers can say until they are blue that their downtime is on a Sunday at 3 am because it doesn't impact their customers it doesn't seem to matter.
Perz1val 1y ago
That's even worse when AWS goes down and myth of it never going down should be more than shattered by now
juliusceasar 1y ago
When your system goes down on AWS and your AWS admin is on holiday, you'll have the same problem. What is your point?
- aurareturn 1y ago
  I don't have an AWS admin. I assume a $2.4 trillion dollar company always has dev ops on call?
- wink 1y ago
  Every team I have worked on so far, if using AWS you had 50-100% of the developers with the knowledge and credentials (and usually the confidence) to troubleshoot/just fix it/replace it. Every team with dedicated hardware in a data center it was generally 1-2 people who would have fixed stuff quickly, no matter the size of the company (small ones, of course - so 10-50 devs). And that's with available replacement hardware. I'm not even one of the "cloud is so great" people - but it you're generally doing software it's actually a lot less friction. And while the ratio of cost difference may sound bad, it's generally not. Unless we're talkign huge scale, you can buy a lot of AWS crap for the yearly salary of a single person.
  - tpxl 1y ago
    You said developers have the knowledge and credentials (and thus the work) of managing your infra, and a moment later basically asserted you're saving money on the salary for the sysadmin. This is the actual lie you got sold on. AWS isn't going to help you setup your security, you have to do it yourself. Previously a sysadmin would do this, now it's the devs. They aren't going to monitor your database performance. Previously a sysadmin would do this, now it's the devs. They aren't going to setup your networking. Previously a sysadmin would do this, ... Managing hardware and updating hosts is maybe 10% of the work of a sysadmin. You can't buy much on 1/10th of a sysadmins salary, and even the things you can, the quality and response time are generally going to be shit compared to someone who cares about your company (been there).
    - wink 1y ago
oersted 1y ago
After 8 years operating like this, I have had approximately the same number of critical outages in standard Cloud as with these providers. One included a whole OVH building burning down with our server in it, and recovery was faster than the recent AWS and Cloudflare outages. We felt less impotent and we could do more to mitigate the situation. If you want to, these providers also offer VMs, object storage and other virtualized services for way cheaper with similar guarantees, they are not stuck in the last century. And I don’t know how people are using cloud, but most config issues happen above the VM/Docker/Kubernetes level, which is the same wether you are on cloud or not. Even fully managed database deployments or serverless backends are not really that much simpler or less error-prone than deploying the containers yourself. Actually the complexity of Cloud is often a worse minefield of footguns, with their myriad artificial quirks and limitations. Often dealing with the true complexities of the underlying open-source technologies they are reselling ends up being easier and more predictable. This fearmongering is really weakening us as an industry. Just try it, it is not as complex or dangerous as they claim.
- aurareturn 1y ago
  I'm a geek and I like to tinker with hardware. I want to maximum my $/hardware and have built a ton of DIY computers myself since I was young. I'm all about getting the most hardware for the money. But I'd like to sleep at night and the cost of AWS is not a significant issue to the business.
  - oersted 1y ago
    That’s fair enough but that’s a luxury position, if costs are not concern to you then there’s not much point in discussing the merits of different methods to manage infrastructure efficiently. And yes of course such costs are nothing if you are thinking of $300K just on a couple sysadmins. But this is just a bizarre bubble in a handful of small areas in the US and I am not sure how it can stay like that for much longer in this era of remote work. We built a whole business with $100K in seed and a few government grants. I have worked with quite a few world-class senior engineers happily making 40K-70K.
bjourne 1y ago
What prevents an EC2 instance from going down in exactly the same way? Other hosting providers offer automatic backup too - it's not an AWS exclusive feature.
- aurareturn 1y ago
  Nothing. It's just that I'm not the one responsible to fix it at 3am.
  - bjourne 1y ago
    Then who is responsible to fix it?
    - aurareturn 1y ago
      AWS if it's AWS' fault.
      - bjourne 1y ago
        You do realize that a server can "go down" for many other reasons than "the intern pulled the plug on it", right?
  - gizzlon 1y ago
    So if you app enters a crash-loop and fails to start an AWS engineer comes in and fixes it? Because that has not been my experience.. The truth is that there's still a lot of things you have to handle, including cloud bugs and problems. And other problems you don't have to think about anymore, especially with fully managed, high-level PaaS- like services. I ran a cloud backend service for a startup with users, using manged services, and we still had an on-call team. The cloud is not magic.
tiew9Vii 1y ago
> When your cheap dedicated server goes down and your admin is on holiday and you have hundreds of angry customers calling you, you'll get it. Or when you need to post on Hackernews to get support from your cloud provider as locked out of your account, being ignored and the only way to get access is try to create as much noise as possible it gets spotted. Or your cloud provider wipes your account and you are a $135B pension fund [1] Or your cloud portfolio is so big you need a "platform" team of multiple devops/developer staff to build wrappers around/package up your cloud provider for you and your platform team is now the bottleneck. Cloud is useful but it's not as pain free as everyone says when comparing with managing your own, it still costs money and work. Having worked on several cloud transformations they've all cost more and taken more effort than expected. A large proportion have also been canned/postponed/re-evaluated due to cost/size/time/complexity. Unless you are a big spender with dedicated technical account manager, your support is likely to be as bad as a no name budget VPS provider. Both cloud and traditional hosting have their merits and place. [1] https://arstechnica.com/gadgets/2024/05/google-cloud-accidentally-nukes-customer-account-causes-two-weeks-of-downtime/ https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...
- aurareturn 1y ago
  Or when you need to post on Hackernews to get support from your cloud provider as locked out of your account, being ignored and the only way to get access is try to create as much noise as possible it gets spotted. https://news.ycombinator.com/item?id=42365295 https://news.ycombinator.com/item?id=42365295 https://www.reddit.com/r/hetzner/comments/1ha5qgk/hetzner_cancelled_our_account_without_prior/ https://www.reddit.com/r/hetzner/comments/1ha5qgk/hetzner_ca...
isoprophlex 1y ago
When your AWS bill suddenly spikes to $69k because some data science intern left a huge gpu backed EC2 instance running in ap-southeast-2 with a misconfigured auto-scaling group, and your CTO is at a "digital transformation" conference, and you have hundreds of angry investors asking why your burn rate tripled, you’ll get it. Or you need to debug why your Lambda function is throttling and you find out that the CloudWatch logs were never properly configured and you’ve been flying blind for three months. And finally you have a brilliant idea of hiring a second $150k/year AWS solutions architect so that at least one person can actually understand the bill and they can check each other’s Terraform configs. Suddenly, you’re spending $300k on two cloud wizards alone and the cost savings of "not managing your own infrastructure" are completely gone. The snide rebuttal basically writes itself.
- fatata123 1y ago
  [dead]
- fatata123 1y ago
  [dead]
- bdangubic 11mo ago
  each rebuttal is about hiring shitty/icompetent people which also is same if you gonna manage your own (except of course much, much worse) - so not much or a rebuttal
  - array_key_first 11mo ago
    AWS facilitates the shittiness and incompetence because it, supposedly, handles all this stuff for you so you don't have to think about it. Except - wait, you do have to think about, because of course you. So the promise of AWS is gone.
veeti 1y ago
You guys aren't relying on only RDS backups, right? Something off AWS, right?
jjav 1y ago
> And finally you have a brilliant idea of hiring a second $150k/year dev ops admin The initial impression that we don't need to hire many people because AWS takes care of everthing, fades away pretty quick. You still need to hire the same people, they just do the same things in a different way.
_rwo 1y ago
> And finally you have a brilliant idea of hiring a second $150k/year dev ops admin (...) in my experience you always need a "Devops team" to operate all that cloud stuff; so to paraphrase - suddenly you're spending $400k on three devops to operate $500k cloud I think The Promise behind the cloud was you just pay for the service and not worry about it, but in practice you need some team to maintain it
utopiah 11mo ago
> your admin is on holiday and you have hundreds of angry customers Single human point of failure for something tied to the bottom line is NOT a technical problem, it's management.

esseph 1y ago

Sure, but AWS has more downtime than I do :-)

aurareturn 1y ago
Yes but you go on holidays but AWS does not.
fragmede 1y ago
If we assume that you're a human being that sleeps, say 8 hrs/day, and not an LLM, that leaves you with 16 hours of consciousness a day, for an uptime of 66%. That's upsidedown nines. You don't even crack one nine of uptime. If we assume you've a diet of meth and cocaine, and only sleep 2 hours a day, that still puts you at only like 92% uptime.
- esseph 1y ago
  > If we assume that you're a human being I'll have you know I am a cantaloupe, you insensitive clod!

grey-area 1y ago

It is statistically far more likely that your cloud service will go down for hours or days, and you will have no recourse and will just have to wait till AWS manage to resolve it.

aurareturn 1y ago
The difference is that if AWS goes down, I know for a fact that it'll be back up without me doing anything. If my own dedicated server goes down, I'm going to need to call my admin at 3am 10 times just to wake him up.
- jrk 1y ago
  You know that AWS will come back up. You definitely don’t know whether your own instances will come back or if you’ll need to redeploy it all.
  - BrandoElFollito 11mo ago
    Why do you assume that the small dedicated server has a higher probability to come back?
- KetoManx64 11mo ago
  If youe admin isn't competant enough to setup logging or notifications, how is it going to be better when your Cloud VM runs out of storage or doesn't reboot properly due to AWS swapping out hardware?
n4r9 1y ago
I suspect that this is really about liability. When AWS goes down you can just throw up your hands, everyone's in the same boat. If your own server goes down you worry that your customers doubt your competence. It's actually kinda frustrating - as an industry we're accepting worse outcomes due to misperceptions. That's how the free market goes sometimes.
- deleted 1y ago
  [deleted]
- PaulKeeble 1y ago
  Nobody gets fired for hiring IBM. This is the new version, when you go down because AWS did its someone else’s fault. Of course AWS will compare their downtime to industry standards for on premise and conclude they are down less often. On Premise engineers can say until they are blue that their downtime is on a Sunday at 3 am because it doesn't impact their customers it doesn't seem to matter.

Perz1val 1y ago

That's even worse when AWS goes down and myth of it never going down should be more than shattered by now

juliusceasar 1y ago

When your system goes down on AWS and your AWS admin is on holiday, you'll have the same problem. What is your point?

aurareturn 1y ago
I don't have an AWS admin. I assume a $2.4 trillion dollar company always has dev ops on call?
wink 1y ago
Every team I have worked on so far, if using AWS you had 50-100% of the developers with the knowledge and credentials (and usually the confidence) to troubleshoot/just fix it/replace it. Every team with dedicated hardware in a data center it was generally 1-2 people who would have fixed stuff quickly, no matter the size of the company (small ones, of course - so 10-50 devs). And that's with available replacement hardware. I'm not even one of the "cloud is so great" people - but it you're generally doing software it's actually a lot less friction. And while the ratio of cost difference may sound bad, it's generally not. Unless we're talkign huge scale, you can buy a lot of AWS crap for the yearly salary of a single person.
- tpxl 1y ago
  You said developers have the knowledge and credentials (and thus the work) of managing your infra, and a moment later basically asserted you're saving money on the salary for the sysadmin. This is the actual lie you got sold on. AWS isn't going to help you setup your security, you have to do it yourself. Previously a sysadmin would do this, now it's the devs. They aren't going to monitor your database performance. Previously a sysadmin would do this, now it's the devs. They aren't going to setup your networking. Previously a sysadmin would do this, ... Managing hardware and updating hosts is maybe 10% of the work of a sysadmin. You can't buy much on 1/10th of a sysadmins salary, and even the things you can, the quality and response time are generally going to be shit compared to someone who cares about your company (been there).
  - wink 1y ago

oersted 1y ago

After 8 years operating like this, I have had approximately the same number of critical outages in standard Cloud as with these providers. One included a whole OVH building burning down with our server in it, and recovery was faster than the recent AWS and Cloudflare outages. We felt less impotent and we could do more to mitigate the situation. If you want to, these providers also offer VMs, object storage and other virtualized services for way cheaper with similar guarantees, they are not stuck in the last century. And I don’t know how people are using cloud, but most config issues happen above the VM/Docker/Kubernetes level, which is the same wether you are on cloud or not. Even fully managed database deployments or serverless backends are not really that much simpler or less error-prone than deploying the containers yourself. Actually the complexity of Cloud is often a worse minefield of footguns, with their myriad artificial quirks and limitations. Often dealing with the true complexities of the underlying open-source technologies they are reselling ends up being easier and more predictable. This fearmongering is really weakening us as an industry. Just try it, it is not as complex or dangerous as they claim.

aurareturn 1y ago
I'm a geek and I like to tinker with hardware. I want to maximum my $/hardware and have built a ton of DIY computers myself since I was young. I'm all about getting the most hardware for the money. But I'd like to sleep at night and the cost of AWS is not a significant issue to the business.
- oersted 1y ago
  That’s fair enough but that’s a luxury position, if costs are not concern to you then there’s not much point in discussing the merits of different methods to manage infrastructure efficiently. And yes of course such costs are nothing if you are thinking of $300K just on a couple sysadmins. But this is just a bizarre bubble in a handful of small areas in the US and I am not sure how it can stay like that for much longer in this era of remote work. We built a whole business with $100K in seed and a few government grants. I have worked with quite a few world-class senior engineers happily making 40K-70K.

bjourne 1y ago

What prevents an EC2 instance from going down in exactly the same way? Other hosting providers offer automatic backup too - it's not an AWS exclusive feature.

aurareturn 1y ago
Nothing. It's just that I'm not the one responsible to fix it at 3am.
- bjourne 1y ago
  Then who is responsible to fix it?
  - aurareturn 1y ago
    AWS if it's AWS' fault.
    - bjourne 1y ago
      You do realize that a server can "go down" for many other reasons than "the intern pulled the plug on it", right?
- gizzlon 1y ago
  So if you app enters a crash-loop and fails to start an AWS engineer comes in and fixes it? Because that has not been my experience.. The truth is that there's still a lot of things you have to handle, including cloud bugs and problems. And other problems you don't have to think about anymore, especially with fully managed, high-level PaaS- like services. I ran a cloud backend service for a startup with users, using manged services, and we still had an on-call team. The cloud is not magic.

tiew9Vii 1y ago

> When your cheap dedicated server goes down and your admin is on holiday and you have hundreds of angry customers calling you, you'll get it. Or when you need to post on Hackernews to get support from your cloud provider as locked out of your account, being ignored and the only way to get access is try to create as much noise as possible it gets spotted. Or your cloud provider wipes your account and you are a $135B pension fund [1] Or your cloud portfolio is so big you need a "platform" team of multiple devops/developer staff to build wrappers around/package up your cloud provider for you and your platform team is now the bottleneck. Cloud is useful but it's not as pain free as everyone says when comparing with managing your own, it still costs money and work. Having worked on several cloud transformations they've all cost more and taken more effort than expected. A large proportion have also been canned/postponed/re-evaluated due to cost/size/time/complexity. Unless you are a big spender with dedicated technical account manager, your support is likely to be as bad as a no name budget VPS provider. Both cloud and traditional hosting have their merits and place. [1] https://arstechnica.com/gadgets/2024/05/google-cloud-accidentally-nukes-customer-account-causes-two-weeks-of-downtime/ https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...

aurareturn 1y ago
Or when you need to post on Hackernews to get support from your cloud provider as locked out of your account, being ignored and the only way to get access is try to create as much noise as possible it gets spotted. https://news.ycombinator.com/item?id=42365295 https://news.ycombinator.com/item?id=42365295 https://www.reddit.com/r/hetzner/comments/1ha5qgk/hetzner_cancelled_our_account_without_prior/ https://www.reddit.com/r/hetzner/comments/1ha5qgk/hetzner_ca...

isoprophlex 1y ago

When your AWS bill suddenly spikes to $69k because some data science intern left a huge gpu backed EC2 instance running in ap-southeast-2 with a misconfigured auto-scaling group, and your CTO is at a "digital transformation" conference, and you have hundreds of angry investors asking why your burn rate tripled, you’ll get it. Or you need to debug why your Lambda function is throttling and you find out that the CloudWatch logs were never properly configured and you’ve been flying blind for three months. And finally you have a brilliant idea of hiring a second $150k/year AWS solutions architect so that at least one person can actually understand the bill and they can check each other’s Terraform configs. Suddenly, you’re spending $300k on two cloud wizards alone and the cost savings of "not managing your own infrastructure" are completely gone. The snide rebuttal basically writes itself.

fatata123 1y ago
[dead]
fatata123 1y ago
[dead]
bdangubic 11mo ago
each rebuttal is about hiring shitty/icompetent people which also is same if you gonna manage your own (except of course much, much worse) - so not much or a rebuttal
- array_key_first 11mo ago
  AWS facilitates the shittiness and incompetence because it, supposedly, handles all this stuff for you so you don't have to think about it. Except - wait, you do have to think about, because of course you. So the promise of AWS is gone.

veeti 1y ago

You guys aren't relying on only RDS backups, right? Something off AWS, right?

jjav 1y ago

> And finally you have a brilliant idea of hiring a second $150k/year dev ops admin The initial impression that we don't need to hire many people because AWS takes care of everthing, fades away pretty quick. You still need to hire the same people, they just do the same things in a different way.

_rwo 1y ago

> And finally you have a brilliant idea of hiring a second $150k/year dev ops admin (...) in my experience you always need a "Devops team" to operate all that cloud stuff; so to paraphrase - suddenly you're spending $400k on three devops to operate $500k cloud I think The Promise behind the cloud was you just pay for the service and not worry about it, but in practice you need some team to maintain it

utopiah 11mo ago

> your admin is on holiday and you have hundreds of angry customers Single human point of failure for something tied to the bottom line is NOT a technical problem, it's management.