8 ms·

If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swe

If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go back on prem, or you just don't give a shit about reliability (start pricing up DDoS protection costs at anything higher than 10G and tell me the cloud is more expensive). We spend 100k+ on AWS bandwidth and it's still cheaper than our DIA circuits because we don't have to pay network engineers to manage 3 different AZs.

qvrjuec 3y ago
>someone who will not treat their datacenter like a home lab What does this mean? They steal company resources for themselves, or just configure things incompetently?
- ttul 3y ago
  Incompetence. Take my friend’s company for instance. They were frustrated paying $60K/mo to Amazon so their brilliant sysadmin bought $600K of servers and moved them into a cheap colo. Over Christmas, everything died, and the brilliant sysadmin was on holiday. Nobody could get things going again for many days and so their entire SaaS business was failing. They lost a lot of business and trust as a result. The sysadmin is now gone and they are back on AWS.
  - lijok 3y ago
    No key person risk management -> no risk register -> no management. Your friends company will fail regardless of poor sysadmin decision making or not. They need to hire competent management ASAP.
    - mdale 3y ago
      With cloud and SaaS services you are paying to reduce person risk profile. Your forming a larger dependency on a team lead against a custom system that now is a liability as new people come to the organization don't want to adopt an abandoned poorly understood project.
    - overstay8930 3y ago
      This is basically the logic of people who say the cloud is too expensive, you have to ignore so many things to make being on premise logical. Basically you are lying to yourself if you think you can run a datacenter cheaper and better than Amazon or Microsoft can, because if you can you are just making huge sacrifices somewhere (usually time, which is why reddit sysadmins complain about how much work they have while defending being on-premise because they couldn't possibly be wrong).
      - project2501a 3y ago
        you must be management, cuz 1. you think it's the sysadmin's fault 2. there are no competent sysadmins out there
      - 123pie123 3y ago
        >Basically you are lying to yourself if you think you can run a datacenter cheaper and better than Amazon or Microsoft what magical things do they have? that every single reasonably sized enterprise doesn't have? it should be extremly easy for a small enterprise to beat any of the main clouds* - they make crap ton of profit from you *making an assumption that your needs are reasonably static and not MASSIVELY busting up and down your infrastructure
        plywoodtrees 3y ago
        The "magical" thing they have is thousands and thousands of people thinking about how to improve the performance/efficiency/availability of their datacenters. And yes, they pay the costs of those people and take a good profit margin, and yes there are in some ways diminishing returns to go from 3 nines to 6. But most enterprises can't match that depth of concentrated expertise, certainly not most small enterprises.
  - OrvalWintermute 3y ago
    > brilliant sysadmin was on holiday > entire SaaS business > [ Unmentioned - Single Point of Failure Service dependent on a single admin ] If you are fully accounting for vacation, training, sleep etc then you need a minimum of 5 admins for mission critical services. Now, you can engineer around this to reduce your staffing requirement but I wouldn't recommend going under 2 ever because accidents happen. This business seemed one below that, without the engineering, and I would point to the mgmt, not the brilliant admin as the problem.
  - jjav 3y ago
    > The sysadmin is now gone and they are back on AWS. This story has nothing to do with AWS or on-prem. It's a story about incompetent management allowing a single human point of failure. If they don't change that, they'll have the same problem wherever they go.
- overstay8930 3y ago
  Non-scalable incompetence or basically pretending that the datacenter will never go down. Any high schooler with an iPhone can set up and maintain a datacenter full of servers. But if you want something reliable that I can spend 30 seconds writing some terraform for, it will take an entire infra team to set up and maintain it, not to mention an entire procurement process and now having to integrate a new supply chain just for a basic multi-az setup (probably without things like backups and still without basic features the cloud gives you automatically).
echelon 3y ago
You act like these problems are especially hard. Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder. This is what we're paid for.
- hobs 3y ago
  Managing on prem is definitely harder because you are benefiting from the economics of scale of all the management problems that you have to pay yourself, and if you don't have scale then you will be significantly overpaying to get the same type of quality, reliability, or responsiveness. Most people are not paid to manage infra, they are paid to talk to customers, ship features, fix bugs, and other "core business" items; just like most businesses don't build roads, they pay taxes and utilize them because the cost of doing it themselves for their preferred traffic patterns would be much more than they could justify (for now.)
  - BackBlast 3y ago
    If you don't have scale, you don't need most of the features. Fire up PC, load application. Setup egress port open to internet. Setup application backup on cron job. Done until scale problems arise.
    - hobs 3y ago
      Correct, my point absolutely doesn't apply to someone who is just doing their thing, even maybe 2 orders of magnitude more stuff than their thing. But when your local IT goon says its going to be 8 months to procure the next set of hard drives for your next order of magnitude, it's a real problem and you have real money to invest in solving it, just not owning a data center money.
- kevin_nisbet 3y ago
  Managing on-prem hardware may not necessarily be hard, but it can be extremely time consuming. To me, the nice thing about dropping a bunch of hardware in a colo is you get to take a lot of shortcuts and take risks that you cannot buy from the public cloud providers. I worked for a company and would do it again that did the colo route, and it gave immense cost savings compared to public cloud, taking on risks that you can't do elsewhere. Before they started investing in having folks take care of the infra as a raw startup, it was just some servers and some desktop unmanaged switches. But that gave the company breathing room to survive as the business model probably didn't work without it. But also had a reputation for unreliable service. I've also built the five nines infra at telcos, and yes you can do it with average engineers, but it's going to be time consuming, slow, and expensive in costs and labor. To allow 26 seconds of unplanned outage a month, you're going to be testing every firmware update for every piece of equipment on an ongoing basis, and practicing every operation and change as best as possible. And you need the scale that you get that 26s by having most outages only impact a subset of your customer base, otherwise you're going to blow that outage budget fast.
holoduke 3y ago
My small business once spend 50k per month on AWS. We brought that back to 800 dollars for a similar setup at Hetzner. I find this a significant number.
- glintik 3y ago
  Reliability differs, right?
  - simplyinfinity 3y ago
    I have hetzner vms going on 500+ DAYS without restart. I've been a hetzner customer for about 8 years now. Aws has been down x10 times more than hetzner has. And even when hetzner has issues they are localized to a single DC at worst, most often a single host is down. When aws has issue.. The whole internet has issues. Or their central region goes out, also can affect their other regions. For the small/medium businesses infra I manage here in bulgaria , the same thing would cost 5-10 times as much on aws just for the compute, throw in 1tb of bandwidth.. And this makes no financial sense. On hetzner I pay 40 euros for 2 vms, dedicated ips, daily backups, 100gb of ssd external storage, and firewall.
    - mad182 3y ago
      I have more than 10 servers on Hetzner (some dedicated, some VPS), for 5+ years and the same experience, once one of the dedicated servers had some hardware issues, and an hour later the drives were moved to another box and it was running again. Other than that time, I had downtime only because of my own fault. Pretty sure over these years AWS has experienced a lot more issues overall.
- overstay8930 3y ago
  "Yea this small VPS provider with 1% of the features is just as good as AWS to us" yea that's because you aren't using features as basic as AWS Nitro Enclaves and you are years behind even basic cloud security. Hetzner is for running homelabs and basic compute, not businesses. That's why EU companies constantly ignore EU rulings on US-EU privacy shield because there's just no alternative to American cloud providers yet.
dzikimarian 3y ago
Apparently we're doing the impossible for over 12 years now. Who knew? Some people act like it's some kind of black magic. It's not. We've some customers in our DC and some on AWS for various reasons. AWS isn't less problematic. AWS is about 10x more expensive. Both on prem and cloud require people familiar with them and cloud-engineers are in no way cheaper. Only meaningful problem is that on-prem requires some up front cost&time. That can be mitigated by leasing and other means, but indeed can be an issue for small businesses.
- chongli 3y ago
  Both on prem and cloud require people familiar with them and cloud-engineers are in no way cheaper. I think the real story is a bit sordid: office politics. On-prem and cloud are different skillsets. Companies that have been around for a while can end up with both on-prem and cloud experts who end up competing with each other, often on separate teams. Throw in some slick consultants from Amazon who are able to bend the ear of the VP and you've got a real problem. From what I've seen it doesn't end well for the on-prem team!
  - hirako2000 3y ago
    I concur. Many VPs bend ears so easily you got to wonder. do they also get invited to some private dinner by those so called slick consultants, who pay the bill in a rush and leaves after forgetting some thick envelope on the table.
    - chongli 3y ago
      It’s a story as old as time itself. IBM has been doing it at least as far back as the 60’s. Fancy consultants who know the tech and also know how to sell and make themselves seem way smarter than the VP’s reports. Do one slick presentation and the VP is asking his team “why didn’t you guys come up with this stuff?” Next thing you know these multi-million-dollar contracts are signed and the existing teams are just shaking their heads. The smart ones have already put out their resumes and started interviewing elsewhere.
snug 3y ago
AWS and GCP are giving companies like Apple huge discounts so someone could say something like, "Even Apple iCloud is in AWS and GCP because of how economical it is" There is too much nuance to say one is better than the other. In some cases using a IaaS is more economical, in other cases it's not. For Apple, the same is also true[0] to say "Even Apple is running their own datacenters because of how economical it is" 0 - https://dgtlinfra.com/apple-data-center-locations/#:~:text=and%20international%20markets.-,United%20States%20%E2%80%93%20Apple%20Data%20Centers,Oregon%3B%20and%20Newark%2C%20California https://dgtlinfra.com/apple-data-center-locations/#:~:text=a....
- overstay8930 3y ago
  You know everyone gets those discounts right? Like that's why the cloud is so much more economical than a datacenter, once you are at scale AWS will give you MASSIVE (I'm talking 30-60% discounts) on compute and other compute-adjacent resources, and I've seen 99% discounts on bandwidth with multi year agreements too. I think that's where the disconnect is, a lot of people don't actually realize how cheap cloud compute is because they're only seeing the price for like... 20-30 servers and some basic S3 or load balancer usage. There are entire departments at Amazon that run those numbers on a daily basis to make sure AWS is always competitive with building your own datacenter.
tiffanyh 3y ago
> on premise have no idea how much the salary costs of someone I haven’t yet meet anyone at a company that heavily uses cloud and still doesn’t have the same number of salaried infra people as on-prem.

qvrjuec 3y ago

>someone who will not treat their datacenter like a home lab What does this mean? They steal company resources for themselves, or just configure things incompetently?

ttul 3y ago
Incompetence. Take my friend’s company for instance. They were frustrated paying $60K/mo to Amazon so their brilliant sysadmin bought $600K of servers and moved them into a cheap colo. Over Christmas, everything died, and the brilliant sysadmin was on holiday. Nobody could get things going again for many days and so their entire SaaS business was failing. They lost a lot of business and trust as a result. The sysadmin is now gone and they are back on AWS.
- lijok 3y ago
  No key person risk management -> no risk register -> no management. Your friends company will fail regardless of poor sysadmin decision making or not. They need to hire competent management ASAP.
  - mdale 3y ago
    With cloud and SaaS services you are paying to reduce person risk profile. Your forming a larger dependency on a team lead against a custom system that now is a liability as new people come to the organization don't want to adopt an abandoned poorly understood project.
  - overstay8930 3y ago
    This is basically the logic of people who say the cloud is too expensive, you have to ignore so many things to make being on premise logical. Basically you are lying to yourself if you think you can run a datacenter cheaper and better than Amazon or Microsoft can, because if you can you are just making huge sacrifices somewhere (usually time, which is why reddit sysadmins complain about how much work they have while defending being on-premise because they couldn't possibly be wrong).
    - project2501a 3y ago
      you must be management, cuz 1. you think it's the sysadmin's fault 2. there are no competent sysadmins out there
    - 123pie123 3y ago
      >Basically you are lying to yourself if you think you can run a datacenter cheaper and better than Amazon or Microsoft what magical things do they have? that every single reasonably sized enterprise doesn't have? it should be extremly easy for a small enterprise to beat any of the main clouds* - they make crap ton of profit from you *making an assumption that your needs are reasonably static and not MASSIVELY busting up and down your infrastructure
      - plywoodtrees 3y ago
        The "magical" thing they have is thousands and thousands of people thinking about how to improve the performance/efficiency/availability of their datacenters. And yes, they pay the costs of those people and take a good profit margin, and yes there are in some ways diminishing returns to go from 3 nines to 6. But most enterprises can't match that depth of concentrated expertise, certainly not most small enterprises.
- OrvalWintermute 3y ago
  > brilliant sysadmin was on holiday > entire SaaS business > [ Unmentioned - Single Point of Failure Service dependent on a single admin ] If you are fully accounting for vacation, training, sleep etc then you need a minimum of 5 admins for mission critical services. Now, you can engineer around this to reduce your staffing requirement but I wouldn't recommend going under 2 ever because accidents happen. This business seemed one below that, without the engineering, and I would point to the mgmt, not the brilliant admin as the problem.
- jjav 3y ago
  > The sysadmin is now gone and they are back on AWS. This story has nothing to do with AWS or on-prem. It's a story about incompetent management allowing a single human point of failure. If they don't change that, they'll have the same problem wherever they go.
overstay8930 3y ago
Non-scalable incompetence or basically pretending that the datacenter will never go down. Any high schooler with an iPhone can set up and maintain a datacenter full of servers. But if you want something reliable that I can spend 30 seconds writing some terraform for, it will take an entire infra team to set up and maintain it, not to mention an entire procurement process and now having to integrate a new supply chain just for a basic multi-az setup (probably without things like backups and still without basic features the cloud gives you automatically).

echelon 3y ago

You act like these problems are especially hard. Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder. This is what we're paid for.

hobs 3y ago
Managing on prem is definitely harder because you are benefiting from the economics of scale of all the management problems that you have to pay yourself, and if you don't have scale then you will be significantly overpaying to get the same type of quality, reliability, or responsiveness. Most people are not paid to manage infra, they are paid to talk to customers, ship features, fix bugs, and other "core business" items; just like most businesses don't build roads, they pay taxes and utilize them because the cost of doing it themselves for their preferred traffic patterns would be much more than they could justify (for now.)
- BackBlast 3y ago
  If you don't have scale, you don't need most of the features. Fire up PC, load application. Setup egress port open to internet. Setup application backup on cron job. Done until scale problems arise.
  - hobs 3y ago
    Correct, my point absolutely doesn't apply to someone who is just doing their thing, even maybe 2 orders of magnitude more stuff than their thing. But when your local IT goon says its going to be 8 months to procure the next set of hard drives for your next order of magnitude, it's a real problem and you have real money to invest in solving it, just not owning a data center money.
kevin_nisbet 3y ago
Managing on-prem hardware may not necessarily be hard, but it can be extremely time consuming. To me, the nice thing about dropping a bunch of hardware in a colo is you get to take a lot of shortcuts and take risks that you cannot buy from the public cloud providers. I worked for a company and would do it again that did the colo route, and it gave immense cost savings compared to public cloud, taking on risks that you can't do elsewhere. Before they started investing in having folks take care of the infra as a raw startup, it was just some servers and some desktop unmanaged switches. But that gave the company breathing room to survive as the business model probably didn't work without it. But also had a reputation for unreliable service. I've also built the five nines infra at telcos, and yes you can do it with average engineers, but it's going to be time consuming, slow, and expensive in costs and labor. To allow 26 seconds of unplanned outage a month, you're going to be testing every firmware update for every piece of equipment on an ongoing basis, and practicing every operation and change as best as possible. And you need the scale that you get that 26s by having most outages only impact a subset of your customer base, otherwise you're going to blow that outage budget fast.

holoduke 3y ago

My small business once spend 50k per month on AWS. We brought that back to 800 dollars for a similar setup at Hetzner. I find this a significant number.

glintik 3y ago
Reliability differs, right?
- simplyinfinity 3y ago
  I have hetzner vms going on 500+ DAYS without restart. I've been a hetzner customer for about 8 years now. Aws has been down x10 times more than hetzner has. And even when hetzner has issues they are localized to a single DC at worst, most often a single host is down. When aws has issue.. The whole internet has issues. Or their central region goes out, also can affect their other regions. For the small/medium businesses infra I manage here in bulgaria , the same thing would cost 5-10 times as much on aws just for the compute, throw in 1tb of bandwidth.. And this makes no financial sense. On hetzner I pay 40 euros for 2 vms, dedicated ips, daily backups, 100gb of ssd external storage, and firewall.
  - mad182 3y ago
    I have more than 10 servers on Hetzner (some dedicated, some VPS), for 5+ years and the same experience, once one of the dedicated servers had some hardware issues, and an hour later the drives were moved to another box and it was running again. Other than that time, I had downtime only because of my own fault. Pretty sure over these years AWS has experienced a lot more issues overall.
overstay8930 3y ago
"Yea this small VPS provider with 1% of the features is just as good as AWS to us" yea that's because you aren't using features as basic as AWS Nitro Enclaves and you are years behind even basic cloud security. Hetzner is for running homelabs and basic compute, not businesses. That's why EU companies constantly ignore EU rulings on US-EU privacy shield because there's just no alternative to American cloud providers yet.

dzikimarian 3y ago

Apparently we're doing the impossible for over 12 years now. Who knew? Some people act like it's some kind of black magic. It's not. We've some customers in our DC and some on AWS for various reasons. AWS isn't less problematic. AWS is about 10x more expensive. Both on prem and cloud require people familiar with them and cloud-engineers are in no way cheaper. Only meaningful problem is that on-prem requires some up front cost&time. That can be mitigated by leasing and other means, but indeed can be an issue for small businesses.

chongli 3y ago
Both on prem and cloud require people familiar with them and cloud-engineers are in no way cheaper. I think the real story is a bit sordid: office politics. On-prem and cloud are different skillsets. Companies that have been around for a while can end up with both on-prem and cloud experts who end up competing with each other, often on separate teams. Throw in some slick consultants from Amazon who are able to bend the ear of the VP and you've got a real problem. From what I've seen it doesn't end well for the on-prem team!
- hirako2000 3y ago
  I concur. Many VPs bend ears so easily you got to wonder. do they also get invited to some private dinner by those so called slick consultants, who pay the bill in a rush and leaves after forgetting some thick envelope on the table.
  - chongli 3y ago
    It’s a story as old as time itself. IBM has been doing it at least as far back as the 60’s. Fancy consultants who know the tech and also know how to sell and make themselves seem way smarter than the VP’s reports. Do one slick presentation and the VP is asking his team “why didn’t you guys come up with this stuff?” Next thing you know these multi-million-dollar contracts are signed and the existing teams are just shaking their heads. The smart ones have already put out their resumes and started interviewing elsewhere.

snug 3y ago

AWS and GCP are giving companies like Apple huge discounts so someone could say something like, "Even Apple iCloud is in AWS and GCP because of how economical it is" There is too much nuance to say one is better than the other. In some cases using a IaaS is more economical, in other cases it's not. For Apple, the same is also true[0] to say "Even Apple is running their own datacenters because of how economical it is" 0 - https://dgtlinfra.com/apple-data-center-locations/#:~:text=and%20international%20markets.-,United%20States%20%E2%80%93%20Apple%20Data%20Centers,Oregon%3B%20and%20Newark%2C%20California https://dgtlinfra.com/apple-data-center-locations/#:~:text=a....

overstay8930 3y ago
You know everyone gets those discounts right? Like that's why the cloud is so much more economical than a datacenter, once you are at scale AWS will give you MASSIVE (I'm talking 30-60% discounts) on compute and other compute-adjacent resources, and I've seen 99% discounts on bandwidth with multi year agreements too. I think that's where the disconnect is, a lot of people don't actually realize how cheap cloud compute is because they're only seeing the price for like... 20-30 servers and some basic S3 or load balancer usage. There are entire departments at Amazon that run those numbers on a daily basis to make sure AWS is always competitive with building your own datacenter.

tiffanyh 3y ago

> on premise have no idea how much the salary costs of someone I haven’t yet meet anyone at a company that heavily uses cloud and still doesn’t have the same number of salaried infra people as on-prem.