4 ms·

> Those engineers' salaries almost always outweigh all of your cloud costs I think there is a divide here. Most people in this thread that mention cost as an i

> Those engineers' salaries almost always outweigh all of your cloud costs I think there is a divide here. Most people in this thread that mention cost as an issue are running some serious gear, and that does not come cheap. Your parent is running 70 servers plus some serious networking equipment, which is easily a couple million dollars. And he said that a cloud provider would 10x that cost. If all you need is a little VM to run a website, cloud hosting is cheap. If you are running real infrastructure, a couple hours of developer time will never out-weight the astronomical costs of cloud hosting.

kelnos 7y ago
Not sure that's universally true. We have probably tens of thousands of EC2 instances in our infra right now. Just the cost alone to build out on-prem infra and migrate things over would likely wipe out a half-decade or more of cost savings. And in the meantime our shift in focus would hurt our market position. If we'd started in the beginning (11 or 12 years ago) with our own infra, I imagine our recurring infra costs would be lower right now, but I also suspect that our 3-person founding team would have failed to produce an MVP before their initial time and money ran out. If I were starting a company now, I'd probably do things in a more "platform agnostic" way such that a cloud->on-prem migration might be easier. But I still never expect it'd be easy.
- pvorb 7y ago
  If you implement things in a cloud agnostic way you can also save a lot by choosing a traditional webhoster/server hoster. If you want your product to be cloud agnostic, you lose almost every advantage of the cloud. That's the whole problem why we haven't seen a competitive reduction in prices between cloud providers, yet.
Aeolun 7y ago
> If you are running real infrastructure, a couple hours of developer time will never out-weight the astronomical costs of cloud hosting. Over time, buying things yourself will always win out on cloud architecture, or the cloud business would be bankrupt.
- waterhouse 7y ago
  It would be theoretically possible for cloud providers, specializing in what they do, to gain from economies of scale of various sorts (having their own large purpose-built datacenters, needing less total redundant equipment to have the same replacement speed, having local servers in all parts of the world, being able to use nighttime idle cycles in servers in one part of the world for daytime customers on the other side), which could theoretically yield savings greater than the profit margin that they charge. Not saying this is what happens in practice.
  - worldsayshi 7y ago
    To me the economics of scale is the whole point of cloud. If vendors don't deliver on that promise and makes it more expensive than hosting it yourself they have fundamentally failed and there's no reason, market wise or utility wise, for them to continue in the long run.
  - twic 7y ago
    These economies could only put cloud ahead if they were larger than the provider's profit margin. If cloud saves 10% in costs but charges you 30% in markup, it costs you 20% more. AWS's margin is 20-25% (eg [1]). Is AWS's cost of goods sold really 25% lower? Admittedly this calculation is tricky, because it's not clear what to include. Does that figure for AWS's margin include R&D? Should it? [1] https://www.forbes.com/sites/greatspeculations/2018/10/17/how-much-will-aws-contribute-to-amazons-near-term-revenue-growth/ https://www.forbes.com/sites/greatspeculations/2018/10/17/ho...
RhodesianHunter 7y ago
That reads as fairly dismissive to me. Most people running serious gear have serious engineering payroll overhead. Sure, some folks are managing something fairly static like game servers, but many of us work for bigco with two dozen products and massive teams. Even with thousands of ec2 instances we continue to move our on-prem infrastructure to the cloud because developer productivity saves us big money in the long run.
- kortilla 7y ago
  Does it actually save or does it just encourage waste by the developers and poor system design? I’ve heard stories of companies having each PR spin up something like 20 ec2 instances to build up a whole deployment. A CICD design like that used to be a fireable offense. Now people see that and assume it’s saving them money because it would have bottlenecked before.
  - sokoloff 7y ago
    If it spins those EC2 instances up, runs for a hour of “waste” and then turns off (or the alternate cluster turns off when no longer in use), it very likely is saving money. It might be saving money in that very moment; it might be saving money invisibly when “time to patch the servers again” is a complete non-event for staff time.
  - troupe 7y ago
    So I've setup processes like you described where every PR would spin up 20 ec2 instances, run a huge number of parallel testing, and then shut itself down. The savings associated with developers getting feedback on a full regression test of the system in 20 minutes vs. 400 minutes was significant.
    - kortilla 7y ago
      That’s parallelizing, which is not what I’m talking about. I’m talking about pointlessly building up huge environments that “match production”.
  - 7y ago
winter_blue 7y ago
If one could come up with an estimate of how many hours per year it would cost to maintain (plus initially setup cost), and multiple that by $75/hr (a reasonable rate for a good engineer), then you could estimate how much it all costs. I'd wager a guess that a properly-configured server won't actually cost that much to maintain. Unless you're frequently updating the OS and other stuff - but that's a software issue. I don't expect the hardware will fail that frequently at all. If there's a HDD failure, a good RAID system (with tolerance for 3 or 4 disks failing simultaneously) will alert you to it, and you'd grab the spare HDD (you should have a few spares), and pop it in, and the RAID would recover from the failure. What other hardware components frequently fail? RAM? CPU? Not really. The engineer who sets up the server should be someone who is a generalist, and a full-time employee who when not tending to the server, works on other stuff (like the product). Then you have someone on staff who is familiar with the system, and can fix something that goes wrong, without needing to ramp up on it first. I know a lot of programmers would love to set up a server. Any who enjoys buildings PCs (which many generalist programmers do) would probably love to have a one-month project where they pick parts and set up a powerful server.
- sokoloff 7y ago
  In quarantine, I’m doing exactly that, fooling about at home with old enterprise gear you can buy seemingly by the pound on EBay now. It’s super fun and I somewhat enjoy getting all the power management right, all the networking setup, and ultimately running Proxmox and a bunch of small containers and VMs for things like Minecraft servers and file storage. In my day job and my side projects, I’m 100% paying for AWS for 99% of our workloads. I choose to let Amazon deal with all the staff issues and all the other, to use their words, undifferentiated heavy lifting. There’s some scale point at which you have to ask the question whether you ought to be in the cloud, but IMO if you think a cloud provider is only 50% more than you could DIY, you should probably be in the cloud.
C1sc0cat 7y ago
In that case Use Linode and not AWS - I ported a little POC system I had built to Linode and it cost 10x to run it on AWS. Though I id get a understanding of the basics of AWS out of it probably not the best use of the share holders money

kelnos 7y ago

Not sure that's universally true. We have probably tens of thousands of EC2 instances in our infra right now. Just the cost alone to build out on-prem infra and migrate things over would likely wipe out a half-decade or more of cost savings. And in the meantime our shift in focus would hurt our market position. If we'd started in the beginning (11 or 12 years ago) with our own infra, I imagine our recurring infra costs would be lower right now, but I also suspect that our 3-person founding team would have failed to produce an MVP before their initial time and money ran out. If I were starting a company now, I'd probably do things in a more "platform agnostic" way such that a cloud->on-prem migration might be easier. But I still never expect it'd be easy.

pvorb 7y ago
If you implement things in a cloud agnostic way you can also save a lot by choosing a traditional webhoster/server hoster. If you want your product to be cloud agnostic, you lose almost every advantage of the cloud. That's the whole problem why we haven't seen a competitive reduction in prices between cloud providers, yet.

Aeolun 7y ago

> If you are running real infrastructure, a couple hours of developer time will never out-weight the astronomical costs of cloud hosting. Over time, buying things yourself will always win out on cloud architecture, or the cloud business would be bankrupt.

waterhouse 7y ago
It would be theoretically possible for cloud providers, specializing in what they do, to gain from economies of scale of various sorts (having their own large purpose-built datacenters, needing less total redundant equipment to have the same replacement speed, having local servers in all parts of the world, being able to use nighttime idle cycles in servers in one part of the world for daytime customers on the other side), which could theoretically yield savings greater than the profit margin that they charge. Not saying this is what happens in practice.
- worldsayshi 7y ago
  To me the economics of scale is the whole point of cloud. If vendors don't deliver on that promise and makes it more expensive than hosting it yourself they have fundamentally failed and there's no reason, market wise or utility wise, for them to continue in the long run.
- twic 7y ago
  These economies could only put cloud ahead if they were larger than the provider's profit margin. If cloud saves 10% in costs but charges you 30% in markup, it costs you 20% more. AWS's margin is 20-25% (eg [1]). Is AWS's cost of goods sold really 25% lower? Admittedly this calculation is tricky, because it's not clear what to include. Does that figure for AWS's margin include R&D? Should it? [1] https://www.forbes.com/sites/greatspeculations/2018/10/17/how-much-will-aws-contribute-to-amazons-near-term-revenue-growth/ https://www.forbes.com/sites/greatspeculations/2018/10/17/ho...

RhodesianHunter 7y ago

That reads as fairly dismissive to me. Most people running serious gear have serious engineering payroll overhead. Sure, some folks are managing something fairly static like game servers, but many of us work for bigco with two dozen products and massive teams. Even with thousands of ec2 instances we continue to move our on-prem infrastructure to the cloud because developer productivity saves us big money in the long run.

kortilla 7y ago
Does it actually save or does it just encourage waste by the developers and poor system design? I’ve heard stories of companies having each PR spin up something like 20 ec2 instances to build up a whole deployment. A CICD design like that used to be a fireable offense. Now people see that and assume it’s saving them money because it would have bottlenecked before.
- sokoloff 7y ago
  If it spins those EC2 instances up, runs for a hour of “waste” and then turns off (or the alternate cluster turns off when no longer in use), it very likely is saving money. It might be saving money in that very moment; it might be saving money invisibly when “time to patch the servers again” is a complete non-event for staff time.
- troupe 7y ago
  So I've setup processes like you described where every PR would spin up 20 ec2 instances, run a huge number of parallel testing, and then shut itself down. The savings associated with developers getting feedback on a full regression test of the system in 20 minutes vs. 400 minutes was significant.
  - kortilla 7y ago
    That’s parallelizing, which is not what I’m talking about. I’m talking about pointlessly building up huge environments that “match production”.
- 7y ago

winter_blue 7y ago

If one could come up with an estimate of how many hours per year it would cost to maintain (plus initially setup cost), and multiple that by $75/hr (a reasonable rate for a good engineer), then you could estimate how much it all costs. I'd wager a guess that a properly-configured server won't actually cost that much to maintain. Unless you're frequently updating the OS and other stuff - but that's a software issue. I don't expect the hardware will fail that frequently at all. If there's a HDD failure, a good RAID system (with tolerance for 3 or 4 disks failing simultaneously) will alert you to it, and you'd grab the spare HDD (you should have a few spares), and pop it in, and the RAID would recover from the failure. What other hardware components frequently fail? RAM? CPU? Not really. The engineer who sets up the server should be someone who is a generalist, and a full-time employee who when not tending to the server, works on other stuff (like the product). Then you have someone on staff who is familiar with the system, and can fix something that goes wrong, without needing to ramp up on it first. I know a lot of programmers would love to set up a server. Any who enjoys buildings PCs (which many generalist programmers do) would probably love to have a one-month project where they pick parts and set up a powerful server.

sokoloff 7y ago
In quarantine, I’m doing exactly that, fooling about at home with old enterprise gear you can buy seemingly by the pound on EBay now. It’s super fun and I somewhat enjoy getting all the power management right, all the networking setup, and ultimately running Proxmox and a bunch of small containers and VMs for things like Minecraft servers and file storage. In my day job and my side projects, I’m 100% paying for AWS for 99% of our workloads. I choose to let Amazon deal with all the staff issues and all the other, to use their words, undifferentiated heavy lifting. There’s some scale point at which you have to ask the question whether you ought to be in the cloud, but IMO if you think a cloud provider is only 50% more than you could DIY, you should probably be in the cloud.

C1sc0cat 7y ago

In that case Use Linode and not AWS - I ported a little POC system I had built to Linode and it cost 10x to run it on AWS. Though I id get a understanding of the basics of AWS out of it probably not the best use of the share holders money