8 ms·

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for m

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure.

icebraining 7y ago
It's kinda annoying how the valid criticism of cloud solutions is getting drowned out by criticism on strawmen. Who exactly is saying that no one can afford their own infra anymore? What some people say is that cloud is usually cheaper than running your own infra. They may be wrong (I'm doubtful myself), but it's a wholly different argument.
- rhizome 7y ago
  It usually takes the form of "ugh then you have to hire dedicated people to manage it."
  - RussianCow 7y ago
    Which is a valid argument, isn't it? Engineers are expensive! Whether they're more expensive than AWS depends entirely on what you're doing, but the answer is almost always "yes" at a small enough scale.
    - rhizome 7y ago
      See my other comment.
      - acdha 7y ago
        At least link it
        rhizome 7y ago
        Here ya go: https://news.ycombinator.com/item?id=19842802 https://news.ycombinator.com/item?id=19842802
  - antonvs 7y ago
    You don't just have to hire dedicated people. You also have to manage those people effectively. That tends to lead to the typical old-style IT organization which is nowhere near as responsive to the needs of the business as it could be, and this is a competitive disadvantage. The value proposition of cloud is much more than just technical. Engineers who don't understand why cloud is "winning" are focusing on the wrong factors. The technical aspect is only a part of the picture.
rhizome 7y ago
Verio was a harbinger.
acdha 7y ago
> The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. That conspiracy theory depends on not asking about the major costs areas you left out (e.g. ops, security, reliability) which are the most significant until you're at a fairly large scale. The places I saw switching to cloud providers did so after comparing the quality & cost of the options available, not because they were following the cool kids.
- kortilla 7y ago
  Ops, security, and reliability don’t go away when using EC2 either, which is what led the massive cloud explosion. Places switched because they didn’t like having to forecast capacity so hourly billed resources turned on and off at the drop of a hat were super attractive.
  - RussianCow 7y ago
    > Ops, security, and reliability don’t go away when using EC2 either They don't go away, but you have to spend considerably fewer resources on it because Amazon does a lot of it for you. Edit: This effect is even more pronounced when you use Amazon's more specialized managed services, like S3, RDS, Lambda, etc. Then the things you mentioned almost completely go away, as you're letting Amazon completely maintain your infrastructure.
    - devonkim 7y ago
      What used to be behind firewalls oftentimes became accidentally public through the many, many blunders of publicly available S3 buckets that people started to use because they were frustrated with AWS IAM. This is ironic because IAM was supposed to help you secure your data and such, not make things worse for your security profile. Even with all the tooling that exists to help people write better IAM policies and figure things out, cross-account IAM permissions and roles, S3 bucket policies, etc. are a nightmare for most people beyond fairly seasoned engineers in AWS and that's a problem. Secure-by-default is not quite 100% true with most AWS services in practice unfortunately.
spamizbad 7y ago
There's no conspiracy: operating your own hardware in a datacenter requires expertise, start-up costs, and additional time. That power, control and (later) cost-savings doesn't come without a few drawbacks. Short-term budgets and time windows to meet deliverables shrank along with the costs. That's why so many are on the cloud.
duxup 7y ago
I worked in a shop that had dedicated lab equipment and resources. Suddenly buying new equipment that we would use for years became like pulling teeth, no money they said. They couldn't explain what changed. Then just as suddenly they were willing to approve spending more than our one time equipment purchase... per month in monthly cloud costs / some poorly thought out cloud deployments.... Result: -1 Dedicated person to manage the lab. +2 two new way more expensive dev ops guys. -X Occasional new equipment costs / service contracts. +Y A metric ton of reoccurring monthly costs.
- jethro_tell 7y ago
  Sure, but the accountants got to move it from capEx to opEx and then everyone went out for a beer.
  - duxup 7y ago
    Also someone got to put it on their resume... probably someone who didn't have to deal with the headaches ;)
  - z3t4 7y ago
    And the boss get to control a bigger budget, instead of having to ask for the money every time. It would probably be much nicer to work for a company where all exec and middle management where replaced by some AI mastermind, that would give compiler like answers: - "sorry, we can't order this equipment because the inquiry does not end with a new-line character". Replace HR too, and the answer to your C++ job application would be "Sorry, we could not find c++ in your resume" (case sensitive, it has to be in lower-case). Think of how much money and grief that would save.
    - duxup 7y ago
      Add's c++ to resume. "job added!" This is awesome!
- 420codebro 7y ago
  Big company politics my friend. I was just told to find a way to move 1M of capex to opex in the next 6 months for my project. Dumping shit into Cloud Provider X instead of buying our own stuff just because? Yeah, I agree, short-sighted. But then again everything in life is technically short-sighted on a long enough timeline.
rhinoceraptor 7y ago
The hardware isn’t the expensive part, it’s the people to run it that is.
- hotsauceror 7y ago
  Using publicly available costing data from last November, I see that an m4.xlarge RDS reserved instance of SQL Server Enterprise Edition with multi-AZ failover is $39,000 / year. If I have 4 DBAs at a cost of $150K / yr salary + benefits, that's the equivalent of around 15 smallish, managed SQL Server instances. My current SQL footprint is on the order of 600 instances, which works out to be a bit over $23M a year. It is true that I've saved the $600K on the expensive personnel, though...
  - user5994461 7y ago
    How much is the license for SQL enterprise nowadays? still 5k per node or does Microsoft charge per socket now?
    - hotsauceror 7y ago
      Depends on your organization, the licensing model you pick, etc. They generally license per core now, and you have to run at least 4 cores. Without giving too much away I'd say the cost of licensing a 4-core box running Enterprise is on the order of $20K, which you pay one time, up front. Then you just pay for your support contract. You can also do stuff like license an entire VMware farm or failover cluster, and then you can overprovision services for extra cost savings.
  - RussianCow 7y ago
    As your operation gets bigger, the scale generally starts to tip in favor of running your own hardware. But 600 SQL Server instances is fairly large—when that number is in the single digits, the cost of DBAs/DevOps dwarfs that of the hardware. So the scale of your business matters a lot, and it's not useful to generalize this to all companies. Edit: Your math seems off as well. Don't forget that you'd have to pay the SQL Server license regardless of where it's hosted, as well as the cost of dedicated hardware if you go that route, so the difference between the two options will be much smaller than $39k per instance. And I don't know if that salary number is real, but $150k including overhead seems really low for a good DBA (at least in the US)—I would expect at least $200k.
jasode 7y ago
>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization". In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house IT employees' speed of tech innovation vs Amazon's engineers'. An example of this disparity was Guardian's disaster with its in-house organization trying to use OpenStack.[1] For many non-tech companies where IT computing is a cost center, their employees won't be able to match the iteration speed of Google's engineers constantly improving on GCP or Amazon's employees enhancing the features of AWS. We've all heard the stories where a company's project submits a requisition to internal IT department for 2 development servers for their programmers -- and then the IT bureaucracy tells them that it will take 2 weeks. Over time, the internal IT dept treats the other departments as adversaries instead of customers. Executives get fed up with slow IT departments and get excited when a few clicks on AWS dashboard gets them servers spun up in 10 minutes. It's not just a cpu+hardware comparison. Companies outsource to AWS/GCP/Azure because it's quicker turnaround with more datacenter features than their internal IT teams can deliver. Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. [1] https://www.computerworlduk.com/cloud-computing/guardian-goes-all-in-on-aws-public-cloud-after-openstack-disaster-3629790/ https://www.computerworlduk.com/cloud-computing/guardian-goe...
- gambler 7y ago
  >Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list of self-hosting companies shrunk by a huge degree. Doesn't that seem interesting?
  - lixtra 7y ago
    Ok, I will try to counter that. The stack grows higher. More specialized employees become more effective. In 200X you just needed a Linux admin. Now on top of being good a good Linux admin you also need to know docker, kubernetes (the admin part, not the user part) etc. If you don’t need that daily, because you also need to maintain some apps, you will be less professional than a cloud provider. Of course it’s a trade-off. If you’re large, with constant high resource needs and benefit from custom optimization, you may be better off with your own team/infrastructure. If you’re small (2 it guys), fast changing, normal speed requirements / easy to scale out then the cloud should be interesting to you.
c3534l 7y ago
That's the difference between comparative advantage and absolute advantage.
jononor 7y ago
Back then you basically had to do it, so everyone did. These days you have to compete with companies that make use of the flexible offerings of AWS/GoogleCloud/Azure. Which for small companies is a large advantage, since it makes it possible to start up with almost no money, and scale up smoothly, while letting your employees be focused on the product you offer for your customers.
technological 7y ago
I think one of the issue is availability or distributed data centers. Its indeed cheaper to buy hardware and setup server but to ensure you have datacenters in many locations is the problem.
Thaxll 7y ago
Yep and we were running bare metal servers with pretty bad behaviour sec wise.
nijave 7y ago
Yeah and how long did it take to update software back then? Weeks, months, years? Now changes can reliably get pushed to production in hours or even minutes which has incredible business value
radium3d 7y ago
I dunno, you can get a lot of dedicated power for a small amount of money from great hosting companies such as Linode.com. I have been working with them for the last few years and they've been great. They even roll out new hardware to existing customers for no additional cost. Blows bad hosting companies out of the water. For example Media Temple who never upgrade their hardware for existing customers at no additional cost.
neop1x 7y ago
Here is an awesome presentation from Rancher about how cloud providers are not providing resources but convenience and want people to buy more and more services. https://m.youtube.com/watch?v=mnu477057W0 https://m.youtube.com/watch?v=mnu477057W0 full slides etc are here after registration if interested https://info.rancher.com/single-node-standalone-servers-scalable-secure-storage https://info.rancher.com/single-node-standalone-servers-scal...

icebraining 7y ago

It's kinda annoying how the valid criticism of cloud solutions is getting drowned out by criticism on strawmen. Who exactly is saying that no one can afford their own infra anymore? What some people say is that cloud is usually cheaper than running your own infra. They may be wrong (I'm doubtful myself), but it's a wholly different argument.

rhizome 7y ago
It usually takes the form of "ugh then you have to hire dedicated people to manage it."
- RussianCow 7y ago
  Which is a valid argument, isn't it? Engineers are expensive! Whether they're more expensive than AWS depends entirely on what you're doing, but the answer is almost always "yes" at a small enough scale.
  - rhizome 7y ago
    See my other comment.
    - acdha 7y ago
      At least link it
      - rhizome 7y ago
        Here ya go: https://news.ycombinator.com/item?id=19842802 https://news.ycombinator.com/item?id=19842802
- antonvs 7y ago
  You don't just have to hire dedicated people. You also have to manage those people effectively. That tends to lead to the typical old-style IT organization which is nowhere near as responsive to the needs of the business as it could be, and this is a competitive disadvantage. The value proposition of cloud is much more than just technical. Engineers who don't understand why cloud is "winning" are focusing on the wrong factors. The technical aspect is only a part of the picture.

rhizome 7y ago

Verio was a harbinger.

acdha 7y ago

> The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. That conspiracy theory depends on not asking about the major costs areas you left out (e.g. ops, security, reliability) which are the most significant until you're at a fairly large scale. The places I saw switching to cloud providers did so after comparing the quality & cost of the options available, not because they were following the cool kids.

kortilla 7y ago
Ops, security, and reliability don’t go away when using EC2 either, which is what led the massive cloud explosion. Places switched because they didn’t like having to forecast capacity so hourly billed resources turned on and off at the drop of a hat were super attractive.
- RussianCow 7y ago
  > Ops, security, and reliability don’t go away when using EC2 either They don't go away, but you have to spend considerably fewer resources on it because Amazon does a lot of it for you. Edit: This effect is even more pronounced when you use Amazon's more specialized managed services, like S3, RDS, Lambda, etc. Then the things you mentioned almost completely go away, as you're letting Amazon completely maintain your infrastructure.
  - devonkim 7y ago
    What used to be behind firewalls oftentimes became accidentally public through the many, many blunders of publicly available S3 buckets that people started to use because they were frustrated with AWS IAM. This is ironic because IAM was supposed to help you secure your data and such, not make things worse for your security profile. Even with all the tooling that exists to help people write better IAM policies and figure things out, cross-account IAM permissions and roles, S3 bucket policies, etc. are a nightmare for most people beyond fairly seasoned engineers in AWS and that's a problem. Secure-by-default is not quite 100% true with most AWS services in practice unfortunately.

spamizbad 7y ago

There's no conspiracy: operating your own hardware in a datacenter requires expertise, start-up costs, and additional time. That power, control and (later) cost-savings doesn't come without a few drawbacks. Short-term budgets and time windows to meet deliverables shrank along with the costs. That's why so many are on the cloud.

duxup 7y ago

I worked in a shop that had dedicated lab equipment and resources. Suddenly buying new equipment that we would use for years became like pulling teeth, no money they said. They couldn't explain what changed. Then just as suddenly they were willing to approve spending more than our one time equipment purchase... per month in monthly cloud costs / some poorly thought out cloud deployments.... Result: -1 Dedicated person to manage the lab. +2 two new way more expensive dev ops guys. -X Occasional new equipment costs / service contracts. +Y A metric ton of reoccurring monthly costs.

jethro_tell 7y ago
Sure, but the accountants got to move it from capEx to opEx and then everyone went out for a beer.
- duxup 7y ago
  Also someone got to put it on their resume... probably someone who didn't have to deal with the headaches ;)
- z3t4 7y ago
  And the boss get to control a bigger budget, instead of having to ask for the money every time. It would probably be much nicer to work for a company where all exec and middle management where replaced by some AI mastermind, that would give compiler like answers: - "sorry, we can't order this equipment because the inquiry does not end with a new-line character". Replace HR too, and the answer to your C++ job application would be "Sorry, we could not find c++ in your resume" (case sensitive, it has to be in lower-case). Think of how much money and grief that would save.
  - duxup 7y ago
    Add's c++ to resume. "job added!" This is awesome!
420codebro 7y ago
Big company politics my friend. I was just told to find a way to move 1M of capex to opex in the next 6 months for my project. Dumping shit into Cloud Provider X instead of buying our own stuff just because? Yeah, I agree, short-sighted. But then again everything in life is technically short-sighted on a long enough timeline.

rhinoceraptor 7y ago

The hardware isn’t the expensive part, it’s the people to run it that is.

hotsauceror 7y ago
Using publicly available costing data from last November, I see that an m4.xlarge RDS reserved instance of SQL Server Enterprise Edition with multi-AZ failover is $39,000 / year. If I have 4 DBAs at a cost of $150K / yr salary + benefits, that's the equivalent of around 15 smallish, managed SQL Server instances. My current SQL footprint is on the order of 600 instances, which works out to be a bit over $23M a year. It is true that I've saved the $600K on the expensive personnel, though...
- user5994461 7y ago
  How much is the license for SQL enterprise nowadays? still 5k per node or does Microsoft charge per socket now?
  - hotsauceror 7y ago
    Depends on your organization, the licensing model you pick, etc. They generally license per core now, and you have to run at least 4 cores. Without giving too much away I'd say the cost of licensing a 4-core box running Enterprise is on the order of $20K, which you pay one time, up front. Then you just pay for your support contract. You can also do stuff like license an entire VMware farm or failover cluster, and then you can overprovision services for extra cost savings.
- RussianCow 7y ago
  As your operation gets bigger, the scale generally starts to tip in favor of running your own hardware. But 600 SQL Server instances is fairly large—when that number is in the single digits, the cost of DBAs/DevOps dwarfs that of the hardware. So the scale of your business matters a lot, and it's not useful to generalize this to all companies. Edit: Your math seems off as well. Don't forget that you'd have to pay the SQL Server license regardless of where it's hosted, as well as the cost of dedicated hardware if you go that route, so the difference between the two options will be much smaller than $39k per instance. And I don't know if that salary number is real, but $150k including overhead seems really low for a good DBA (at least in the US)—I would expect at least $200k.

jasode 7y ago

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization". In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house IT employees' speed of tech innovation vs Amazon's engineers'. An example of this disparity was Guardian's disaster with its in-house organization trying to use OpenStack.[1] For many non-tech companies where IT computing is a cost center, their employees won't be able to match the iteration speed of Google's engineers constantly improving on GCP or Amazon's employees enhancing the features of AWS. We've all heard the stories where a company's project submits a requisition to internal IT department for 2 development servers for their programmers -- and then the IT bureaucracy tells them that it will take 2 weeks. Over time, the internal IT dept treats the other departments as adversaries instead of customers. Executives get fed up with slow IT departments and get excited when a few clicks on AWS dashboard gets them servers spun up in 10 minutes. It's not just a cpu+hardware comparison. Companies outsource to AWS/GCP/Azure because it's quicker turnaround with more datacenter features than their internal IT teams can deliver. Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. [1] https://www.computerworlduk.com/cloud-computing/guardian-goes-all-in-on-aws-public-cloud-after-openstack-disaster-3629790/ https://www.computerworlduk.com/cloud-computing/guardian-goe...

gambler 7y ago
>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list of self-hosting companies shrunk by a huge degree. Doesn't that seem interesting?
- lixtra 7y ago
  Ok, I will try to counter that. The stack grows higher. More specialized employees become more effective. In 200X you just needed a Linux admin. Now on top of being good a good Linux admin you also need to know docker, kubernetes (the admin part, not the user part) etc. If you don’t need that daily, because you also need to maintain some apps, you will be less professional than a cloud provider. Of course it’s a trade-off. If you’re large, with constant high resource needs and benefit from custom optimization, you may be better off with your own team/infrastructure. If you’re small (2 it guys), fast changing, normal speed requirements / easy to scale out then the cloud should be interesting to you.

c3534l 7y ago

That's the difference between comparative advantage and absolute advantage.

jononor 7y ago

Back then you basically had to do it, so everyone did. These days you have to compete with companies that make use of the flexible offerings of AWS/GoogleCloud/Azure. Which for small companies is a large advantage, since it makes it possible to start up with almost no money, and scale up smoothly, while letting your employees be focused on the product you offer for your customers.

technological 7y ago

I think one of the issue is availability or distributed data centers. Its indeed cheaper to buy hardware and setup server but to ensure you have datacenters in many locations is the problem.

Thaxll 7y ago

Yep and we were running bare metal servers with pretty bad behaviour sec wise.

nijave 7y ago

Yeah and how long did it take to update software back then? Weeks, months, years? Now changes can reliably get pushed to production in hours or even minutes which has incredible business value

radium3d 7y ago

I dunno, you can get a lot of dedicated power for a small amount of money from great hosting companies such as Linode.com. I have been working with them for the last few years and they've been great. They even roll out new hardware to existing customers for no additional cost. Blows bad hosting companies out of the water. For example Media Temple who never upgrade their hardware for existing customers at no additional cost.

neop1x 7y ago

Here is an awesome presentation from Rancher about how cloud providers are not providing resources but convenience and want people to buy more and more services. https://m.youtube.com/watch?v=mnu477057W0 https://m.youtube.com/watch?v=mnu477057W0 full slides etc are here after registration if interested https://info.rancher.com/single-node-standalone-servers-scalable-secure-storage https://info.rancher.com/single-node-standalone-servers-scal...