5 ms·

The hidden cost of all cloud based software. Even the cheapest cloud (seems to be Google at the moment) is 5x or more of the price of an equivalent dedicated se

The hidden cost of all cloud based software. Even the cheapest cloud (seems to be Google at the moment) is 5x or more of the price of an equivalent dedicated server (2x for "compute", >20x for bandwidth, so 5x is somewhat average. And that's compared to "normal" dedi providers. Ok to use Leaseweb, Hetzner and OVH ? Make it 10x more expensive). Since VPS's exist cloud doesn't even make sense for the smallest websites anymore. Ironically VPS's predate and are even the basis of cloud systems. And there's the intangible technical debt that cloud imparts. Whichever cloud you pick, in the future you'll change. That's how the world works. Switching dedicated service providers is a complicated ops problem. Switching cloud providers is a full rewrite of everything AND a complicated ops problem. Just being on the cloud, by itself, forces you to put in serious technical debt. I don't understand, even after using cloud at work, what those cloud systems provide that can't be done, better, cheaper, and with less relying on a single organisation, on dedicated service providers. I mean Google's hypervisor is good (very good), but it still imparts significant costs compared to bare metal.

bamboozled 9y ago
It's fashionable to use cloud services and people don't really have to know how to use configuration management tools, or know almost anything about operating system their software runs on. What more could you want besides paying 5x more for the privilege ? It's just outsourcing. Edit: I wanted to add that from what I've seen recently. "Engineers" who aren't familiar with how the various layers of the OSI model work and know a little about operating systems seem to struggle to build useful, secure solutions at higher levels which function at scale. Ultimately, we're still writing code to run on Von Neuman computers.
- throwaway2048 9y ago
  except its not outsourcing, not really, you still need a strong understanding of what you are doing at all levels, hardware and software for any kind of serious setup.
  - bamboozled 9y ago
    Sorry, see my update. I agree with you, but what's the incentive to learn that stuff these days?
    - throwaway2048 9y ago
      not having your stuff run like garbage while you overpay out the ass? I mean I know its not a concern for startups fueled by some one else's money, but it still matters.
cdoxsey 9y ago
They provide many things: - managed systems like s3, dynamodb, etc, which are not trivial to run yourself - on-demand and flexible instance types which allow you to try something without committing - spot instances, autoscaling groups, access to job scheduling like kubernetes. At scale (thousands of nodes) technologies like this are extremely important - redundant and reliable service. It also gives you someone to complain too and blame (they provide slas) - powerful management tooling and APIs that are well documented and widely understood (you can hire an engineer that has used AWS) - security: audit trails, integrated single sign on, private networks, vpns, role based access keys I could keep going. Also keep in mind that reserved pricing (or googles sustained use discount) lowers the cost.
- candiodari 9y ago
  > managed systems like s3, dynamodb, etc, which are not trivial to run yourself How many organisations need a database to scale like that ? Maybe 1000 worldwide, and at least 900 of them are forbidden by law to run them on clouds (e.g. banks). Also, cloud is essentially outsourcing server management. If you're of a size that needs these sorts of databases, you need a big IT department, cloud or no cloud. At that point, cloud is just vastly more expensive without any monetary advantage. And of course, if you use any of these, the lock in is ridiculous. Also known as "they have you by the b....", and if one thing is damned sure the result will not be saving money, complexity or effort. Or savings of any kind whatsoever. > on-demand and flexible instance types which allow you to try something without committing And ever more management systems that let management "control spending", "control access", ... and so on, designed to prevent that. The problem with the old system was management and processes getting in the way of experimenting, and the only advantage cloud has is that it doesn't have decent support for such processes yet. Every vendor is racing to build them in, and it's getting worse week by week. It won't be long before experimenting is impossible in cloud just like it is in owned datacenter and dedicated setups. The problem with experiments is management in every company I've ever consulted for, and cloud doesn't change a thing, aside from, at the moment, management incompetence getting in the way of their usual sabotage. That will stop once current engineers that know cloud start getting a few promos and is already happening. > spot instances, autoscaling groups, access to job scheduling like kubernetes. At scale (thousands of nodes) technologies like this are extremely important There is no shortage of system supporting this sort of load on dedicated machines, servers, ... > redundant and reliable service. It also gives you someone to complain too and blame (they provide slas) Have you read those slas ? They'll return 10% or so of what you normally pay (not in cash of course, in vouchers) if their service is out for 33% of the time or so. That's not an sla, that's just laughable. If they are out for 5% of the total time, I'd be so pissed I'd start a chargeback. As for redundancy and reliability, there is no shortage of vendors with similar reliability track records, and it is well documented how to make server setups that provide reliability. > powerful management tooling and APIs that are well documented and widely understood (you can hire an engineer that has used AWS) And dedicated servers are linux servers. You can hire engineers, I believe, that have used those. > security: audit trails, integrated single sign on, private networks, vpns, role based access keys Yep, these controls are part of what's preventing that experimenting advantage that currently exists. In reality, all of this relies on well designed policies at the customer side, and I've yet to see that done properly for more than a single team in any company. All of these are of course critically dependent on Google's network, tapped by the NSA as documented in the famous leaked "SSL inserted and removed here" slide, and for numerous privacy breaches, Amazon, famous for "somehow" finding out about how the businesses hosted on their cloud work and taking their customers away (apparently more than 100 court cases against Amazon currently going on), or Microsoft, famous for so many things, including pushing through centralization in Skype for the admitted reason of gaining the ability to spy on their customers. I'm sure their ethics in the cloud hosting department are much better. And please don't even mention those "encrypted storage" assurances. They're not even worth being called bullshit. If I get to write the code that you use to encrypt/decrypt, I can obviously trivially break any key you use. If you don't believe me, I'll give you a webbrowser to download, and I assure you it encrypts all your passwords, then just log into your web banking a few times and see what happens (note: webbrowsers inserting "extra" transactions when doing webbanking are a common thing to find in security work). So secure ... yeah right. Sure, it can work. It won't, of course. > I could keep going. Yeah you could mention that cloud's real advantage is that engineers believe that if they get a web-based non-multiplayer version of minesweeper implemented on kubernetes, they'll get a $200k+ GOOG/FB job. Not true, of course, but nobody wants to tell em that. In fact, implementing a linked list in C on a DOS 1.0 C compiler will help them a LOT more.
Niksko 9y ago
Using an infrastructure as code platform like Terraform makes switching clouds vastly easier, but your other points still stand.
- bonesss 9y ago
  Kubernetes, along with an architecture informed by the salient points the GP, also provides cloud provider mobility while opening up an on-premise story to match. As with all dependencies, though, you either manage them or they manage you... My systems generally wrap their usage of third-party components with internal facades -- an interface that I control that provides a clear contract for replacing the component, generally as part of an AntiCorruption Layer [0]. The more costly the replacement, the more need for abstraction, so I can't imagine why I would treat multi-system communication with Googles or Microsofts cloud-ESB as something less prone to change, replacement, or parallel requirements than whatever PDF-generator we're using this week. - [0] - http://www.markhneedham.com/blog/2009/07/07/domain-driven-design-anti-corruption-layer/ http://www.markhneedham.com/blog/2009/07/07/domain-driven-de...
sitharus 9y ago
You're completely right, if your workload is even so you can get 15 servers installed and use them all year you're better off using bare metal. However if you have uneven load - say you're in accounting and you need 5 servers most of the time, but 50 in the runup to year end, you will save money in the cloud rather than having 50 servers requiring power and patching and 45 idle. There are also other benefits to doing things in the cloud. When you start treating machines as disposable you're forced to automate setup. When you've automated setup you no longer patch machines - you get a new patched base image, run the setup, then simply replace the old machines.
bonesss 9y ago
> I don't understand, even after using cloud at work, what those cloud systems provide that can't be done, better, cheaper, and with less relying on a single organisation, on dedicated service providers. You're pretty much spot on, technically. Anything they can do, you can do better, faster, smarter, more streamlined, etc, outside of some core competencies. It's hard to do BigTable better than Google, and my Paxos is just gonna be worse than what a few decades of 'mission critical' at Microsoft will create. The reason these are viable in the market, though, isn't technical: it's Time To Market and capital expenditures. For a SaaS company "The Cloud" is a dead end in the long run because it costs too much, but it's golden in the short term because you throw a few credit card numbers at the issue and you're up and running. You scale, make money, scale more, make more money, and then use your half-decades head start on the competition to bring down your costs. Anything that's not working gets turned off, and you've only lost operating costs without some huge hardware investment. The commercial cloud is not about running things 24/7 cheaper than your datacenter, the commercial cloud is about letting you pretend you have a datacenter while you get your unicorn to an IPO ;) For investors, CTOs, CIOs, and CEOs the picture is one of transition, not permanence [0]: > Even if we believe the world is going gaga for cloud - and I do - there are different flavours of cloud, with SaaS comprising a two-thirds of all public cloud spending, by IDC’s estimates. As for those SaaS vendors, they’re running lots (and lots) of VMware. As one insider at a large SaaS provider shared with me, their model is to run on public cloud when they enter a new region, then switch to private data centre once they hit a revenue threshold ($50m). - [0] - https://www.theregister.co.uk/2017/09/11/kubernetes_envy/ https://www.theregister.co.uk/2017/09/11/kubernetes_envy/
oelmekki 9y ago
I've thought like this for a decade, then the hdd of one of my dedicated server at ovh, on which a side-project with paying users runs, failed. I was confident at the moment: I had backups, I knew the server was provided with raid support, so it was just about replacing the hdd. It took one hour of downtime to replace it, and then I got a message from ovh: "the hdd has been replaced. The server still doesn't boot. Customer has to start the server in rescue mode and fix raid support". Now, I wouldn't mind learning about managing raid, but not while my service is down and I keep answering users mails about it. I switched to aws, despite the additional cost, because it's less dangerous for a small team (or one person team, like me). I realized the additional cost was actually paying for additional services (managing everything related to hardware).
stu432 9y ago
For the majority of use cases, involving relatively static workloads, the cloud doesn't offer many financial benefits, as you say, particularly if you have access to a competent infrastructure engineer (which are harder to find than you'd think). My experience of cloud has generally been around gambling websites, which have a very distinctive use case pattern, with daily peaks and troughs as well as seasonal peaks and troughs. There can be as much as a 25x difference in throughput between these occasions. For these sites it's a case of implement scaling natively in the cloud and stop worrying about capacity forecasting or invest millions in infrastructure upfront and watch it idle 90% of the time. The financials simply work in this field, as I'd guess they do in serious online retailers with seasonal demand. However your average 2 web server, load balancer and single db server, not so much.
lovich 9y ago
In my companies where we used the cloud it just came down to internal pragmatism on our part. If we wanted to hire someone that could take over a year between convincing management to allocate the budget for it and going through HR's hiring process. The cloud was something where the recurring monthly costs were small enough to be expensed. We were given a single IT contractor who worked at multiple sites for the company and would show up 1-2 weeks after you notified him of an issue. So we ended up using the cloud, the company has a giant liability now that everything is dependent on the cloud providers, but management ends up happy because they only see 4-5figure charges per month and not a 6 figure cost per year they'd see for a new employee. I'm pretty sure if the company's budgets reported employees as monthly costs we could have just gotten another employee, but there's not much you can do when the decisions are made that far above your head

bamboozled 9y ago

It's fashionable to use cloud services and people don't really have to know how to use configuration management tools, or know almost anything about operating system their software runs on. What more could you want besides paying 5x more for the privilege ? It's just outsourcing. Edit: I wanted to add that from what I've seen recently. "Engineers" who aren't familiar with how the various layers of the OSI model work and know a little about operating systems seem to struggle to build useful, secure solutions at higher levels which function at scale. Ultimately, we're still writing code to run on Von Neuman computers.

throwaway2048 9y ago
except its not outsourcing, not really, you still need a strong understanding of what you are doing at all levels, hardware and software for any kind of serious setup.
- bamboozled 9y ago
  Sorry, see my update. I agree with you, but what's the incentive to learn that stuff these days?
  - throwaway2048 9y ago
    not having your stuff run like garbage while you overpay out the ass? I mean I know its not a concern for startups fueled by some one else's money, but it still matters.

cdoxsey 9y ago

They provide many things: - managed systems like s3, dynamodb, etc, which are not trivial to run yourself - on-demand and flexible instance types which allow you to try something without committing - spot instances, autoscaling groups, access to job scheduling like kubernetes. At scale (thousands of nodes) technologies like this are extremely important - redundant and reliable service. It also gives you someone to complain too and blame (they provide slas) - powerful management tooling and APIs that are well documented and widely understood (you can hire an engineer that has used AWS) - security: audit trails, integrated single sign on, private networks, vpns, role based access keys I could keep going. Also keep in mind that reserved pricing (or googles sustained use discount) lowers the cost.

candiodari 9y ago
> managed systems like s3, dynamodb, etc, which are not trivial to run yourself How many organisations need a database to scale like that ? Maybe 1000 worldwide, and at least 900 of them are forbidden by law to run them on clouds (e.g. banks). Also, cloud is essentially outsourcing server management. If you're of a size that needs these sorts of databases, you need a big IT department, cloud or no cloud. At that point, cloud is just vastly more expensive without any monetary advantage. And of course, if you use any of these, the lock in is ridiculous. Also known as "they have you by the b....", and if one thing is damned sure the result will not be saving money, complexity or effort. Or savings of any kind whatsoever. > on-demand and flexible instance types which allow you to try something without committing And ever more management systems that let management "control spending", "control access", ... and so on, designed to prevent that. The problem with the old system was management and processes getting in the way of experimenting, and the only advantage cloud has is that it doesn't have decent support for such processes yet. Every vendor is racing to build them in, and it's getting worse week by week. It won't be long before experimenting is impossible in cloud just like it is in owned datacenter and dedicated setups. The problem with experiments is management in every company I've ever consulted for, and cloud doesn't change a thing, aside from, at the moment, management incompetence getting in the way of their usual sabotage. That will stop once current engineers that know cloud start getting a few promos and is already happening. > spot instances, autoscaling groups, access to job scheduling like kubernetes. At scale (thousands of nodes) technologies like this are extremely important There is no shortage of system supporting this sort of load on dedicated machines, servers, ... > redundant and reliable service. It also gives you someone to complain too and blame (they provide slas) Have you read those slas ? They'll return 10% or so of what you normally pay (not in cash of course, in vouchers) if their service is out for 33% of the time or so. That's not an sla, that's just laughable. If they are out for 5% of the total time, I'd be so pissed I'd start a chargeback. As for redundancy and reliability, there is no shortage of vendors with similar reliability track records, and it is well documented how to make server setups that provide reliability. > powerful management tooling and APIs that are well documented and widely understood (you can hire an engineer that has used AWS) And dedicated servers are linux servers. You can hire engineers, I believe, that have used those. > security: audit trails, integrated single sign on, private networks, vpns, role based access keys Yep, these controls are part of what's preventing that experimenting advantage that currently exists. In reality, all of this relies on well designed policies at the customer side, and I've yet to see that done properly for more than a single team in any company. All of these are of course critically dependent on Google's network, tapped by the NSA as documented in the famous leaked "SSL inserted and removed here" slide, and for numerous privacy breaches, Amazon, famous for "somehow" finding out about how the businesses hosted on their cloud work and taking their customers away (apparently more than 100 court cases against Amazon currently going on), or Microsoft, famous for so many things, including pushing through centralization in Skype for the admitted reason of gaining the ability to spy on their customers. I'm sure their ethics in the cloud hosting department are much better. And please don't even mention those "encrypted storage" assurances. They're not even worth being called bullshit. If I get to write the code that you use to encrypt/decrypt, I can obviously trivially break any key you use. If you don't believe me, I'll give you a webbrowser to download, and I assure you it encrypts all your passwords, then just log into your web banking a few times and see what happens (note: webbrowsers inserting "extra" transactions when doing webbanking are a common thing to find in security work). So secure ... yeah right. Sure, it can work. It won't, of course. > I could keep going. Yeah you could mention that cloud's real advantage is that engineers believe that if they get a web-based non-multiplayer version of minesweeper implemented on kubernetes, they'll get a $200k+ GOOG/FB job. Not true, of course, but nobody wants to tell em that. In fact, implementing a linked list in C on a DOS 1.0 C compiler will help them a LOT more.

Niksko 9y ago

Using an infrastructure as code platform like Terraform makes switching clouds vastly easier, but your other points still stand.

bonesss 9y ago
Kubernetes, along with an architecture informed by the salient points the GP, also provides cloud provider mobility while opening up an on-premise story to match. As with all dependencies, though, you either manage them or they manage you... My systems generally wrap their usage of third-party components with internal facades -- an interface that I control that provides a clear contract for replacing the component, generally as part of an AntiCorruption Layer [0]. The more costly the replacement, the more need for abstraction, so I can't imagine why I would treat multi-system communication with Googles or Microsofts cloud-ESB as something less prone to change, replacement, or parallel requirements than whatever PDF-generator we're using this week. - [0] - http://www.markhneedham.com/blog/2009/07/07/domain-driven-design-anti-corruption-layer/ http://www.markhneedham.com/blog/2009/07/07/domain-driven-de...

sitharus 9y ago

You're completely right, if your workload is even so you can get 15 servers installed and use them all year you're better off using bare metal. However if you have uneven load - say you're in accounting and you need 5 servers most of the time, but 50 in the runup to year end, you will save money in the cloud rather than having 50 servers requiring power and patching and 45 idle. There are also other benefits to doing things in the cloud. When you start treating machines as disposable you're forced to automate setup. When you've automated setup you no longer patch machines - you get a new patched base image, run the setup, then simply replace the old machines.

bonesss 9y ago

> I don't understand, even after using cloud at work, what those cloud systems provide that can't be done, better, cheaper, and with less relying on a single organisation, on dedicated service providers. You're pretty much spot on, technically. Anything they can do, you can do better, faster, smarter, more streamlined, etc, outside of some core competencies. It's hard to do BigTable better than Google, and my Paxos is just gonna be worse than what a few decades of 'mission critical' at Microsoft will create. The reason these are viable in the market, though, isn't technical: it's Time To Market and capital expenditures. For a SaaS company "The Cloud" is a dead end in the long run because it costs too much, but it's golden in the short term because you throw a few credit card numbers at the issue and you're up and running. You scale, make money, scale more, make more money, and then use your half-decades head start on the competition to bring down your costs. Anything that's not working gets turned off, and you've only lost operating costs without some huge hardware investment. The commercial cloud is not about running things 24/7 cheaper than your datacenter, the commercial cloud is about letting you pretend you have a datacenter while you get your unicorn to an IPO ;) For investors, CTOs, CIOs, and CEOs the picture is one of transition, not permanence [0]: > Even if we believe the world is going gaga for cloud - and I do - there are different flavours of cloud, with SaaS comprising a two-thirds of all public cloud spending, by IDC’s estimates. As for those SaaS vendors, they’re running lots (and lots) of VMware. As one insider at a large SaaS provider shared with me, their model is to run on public cloud when they enter a new region, then switch to private data centre once they hit a revenue threshold ($50m). - [0] - https://www.theregister.co.uk/2017/09/11/kubernetes_envy/ https://www.theregister.co.uk/2017/09/11/kubernetes_envy/

oelmekki 9y ago

I've thought like this for a decade, then the hdd of one of my dedicated server at ovh, on which a side-project with paying users runs, failed. I was confident at the moment: I had backups, I knew the server was provided with raid support, so it was just about replacing the hdd. It took one hour of downtime to replace it, and then I got a message from ovh: "the hdd has been replaced. The server still doesn't boot. Customer has to start the server in rescue mode and fix raid support". Now, I wouldn't mind learning about managing raid, but not while my service is down and I keep answering users mails about it. I switched to aws, despite the additional cost, because it's less dangerous for a small team (or one person team, like me). I realized the additional cost was actually paying for additional services (managing everything related to hardware).

stu432 9y ago

For the majority of use cases, involving relatively static workloads, the cloud doesn't offer many financial benefits, as you say, particularly if you have access to a competent infrastructure engineer (which are harder to find than you'd think). My experience of cloud has generally been around gambling websites, which have a very distinctive use case pattern, with daily peaks and troughs as well as seasonal peaks and troughs. There can be as much as a 25x difference in throughput between these occasions. For these sites it's a case of implement scaling natively in the cloud and stop worrying about capacity forecasting or invest millions in infrastructure upfront and watch it idle 90% of the time. The financials simply work in this field, as I'd guess they do in serious online retailers with seasonal demand. However your average 2 web server, load balancer and single db server, not so much.

lovich 9y ago

In my companies where we used the cloud it just came down to internal pragmatism on our part. If we wanted to hire someone that could take over a year between convincing management to allocate the budget for it and going through HR's hiring process. The cloud was something where the recurring monthly costs were small enough to be expensed. We were given a single IT contractor who worked at multiple sites for the company and would show up 1-2 weeks after you notified him of an issue. So we ended up using the cloud, the company has a giant liability now that everything is dependent on the cloud providers, but management ends up happy because they only see 4-5figure charges per month and not a 6 figure cost per year they'd see for a new employee. I'm pretty sure if the company's budgets reported employees as monthly costs we could have just gotten another employee, but there's not much you can do when the decisions are made that far above your head