5 ms·

So I don't get this. 1gbps and 2 million events/s could be comfortably handled on a single beefy 10gbps server. Add a second for failover. That's maximum $20K/y

by eloff 11y ago

So I don't get this. 1gbps and 2 million events/s could be comfortably handled on a single beefy 10gbps server. Add a second for failover. That's maximum $20K/year which is far cry from $3.5 million. How does this even begin to make economic sense? The economy of the cloud has never made sense to me, since it's cheaper to overbuy your capacity by 5x in dedicated machines than use AWS or GCE. After just one year you've paid for all your capital expenditures and then you're minting it. For a startup the appeal of the cloud is obvious, but for a big established company with known technical requirements I don't get it at all.

lobster_johnson 11y ago
Virtualizing your server resources is essentially outsourcing a whole lot of sysadmin work to a vendor. Dedicated servers create a different kind of lock-in: Dynamically scaling up dedicated servers is like laying down the tracks while the train is moving. Acceptable to some, but not to a company like Spotify.
- eloff 11y ago
  You outsource some of the sysadmin work, but not all of it. Cloud servers have plenty of their own failure cases, some of them unique. Physical hardware is actually quite reliable and if you follow the same practices you should in the cloud - namely automatic failover, eliminate single points of failure, etc then the differences decrease. Dynamically scaling load is only really interesting if the load is extremely variable. You could just buy 5x the dedicated hardware, which will likely be able to serve 10x the traffic and may still end up paying less. Who cares if it sits idle most of the time. Now Spotify, like Netflix probably has extreme variance in load, much more so than a normal company. So maybe the cloud works for them. But that's not a common situation at all.
  - _delirium 11y ago
    It's definitely common for IT staffing costs to increase rather than decrease with a cloud transition, at least so far. Part of the problem is that you often are trading off sysadmins (relatively cheap) for engineers (relatively expensive), as you reengineer your system to run on top of ephemeral, distributed cloud servers, with new failure cases and new technology stacks. This is often justified on the basis that a large chunk is one-time engineering cost that will pay for itself in the long run. But that's a speculative bet, especially considering how much technology churn is happening so far (e.g. a bunch of companies that invested heavily in early 2010s cloud stacks are now re-investing in Docker). It's possible things will stabilize enough for such companies to eventually lay off many of their distributed-systems engineers and devops staff, and go more into cloud-autopilot mode with low IT headcount and lower staffing cost than pre-cloud-transition. But I don't think many companies are there yet, and I wouldn't bet a lot of money on it myself.
    - jacques_chester 11y ago
      I work for a company (Pivotal) where a major revenue source is selling a packaged version of Cloud Foundry. Our customers like it because their existing ops staff can greatly reduce their workload and allow development staff to move much faster.
  - coldtea 11y ago
    >You outsource some of the sysadmin work, but not all of it. Cloud servers have plenty of their own failure cases, some of them unique. The thing with them is if they go down you have Google to worry just as much as you (if not more) for bringing them back up. You don't have that with your own servers.
    - toyg 11y ago
      Google and their famous customer support, yes. Worried they might be, but more than you? For them, the cloud is one of many revenue streams, and nowhere near their most profitable. For you-cloud-customer, a website outage might mean you don't have a revenue stream anymore. It could make the difference between making payroll and shutting down next week. I bet you'll be much more worried than Google will ever be. Googlers might be more skilled than your ops team, yeah, but I would never bet on huge companies to worry about the fate of something that is not their main revenue generator.
kpil 11y ago
You think like an engineer :-) I think cloud services are appealing for a couple of reasons from a top-brass perspective. a) You free the capital, and the balance sheet looks nicer. b) It's less complicated on a PowerPoint level. Just a box. (probably not the reason Spotify opted for google though.) c) You don't think it's your core competence to keep servers safe and running, and rather would like to spend the time and effort on developing services while enjoying reason a) although you know it won't be cheaper. d) Maybe it's hard to find the right people
- Pyxl101 11y ago
  I don't think he thinks like an engineer (not to insult him), but I think he's overlooking the challenges that exist to running large-scale systems. > Add a second for failover. That's not how it works. How does the second server pick up where the first server left off, when the requests are stateful? How does it know when to take over? (What detects that and how?) Or if it's a human, how does that human make the judgment call? Cloud services are absolutely appealing from an engineering perspective. I am capable of building an application and running it on a fleet of machines such that the infrastructure takes care of itself completely and 100% by itself. Essentially no failure mode can involve me, unless either (A) my application has a defect in its logic, in which case it's unavoidable under any solution (B) probably a lot of customers are similarly affected, and someone is working on it. The data stores that I use are hosted and will not be impacted by any typical failure mode. The application that I've designed is deployed onto machines automatically, and if any of them encounter a failure that takes them offline, they'll be replaced automatically in a way that's a non-event. Furthermore, I have a blueprint for such systems such that building a new application in this way requires no more effort than filling out a single web form, after which an automation layer sets everything up, from the source control to the continuous build and deployment onwards. Not all systems are simple enough to fit into this model, but if you account for it while designing new software, quite a lot of it can be. With the right building blocks you can operate very operations-free infrastructure at the application layer even at large scale. These problems might start to look different once you consider challenges that require tens, hundreds, thousands, or more individual machines to solve. Another challenge is ensuring that new systems are built in a way from the beginning that can scale and that will be easy for other people to maintain. You don't want some random new-hire at your company setting up the "opaque box from hell" that has all the data on its disk (and, ahem, a "second for failover", that through a mysterious and unknown mechanism somehow takes over for the first) and that no one else will be able to maintain during failure. You also don't want to be surprised a year or two down the line when you're scaling up for the Super Bowl or Tax Day or Black Friday, or whatever, and your software falls over because "Bob the Intern, who thought every software system can be solved with a single box and 100 lines of assembly code" didn't anticipate that the business would be larger one day. If you standardize onto a certain set of building blocks, you can build software that, in addition to being operations-free, can scale to a large degree. Note: I'm speaking hyperbolically. No software is truly operations-free. It's all a matter of degree. I have worked however with teams that are expected to hand-manage their own machines, and teams that use effective cloud automation, and the pattern I've seen is a huge difference in their operational load from routine engineering tasks. I have also never seen systems that completely 100% automatically scale themselves, but there is still a huge difference between systems where you simply say "I want 10 machines" / "I want 10 requests per second to my data store", and a system like, "I guess it's time to tell our DBA to figure out how to shard our MySQL server". I also think that a large company could achieve a lot of the benefits provided by the cloud in-house, with its own data centers, if it had really strong internal automation and APIs, and strong teams who operate shared services for other teams (e.g., managed data layers). The really strong software I'm aware of for solving these problems has been kept as a competitive advantage, not released open source. So the practical reality is that few companies have through-and-through automation of the quality and degree that you can get from top cloud providers. > a) You free the capital, and the balance sheet looks nicer. Yes, I agree that this is a genuine reason for some companies to prefer the cloud, but these are largely not companies who have "top brass". Companies that are large enough to be "top brass" are typically not so capital-constrained as that this would majorly affect their purchasing decisions. The companies that care about this, rather, tend to be smaller and more capital-constrained. I think the more mundane reason why companies prefer the cloud, which is frequently overlooked, is that individual business units can purchase cloud services with more autonomy than they could traditionally order IT services in-house. Cloud budgets often fit within what an executive can expense within their own authority on a month-to-month basis, where as the same capital purchase would require more review and approval. So some companies tend toward the cloud because it makes them more agile by providing decisions they can make that are smaller and more easily fit within the scope of less-senior people.
dogma1138 11y ago
ISP cost, backups, patching, hardware upgrades, co-location, DR, insurance, paying for physical and cyber-security, personnel etc. outsourcing allot of those things saves money it's also a different expense of you buy infrastructure it has value which means depreciation overtime which affects how your accounting is done. Cloud makes sense for allot of companies not just startups the biggest reason why companies with eatablished infrastructure don't migrate is the initial cost and regulation.
- sharpy 11y ago
  +1 Not to mention that often it can takes weeks for new hardware order to be approved / delivered / provisioned. Cloud? Just dial it up... Well, if you are at Netflix scale, you probably couldn't just dial it up, and would have to work with the vendor to ensure adequate capacity was available for your use.
  - taf2 11y ago
    And keep in mind that adding those new servers even in managed eviromnents can fail and fail in unexpected ways like taking out your whole network. We had this happen with rackspace - hence we moved to Amazon
    - hhw 11y ago
      If a single point of failure took down your entire network, then your solution wasn't engineered properly. Just because on provider messed up big time doesn't mean every other provider would. The hardwdare design/architecture can also be provided to you for free as part of the sales process, no extra service required. As for hardware failures in general, you can afford to have some complete extra servers on hand doing nothing most of the time with the amount of money you'd save on renting bare metal instead of paying for cloud. Or if engineered properly, you'd be using all the machines at once but be able to afford to lose half of them, and failover automatically. Yes, a cloud (bunch of VMs on a cluster) is going to be more reliable than a single machine, as you get hardware redundancy. But a cluster of machines with hardware redundancy is going to be much more reliable than a bunch of VMs on a cluster, as there's less complexity involved without the cloud stack, and thus fewer points of failure.
kenshaw 11y ago
On a balance sheet, you are trading CapEx for OpEx. That's it. For certain organizations with specific types of computing loads, that may make the most sense. (btw -- I disagree with your math there on the actual cost -- I agree it could ultimately be cheaper to run things in house, there's no way it's going to go from $3.5 million to $20k annually. It'd be more like $3.5 million to $2 million annually. At that point, it's a question of whether or not you want the opportunity cost of having your tech staff engaged in deploying/operating/maintaining server infrastructure as opposed to building new features/etc)
kordless 11y ago
It's for the same reason MSPs were a thing in the 90s. People simply don't want to screw with machines or the people who run them. Given Spotify's use case isn't that security sensitive, it makes perfect sense to let someone else worry about keeping your machines up. In short, it's insurance.
robbiemitchell 11y ago
Redundancy and spare capacity everywhere in production, staging environment, QA environment, development environments ...
marcoperaza 11y ago
You might be underestimating just how valuable it is to make things somebody else's problem. More employees means more real estate, regulations, taxes, recruiters, HR drones, lawyers, personnel issues, bureaucratic decay, and all of the other challenges of running a company. And if Spotify wants to change strategy tomorrow, they can just end their contract with Google. Winding down internal operations is way more complicated.

by eloff 11y ago

lobster_johnson 11y ago

Virtualizing your server resources is essentially outsourcing a whole lot of sysadmin work to a vendor. Dedicated servers create a different kind of lock-in: Dynamically scaling up dedicated servers is like laying down the tracks while the train is moving. Acceptable to some, but not to a company like Spotify.

eloff 11y ago
You outsource some of the sysadmin work, but not all of it. Cloud servers have plenty of their own failure cases, some of them unique. Physical hardware is actually quite reliable and if you follow the same practices you should in the cloud - namely automatic failover, eliminate single points of failure, etc then the differences decrease. Dynamically scaling load is only really interesting if the load is extremely variable. You could just buy 5x the dedicated hardware, which will likely be able to serve 10x the traffic and may still end up paying less. Who cares if it sits idle most of the time. Now Spotify, like Netflix probably has extreme variance in load, much more so than a normal company. So maybe the cloud works for them. But that's not a common situation at all.
- _delirium 11y ago
  It's definitely common for IT staffing costs to increase rather than decrease with a cloud transition, at least so far. Part of the problem is that you often are trading off sysadmins (relatively cheap) for engineers (relatively expensive), as you reengineer your system to run on top of ephemeral, distributed cloud servers, with new failure cases and new technology stacks. This is often justified on the basis that a large chunk is one-time engineering cost that will pay for itself in the long run. But that's a speculative bet, especially considering how much technology churn is happening so far (e.g. a bunch of companies that invested heavily in early 2010s cloud stacks are now re-investing in Docker). It's possible things will stabilize enough for such companies to eventually lay off many of their distributed-systems engineers and devops staff, and go more into cloud-autopilot mode with low IT headcount and lower staffing cost than pre-cloud-transition. But I don't think many companies are there yet, and I wouldn't bet a lot of money on it myself.
  - jacques_chester 11y ago
    I work for a company (Pivotal) where a major revenue source is selling a packaged version of Cloud Foundry. Our customers like it because their existing ops staff can greatly reduce their workload and allow development staff to move much faster.
- coldtea 11y ago
  >You outsource some of the sysadmin work, but not all of it. Cloud servers have plenty of their own failure cases, some of them unique. The thing with them is if they go down you have Google to worry just as much as you (if not more) for bringing them back up. You don't have that with your own servers.
  - toyg 11y ago
    Google and their famous customer support, yes. Worried they might be, but more than you? For them, the cloud is one of many revenue streams, and nowhere near their most profitable. For you-cloud-customer, a website outage might mean you don't have a revenue stream anymore. It could make the difference between making payroll and shutting down next week. I bet you'll be much more worried than Google will ever be. Googlers might be more skilled than your ops team, yeah, but I would never bet on huge companies to worry about the fate of something that is not their main revenue generator.

kpil 11y ago

You think like an engineer :-) I think cloud services are appealing for a couple of reasons from a top-brass perspective. a) You free the capital, and the balance sheet looks nicer. b) It's less complicated on a PowerPoint level. Just a box. (probably not the reason Spotify opted for google though.) c) You don't think it's your core competence to keep servers safe and running, and rather would like to spend the time and effort on developing services while enjoying reason a) although you know it won't be cheaper. d) Maybe it's hard to find the right people

Pyxl101 11y ago
I don't think he thinks like an engineer (not to insult him), but I think he's overlooking the challenges that exist to running large-scale systems. > Add a second for failover. That's not how it works. How does the second server pick up where the first server left off, when the requests are stateful? How does it know when to take over? (What detects that and how?) Or if it's a human, how does that human make the judgment call? Cloud services are absolutely appealing from an engineering perspective. I am capable of building an application and running it on a fleet of machines such that the infrastructure takes care of itself completely and 100% by itself. Essentially no failure mode can involve me, unless either (A) my application has a defect in its logic, in which case it's unavoidable under any solution (B) probably a lot of customers are similarly affected, and someone is working on it. The data stores that I use are hosted and will not be impacted by any typical failure mode. The application that I've designed is deployed onto machines automatically, and if any of them encounter a failure that takes them offline, they'll be replaced automatically in a way that's a non-event. Furthermore, I have a blueprint for such systems such that building a new application in this way requires no more effort than filling out a single web form, after which an automation layer sets everything up, from the source control to the continuous build and deployment onwards. Not all systems are simple enough to fit into this model, but if you account for it while designing new software, quite a lot of it can be. With the right building blocks you can operate very operations-free infrastructure at the application layer even at large scale. These problems might start to look different once you consider challenges that require tens, hundreds, thousands, or more individual machines to solve. Another challenge is ensuring that new systems are built in a way from the beginning that can scale and that will be easy for other people to maintain. You don't want some random new-hire at your company setting up the "opaque box from hell" that has all the data on its disk (and, ahem, a "second for failover", that through a mysterious and unknown mechanism somehow takes over for the first) and that no one else will be able to maintain during failure. You also don't want to be surprised a year or two down the line when you're scaling up for the Super Bowl or Tax Day or Black Friday, or whatever, and your software falls over because "Bob the Intern, who thought every software system can be solved with a single box and 100 lines of assembly code" didn't anticipate that the business would be larger one day. If you standardize onto a certain set of building blocks, you can build software that, in addition to being operations-free, can scale to a large degree. Note: I'm speaking hyperbolically. No software is truly operations-free. It's all a matter of degree. I have worked however with teams that are expected to hand-manage their own machines, and teams that use effective cloud automation, and the pattern I've seen is a huge difference in their operational load from routine engineering tasks. I have also never seen systems that completely 100% automatically scale themselves, but there is still a huge difference between systems where you simply say "I want 10 machines" / "I want 10 requests per second to my data store", and a system like, "I guess it's time to tell our DBA to figure out how to shard our MySQL server". I also think that a large company could achieve a lot of the benefits provided by the cloud in-house, with its own data centers, if it had really strong internal automation and APIs, and strong teams who operate shared services for other teams (e.g., managed data layers). The really strong software I'm aware of for solving these problems has been kept as a competitive advantage, not released open source. So the practical reality is that few companies have through-and-through automation of the quality and degree that you can get from top cloud providers. > a) You free the capital, and the balance sheet looks nicer. Yes, I agree that this is a genuine reason for some companies to prefer the cloud, but these are largely not companies who have "top brass". Companies that are large enough to be "top brass" are typically not so capital-constrained as that this would majorly affect their purchasing decisions. The companies that care about this, rather, tend to be smaller and more capital-constrained. I think the more mundane reason why companies prefer the cloud, which is frequently overlooked, is that individual business units can purchase cloud services with more autonomy than they could traditionally order IT services in-house. Cloud budgets often fit within what an executive can expense within their own authority on a month-to-month basis, where as the same capital purchase would require more review and approval. So some companies tend toward the cloud because it makes them more agile by providing decisions they can make that are smaller and more easily fit within the scope of less-senior people.

dogma1138 11y ago

ISP cost, backups, patching, hardware upgrades, co-location, DR, insurance, paying for physical and cyber-security, personnel etc. outsourcing allot of those things saves money it's also a different expense of you buy infrastructure it has value which means depreciation overtime which affects how your accounting is done. Cloud makes sense for allot of companies not just startups the biggest reason why companies with eatablished infrastructure don't migrate is the initial cost and regulation.

sharpy 11y ago
+1 Not to mention that often it can takes weeks for new hardware order to be approved / delivered / provisioned. Cloud? Just dial it up... Well, if you are at Netflix scale, you probably couldn't just dial it up, and would have to work with the vendor to ensure adequate capacity was available for your use.
- taf2 11y ago
  And keep in mind that adding those new servers even in managed eviromnents can fail and fail in unexpected ways like taking out your whole network. We had this happen with rackspace - hence we moved to Amazon
  - hhw 11y ago
    If a single point of failure took down your entire network, then your solution wasn't engineered properly. Just because on provider messed up big time doesn't mean every other provider would. The hardwdare design/architecture can also be provided to you for free as part of the sales process, no extra service required. As for hardware failures in general, you can afford to have some complete extra servers on hand doing nothing most of the time with the amount of money you'd save on renting bare metal instead of paying for cloud. Or if engineered properly, you'd be using all the machines at once but be able to afford to lose half of them, and failover automatically. Yes, a cloud (bunch of VMs on a cluster) is going to be more reliable than a single machine, as you get hardware redundancy. But a cluster of machines with hardware redundancy is going to be much more reliable than a bunch of VMs on a cluster, as there's less complexity involved without the cloud stack, and thus fewer points of failure.

kenshaw 11y ago

On a balance sheet, you are trading CapEx for OpEx. That's it. For certain organizations with specific types of computing loads, that may make the most sense. (btw -- I disagree with your math there on the actual cost -- I agree it could ultimately be cheaper to run things in house, there's no way it's going to go from $3.5 million to $20k annually. It'd be more like $3.5 million to $2 million annually. At that point, it's a question of whether or not you want the opportunity cost of having your tech staff engaged in deploying/operating/maintaining server infrastructure as opposed to building new features/etc)

kordless 11y ago

It's for the same reason MSPs were a thing in the 90s. People simply don't want to screw with machines or the people who run them. Given Spotify's use case isn't that security sensitive, it makes perfect sense to let someone else worry about keeping your machines up. In short, it's insurance.

robbiemitchell 11y ago

Redundancy and spare capacity everywhere in production, staging environment, QA environment, development environments ...

marcoperaza 11y ago

You might be underestimating just how valuable it is to make things somebody else's problem. More employees means more real estate, regulations, taxes, recruiters, HR drones, lawyers, personnel issues, bureaucratic decay, and all of the other challenges of running a company. And if Spotify wants to change strategy tomorrow, they can just end their contract with Google. Winding down internal operations is way more complicated.