8 ms·

Maybe. Running your own infrastructure at large scale is actually really hard and therefore surprisingly expensive in risk and talent cost. If you look at how b

Maybe. Running your own infrastructure at large scale is actually really hard and therefore surprisingly expensive in risk and talent cost. If you look at how big the internal infra teams are at companies that host their own infrastructure, they're often individually the size of growth stage companies. The problem is that provisioning, reliability, and security are by themselves really tough problems. If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. I look at it as the question: can you get the same set of agility/reliability/security guarantees for your narrower set of use cases by paying for your own hardware and engineering? I won't even begin to pretend I have any answers there, but I think that's the calculus.

doctorpangloss 8y ago
> If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. Maybe that's just the story cloud providers tell you. Until you try, do you really know if it's all that complicated? People have been running datacenters for a long time, and not all of them work for Amazon. But there may be also a beneficial side effect of having gearheads around, and maybe that's the real cost to going cloud.
- munchbunny 8y ago
  I work for one of said teams, doing datacenter cyber-security. You could say "trying" is my day job. Since we're internal and we manage a lot of capacity, we do often provision and roll our own equivalents of things that cloud providers will sell you, rather than just buying a cloud solution. It's often ambiguous whether it was a good use of time/money. If it weren't for the economies of scale that kick in at the sheer size of this operation, it would definitely not be worth it.
- theossuary 8y ago
  Yes, I've worked with a few of those datacenters. A few examples: - Recently had to purchase new servers, because of signed contracts the only servers we were allowed to purchase and put in the datacenter were four years old and technically EOF. - Firewall changes, AD changes, provisioning a VM, etc. are 48 hour turnaround. Purchasing new hardware requires 4-6 weeks. - Had an intermittent issue with their edge firewall, it'd slow certain connections to a crawl and eventually they'd timeout. Took six months to fix it, for the first three months they told us it wasn't their fault (turning off their deep packet inspection ended up fixing it). I still remember when we opened the first ticket about it, and the reply was "no other customers are experiencing problems" and it was closed. That's just a few examples of how painful it can be. To give you the other side of the coin, having worked with an enterprise contract in AWS, we were having an intermittent issue with DNS resolving failing for a few seconds every few days. They put an engineer on it full time till they found the problem (we misconfigured it), and it didn't cost us anything more than the enterprise support. I was actually shocked they'd invest that much on such a vague issue. Yes AWS is expensive, but you're getting world class engineering proven at scale, and access to some very smart/motivated people to support it (and they have access to the teams who built it, when they can't solve it). I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Either do it in-house where there's accountability, or use cloud providers who have proven they're competency. To be clear, I'm talking about VPC/EC2/etc. I can't really discuss a lot of their higher level and newer managed services; they either weren't as good, or I haven't tried them. But the bedrock these clouds are built on is solid, and that's worth paying good money for.
  - vonmoltke 8y ago
    > Yes, I've worked with a few of those datacenters. > I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Who mentioned managed datacenters? I'm pretty sure people are talking about leasing space and doing everything else in-house.
    - eeeeeeeeeeeee 8y ago
      I've done that too. It's obviously easier than running an entire datacenter, but you still need to manage all the underlying services that you deliver to your development team. For example, running things like this on your own: - storage clusters - database clusters - compute clusters They are often very easy to setup, but when things go wrong, they go very wrong. And welcome to a stressful environment because if you can't figure it out and your people can't, well, your business just sits and burns while you do. Even when AWS has a system-wide outage, it's nice to know that I don't have to be dealing with those underlying problems anymore and I know they have the best people working on them. I cannot put into words, after operating MySQL clusters on my own and playing back transactions after failures, how nice it is to use AWS RDS and how it's just been zero problems. Zero. I sleep through automatic updates of our database system with RDS. I would have never done that on our own system. And in most places, even "managed" leased hardware, you still will need to purchase/lease and run your own hardware firewalls and ddos mitigation. The datacenter might offer that protection "built-in" but you'll soon find the limitations of that offering when you face a substantial attack.
  - llama052 8y ago
    This honestly sounds like a company specific problem more than a datacenter issue.
    - spookthesunset 8y ago
      No. That is my experience too. If your company’s core value prop isn’t running and managing a datacenter, your datacenter is gonna suck when compared to folks who do it for a living.
  - 8y ago
- halbritt 8y ago
  Having done a bunch of bare metal, I can tell you the calculus isn't really that hard. Bare metal will save you money. Operating bare metal at scale requires talent that doesn't exist, not necessarily at an engineering level, but at all levels. As an example, I worked at a place that had a large bare metal deployment, i.e. >1MW worth of compute. It was woefully inefficient and costly to operate. The product that they offered required network QOS and compute with real time capabilities, neither of which was available from any cloud provider at the time. One of our executives (formerly a leader in the DC ops org at AWS) left the company to be replaced by another executive by another well-known silicon valley org who then insisted we should migrate everything to the cloud. I showed him the relatively easy math that efficiently utilized bare metal was way less costly and that the aforementioned QOS and RT requirements would be a deal breaker anyway. He failed to fully grok this and remained insistent. When I quite, he seemed surprised. After the fact, I discovered that they'd made a deal with IBM to move everything into their cloud. A year later it was an utter failure and they abandoned the project. There are lots of folks in the valley with lots of experience on their resumes that suggests that they should be capable of understanding these kinds of things that simply don't. Lacking that understanding leads to poor decision-making, which leads to failure, which leads to risk-aversion, which leads to everyone believing that it must be cheaper in the cloud. Or so goes the old adage, "nobody ever got fired for buying IBM."
  - throwawaymath 8y ago
    Feel free to ignore this, but would you shoot me an email? I'm working on a project that uses bare metal and would like to pick your brain. EDIT: To whoever downvoted this, the commenter hasn't listed an email address, or I would have reached out directly. This is an honest attempt at communication that doesn't require someone to break anonymity.
    - asdfafd2sdf 8y ago
      You were probably downvoted for using the term 'pick your brain'. I am sure they want their brain 'picked'.
- woolvalley 8y ago
  I've worked at 2 companies that decided to do their 'own cloud' vs just use stuff like AWS, and they were not devops tool companies. It slowed down both them. Remember that staff cost money too!
- privateSFacct 8y ago
  I'm actually curious how AWS is able to scale support so well with what seems reasonable quality. I had an issue (my fault in end), got excellent support - and they didn't tell us to take a hike at end when it turned out not AWS fault. Conversely, with GCP 4 years ago now had some support issues - didn't come away impressed - I'm convinced even internally GCP isn't well doc'd or something.
  - munchbunny 8y ago
    I think it comes down to Amazon vs. Google's company cultures around customer support, which informs attention and resource budgeting. GCP isn't the only Google product where people hate its customer support, and AWS isn't the only Amazon product where people love its customer support.
    - bm1362 8y ago
      I think Amazon is the only company where values actually created a culture.
    - privateSFacct 8y ago
      Amazon takes the customer support very far - with one exception - bogus products on marketplace seems like a big miss. But what I paid for and got on aws support is so far out of whack there is NO way they made money on my account for that whole year. And the person was actually competant which was a shock. So many "technical support" folks seem like idiots. Comcast for example, I'd purchased my modem, they started charging a rental fee - I had to call these bozos every month to reverse the charge - a total waste of time. I cancelled finally - I just couldn't take it, and each one lied to me or didn't have a clue. Things like condescendingly saying - you have to pay for the modem.
- onlyrealcuzzo 8y ago
  I think the point is, Lyft doesn't want to be in the business of running enterprise could infrastructure. They want to make money brokering rides. Taking on their own cloud infrastructure -- in theory -- could economically make sense. But that's just an extra layer of risk and complexity they'd rather forego to focus on their core business. After all, their core business is already losing $930M on $2B in revenue. They're cash-flow doesn't put them in a good position to make large up-front investments on data centers. So, yeah, like a broke renter in an expensive city. In theory, it might be better to buy a house, but you don't have the down payment, and maybe you should be focused on increasing your earning power rather than saving money anyway...
  - sjg007 8y ago
    Great analogy.
kerng 8y ago
Not really, at that scale it eventually will be a lot cheaper running your own thing - Lyft might still be too small, but if they go internationally and grow ×10 then AWS seems like a choice to reconsider.

doctorpangloss 8y ago

> If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. Maybe that's just the story cloud providers tell you. Until you try, do you really know if it's all that complicated? People have been running datacenters for a long time, and not all of them work for Amazon. But there may be also a beneficial side effect of having gearheads around, and maybe that's the real cost to going cloud.

munchbunny 8y ago
I work for one of said teams, doing datacenter cyber-security. You could say "trying" is my day job. Since we're internal and we manage a lot of capacity, we do often provision and roll our own equivalents of things that cloud providers will sell you, rather than just buying a cloud solution. It's often ambiguous whether it was a good use of time/money. If it weren't for the economies of scale that kick in at the sheer size of this operation, it would definitely not be worth it.
theossuary 8y ago
Yes, I've worked with a few of those datacenters. A few examples: - Recently had to purchase new servers, because of signed contracts the only servers we were allowed to purchase and put in the datacenter were four years old and technically EOF. - Firewall changes, AD changes, provisioning a VM, etc. are 48 hour turnaround. Purchasing new hardware requires 4-6 weeks. - Had an intermittent issue with their edge firewall, it'd slow certain connections to a crawl and eventually they'd timeout. Took six months to fix it, for the first three months they told us it wasn't their fault (turning off their deep packet inspection ended up fixing it). I still remember when we opened the first ticket about it, and the reply was "no other customers are experiencing problems" and it was closed. That's just a few examples of how painful it can be. To give you the other side of the coin, having worked with an enterprise contract in AWS, we were having an intermittent issue with DNS resolving failing for a few seconds every few days. They put an engineer on it full time till they found the problem (we misconfigured it), and it didn't cost us anything more than the enterprise support. I was actually shocked they'd invest that much on such a vague issue. Yes AWS is expensive, but you're getting world class engineering proven at scale, and access to some very smart/motivated people to support it (and they have access to the teams who built it, when they can't solve it). I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Either do it in-house where there's accountability, or use cloud providers who have proven they're competency. To be clear, I'm talking about VPC/EC2/etc. I can't really discuss a lot of their higher level and newer managed services; they either weren't as good, or I haven't tried them. But the bedrock these clouds are built on is solid, and that's worth paying good money for.
- vonmoltke 8y ago
  > Yes, I've worked with a few of those datacenters. > I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Who mentioned managed datacenters? I'm pretty sure people are talking about leasing space and doing everything else in-house.
  - eeeeeeeeeeeee 8y ago
    I've done that too. It's obviously easier than running an entire datacenter, but you still need to manage all the underlying services that you deliver to your development team. For example, running things like this on your own: - storage clusters - database clusters - compute clusters They are often very easy to setup, but when things go wrong, they go very wrong. And welcome to a stressful environment because if you can't figure it out and your people can't, well, your business just sits and burns while you do. Even when AWS has a system-wide outage, it's nice to know that I don't have to be dealing with those underlying problems anymore and I know they have the best people working on them. I cannot put into words, after operating MySQL clusters on my own and playing back transactions after failures, how nice it is to use AWS RDS and how it's just been zero problems. Zero. I sleep through automatic updates of our database system with RDS. I would have never done that on our own system. And in most places, even "managed" leased hardware, you still will need to purchase/lease and run your own hardware firewalls and ddos mitigation. The datacenter might offer that protection "built-in" but you'll soon find the limitations of that offering when you face a substantial attack.
- llama052 8y ago
  This honestly sounds like a company specific problem more than a datacenter issue.
  - spookthesunset 8y ago
    No. That is my experience too. If your company’s core value prop isn’t running and managing a datacenter, your datacenter is gonna suck when compared to folks who do it for a living.
- 8y ago
halbritt 8y ago
Having done a bunch of bare metal, I can tell you the calculus isn't really that hard. Bare metal will save you money. Operating bare metal at scale requires talent that doesn't exist, not necessarily at an engineering level, but at all levels. As an example, I worked at a place that had a large bare metal deployment, i.e. >1MW worth of compute. It was woefully inefficient and costly to operate. The product that they offered required network QOS and compute with real time capabilities, neither of which was available from any cloud provider at the time. One of our executives (formerly a leader in the DC ops org at AWS) left the company to be replaced by another executive by another well-known silicon valley org who then insisted we should migrate everything to the cloud. I showed him the relatively easy math that efficiently utilized bare metal was way less costly and that the aforementioned QOS and RT requirements would be a deal breaker anyway. He failed to fully grok this and remained insistent. When I quite, he seemed surprised. After the fact, I discovered that they'd made a deal with IBM to move everything into their cloud. A year later it was an utter failure and they abandoned the project. There are lots of folks in the valley with lots of experience on their resumes that suggests that they should be capable of understanding these kinds of things that simply don't. Lacking that understanding leads to poor decision-making, which leads to failure, which leads to risk-aversion, which leads to everyone believing that it must be cheaper in the cloud. Or so goes the old adage, "nobody ever got fired for buying IBM."
- throwawaymath 8y ago
  Feel free to ignore this, but would you shoot me an email? I'm working on a project that uses bare metal and would like to pick your brain. EDIT: To whoever downvoted this, the commenter hasn't listed an email address, or I would have reached out directly. This is an honest attempt at communication that doesn't require someone to break anonymity.
  - asdfafd2sdf 8y ago
    You were probably downvoted for using the term 'pick your brain'. I am sure they want their brain 'picked'.
woolvalley 8y ago
I've worked at 2 companies that decided to do their 'own cloud' vs just use stuff like AWS, and they were not devops tool companies. It slowed down both them. Remember that staff cost money too!
privateSFacct 8y ago
I'm actually curious how AWS is able to scale support so well with what seems reasonable quality. I had an issue (my fault in end), got excellent support - and they didn't tell us to take a hike at end when it turned out not AWS fault. Conversely, with GCP 4 years ago now had some support issues - didn't come away impressed - I'm convinced even internally GCP isn't well doc'd or something.
- munchbunny 8y ago
  I think it comes down to Amazon vs. Google's company cultures around customer support, which informs attention and resource budgeting. GCP isn't the only Google product where people hate its customer support, and AWS isn't the only Amazon product where people love its customer support.
  - bm1362 8y ago
    I think Amazon is the only company where values actually created a culture.
  - privateSFacct 8y ago
    Amazon takes the customer support very far - with one exception - bogus products on marketplace seems like a big miss. But what I paid for and got on aws support is so far out of whack there is NO way they made money on my account for that whole year. And the person was actually competant which was a shock. So many "technical support" folks seem like idiots. Comcast for example, I'd purchased my modem, they started charging a rental fee - I had to call these bozos every month to reverse the charge - a total waste of time. I cancelled finally - I just couldn't take it, and each one lied to me or didn't have a clue. Things like condescendingly saying - you have to pay for the modem.
onlyrealcuzzo 8y ago
I think the point is, Lyft doesn't want to be in the business of running enterprise could infrastructure. They want to make money brokering rides. Taking on their own cloud infrastructure -- in theory -- could economically make sense. But that's just an extra layer of risk and complexity they'd rather forego to focus on their core business. After all, their core business is already losing $930M on $2B in revenue. They're cash-flow doesn't put them in a good position to make large up-front investments on data centers. So, yeah, like a broke renter in an expensive city. In theory, it might be better to buy a house, but you don't have the down payment, and maybe you should be focused on increasing your earning power rather than saving money anyway...
- sjg007 8y ago
  Great analogy.

kerng 8y ago

Not really, at that scale it eventually will be a lot cheaper running your own thing - Lyft might still be too small, but if they go internationally and grow ×10 then AWS seems like a choice to reconsider.