15 ms·

At this point isn't it cost-effective for Lyft to just build its own infrastructure?

godzillabrennus 8y ago
Very much so. The cost benefit analysis must also question how much it would cost to educate the teams on the new hosting tools. Management has clearly decided that it’s not worth the retraining and loss of productivity.
- julianozen 8y ago
  Will add if lyft is using things like dynamodb, there is no good onprem alternatives that they can easily migrate to without significant rewrites of core business logic This is expensive and risky and also difficult to do in piece meal Disclaimer: former AWS + Amazon employee
the_watcher 8y ago
Possibly, but if they don't have any of their own infra, they'd need to pay someone like AWS while they build it out, right?
freewilly1040 8y ago
With cloud vs DC you also needs to consider that re-sizing datacenter space is a very slow process, and if you're wrong about how much you need it's a massive pain. Buy too much and you're wasting money, buy too little and you're stuck throwing eng time at scrambling to keep your services from falling over, and bottlenecking your entire org with resource constraints. That's not a judgement on whether it's worth it for Lyft or not, but especially for a growing company with spiky load the decision is not just a dollars to dollars comparison.
- peteradio 8y ago
  But let's say you have a 300m budget.
  - eropple 8y ago
    You don't, though. You have room for $300m of opex, coming in over time and allocated, as mentioned, around $8m a month. Less early on, more later. If you gave me $300m to spend, largely up-front, for significant capex purchases? Sure. We could do it. The team I would build would also probably still make mistakes that AWS et al have already largely learned how to avoid, but we could do it. But capex and opex are very different beasts. By the end of that three years I'm already looking at spending way more to refresh what I bought at the start of that three year period because I'm starting to near the end of early contracts and I'm figuring out how best to wrangle, in a way that makes the rest of the business succeed most optimally, a now-heterogeneous environment, etcetera etcetera and etcetera. It's all solvable. But whether it's cheaper, at scale, and more reliable, and presents a unified tool for use by the business...that's a harder question. Understanding how capex and opex work and how they differ is pretty critical to successfully running an engineering organization, to say nothing of a company.
    - halbritt 8y ago
      If you've built a heterogenous environment at the end of three years, then you've failed. The reason that AWS, Google, Azure, et.al do so well is that they don't just buy some servers. They do actual capacity maangement, and not a very good job of it I might add. They also manage the lifecycle of every component in the infrastructure such that the next iteration of that component is understood and interchangeable. Network architecture, for example, should suit the needs of the application, but should also be decoupled from the underlying hardware as that hardware is going to evolve. Compute is fairly straightforward as well. At the data center level, one makes a bunch of 400W holes. What you fill those 400W holes with is relatively irrelevant.
      - eropple 8y ago
        Fully agreed on all points. But it remains a really hard problem. And when you start to do it out at the scale of something like Lyft, you're gonna blow through your available parts of that $300m (because you can't spend it all up front, obviously) pretty quick. The care and feeding of fleets of (physical) machines is really, really hard and not to be underestimated.
munchbunny 8y ago
Maybe. Running your own infrastructure at large scale is actually really hard and therefore surprisingly expensive in risk and talent cost. If you look at how big the internal infra teams are at companies that host their own infrastructure, they're often individually the size of growth stage companies. The problem is that provisioning, reliability, and security are by themselves really tough problems. If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. I look at it as the question: can you get the same set of agility/reliability/security guarantees for your narrower set of use cases by paying for your own hardware and engineering? I won't even begin to pretend I have any answers there, but I think that's the calculus.
- doctorpangloss 8y ago
  > If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. Maybe that's just the story cloud providers tell you. Until you try, do you really know if it's all that complicated? People have been running datacenters for a long time, and not all of them work for Amazon. But there may be also a beneficial side effect of having gearheads around, and maybe that's the real cost to going cloud.
  - munchbunny 8y ago
    I work for one of said teams, doing datacenter cyber-security. You could say "trying" is my day job. Since we're internal and we manage a lot of capacity, we do often provision and roll our own equivalents of things that cloud providers will sell you, rather than just buying a cloud solution. It's often ambiguous whether it was a good use of time/money. If it weren't for the economies of scale that kick in at the sheer size of this operation, it would definitely not be worth it.
  - theossuary 8y ago
    Yes, I've worked with a few of those datacenters. A few examples: - Recently had to purchase new servers, because of signed contracts the only servers we were allowed to purchase and put in the datacenter were four years old and technically EOF. - Firewall changes, AD changes, provisioning a VM, etc. are 48 hour turnaround. Purchasing new hardware requires 4-6 weeks. - Had an intermittent issue with their edge firewall, it'd slow certain connections to a crawl and eventually they'd timeout. Took six months to fix it, for the first three months they told us it wasn't their fault (turning off their deep packet inspection ended up fixing it). I still remember when we opened the first ticket about it, and the reply was "no other customers are experiencing problems" and it was closed. That's just a few examples of how painful it can be. To give you the other side of the coin, having worked with an enterprise contract in AWS, we were having an intermittent issue with DNS resolving failing for a few seconds every few days. They put an engineer on it full time till they found the problem (we misconfigured it), and it didn't cost us anything more than the enterprise support. I was actually shocked they'd invest that much on such a vague issue. Yes AWS is expensive, but you're getting world class engineering proven at scale, and access to some very smart/motivated people to support it (and they have access to the teams who built it, when they can't solve it). I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Either do it in-house where there's accountability, or use cloud providers who have proven they're competency. To be clear, I'm talking about VPC/EC2/etc. I can't really discuss a lot of their higher level and newer managed services; they either weren't as good, or I haven't tried them. But the bedrock these clouds are built on is solid, and that's worth paying good money for.
djakjxnanjak 8y ago
They have many teams making their own web services, so replacing AWS with in-house infra would require an AWS-like “private cloud” feature set, backed by geographically distributed datacenters. Network segmentation that is managed by configuration, user accounts with permissions to do specific things to specific services, etc. Engineers to build all this stuff are not cheap, and management to build and migrate to it cleanly it is not easy. If you can do this you ought to be selling cloud services. Even if it makes sense for Lyft to implement this, it would take a while. The RIO is probably much better on improving their core product, where there is more than $100M/year at stake.
- metildaa 8y ago
  Sounds like OpenStack...
  - toomuchtodo 8y ago
    Works for Wikipedia, and they’re in the top 5 of Internet websites.
    - djakjxnanjak 8y ago
      I suspect nonprofits like the Wikimedia Foundation have much greater constraints on billables but perhaps relatively less constraints on engineering and management effort. That probably changes the ROI calculation quite a bit. And management of such organizations might also have ideological tendencies that further skew the calculation.
bduerst 8y ago
No. The capex and overhead for rolling and maintaining your own infrastructure is outstanding. The only time it typically makes sense these days is when it's done in quantity and also sold to others, which is where the economies of scale kick in.
zjaffee 8y ago
It's believed that it would be cheaper for Twitter to have used cloud services (Snapchat spends less then they do on data center operations). There are certainly examples for big companies that benefit from having their own infrastructure (i.e. Dropbox since they have relatively specialized hardware needs compared to what cloud providers set prices around), but the number of people you need to hire to build and maintain datacenters is very high.
- netik 8y ago
  Twitter initially ran everything in the cloud. I should know, I was employee 13. It was much cheaper to build our own server hardware and move to our own server and network agreements. Peering and transit is cheaper at scale, as is commodity hardware.
  - rrdharan 8y ago
    Those numbers have changed since you left. And they could change again. It's all dynamic: these comparisons and the economically and strategically "correct" choices change over time depending on talent availability, demand, speed of scaling requirements and so on. See https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/a-new-collaboration-with-google-cloud.html https://blog.twitter.com/engineering/en_us/topics/infrastruc....
fierro 8y ago
if I have one job in this life, it's to hang out on hacker news and repeatedly post about how it's not cost effective to run your own infrastructure. 8MM/month doesn't even come CLOSE to needing your own infra. - person who knows how hard it is to run your own infrastructure
- halbritt 8y ago
  So, convince me? - person who knows how hard it is to run your own infrastructure.
  - fierro 8y ago
    ah I regret this comment lol
    - halbritt 8y ago
      No worries. Generally speaking, you're right.
- Alupis 8y ago
  > person who knows how hard it is to run your own infrastructure > fierro Profile: SWE @ Google Resource & Capacity Planning I think you mean "Person who's job it is to convince others it's really hard and they should just buy your product"...?
  - fierro 8y ago
    ha sure. I just meant I know the sort of operational burden cloud customers are able to outsource by not rolling their own, based on my time at the G inside resource planning and management.
    - tjoff 8y ago
      You know anything of the burden to be reliant on someone else for your core operations? And having to invest and adapt so much just to lock yourself to one vendor? For many that alone is more than the operational burden of doing it yourself.
KerrickStaley 8y ago
Lyft's load varies wildly, with significantly higher traffic on Friday and Saturday nights than e.g. 4 AM on a Tuesday, plus spikes on certain evenings like Halloween and New Year's Eve. Having cloud hosting where we can dynamically grow and shrink based on load saves us a lot compared to having fixed infrastructure that is always provisioned for the NYE peak. Source: I work at Lyft.
- qxi 8y ago
  Why not employ an hybrid architecture of bare metal for base load augmented by cloud-based infrastructure for peaks, constructed via a polyglot union of taped-together tools and lubricated by the daily tears of a hundred college hires only to regret it after the engineers who designed it have successfully used it as a springboard for promotion and departed with their accumulated arcane knowledge (and vested shares) for greener pastures? Worked for us at Twitch.
  - diminoten 8y ago
    Oh come on, you know the performance profile of a streaming service is wildly different from a mobile ridesharing app... Their surges are nothing like yours.
    - deleted 8y ago
      [deleted]
    - Alupis 8y ago
      It still stands... $8mm monthly is over 350 full time engineers making $250k annually (ya that doesn't cover hardware, but it just illustrates the point). That would be a 22% increase in Lyft's total number of employees. If you have engineers making less than $250k annually, then you have a lot more staff. $8mm monthly is a LOT... Amazon's clearly making a profit after $8mm monthly.
      - glass_of_water 8y ago
        But it's not like Lyft would suddenly have 350 full time engineers developing new features. A large chunk of those engineers would be working on building and maintaining infrastructure that AWS provides.
mcqueenjordan 8y ago
Is there a strategic advantage for a shoemaker to make their own hammer?
- petronio 8y ago
  There's a couple. Some samples: * If the hammer manufacturer decides not to sell you any, you'll still have hammers. * If the hammer manufacturer gains enough power to fix prices, you won't be paying them exorbitant prices. * If the hammer manufacturer or their country gets embargoed and you're unable to legally purchase their hammers, you'll still have hammers. All the above grant you a strategic advantage since you'll still have the necessary tools to continue your business while your competitors won't (or will have to pay much higher prices for their supply of hammers).
- rdtsc 8y ago
  > Is there a strategic advantage for a shoemaker to make their own hammer? Sure. If they are paying $100M/y on hammers, it's at least worth running the numbers and investigate alternatives.
  - mcqueenjordan 8y ago
    But that's ignoring the opportunity cost. Strategically, you probably want to focus on what your core competencies are, even if you could in theory do something for cheaper. It's easy to ignore the foregone best alternative of iterating on your own product instead.

Very much so. The cost benefit analysis must also question how much it would cost to educate the teams on the new hosting tools. Management has clearly decided that it’s not worth the retraining and loss of productivity.

julianozen 8y ago
Will add if lyft is using things like dynamodb, there is no good onprem alternatives that they can easily migrate to without significant rewrites of core business logic This is expensive and risky and also difficult to do in piece meal Disclaimer: former AWS + Amazon employee

the_watcher 8y ago

Possibly, but if they don't have any of their own infra, they'd need to pay someone like AWS while they build it out, right?

freewilly1040 8y ago

With cloud vs DC you also needs to consider that re-sizing datacenter space is a very slow process, and if you're wrong about how much you need it's a massive pain. Buy too much and you're wasting money, buy too little and you're stuck throwing eng time at scrambling to keep your services from falling over, and bottlenecking your entire org with resource constraints. That's not a judgement on whether it's worth it for Lyft or not, but especially for a growing company with spiky load the decision is not just a dollars to dollars comparison.

peteradio 8y ago
But let's say you have a 300m budget.
- eropple 8y ago
  You don't, though. You have room for $300m of opex, coming in over time and allocated, as mentioned, around $8m a month. Less early on, more later. If you gave me $300m to spend, largely up-front, for significant capex purchases? Sure. We could do it. The team I would build would also probably still make mistakes that AWS et al have already largely learned how to avoid, but we could do it. But capex and opex are very different beasts. By the end of that three years I'm already looking at spending way more to refresh what I bought at the start of that three year period because I'm starting to near the end of early contracts and I'm figuring out how best to wrangle, in a way that makes the rest of the business succeed most optimally, a now-heterogeneous environment, etcetera etcetera and etcetera. It's all solvable. But whether it's cheaper, at scale, and more reliable, and presents a unified tool for use by the business...that's a harder question. Understanding how capex and opex work and how they differ is pretty critical to successfully running an engineering organization, to say nothing of a company.
  - halbritt 8y ago
    If you've built a heterogenous environment at the end of three years, then you've failed. The reason that AWS, Google, Azure, et.al do so well is that they don't just buy some servers. They do actual capacity maangement, and not a very good job of it I might add. They also manage the lifecycle of every component in the infrastructure such that the next iteration of that component is understood and interchangeable. Network architecture, for example, should suit the needs of the application, but should also be decoupled from the underlying hardware as that hardware is going to evolve. Compute is fairly straightforward as well. At the data center level, one makes a bunch of 400W holes. What you fill those 400W holes with is relatively irrelevant.
    - eropple 8y ago
      Fully agreed on all points. But it remains a really hard problem. And when you start to do it out at the scale of something like Lyft, you're gonna blow through your available parts of that $300m (because you can't spend it all up front, obviously) pretty quick. The care and feeding of fleets of (physical) machines is really, really hard and not to be underestimated.

munchbunny 8y ago

Maybe. Running your own infrastructure at large scale is actually really hard and therefore surprisingly expensive in risk and talent cost. If you look at how big the internal infra teams are at companies that host their own infrastructure, they're often individually the size of growth stage companies. The problem is that provisioning, reliability, and security are by themselves really tough problems. If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. I look at it as the question: can you get the same set of agility/reliability/security guarantees for your narrower set of use cases by paying for your own hardware and engineering? I won't even begin to pretend I have any answers there, but I think that's the calculus.

doctorpangloss 8y ago
> If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. Maybe that's just the story cloud providers tell you. Until you try, do you really know if it's all that complicated? People have been running datacenters for a long time, and not all of them work for Amazon. But there may be also a beneficial side effect of having gearheads around, and maybe that's the real cost to going cloud.
- munchbunny 8y ago
  I work for one of said teams, doing datacenter cyber-security. You could say "trying" is my day job. Since we're internal and we manage a lot of capacity, we do often provision and roll our own equivalents of things that cloud providers will sell you, rather than just buying a cloud solution. It's often ambiguous whether it was a good use of time/money. If it weren't for the economies of scale that kick in at the sheer size of this operation, it would definitely not be worth it.
- theossuary 8y ago
  Yes, I've worked with a few of those datacenters. A few examples: - Recently had to purchase new servers, because of signed contracts the only servers we were allowed to purchase and put in the datacenter were four years old and technically EOF. - Firewall changes, AD changes, provisioning a VM, etc. are 48 hour turnaround. Purchasing new hardware requires 4-6 weeks. - Had an intermittent issue with their edge firewall, it'd slow certain connections to a crawl and eventually they'd timeout. Took six months to fix it, for the first three months they told us it wasn't their fault (turning off their deep packet inspection ended up fixing it). I still remember when we opened the first ticket about it, and the reply was "no other customers are experiencing problems" and it was closed. That's just a few examples of how painful it can be. To give you the other side of the coin, having worked with an enterprise contract in AWS, we were having an intermittent issue with DNS resolving failing for a few seconds every few days. They put an engineer on it full time till they found the problem (we misconfigured it), and it didn't cost us anything more than the enterprise support. I was actually shocked they'd invest that much on such a vague issue. Yes AWS is expensive, but you're getting world class engineering proven at scale, and access to some very smart/motivated people to support it (and they have access to the teams who built it, when they can't solve it). I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Either do it in-house where there's accountability, or use cloud providers who have proven they're competency. To be clear, I'm talking about VPC/EC2/etc. I can't really discuss a lot of their higher level and newer managed services; they either weren't as good, or I haven't tried them. But the bedrock these clouds are built on is solid, and that's worth paying good money for.

djakjxnanjak 8y ago

They have many teams making their own web services, so replacing AWS with in-house infra would require an AWS-like “private cloud” feature set, backed by geographically distributed datacenters. Network segmentation that is managed by configuration, user accounts with permissions to do specific things to specific services, etc. Engineers to build all this stuff are not cheap, and management to build and migrate to it cleanly it is not easy. If you can do this you ought to be selling cloud services. Even if it makes sense for Lyft to implement this, it would take a while. The RIO is probably much better on improving their core product, where there is more than $100M/year at stake.

metildaa 8y ago
Sounds like OpenStack...
- toomuchtodo 8y ago
  Works for Wikipedia, and they’re in the top 5 of Internet websites.
  - djakjxnanjak 8y ago
    I suspect nonprofits like the Wikimedia Foundation have much greater constraints on billables but perhaps relatively less constraints on engineering and management effort. That probably changes the ROI calculation quite a bit. And management of such organizations might also have ideological tendencies that further skew the calculation.

bduerst 8y ago

No. The capex and overhead for rolling and maintaining your own infrastructure is outstanding. The only time it typically makes sense these days is when it's done in quantity and also sold to others, which is where the economies of scale kick in.

zjaffee 8y ago

It's believed that it would be cheaper for Twitter to have used cloud services (Snapchat spends less then they do on data center operations). There are certainly examples for big companies that benefit from having their own infrastructure (i.e. Dropbox since they have relatively specialized hardware needs compared to what cloud providers set prices around), but the number of people you need to hire to build and maintain datacenters is very high.

netik 8y ago
Twitter initially ran everything in the cloud. I should know, I was employee 13. It was much cheaper to build our own server hardware and move to our own server and network agreements. Peering and transit is cheaper at scale, as is commodity hardware.
- rrdharan 8y ago
  Those numbers have changed since you left. And they could change again. It's all dynamic: these comparisons and the economically and strategically "correct" choices change over time depending on talent availability, demand, speed of scaling requirements and so on. See https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/a-new-collaboration-with-google-cloud.html https://blog.twitter.com/engineering/en_us/topics/infrastruc....

fierro 8y ago

if I have one job in this life, it's to hang out on hacker news and repeatedly post about how it's not cost effective to run your own infrastructure. 8MM/month doesn't even come CLOSE to needing your own infra. - person who knows how hard it is to run your own infrastructure

halbritt 8y ago
So, convince me? - person who knows how hard it is to run your own infrastructure.
- fierro 8y ago
  ah I regret this comment lol
  - halbritt 8y ago
    No worries. Generally speaking, you're right.
Alupis 8y ago
> person who knows how hard it is to run your own infrastructure > fierro Profile: SWE @ Google Resource & Capacity Planning I think you mean "Person who's job it is to convince others it's really hard and they should just buy your product"...?
- fierro 8y ago
  ha sure. I just meant I know the sort of operational burden cloud customers are able to outsource by not rolling their own, based on my time at the G inside resource planning and management.
  - tjoff 8y ago
    You know anything of the burden to be reliant on someone else for your core operations? And having to invest and adapt so much just to lock yourself to one vendor? For many that alone is more than the operational burden of doing it yourself.

KerrickStaley 8y ago

Lyft's load varies wildly, with significantly higher traffic on Friday and Saturday nights than e.g. 4 AM on a Tuesday, plus spikes on certain evenings like Halloween and New Year's Eve. Having cloud hosting where we can dynamically grow and shrink based on load saves us a lot compared to having fixed infrastructure that is always provisioned for the NYE peak. Source: I work at Lyft.

qxi 8y ago
Why not employ an hybrid architecture of bare metal for base load augmented by cloud-based infrastructure for peaks, constructed via a polyglot union of taped-together tools and lubricated by the daily tears of a hundred college hires only to regret it after the engineers who designed it have successfully used it as a springboard for promotion and departed with their accumulated arcane knowledge (and vested shares) for greener pastures? Worked for us at Twitch.
- diminoten 8y ago
  Oh come on, you know the performance profile of a streaming service is wildly different from a mobile ridesharing app... Their surges are nothing like yours.
  - deleted 8y ago
    [deleted]
  - Alupis 8y ago
    It still stands... $8mm monthly is over 350 full time engineers making $250k annually (ya that doesn't cover hardware, but it just illustrates the point). That would be a 22% increase in Lyft's total number of employees. If you have engineers making less than $250k annually, then you have a lot more staff. $8mm monthly is a LOT... Amazon's clearly making a profit after $8mm monthly.
    - glass_of_water 8y ago
      But it's not like Lyft would suddenly have 350 full time engineers developing new features. A large chunk of those engineers would be working on building and maintaining infrastructure that AWS provides.

mcqueenjordan 8y ago

Is there a strategic advantage for a shoemaker to make their own hammer?

petronio 8y ago
There's a couple. Some samples: * If the hammer manufacturer decides not to sell you any, you'll still have hammers. * If the hammer manufacturer gains enough power to fix prices, you won't be paying them exorbitant prices. * If the hammer manufacturer or their country gets embargoed and you're unable to legally purchase their hammers, you'll still have hammers. All the above grant you a strategic advantage since you'll still have the necessary tools to continue your business while your competitors won't (or will have to pay much higher prices for their supply of hammers).
rdtsc 8y ago
> Is there a strategic advantage for a shoemaker to make their own hammer? Sure. If they are paying $100M/y on hammers, it's at least worth running the numbers and investigate alternatives.
- mcqueenjordan 8y ago
  But that's ignoring the opportunity cost. Strategically, you probably want to focus on what your core competencies are, even if you could in theory do something for cheaper. It's easy to ignore the foregone best alternative of iterating on your own product instead.