12 ms·

100M/year is ~8M/month. Some perspective on that, it could by you one of: ~400PB of data in S3. ~2600 bare metal "x1 type" ec2 instances running 24/7, 3 year

100M/year is ~8M/month. Some perspective on that, it could by you one of: ~400PB of data in S3. ~2600 bare metal "x1 type" ec2 instances running 24/7, 3 year upfront reservation. ~60M Write IOPS in dynamodb ~300M Read IOPS in dynamodb ~3500 16xl RDS aurora instances Again, each of those is spending the entire budget on a single service, but that seems like a nonsense level of spending. Maybe they really have that much data. Maybe they have 100PB of data in S3. Assuming 1B rides since day 1, that's 100MB per ride, which seems high. If the average ride is 20 minutes, that's 80KB per second. That would be 25% of the budget. But assuming they generate 80KB/s/ride, that's ~1MB/second (assuming 1M rides/day). So maybe all of that hits DynamoDB, and between duplicate data, secondary indexes, and size of dataset we have 6 million write iops. And then we do big data processing jobs and have 5x the read load. That's 20% of the budget. And to process all these events there is a massive EMR cluster of bare metal instances. About 1750 of them. That's 50% of the budget. Leaving 5% (a measly 400k) for load balancers, and the like. Those numbers are all a little outrageous to me, but I can see how they might be using that much.

geezerjay 8y ago
At this point isn't it cost-effective for Lyft to just build its own infrastructure?
- godzillabrennus 8y ago
  Very much so. The cost benefit analysis must also question how much it would cost to educate the teams on the new hosting tools. Management has clearly decided that it’s not worth the retraining and loss of productivity.
  - julianozen 8y ago
    Will add if lyft is using things like dynamodb, there is no good onprem alternatives that they can easily migrate to without significant rewrites of core business logic This is expensive and risky and also difficult to do in piece meal Disclaimer: former AWS + Amazon employee
- the_watcher 8y ago
  Possibly, but if they don't have any of their own infra, they'd need to pay someone like AWS while they build it out, right?
- freewilly1040 8y ago
  With cloud vs DC you also needs to consider that re-sizing datacenter space is a very slow process, and if you're wrong about how much you need it's a massive pain. Buy too much and you're wasting money, buy too little and you're stuck throwing eng time at scrambling to keep your services from falling over, and bottlenecking your entire org with resource constraints. That's not a judgement on whether it's worth it for Lyft or not, but especially for a growing company with spiky load the decision is not just a dollars to dollars comparison.
  - peteradio 8y ago
    But let's say you have a 300m budget.
    - eropple 8y ago
      You don't, though. You have room for $300m of opex, coming in over time and allocated, as mentioned, around $8m a month. Less early on, more later. If you gave me $300m to spend, largely up-front, for significant capex purchases? Sure. We could do it. The team I would build would also probably still make mistakes that AWS et al have already largely learned how to avoid, but we could do it. But capex and opex are very different beasts. By the end of that three years I'm already looking at spending way more to refresh what I bought at the start of that three year period because I'm starting to near the end of early contracts and I'm figuring out how best to wrangle, in a way that makes the rest of the business succeed most optimally, a now-heterogeneous environment, etcetera etcetera and etcetera. It's all solvable. But whether it's cheaper, at scale, and more reliable, and presents a unified tool for use by the business...that's a harder question. Understanding how capex and opex work and how they differ is pretty critical to successfully running an engineering organization, to say nothing of a company.
      - halbritt 8y ago
        If you've built a heterogenous environment at the end of three years, then you've failed. The reason that AWS, Google, Azure, et.al do so well is that they don't just buy some servers. They do actual capacity maangement, and not a very good job of it I might add. They also manage the lifecycle of every component in the infrastructure such that the next iteration of that component is understood and interchangeable. Network architecture, for example, should suit the needs of the application, but should also be decoupled from the underlying hardware as that hardware is going to evolve. Compute is fairly straightforward as well. At the data center level, one makes a bunch of 400W holes. What you fill those 400W holes with is relatively irrelevant.
- munchbunny 8y ago
  Maybe. Running your own infrastructure at large scale is actually really hard and therefore surprisingly expensive in risk and talent cost. If you look at how big the internal infra teams are at companies that host their own infrastructure, they're often individually the size of growth stage companies. The problem is that provisioning, reliability, and security are by themselves really tough problems. If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. I look at it as the question: can you get the same set of agility/reliability/security guarantees for your narrower set of use cases by paying for your own hardware and engineering? I won't even begin to pretend I have any answers there, but I think that's the calculus.
  - doctorpangloss 8y ago
    > If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. Maybe that's just the story cloud providers tell you. Until you try, do you really know if it's all that complicated? People have been running datacenters for a long time, and not all of them work for Amazon. But there may be also a beneficial side effect of having gearheads around, and maybe that's the real cost to going cloud.
    - munchbunny 8y ago
      I work for one of said teams, doing datacenter cyber-security. You could say "trying" is my day job. Since we're internal and we manage a lot of capacity, we do often provision and roll our own equivalents of things that cloud providers will sell you, rather than just buying a cloud solution. It's often ambiguous whether it was a good use of time/money. If it weren't for the economies of scale that kick in at the sheer size of this operation, it would definitely not be worth it.
    - theossuary 8y ago
      Yes, I've worked with a few of those datacenters. A few examples: - Recently had to purchase new servers, because of signed contracts the only servers we were allowed to purchase and put in the datacenter were four years old and technically EOF. - Firewall changes, AD changes, provisioning a VM, etc. are 48 hour turnaround. Purchasing new hardware requires 4-6 weeks. - Had an intermittent issue with their edge firewall, it'd slow certain connections to a crawl and eventually they'd timeout. Took six months to fix it, for the first three months they told us it wasn't their fault (turning off their deep packet inspection ended up fixing it). I still remember when we opened the first ticket about it, and the reply was "no other customers are experiencing problems" and it was closed. That's just a few examples of how painful it can be. To give you the other side of the coin, having worked with an enterprise contract in AWS, we were having an intermittent issue with DNS resolving failing for a few seconds every few days. They put an engineer on it full time till they found the problem (we misconfigured it), and it didn't cost us anything more than the enterprise support. I was actually shocked they'd invest that much on such a vague issue. Yes AWS is expensive, but you're getting world class engineering proven at scale, and access to some very smart/motivated people to support it (and they have access to the teams who built it, when they can't solve it). I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Either do it in-house where there's accountability, or use cloud providers who have proven they're competency. To be clear, I'm talking about VPC/EC2/etc. I can't really discuss a lot of their higher level and newer managed services; they either weren't as good, or I haven't tried them. But the bedrock these clouds are built on is solid, and that's worth paying good money for.
- djakjxnanjak 8y ago
  They have many teams making their own web services, so replacing AWS with in-house infra would require an AWS-like “private cloud” feature set, backed by geographically distributed datacenters. Network segmentation that is managed by configuration, user accounts with permissions to do specific things to specific services, etc. Engineers to build all this stuff are not cheap, and management to build and migrate to it cleanly it is not easy. If you can do this you ought to be selling cloud services. Even if it makes sense for Lyft to implement this, it would take a while. The RIO is probably much better on improving their core product, where there is more than $100M/year at stake.
  - metildaa 8y ago
    Sounds like OpenStack...
    - toomuchtodo 8y ago
      Works for Wikipedia, and they’re in the top 5 of Internet websites.
      - djakjxnanjak 8y ago
        I suspect nonprofits like the Wikimedia Foundation have much greater constraints on billables but perhaps relatively less constraints on engineering and management effort. That probably changes the ROI calculation quite a bit. And management of such organizations might also have ideological tendencies that further skew the calculation.
- bduerst 8y ago
  No. The capex and overhead for rolling and maintaining your own infrastructure is outstanding. The only time it typically makes sense these days is when it's done in quantity and also sold to others, which is where the economies of scale kick in.
- zjaffee 8y ago
  It's believed that it would be cheaper for Twitter to have used cloud services (Snapchat spends less then they do on data center operations). There are certainly examples for big companies that benefit from having their own infrastructure (i.e. Dropbox since they have relatively specialized hardware needs compared to what cloud providers set prices around), but the number of people you need to hire to build and maintain datacenters is very high.
  - netik 8y ago
    Twitter initially ran everything in the cloud. I should know, I was employee 13. It was much cheaper to build our own server hardware and move to our own server and network agreements. Peering and transit is cheaper at scale, as is commodity hardware.
    - rrdharan 8y ago
      Those numbers have changed since you left. And they could change again. It's all dynamic: these comparisons and the economically and strategically "correct" choices change over time depending on talent availability, demand, speed of scaling requirements and so on. See https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/a-new-collaboration-with-google-cloud.html https://blog.twitter.com/engineering/en_us/topics/infrastruc....
- fierro 8y ago
  if I have one job in this life, it's to hang out on hacker news and repeatedly post about how it's not cost effective to run your own infrastructure. 8MM/month doesn't even come CLOSE to needing your own infra. - person who knows how hard it is to run your own infrastructure
  - halbritt 8y ago
    So, convince me? - person who knows how hard it is to run your own infrastructure.
    - fierro 8y ago
      ah I regret this comment lol
      - halbritt 8y ago
        No worries. Generally speaking, you're right.
  - Alupis 8y ago
    > person who knows how hard it is to run your own infrastructure > fierro Profile: SWE @ Google Resource & Capacity Planning I think you mean "Person who's job it is to convince others it's really hard and they should just buy your product"...?
    - fierro 8y ago
      ha sure. I just meant I know the sort of operational burden cloud customers are able to outsource by not rolling their own, based on my time at the G inside resource planning and management.
      - tjoff 8y ago
        You know anything of the burden to be reliant on someone else for your core operations? And having to invest and adapt so much just to lock yourself to one vendor? For many that alone is more than the operational burden of doing it yourself.
- KerrickStaley 8y ago
  Lyft's load varies wildly, with significantly higher traffic on Friday and Saturday nights than e.g. 4 AM on a Tuesday, plus spikes on certain evenings like Halloween and New Year's Eve. Having cloud hosting where we can dynamically grow and shrink based on load saves us a lot compared to having fixed infrastructure that is always provisioned for the NYE peak. Source: I work at Lyft.
  - qxi 8y ago
    Why not employ an hybrid architecture of bare metal for base load augmented by cloud-based infrastructure for peaks, constructed via a polyglot union of taped-together tools and lubricated by the daily tears of a hundred college hires only to regret it after the engineers who designed it have successfully used it as a springboard for promotion and departed with their accumulated arcane knowledge (and vested shares) for greener pastures? Worked for us at Twitch.
    - diminoten 8y ago
      Oh come on, you know the performance profile of a streaming service is wildly different from a mobile ridesharing app... Their surges are nothing like yours.
      - deleted 8y ago
        [deleted]
      - Alupis 8y ago
        It still stands... $8mm monthly is over 350 full time engineers making $250k annually (ya that doesn't cover hardware, but it just illustrates the point). That would be a 22% increase in Lyft's total number of employees. If you have engineers making less than $250k annually, then you have a lot more staff. $8mm monthly is a LOT... Amazon's clearly making a profit after $8mm monthly.
        glass_of_water 8y ago
        But it's not like Lyft would suddenly have 350 full time engineers developing new features. A large chunk of those engineers would be working on building and maintaining infrastructure that AWS provides.
- mcqueenjordan 8y ago
  Is there a strategic advantage for a shoemaker to make their own hammer?
  - petronio 8y ago
    There's a couple. Some samples: * If the hammer manufacturer decides not to sell you any, you'll still have hammers. * If the hammer manufacturer gains enough power to fix prices, you won't be paying them exorbitant prices. * If the hammer manufacturer or their country gets embargoed and you're unable to legally purchase their hammers, you'll still have hammers. All the above grant you a strategic advantage since you'll still have the necessary tools to continue your business while your competitors won't (or will have to pay much higher prices for their supply of hammers).
  - rdtsc 8y ago
    > Is there a strategic advantage for a shoemaker to make their own hammer? Sure. If they are paying $100M/y on hammers, it's at least worth running the numbers and investigate alternatives.
    - mcqueenjordan 8y ago
      But that's ignoring the opportunity cost. Strategically, you probably want to focus on what your core competencies are, even if you could in theory do something for cheaper. It's easy to ignore the foregone best alternative of iterating on your own product instead.
dfgert 8y ago
When you make money more than what you need it irrelevant how you spend it.
nodefortytwo 8y ago
You are also calculating based on list price, with Enterprise discount, certain services can be quite a lot cheaper with spend that high.
- phamilton 8y ago
  Absolutely. And for data processing I can guarantee they use spot prices. Just trying to get a rough ballpark for infra at that level of spend.
buboard 8y ago
Why would they have so much data and so much computing. Would they be really dependent on that? Makes it scary to consider what could happens to their passengers if aws is down / hacked.
- Sohcahtoa82 8y ago
  If AWS went down, we'd be worrying about a lot more than catching a Lyft ride. EDIT: typo
  - 0xffff2 8y ago
    Who's "we" and why would they worry? I'm struggling to think of a side-effect of AWS going down that would worry me more than being unable to get a Lyft (which itself doesn't worry me very much).
    - cronix 8y ago
      Maybe a few here would cause some concern. Also note that not all companies/governments using AWS just let everybody know that they are, so they aren't listed and you can only know when the service goes offline. > Adobe, Airbnb, Alcatel-Lucent, AOL, Acquia, AdRoll, AEG, Alert Logic, Autodesk, Bitdefender, BMW, British Gas, Canon, Capital One, Channel 4, Chef, Citrix, Coinbase, Comcast, Coursera, Docker, Dow Jones, European Space Agency, Financial Times, FINRA, General Electric, GoSquared, Guardian News & Media, Harvard Medical School, Hearst Corporation, Hitachi, HTC, IMDb, International Centre for Radio Astronomy Research, International Civil Aviation Organization, ITV, iZettle, Johnson & Johnson, JustGiving, JWT, Kaplan, Kellogg’s, Lamborghini, Lonely Planet, Lyft, Made.com, McDonalds, NASA, NASDAQ OMX, National Rail Enquiries, National Trust, Netflix, News International, News UK, Nokia, Nordstrom, Novartis, Pfizer, Philips, Pinterest, Quantas, Sage, Samsung, SAP, Schneider Electric, Scribd, Securitas Direct, Siemens, Slack, Sony, SoundCloud, Spotify, Square Enix, Tata Motors, The Weather Company, Ticketmaster, Time Inc., Trainline, Ubisoft, UCAS, Unilever, US Department of State, USDA Food and Nutrition Service, UK Ministry of Justice, Vodafone Italy, WeTransfer, WIX, Xiaomi, Yelp, Zynga [1]. [1] https://www.contino.io/insights/whos-using-aws https://www.contino.io/insights/whos-using-aws Additional Info: http://nymag.com/intelligencer/2018/03/when-amazon-web-services-goes-down-so-does-a-lot-of-the-web.html http://nymag.com/intelligencer/2018/03/when-amazon-web-servi...
onlyrealcuzzo 8y ago
Everything seems reasonable except... ~3500 16xl RDS aurora instances? I worked at one of the 100 biggest websites on the internet (a search engine), and we only had 3. Why/how would Lyft need 1000x that!?
- MuffinFlavored 8y ago
  What'd you use those 3 for?
  - onlyrealcuzzo 8y ago
    Basically, we had three types of data: 1) everything associated with the SERPs, 2) most things associated with payments, and 3) everything else. Category three was relational, IIRC like ~2TB. We originally stored it all on a custom MySQL cluster, and we started running into pretty bad replication lag. Amazon came around with Arora and promised we'd never have any replication lag. We switched over. Still had replication lag.
- phamilton 8y ago
  The numbers weren't "all this together". I was just saying if 100% of spend was in category X, this is how much you'd get. It highlighted pretty well, evidenced by your comment, that RDS was not likely a significant portion of their spend.

geezerjay 8y ago

At this point isn't it cost-effective for Lyft to just build its own infrastructure?

godzillabrennus 8y ago
Very much so. The cost benefit analysis must also question how much it would cost to educate the teams on the new hosting tools. Management has clearly decided that it’s not worth the retraining and loss of productivity.
- julianozen 8y ago
  Will add if lyft is using things like dynamodb, there is no good onprem alternatives that they can easily migrate to without significant rewrites of core business logic This is expensive and risky and also difficult to do in piece meal Disclaimer: former AWS + Amazon employee
the_watcher 8y ago
Possibly, but if they don't have any of their own infra, they'd need to pay someone like AWS while they build it out, right?
freewilly1040 8y ago
With cloud vs DC you also needs to consider that re-sizing datacenter space is a very slow process, and if you're wrong about how much you need it's a massive pain. Buy too much and you're wasting money, buy too little and you're stuck throwing eng time at scrambling to keep your services from falling over, and bottlenecking your entire org with resource constraints. That's not a judgement on whether it's worth it for Lyft or not, but especially for a growing company with spiky load the decision is not just a dollars to dollars comparison.
- peteradio 8y ago
  But let's say you have a 300m budget.
  - eropple 8y ago
    You don't, though. You have room for $300m of opex, coming in over time and allocated, as mentioned, around $8m a month. Less early on, more later. If you gave me $300m to spend, largely up-front, for significant capex purchases? Sure. We could do it. The team I would build would also probably still make mistakes that AWS et al have already largely learned how to avoid, but we could do it. But capex and opex are very different beasts. By the end of that three years I'm already looking at spending way more to refresh what I bought at the start of that three year period because I'm starting to near the end of early contracts and I'm figuring out how best to wrangle, in a way that makes the rest of the business succeed most optimally, a now-heterogeneous environment, etcetera etcetera and etcetera. It's all solvable. But whether it's cheaper, at scale, and more reliable, and presents a unified tool for use by the business...that's a harder question. Understanding how capex and opex work and how they differ is pretty critical to successfully running an engineering organization, to say nothing of a company.
    - halbritt 8y ago
      If you've built a heterogenous environment at the end of three years, then you've failed. The reason that AWS, Google, Azure, et.al do so well is that they don't just buy some servers. They do actual capacity maangement, and not a very good job of it I might add. They also manage the lifecycle of every component in the infrastructure such that the next iteration of that component is understood and interchangeable. Network architecture, for example, should suit the needs of the application, but should also be decoupled from the underlying hardware as that hardware is going to evolve. Compute is fairly straightforward as well. At the data center level, one makes a bunch of 400W holes. What you fill those 400W holes with is relatively irrelevant.
munchbunny 8y ago
Maybe. Running your own infrastructure at large scale is actually really hard and therefore surprisingly expensive in risk and talent cost. If you look at how big the internal infra teams are at companies that host their own infrastructure, they're often individually the size of growth stage companies. The problem is that provisioning, reliability, and security are by themselves really tough problems. If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. I look at it as the question: can you get the same set of agility/reliability/security guarantees for your narrower set of use cases by paying for your own hardware and engineering? I won't even begin to pretend I have any answers there, but I think that's the calculus.
- doctorpangloss 8y ago
  > If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that. Maybe that's just the story cloud providers tell you. Until you try, do you really know if it's all that complicated? People have been running datacenters for a long time, and not all of them work for Amazon. But there may be also a beneficial side effect of having gearheads around, and maybe that's the real cost to going cloud.
  - munchbunny 8y ago
    I work for one of said teams, doing datacenter cyber-security. You could say "trying" is my day job. Since we're internal and we manage a lot of capacity, we do often provision and roll our own equivalents of things that cloud providers will sell you, rather than just buying a cloud solution. It's often ambiguous whether it was a good use of time/money. If it weren't for the economies of scale that kick in at the sheer size of this operation, it would definitely not be worth it.
  - theossuary 8y ago
    Yes, I've worked with a few of those datacenters. A few examples: - Recently had to purchase new servers, because of signed contracts the only servers we were allowed to purchase and put in the datacenter were four years old and technically EOF. - Firewall changes, AD changes, provisioning a VM, etc. are 48 hour turnaround. Purchasing new hardware requires 4-6 weeks. - Had an intermittent issue with their edge firewall, it'd slow certain connections to a crawl and eventually they'd timeout. Took six months to fix it, for the first three months they told us it wasn't their fault (turning off their deep packet inspection ended up fixing it). I still remember when we opened the first ticket about it, and the reply was "no other customers are experiencing problems" and it was closed. That's just a few examples of how painful it can be. To give you the other side of the coin, having worked with an enterprise contract in AWS, we were having an intermittent issue with DNS resolving failing for a few seconds every few days. They put an engineer on it full time till they found the problem (we misconfigured it), and it didn't cost us anything more than the enterprise support. I was actually shocked they'd invest that much on such a vague issue. Yes AWS is expensive, but you're getting world class engineering proven at scale, and access to some very smart/motivated people to support it (and they have access to the teams who built it, when they can't solve it). I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Either do it in-house where there's accountability, or use cloud providers who have proven they're competency. To be clear, I'm talking about VPC/EC2/etc. I can't really discuss a lot of their higher level and newer managed services; they either weren't as good, or I haven't tried them. But the bedrock these clouds are built on is solid, and that's worth paying good money for.
djakjxnanjak 8y ago
They have many teams making their own web services, so replacing AWS with in-house infra would require an AWS-like “private cloud” feature set, backed by geographically distributed datacenters. Network segmentation that is managed by configuration, user accounts with permissions to do specific things to specific services, etc. Engineers to build all this stuff are not cheap, and management to build and migrate to it cleanly it is not easy. If you can do this you ought to be selling cloud services. Even if it makes sense for Lyft to implement this, it would take a while. The RIO is probably much better on improving their core product, where there is more than $100M/year at stake.
- metildaa 8y ago
  Sounds like OpenStack...
  - toomuchtodo 8y ago
    Works for Wikipedia, and they’re in the top 5 of Internet websites.
    - djakjxnanjak 8y ago
      I suspect nonprofits like the Wikimedia Foundation have much greater constraints on billables but perhaps relatively less constraints on engineering and management effort. That probably changes the ROI calculation quite a bit. And management of such organizations might also have ideological tendencies that further skew the calculation.
bduerst 8y ago
No. The capex and overhead for rolling and maintaining your own infrastructure is outstanding. The only time it typically makes sense these days is when it's done in quantity and also sold to others, which is where the economies of scale kick in.
zjaffee 8y ago
It's believed that it would be cheaper for Twitter to have used cloud services (Snapchat spends less then they do on data center operations). There are certainly examples for big companies that benefit from having their own infrastructure (i.e. Dropbox since they have relatively specialized hardware needs compared to what cloud providers set prices around), but the number of people you need to hire to build and maintain datacenters is very high.
- netik 8y ago
  Twitter initially ran everything in the cloud. I should know, I was employee 13. It was much cheaper to build our own server hardware and move to our own server and network agreements. Peering and transit is cheaper at scale, as is commodity hardware.
  - rrdharan 8y ago
    Those numbers have changed since you left. And they could change again. It's all dynamic: these comparisons and the economically and strategically "correct" choices change over time depending on talent availability, demand, speed of scaling requirements and so on. See https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/a-new-collaboration-with-google-cloud.html https://blog.twitter.com/engineering/en_us/topics/infrastruc....
fierro 8y ago
if I have one job in this life, it's to hang out on hacker news and repeatedly post about how it's not cost effective to run your own infrastructure. 8MM/month doesn't even come CLOSE to needing your own infra. - person who knows how hard it is to run your own infrastructure
- halbritt 8y ago
  So, convince me? - person who knows how hard it is to run your own infrastructure.
  - fierro 8y ago
    ah I regret this comment lol
    - halbritt 8y ago
      No worries. Generally speaking, you're right.
- Alupis 8y ago
  > person who knows how hard it is to run your own infrastructure > fierro Profile: SWE @ Google Resource & Capacity Planning I think you mean "Person who's job it is to convince others it's really hard and they should just buy your product"...?
  - fierro 8y ago
    ha sure. I just meant I know the sort of operational burden cloud customers are able to outsource by not rolling their own, based on my time at the G inside resource planning and management.
    - tjoff 8y ago
      You know anything of the burden to be reliant on someone else for your core operations? And having to invest and adapt so much just to lock yourself to one vendor? For many that alone is more than the operational burden of doing it yourself.
KerrickStaley 8y ago
Lyft's load varies wildly, with significantly higher traffic on Friday and Saturday nights than e.g. 4 AM on a Tuesday, plus spikes on certain evenings like Halloween and New Year's Eve. Having cloud hosting where we can dynamically grow and shrink based on load saves us a lot compared to having fixed infrastructure that is always provisioned for the NYE peak. Source: I work at Lyft.
- qxi 8y ago
  Why not employ an hybrid architecture of bare metal for base load augmented by cloud-based infrastructure for peaks, constructed via a polyglot union of taped-together tools and lubricated by the daily tears of a hundred college hires only to regret it after the engineers who designed it have successfully used it as a springboard for promotion and departed with their accumulated arcane knowledge (and vested shares) for greener pastures? Worked for us at Twitch.
  - diminoten 8y ago
    Oh come on, you know the performance profile of a streaming service is wildly different from a mobile ridesharing app... Their surges are nothing like yours.
    - deleted 8y ago
      [deleted]
    - Alupis 8y ago
      It still stands... $8mm monthly is over 350 full time engineers making $250k annually (ya that doesn't cover hardware, but it just illustrates the point). That would be a 22% increase in Lyft's total number of employees. If you have engineers making less than $250k annually, then you have a lot more staff. $8mm monthly is a LOT... Amazon's clearly making a profit after $8mm monthly.
      - glass_of_water 8y ago
        But it's not like Lyft would suddenly have 350 full time engineers developing new features. A large chunk of those engineers would be working on building and maintaining infrastructure that AWS provides.
mcqueenjordan 8y ago
Is there a strategic advantage for a shoemaker to make their own hammer?
- petronio 8y ago
  There's a couple. Some samples: * If the hammer manufacturer decides not to sell you any, you'll still have hammers. * If the hammer manufacturer gains enough power to fix prices, you won't be paying them exorbitant prices. * If the hammer manufacturer or their country gets embargoed and you're unable to legally purchase their hammers, you'll still have hammers. All the above grant you a strategic advantage since you'll still have the necessary tools to continue your business while your competitors won't (or will have to pay much higher prices for their supply of hammers).
- rdtsc 8y ago
  > Is there a strategic advantage for a shoemaker to make their own hammer? Sure. If they are paying $100M/y on hammers, it's at least worth running the numbers and investigate alternatives.
  - mcqueenjordan 8y ago
    But that's ignoring the opportunity cost. Strategically, you probably want to focus on what your core competencies are, even if you could in theory do something for cheaper. It's easy to ignore the foregone best alternative of iterating on your own product instead.

dfgert 8y ago

When you make money more than what you need it irrelevant how you spend it.

nodefortytwo 8y ago

You are also calculating based on list price, with Enterprise discount, certain services can be quite a lot cheaper with spend that high.

phamilton 8y ago
Absolutely. And for data processing I can guarantee they use spot prices. Just trying to get a rough ballpark for infra at that level of spend.

buboard 8y ago

Why would they have so much data and so much computing. Would they be really dependent on that? Makes it scary to consider what could happens to their passengers if aws is down / hacked.

Sohcahtoa82 8y ago
If AWS went down, we'd be worrying about a lot more than catching a Lyft ride. EDIT: typo
- 0xffff2 8y ago
  Who's "we" and why would they worry? I'm struggling to think of a side-effect of AWS going down that would worry me more than being unable to get a Lyft (which itself doesn't worry me very much).
  - cronix 8y ago
    Maybe a few here would cause some concern. Also note that not all companies/governments using AWS just let everybody know that they are, so they aren't listed and you can only know when the service goes offline. > Adobe, Airbnb, Alcatel-Lucent, AOL, Acquia, AdRoll, AEG, Alert Logic, Autodesk, Bitdefender, BMW, British Gas, Canon, Capital One, Channel 4, Chef, Citrix, Coinbase, Comcast, Coursera, Docker, Dow Jones, European Space Agency, Financial Times, FINRA, General Electric, GoSquared, Guardian News & Media, Harvard Medical School, Hearst Corporation, Hitachi, HTC, IMDb, International Centre for Radio Astronomy Research, International Civil Aviation Organization, ITV, iZettle, Johnson & Johnson, JustGiving, JWT, Kaplan, Kellogg’s, Lamborghini, Lonely Planet, Lyft, Made.com, McDonalds, NASA, NASDAQ OMX, National Rail Enquiries, National Trust, Netflix, News International, News UK, Nokia, Nordstrom, Novartis, Pfizer, Philips, Pinterest, Quantas, Sage, Samsung, SAP, Schneider Electric, Scribd, Securitas Direct, Siemens, Slack, Sony, SoundCloud, Spotify, Square Enix, Tata Motors, The Weather Company, Ticketmaster, Time Inc., Trainline, Ubisoft, UCAS, Unilever, US Department of State, USDA Food and Nutrition Service, UK Ministry of Justice, Vodafone Italy, WeTransfer, WIX, Xiaomi, Yelp, Zynga [1]. [1] https://www.contino.io/insights/whos-using-aws https://www.contino.io/insights/whos-using-aws Additional Info: http://nymag.com/intelligencer/2018/03/when-amazon-web-services-goes-down-so-does-a-lot-of-the-web.html http://nymag.com/intelligencer/2018/03/when-amazon-web-servi...

onlyrealcuzzo 8y ago

Everything seems reasonable except... ~3500 16xl RDS aurora instances? I worked at one of the 100 biggest websites on the internet (a search engine), and we only had 3. Why/how would Lyft need 1000x that!?

MuffinFlavored 8y ago
What'd you use those 3 for?
- onlyrealcuzzo 8y ago
  Basically, we had three types of data: 1) everything associated with the SERPs, 2) most things associated with payments, and 3) everything else. Category three was relational, IIRC like ~2TB. We originally stored it all on a custom MySQL cluster, and we started running into pretty bad replication lag. Amazon came around with Arora and promised we'd never have any replication lag. We switched over. Still had replication lag.
phamilton 8y ago
The numbers weren't "all this together". I was just saying if 100% of spend was in category X, this is how much you'd get. It highlighted pretty well, evidenced by your comment, that RDS was not likely a significant portion of their spend.