20 ms·
Amazon makes 14 cents from every Lyft ride
- tomtompl 8y agoThe title is clickbaity. They get paid that ~14 cents but they don't 'make' that much. What they make is what is left after costs.
- tudelo 8y agoIf you were to say how much you get paid, you would in general say you make x an hour. You wouldn't say "Well, after taxes, rent, food, gas, insurance, etc, I make x-y an hour."
- sisk 8y agoThe thread is much more interesting than the initial tweet. https://threadreaderapp.com/thread/1102401615263223809.html https://threadreaderapp.com/thread/1102401615263223809.html
- draw_down 8y agoThe link shows the thread.
- skilled 8y agoGreat app, didn't know it exists! Thank you. And if mods feel it's appropriate, then I don't mind if the link is swapped.
- deleted 8y ago[deleted]
- altmind 8y agoThis thread only compares ROI of hosting in cloud with building own datacenter and laying own oceanic fiber(!!). The option of using a colo and magistral ISP somehow fell through the cracks. Of course building one datacenter and own oceanic fiber to host your own services is a bad option for almost every business - not every business can fill a full DC, and 1 DC is not enough to guarantee global operations.
- altmind 8y agoThe orignal author mentioned how good AWS worked for netflix, but did not mention that netflix rolled their own content delivery system in part because of high traffic cost with all the major cloud providers.
- snarf21 8y agoExactly, AWS is perfect for running the search and UI features but you have to have the content at the edge or get killed on bandwidth.
- kemitche 8y agoHe did mention that about netflix: https://twitter.com/MohapatraHemant/status/1102410173203660800 https://twitter.com/MohapatraHemant/status/11024101732036608... > E.g. if you are +30% of the internet traffic (nflx) it doesn't make sense to pay rent to telcos any more and feed their margins. You have the volume and stable demand to justify ownership. For the rest, cloud is where they'll live and die.
- drewg123 8y agoMy understanding is that most commercial CDNs (Limelight, Akamai, etc) are also more cost-effective than serving directly from AWS. However, commercial CDNs don't know a-priori what files are going to be popular, and how to distribute them in different regions and on different types of storage to maximize efficiency. But that's one of the things that Netflix's CDN team can predict. See https://medium.com/netflix-techblog/distributing-content-to-open-connect-3e3e391d4dc9 https://medium.com/netflix-techblog/distributing-content-to-...
- isostatic 8y agoHurricane Electric currently have a 42U cabinet and 1 gigabit of bandwidth for $400pcm. That's the cost of about 6 AWS t2.large VMs That's 50 cents per hour to colocate dozens of servers, and there's 324TB of data thrown in for free. Obviously you have the tin to buy and maintain which will cost you. Leasing costs for a typical HP server are say $100/month. 10G transit on it's own is around $2k/month with H.E, or 1 cent for 16GB if it's fully saturated.
- the8472 8y agoThe twitter desktop page shows the whole thread already.
- prepend 8y agoAmazon got 2 cents from me typing this note and reading ycombinator for an hour. I love when indirect costs are naively calculated for marginal cost estimates.
- toomuchtodo 8y agoHN runs on a dedicated box colo'd at M5 Hosting :-P
- buboard 8y agoA single box ?
- toomuchtodo 8y agoYes. https://news.ycombinator.com/item?id=11916168 https://news.ycombinator.com/item?id=11916168
- ddorian43 8y agoThese posts always make it like "cloud is better vs building datacenters on mars". There are middlegrounds, like colocation & dedicated servers. If you get dedicated servers 4x cheaper than cloud-shared-vps with remote-hdd, then you can overprovision. Especially now that hardware is getting bigger you need even less space (assuming your software scales vertically). And they ALWAYS make it like the next hour you will have 10x requests and your database will autoscale that quickly.
- kthejoker2 8y agoAs this post makes clear, it's not the things you can control in a dedicated/colo environment you should be worried about. You can always hand them more dollars for more storage and compute. You can never hand them enough dollars to get their SLAs/latency on par with the cloud. There will be outages. There will be delays. There will be scale problems. There And you'll have to address these somewhere else, either in your tech stack, your operating model, or your PR department. Paying a 2-5% TCO premium to have a throat to choke, 5 9s redundancy, GDPR compliance, and the law of large numbers on your side of the court is a pretty fair trade.
- ddorian43 8y agoChange the TCO to 2x-5x then you're right. Good luck with the throat choking. Also what does GDPR have to do with renting vps in cloud ?
- hpkuarg 8y agoAgreed. It's disingenuous of the article OP to: - only parenthetically mention colocation, as if it isn't absolutely normal to rent a rack/cage/suite as needed from an existing DC operator (which in addition would make datacenter rent an OpEx and actual server equipment be amortized over 2-3 years, not the 10 for real property) - somehow present "intercontinental traffic" as something both necessary and tremendously expensive, as if all the major public clouds aren't charging 10x what the market rates for bandwidth are - imply that "the cloud" is immune to outages, as if GCP didn't have multiple major global- or region-wide outages over the past few years I mean, I understand why most companies don't go on-prem despite all that, but this series of tweets is borderline FUD.
- jgalt212 8y agoOK, but that's just the AWS expense. How much per ride does Lyft spend on it's own devs (and admin staff) and boxes (and electric and RE costs)?
- jxi 8y agoThat + business costs are pretty much the rest of the total cost, right?
- craigkilgo 8y agoHow are they paying this much?
- sek 8y agoAre we talking just about machines? A huge chunk of AWS is actually enterprise SAAS. Analytics, authentication, support etc. you don't know what Lyft uses exactly here from AWS. For example if they use Auth0 instead of Cognito they pay that part to Auth0. AWS prices are very competitive most of the time. AWS is the Walmart of enterprise IT, they sell you everything else too.
- oblio 8y agoOne could say: AWS is the... Amazon.com of enterprise IT, they sell you everything else too :D
- syntaxing 8y agoI always see the motivation for the cloud mainly because of financial reasoning but is there a benefit to stay on-site in terms of knowledge retention (for the lack of a better word)? For instance, there is a big push to outsource manufactured parts and good mainly due to "cost savings". However, the are long term cost savings when we keep the parts made in-house which management always overlook. Sure, it cost money to maintain the machines, inventory, and tools but this allows greater flexibility and also retain the knowledge within the company. In addition, it allows the engineers to drive technology within the company faster. Is this the case for data center infrastructure?
- perlgeek 8y agoYou are a bit more flexible in the data center. Like, if you really need very low latency and high bandwidth between two machines, you could buy an infiniband thingy. Expensive AF, but may less expensive than setting a horde of software engineers on it for two years to deal with the higher latency. But, all in all, there aren't all that many practical use cases that really benefits from stuff like that. Most projects are quite happy with some storage, compute, network, and a few managed services. Most companies don't do rocket science with their software and infrastructure. And there are things that you simply can only afford at scale, like security management for your supply chain, dedicated automation for updating firmware of every tiny controller that's in your hardware, and so on.
- msoad 8y agoLyft is horribly inefficient in terms of using their AWS resources. I know this first hand. They don't have to build their own DS. They just need to build a better system. Possibly take advantage of Lambdas because a lot of compute heavy operations like geo en/decoding are done in services that run at like 25% CPU in production.
- ghaff 8y agoNot to pick on you specifically as others are saying more or less the same thing... But if you're such a genius that you know how to slash Lyft's AWS spend without knowing basically a thing about their infrastructure, you should be knocking loudly on their door because I'm sure they have an extremely well-paid job waiting for you. Otherwise, maybe consider voicing opinions like these about things you actually know something about. ADDED: Meant to respond to a different comment that didn't suggest any inside knowledge. (Though I'd still point out that even insiders often can't understand how their own company has so many employees in X department, is so inefficient, etc. And this is more or less the case everywhere.
- ceejayoz 8y ago"I know this first hand" implies they do know something about their infrastructure. Given that they work for "one of few >10b unicorns" per their comment history, I have a suspicion you may be telling a Lyft employee they don't know anything about Lyft.
- deleted 8y ago[deleted]
- ghaff 8y agoI actually meant to respond to a different comment which merely said this was a few hours of an AWS instance.
- ceejayoz 8y agoWell, that's factual - you can get a couple hours time on an entire AWS instance for $0.14. That certainly seems to point to an cost-inefficient infrastructure.
- amaccuish 8y ago"...you'll never have hot-swappable everything managed by ultra fast robotic arms that replace hard drives in seconds." Does anyone know anything more about these robotic hard drive replacers?
- ceejayoz 8y agoI've seen it done for tape libraries. My understanding is that the biggies - Google, Amazon, Facebook, etc. - don't bother with individual drive replacements (and certainly not with an expensive robotic arm system). They wait for an entire rack to fault over a certain percentage and then just swap out the whole thing.
- q3k 8y agoRobotic tape libraries (like the StorageTek/Sun/Oracle SL8500) are standard. However, the robotic part is not to swap parts in case of failure - instead, it is to let ~64 tape drives access ~100k individual tapes within the library and ~inf tapes in off-site storage (as the data stored is not, and _shouldn't_ be, directly accessible).
- ADefenestrator 8y agoDepends on the case. Facebook at least generally replaces drives as they go. In a pipelined fashion rather than as they fail, though. So, a given drive failure doesn't lead to immediate emergency replacement, but sometime in the next few days as someone makes a drive-replacement pass through that area.
- ProAm 8y agoThis is why Twitter is terrible for having real conversations. This was painful to read because of formatting and structure.
- chomp 8y agoThis is a bunch of FUD. * No, using a colo saves you way more than just 20%. In one of our facilities, our org has maybe ~15k servers under management and dc costs are ~ 1 million/mo. Build out of the cages and racks wasn't incredibly expensive, I think ~2 million. Power isn't insane either. * I have no idea what he's talking about re: the fiber statement. We have a blend of different backbone providers that give us ~100 gigabit for ~$0.60 per gig. Compared to bandwidth costs for AWS, it's dirt cheap. * You don't need ultra fast hot swappable robotic arms if you're not FAANG. With a handful of dc techs and a simple monitoring system, you can swap out disks just as easily, at the expense of a little more opex. * PCI requires no consultants, its terms are pretty black and white - you will need an ISA though. HIPAA and can be more of a bear, but your org will need this consultation regardless of whether you own a dc or put it in the cloud. AWS does not give you automatic compliance. * Yes I can totally believe that working at Google, you'd give TCOs that make Google look good. Literally none of us except Lyft's operations knows what's good for Lyft. 100 million a year is a ton of cash, but gives them presence around the world. My company has datacenters around the world and it is very operationally complex to maintain them. It's definitely not for everyone, but this guy is making it sound worse than it actually is.
- jaredwiener 8y agoalso this thinking seems very one-size-has-to-fit-all, cloud/datacenter focused. why would lyft need HIPAA compliance??
- skue 8y agoBoth Lyft & Uber have begun partnering with healthcare providers to provide non-urgent patient transportation.[1] [1] https://www.fiercehealthcare.com/tech/lyft-announces-integration-allscripts-ehr-system-letting-180-000-docs-hail-rides-for-patients https://www.fiercehealthcare.com/tech/lyft-announces-integra...
- kennywinker 8y agoDystopian af
- devit 8y agoMaybe they could have a reasonably efficient setup instead? Not sure how you can possibly pay 0.14$ in computing for a single ride (if that's accurate). That's more than 3 hours of a t3.medium instance for example...
- kemitche 8y agoThe tweet took Lyft's total AWS costs and divided it by total rides. It's not a literal accounting of the compute cost for a single ride. AWS costs that wouldn't really fit into a per-ride accounting: * Redundancy of instances (regions/AZs) * Data-storage/duplication/backups * Non-ride related AWS costs (hosting/processing of analytics, test & automation infrastructure, etc.) I'm sure there are others. I'm not saying that there's no way they could get their AWS costs down, btw.
- ousta 8y agoWhy would you need to build underwater fiber????????? I don't understand.
- mattkrause 8y agoEspecially for Lyft, where you can be pretty sure most stuff won’t cross oceanic boundaries: “Raoul is arriving in a white Cunard QM2-G32”
- aserafini 8y agoLook out for ‘Uber for Ferries’ in this summer’s YC batch!
- mv4 8y agoSome quick math: last I heard, Lyft was doing 1M rides a day. That's 1M transactions per day, or "only" 11 transactions per second. Of course there's lot more (assuming 10x) supporting transactions to enable one ride - search, matching, status updates. Let's say that adds another 100 transactions per second. Rounding up to 150/sec. What am I missing? In terms of data usage and storage, matching is by geography, meaning their data is easy to shard. So, how's this $8M per month? edit: fixed a typo
- thinkingkong 8y agoYour math isnt wrong but the load isnt simply averaged out over the entire day. There will be peak capacity times way way higher than the lowest times. I would guess orders of magnitude larger. You have to build for peak capacity. If their target response time is 100ms or something then that adds in lots of complexity.
- mv4 8y agoI agree the load will fluctuate - but will mostly follow predictable patterns daily and weekly. However, that's what the cloud's elasticity is for - and AWS makes it easy (and inexpensive) to expand and contract when implemented correctly. Man, I would love an opportunity to optimize something like that!
- MisterPea 8y agoRight?! The cost is way too high for their use case, even considering a large padding to OPs quick maths. Part of my team one month stopped all development and solely focused on reducing AWS costs and we cut our monthly bill by 50% (probably around 4 dev's yearly salary) It's not easy, but optimizing AWS resources is an absolute must.
- kennywinker 8y agoIdk, but I will say that there is definitely stuff going on beyond the ride. I.e. for every n people who open the app only a small number will book a ride. Billing services, driver location tracking, analytics stuff, etc are all going to balloon the amount of computing that the service uses.
- quantumhobbit 8y ago14 cents per ride is nothing! Maybe 1% of the cost of a ride. Maybe they could save a bunch by colo or something else. But would 14 cents per ride really matter at all for their competitiveness. I’m not going to notice a 14 cent difference even if I do bother to price compare Uber with Lyft. This is a VC fueled market. It isn’t really about small margins of this size.
- ceejayoz 8y agoIt may be 1% of the cost of the ride, but it's likely a substantially larger part of the profits of the ride. Lyft lost a billion dollars on $2B of revenue in 2018.
- isostatic 8y agoSo saving $8m a month, or $100m a year, isn't going to turn that round.
- ceejayoz 8y agoCutting 10% of losses is hardly something to sneer at.
- tdb7893 8y agoYou wouldn't be able to cut all of it. Also making those major infrastructure changes are really expensive.
- ivalm 8y agoMore like 5% if they cost costs to 0. But realistically they can cut costs by maybe half.... so you're taking significant risk for cutting losses by... 2.5%?
- isostatic 8y agoIf you're still losing $100m a month you have bigger problems then saving a couple million in infrastructure costs. Moving from AWS, or doubling AWS spending, will make no difference to the company's viability, it's not worth the time in meetings to discuss it.
- so_tired 8y agoWell, each driver needs a mobile+plan, at ~$100/month fully amortized. Definitely does <1000 rides/month. APPL, VRZN each get 5c/ride - just for showing up. I started this comment as a joke. But now i am not so sure.
- so_tired 8y agoAnd while i am at it... Suggesting lyft needs 5-9s is ridiculous for 1000s of servers is ridiculous. Its entire event stream comes from a mobile network which is probably less reliable.
- charleslmunger 8y agoIt's not that simple - if Lyft won't connect and everything else works, riders and drivers will turn to competitors. Once they've installed the app and signed up for an account with Uber, they're a lot less likely to stick exclusively or primarily with Lyft.
- marcosdumay 8y agoWell, 5 9's are 5 minutes of disruption an year. Drivers won't be able to get into Uber on that timeframe. 4 9's would be 50 minutes an year. Unless it is all concentrated on a couple of events it may not even make the clients time-out.
- SilasX 8y agoI don't think the title is a fair representation of what's going on. Hear me out. The way it's phrased, it's like each ride directly translates into 14 more cents for Amazon. But really, the figure comes from averaging Lyft's Amazon costs over Lyft's total rides. Now, in some cases, that would be valid accounting, if all (or nearly all) of your costs (with respect to that service) are variable. For example, most of the cost of sending a letter is the truck/plane/employee, which are variable, so it makes sense to divide total letters over total truck etc costs. But would Lyft pay twice as much to Amazon if they served twice as many rides? Would they have to double every AWS service they're buying? I don't think so. They might have to add more of one or two kinds of server/services, but most of that 14 cents figure comes from amortizing fixed costs, which will go down if they scale up further. Am I off base here?
- raiyu 8y agoYou aren't off base as this is a snapshot in time so it's no way to know if those costs are linear and scale with more rides or not without seeing historical figures on AWS spend vs revenue/# of rides. But it could very likely be true that doubling the number of rides would have a potentially material impact on additional AWS spend for Lyft.
- mv4 8y agoTheir model should scale linearly, because everything can be clustered/sharded by geo so nicely, and the scope of a typical transaction is one passenger vs a few dozen potential drivers. That does not significantly change as they grow. Unlike someone like FB, with their many-to-many everything, Lyft/Uber is a data architect's dream.
- SilasX 8y agoThere's definitely something that has to scale linearly as they serve more rides; the question is how much of that 14 cents the linear-scaling stuff accounts for.
- 8y ago
- rhacker 8y agoOur (small) company has a $500-$1000 AWS bill. About 2 months ago it was double that. This is a 3-4 developer company. When we hire another person we'll probably increase our EKS instances by one more box. We already paid down our debt - and it definitely cost us about 2 months with half our dev staff. But our goal wasn't to reduce the AWS cost, it was to reduce the time it takes to replicate an environment in K8s with one script. I'm guessing Lyft has not taken a path like that.
- gesman 8y agoSomething to consider in all estimates - if cloud provider could run into conflict of interest with you.
- thisisit 8y agoI am surprised no one is talking about Lyft's risk assessment of their business. Specifically, this line: Our results of operations vary and are unpredictable from period-to-period, which could cause the trading price of our Class A common stock to decline. Our results of operations have historically varied from period-to-period and we expect that our results of operations will continue to do so for a variety of reasons, many of which are outside of our control and difficult to predict. I find the term "period-to-period" rather vague. It could mean quarter-to-quarter or even year-to-year. Still what they are effectively saying is that Lyft cannot be trusted to provide any growth assessment.
- _bxg1 8y agoA really elaborate (and admittedly interesting) way of repeating the constant refrain: Things that work for the extremely large-scale usually aren't ideal for the small- or medium- scale. This applies to everything from codebase size to infrastructure size, and the opposing force tends to be engineers' inherent enthusiasm for engineering things. Be honest about your actual needs and know when not to chase the N=∞ case.
- buboard 8y agoIs lyft considered extremely large scale?
- _bxg1 8y agoWell, that's the question. The guy basically makes the case that it isn't, for this purpose, and that Netflix and Dropbox are. Good insight into the hairy details, but the overall thesis is pretty simple.
- solidasparagus 8y agoI would say he's making the case that Lyft doesn't have a single component with ridiculous scale like Netflix CDN or Dropbox storage. That could just mean Lyft's costs are more evenly distributed.
- OrgNet 8y agohow much per ride goes in taxes?
- g9yuayon 8y agoPeople seem have forgotten the productivity loss of running their own data center. When I was in my previous company, most of things are just excruciating compared with what I enjoyed in Netflix. Dockers did not come with persistent volumes, provisioning machines involved lengthy approvals and sometimes long waiting time; building and deploying services that involved multiple clusters that talked to each other required serious devop-fu; production clusters got broken because infra upgraded system dependencies using a background job and mitigating such simple mistakes may take hours if not days. Deploying any non-trivial service required multiple meetings with some infra teams, for resources, for concerns with stability, or whatever. PS, did I mention that we built tens of thousands of Puppet files that very few people understood and any updates to my system took half an hour to refresh yet the early members of the infra team thought Puppet was the best f@#$ tool in the world? The list can go on and on. PPS, did I mention that we had a 500-person infra team and later probably more than a thousand for a fraction of traffic/complexity compared with what Netflix had to deal with, while Netflix didn't even have an infra team (to be fair, Netflix had a build team and a platform team and a monitoring team, which had fewer than 30 people combined)? In contrast, EC2 alone would have to save us lots of pains. You may say that my previous company was not technically great. Maybe. If that's the case, how many companies can be really better? I think the fundamental problem is that companies tend to underestimate the effort to build a world-class infrastructure from scratch as well as the opportunity cost due to loss of productivity. Netflix's leaders were truly wise by claiming that Netflix didn't want to work on undifferentiated heavy lifting early on.
- yalogin 8y agoCertification has nothing to do with DC right? It depends on the product. You have to get pci done even if you use AWS
- buboard 8y agoI dunno, hetzner and contabo sell boxes for $20/mo I still find it weird thaf lyft needs so much processing per ride, it just doesn’t sound efficient And a lot of ppl seem to find this normal. Its not, this could be money that pays the driver more
- somethoughts 8y agoI think what's interesting to know is of that 14 cents, how much of that 14 cents does Amazon keep as profits and how much goes directly to costs. Some analysis suggests AWS has 25% profit margin which means of that 14 cents, Amazon really only pockets ~2-3 cents per Lyft ride. Another interesting question is whether that profit margin includes the original capex of when AWS was not reporting the revenue/profit of AWS in its early years. There's also all sorts of creative accounting methods to hide capex such that I wouldn't be surprised if Amazon is selling at cost to buy marketshare in a way somewhat analogous to Uber subsidizes rides to gain early marketshare.
- encoderer 8y agoTheir payment processor also takes a big cut. This is just a cost of doing business.