4 ms·
I don't think the title is a fair representation of what's going on. Hear me out. The way it's phrased, it's like each ride directly translates into 14 more c
by SilasX 8y ago
I don't think the title is a fair representation of what's going on. Hear me out.
The way it's phrased, it's like each ride directly translates into 14 more cents for Amazon. But really, the figure comes from averaging Lyft's Amazon costs over Lyft's total rides.
Now, in some cases, that would be valid accounting, if all (or nearly all) of your costs (with respect to that service) are variable. For example, most of the cost of sending a letter is the truck/plane/employee, which are variable, so it makes sense to divide total letters over total truck etc costs.
But would Lyft pay twice as much to Amazon if they served twice as many rides? Would they have to double every AWS service they're buying? I don't think so. They might have to add more of one or two kinds of server/services, but most of that 14 cents figure comes from amortizing fixed costs, which will go down if they scale up further.
Am I off base here?
- raiyu 8y agoYou aren't off base as this is a snapshot in time so it's no way to know if those costs are linear and scale with more rides or not without seeing historical figures on AWS spend vs revenue/# of rides. But it could very likely be true that doubling the number of rides would have a potentially material impact on additional AWS spend for Lyft.
- mv4 8y agoTheir model should scale linearly, because everything can be clustered/sharded by geo so nicely, and the scope of a typical transaction is one passenger vs a few dozen potential drivers. That does not significantly change as they grow. Unlike someone like FB, with their many-to-many everything, Lyft/Uber is a data architect's dream.
- SilasX 8y agoThere's definitely something that has to scale linearly as they serve more rides; the question is how much of that 14 cents the linear-scaling stuff accounts for.
- mv4 8y agoI would think the bulk of it. Given their ride volume, they should be way past the fixed costs for stuff that has to be there before the 1st ride takes place. API load should be mostly linear as well. Back office, internal reporting, etc anything that deals with data in aggregate can be non-linear, but negligible in the grand scheme of things.
- SilasX 8y agoIn theory, yes, but you might be underestimating how much technical debt can be left behind in a scale-up. For all we know, there's a bunch of stuff that just hasn't been cleaned up. As other comments note, I can totally imagine someone saying "yeah, not worth the risk of breaking something as long as it only comes out to 14 cents a ride". Edit: And, for that matter, they may be using AWS for things other than serving rides, e.g. dev tools and internal service that scale with the number of office employees, not rides.
- mv4 8y agoYeah that $0.14 per ride may sound reasonable, until you realize (as someone pointed out earlier) that it's essentially a EC2 t3.xlarge for one whole hour. Now, imagine that your model requires a 4CPU, 16GB RAM server to handle just one ride per hour - is that really efficient?
- SilasX 8y agoOh, no, I agree it's unreasonable for what they're getting; I'm saying that it might seem small relative to the business priorities.
- solarkraft 8y agoIt didn't sound reasonable to me before, but you really gave me some perspective here. What the the hell are they doing? So much geo processing?
- mv4 8y ago
- hinkley 8y agoThe more customers I have the more shards I have to run. Managing 5 shards is not the same amount of work as managing 50. For one the surface area for cross shard communication goes up. What happens if London or Manhattan are too big for one shard? Multi-tenant providers have been wrestling with the Big Customer problem forever, and sharding is only a bandaid.
- jacurtis 8y agoOn a traditional model (ie building a datacenter or colocating in a datacenter) then I think what you are saying is 100% accurate. You would have fixed costs and variable costs which make it difficult to break overall costs into a "per-ride" expense. This is because, as you mentioned, some fixed costs in their current state may be able to handle higher volume than they are currently utilized on. Thus, as rides go up, not all costs would go up linearly. However, in the case of AWS and other cloud service providers, virtually all products and services are charged on a per-usage basis. Meaning that if the average Lyft ride requires 60 database api calls (say a 10 minute ride with location updates every 10 seconds = 60 location updates/db writes) then AWS charges you for 60 api calls. If the average ride takes 11 minutes, then it would take 66 api calls and so forth. In this case, this cost is a variable cost and maps directly to a specific ride. If that ride had not existed, then that cost would not have been incurred. Take a look at AWS pricing pages and you will see that all prices are broken down into very specific per-usage costs. Some services break this up very deeply, on ingress usage, egress usage, storage usage, api usage, etc all together calculate the cost that the service uses. Looking at the Lyft Case Study that AWS published (you can view here: https://aws.amazon.com/solutions/case-studies/lyft/ https://aws.amazon.com/solutions/case-studies/lyft/), it appears they primarily use Redshift, Kinesis, DynamoDB, and EC2/ECR services. All of these are heavily usage based. The least usage-based service they use would be their EC2 servers. Since a server is rarely running at max-capacity it means that one extra lyft ride doesn't necessarily mean another server. So the EC2 costs for that ride would be the same if it had or had not existed. But it looks like they are heavily reliant on using ECR to auto-scale their servers up and down based on usage. So they are probably running many clusters of smaller servers that move up and down as rides increase. This means that even their EC2 servers are fairly closely mapped to actual usage of their platform. During a slow period they will be running far fewer EC2 servers than during a peak period. So even in this case their costs are relatively variable. All in all, I would say that you actually could extrapolate the total cost of AWS on a per-ride basis, because that is the variable that effects their costs. If they had 1 less ride this month, their bill would ACTUALLY be 14 cents lower (or somewhat close to it). They do incur a direct cost with each individual ride. If there are fixed AWS costs in there, it probably accounts for less than 1 penny of that 14 cents. For example there are a few fixed costs if they are using AWS DirectConnect or AWS Private Link to connect to AWS Datacenters. But this accounts for probably a few thousand dollars a month at most and in total would have minimal effect on thier per-ride cost at the current scale that Lyft is at. edit: spelling
- hinkley 8y agoThis theory has unfortunately never panned out for me. To the extent that I get a little anxious about my employer getting too aggressive with PR. Organic growth always plays out better for the engineers in the short to medium term. IT's version of the Laws of Thermodynamics is that nearly every interesting action you can do programatically is going to take at least O(log n) time, where n is the amount of data you have (at rest, in flight, or both). If I have 16x as much traffic as I used to deal with, my load isn't 16x higher, it's 64x higher - if I'm fortunate and we've engineered for scale. But it could easily be three orders of magnitude higher (16164) if I'm not. For the 64x scenario I tell my boss to buy more hardware and I put medium to high priority tickets in the backlog to improve the worst cases. In the 3 orders situation, those tickets bump all of our other priorities, and there are more of them. Performance is now a feature. It sucks up a significant and very visible fraction of our engineering budget, which comes with its own sort of opportunity costs. If you try to scale vertically, we all know that except at the low end, buying a server that's twice as beefy costs far more than twice as much. If you go horizontal, there are plateaus where your network topology has to get more complex (hardware cost, maintenance cost, speed, pick two). Personally, I think the biggest lie in cloud computing is that 10G ethernet fixes all of your network topology problems (ie, it's treated as magic that you don't have to worry about). As disks and PCIe get faster over the next couple years I think that'll be back on people's radars.
- hexane360 8y agoIf your complexity is O(log n), an increase of 16x only increases your time by log(16)=1.2 (constant, not a multiple).
- naniwaduni 8y agoI'll say, I'm so used to seeing "log" denote either lg or ln (depending on whether the context is closer to maths or computing) that seeing the claim that log(16)=1.2 threw me for a loop. This despite it being fundamentally meaningless because of course it needs to be multiplied by some time-dimensioned value anyway.
- hinkley 8y ago