18 ms·
Dropbox saved almost $75M over two years by moving out of AWS
- meritt 9y agoWhile S3 storage costs are fairly reasonable, request costs are unrelated to file size, which makes it an absolutely terrible choice for filesystem-like activities. I say this from experience: S3 PUT requests make up a very significant portion of our AWS bill as we write a massive number of very small files to S3. Does anyone know of a middle-layer solution that automatically concatenates files into larger chunks (similar to HDFS) prior to persisting to S3 and for retrieval utilizes HTTP Range requests to fetch the individual file? I'm going to build it if it doesn't exist. 100MB chunks would reduce our S3 PUT costs by ~1000x.
- deleted 9y ago[deleted]
- toomuchtodo 9y agoCouldn’t find anything from a google search, but seems you could buffer files for writes, tar them, PUT the tar, and then use the tar index to perform range reads against the tar file. Edit: “dar”, “zip”, or “dump/restore” might be able to do this: https://serverfault.com/questions/59795/is-there-a-smarter-tar-or-cpio-out-there-for-efficiently-retrieving-a-file-store https://serverfault.com/questions/59795/is-there-a-smarter-t... (will create an archive with an index file for random access, vs tar streaming requiring reading the entire file) If you come up with another solution, I’d love to hear it!
- shortj 9y agoEntirely untested and just spitballing... use Firehose to concat a bunch of small files and store based on size or time passage as a larger file to s3. Then use Lambda transform in firehose to index the byte ranges for each record (file)? (Biggest q in the process would be here... you might have to index after it hits s3) S3 also allows the Range request header out of the box as well. Interesting problem, and I can totally related with s3 put costs be astronomical.
- lars_francke 9y agoHDFS does no such thing. If you store a million small files it'll store a million small files on disk as well.
- dvfjsdhgfv 9y agoI think the parent might mean HDF5.
- deleted 9y ago[deleted]
- idunno246 9y agoI POCd dynamodb(+api gateway+lambda for reads) for a project that was tons of tiny files dominated by puts. was a pretty trivial switch(writes was one extra line of code, reads was like 10 total lines), estimated cost savings was large(though we didnt fully calculate it). it didnt exist at the time, but i think lambda@edge might work really well for reads.
- sigil 9y agocperciva’s tarsnap server implements a log structured filesystem on S3. [1] This enables bundling of multiple writes into large objects, with the explicit goal of reducing per-request costs. It’s not open source, but maybe some of the high level ideas can be cribbed? [1] http://www.daemonology.net/blog/2008-12-14-how-tarsnap-uses-aws.html http://www.daemonology.net/blog/2008-12-14-how-tarsnap-uses-...
- RantyDave 9y agoDunno if it would work in your case but ... I've had fun mounting a ZFS file system using an entire EFS volume as the "physical" back end. You have to give up on writes actually being on disk when you think they are, but the payback is that's it's really kinda fast.
- daviesliu 9y agoMy friend also got a real big bill for S3 PUT requests, they are just too expensive. When we built JuiceFS [1], a POSIX filesystem on top of S3, there is feature in the roadmap to combine multiple small files together before uploading into S3 (each file is a slice of chunks in S3). We will let you know when it's ready. [1] https://juicefs.io/ https://juicefs.io/
- idunno246 9y agoheh, when s3 first came out they didnt have per request charges. then someone built a fuse fs on top of s3, and adding those charges was largely to convince people not to use s3 as a filesystem
- dvfjsdhgfv 9y agoDo yourself a favor and use a Hetzner Storage Box: https://www.hetzner.com/storage-box https://www.hetzner.com/storage-box Even if you don't include any extra costs, just keeping 10 TB on Hetzner will cost you 40€ max, whereas with S3 it will cost you at least $230.
- blfr 9y agoA single Hetzner box is not really comparable to S3 in terms of durability and availability.
- dvfjsdhgfv 9y agoMy experience is that they go down every 2-3 years, usually the problem is with the hard rive, occasionally with some other component. Under normal circumstances it takes them up to one hour to replace the hard drive, and then the array gets rebuilt for a few hours with the system being online. That means something like 2 hours downtime every 2-3 years, that's quite acceptable for my usage scenarios. In terms of money spent during that time, the difference is enormous.
- meritt 9y agoLike most businesses on a cloud provider, we're fairly locked-in to AWS without major changes. Going offsite would replace S3 PUT costs with significant bandwidth costs.
- nolite 9y agoDepends on how you’re writing. You could send everything to a kinesis or firehouse stream and use that to batch inputs, or you could write to an Elastic File System and batch there with a scheduled function
- Someone1234 9y agoThere's two good reasons to use a service like AWS: - You're too small for efficient economies of scale on your own equipment (i.e. AWS is cheaper when considering total cost of ownership). - You need to scale rapidly to meet demand The second one is largely a data issue, if you have enough historical data on your customers, their habits, usage, and so on then scaling becomes predictable and even when it isn't you could offload only part of your infrastructure to a cloud vendor. What's interesting is that several companies that I know which rely on AWS/Azure/et al aren't on it for either of the two above stated "good" reasons. They are large businesses and do almost no automated scaling. They're on it for what can only be described as internal political limitations, meaning that they are on these services to remove the politics of technology infrastructure, one less manager at the top, a shorter chain of communications, an external party to blame when something does go wrong, and issues like HR/benefits/etc for infrastructure employees is outside the scope. In effect they view themselves as "not a technology company" so look to employ the fewest technology employees as they can. Even in cases where technology is paramount to their success. It is very interesting to watch, and I'm not even claiming they're "wrong" to handle their infrastructure this way, just that it is hard to quantify the exact reasoning for it.
- rocky1138 9y agoIt's also easy to hire people who have AWS experience. It's getting harder and harder these days to find anyone who has actually seen the inside of a data centre.
- Bombthecat 9y agoThe lock in is even worse than people think. Now more and more companies are locked in in Amazon. Hard to find a good old Datacenter Admin.
- FreedomWarrior 9y agoAs an "old-school" sysadmin I have the opposite view: it's difficult to find jobs that don't require AWS these days. I know perfectly well how to provision and scale a large infrastructure and can give you 99,999% availability in any application, BGP if needed. Yet no one is interested in that. Sure I can write Ansible scripts and Terraform policies, but it's a miniscule part of my skillset and doing it on AWS is just boring compared to building the backend that powers it.
- hayd 9y agoDo we know how much they were spending on AWS before? Not that $75m isn't a lot but it would be interesting as a %!
- mdasen 9y ago> which reduced spending on “our third-party datacenter service provider” by $92.5 million offset by increased expenses of $53 million for its own data centers. They reduced spending on AWS by $92.5M, but still store 10% of data there so ~$103M? Looking at the $53M cost for their data center vs the $92.5M decrease to the "third-party datacenter", they saved around 43% moving to their own data centers.
- jeffwilcox 9y agoData centers aren't cheap, so unless you have the economies of scale to offer R&D investment and stock-based compensation to your employees to build a modern cloud DC, good luck with that... done right you can save operating expenses, but it'll take a huge investment that would not scale for others.
- toomuchtodo 9y agoIf the revenue is there, it’s a no brainer to build out your own data center or acquire prebuilt space. Cloud provider margins become your cost savings (clearly, as evidenced by this article). You’re going to need infrastructure people regardless, whether it’s on-prem expertise or AWS/Azure. It’s disingenuous to inflate the compensation for datacenter employees as a boogie man, or to wave away a “modern cloud DC” as a Herculean undertaking. There rarely is any R&D required, and stock compensation isn’t always necessary.
- jeffwilcox 9y agoFair points, appreciate the insight. I think living in a tech bubble city clouds my thinking a bit much. The only real example I have good knowledge of (1 of 1 cases) is a tech friend who worked for a firm acquired by a company wanting to build out data centers, and in the end everyone left after their acquisition stock awards vested, before really adding value to the corporation and delivering on the promise, because the company did not have a "tech r&d" comp plan in place to keep the subject matter experts employed. In the end the project and the DC expansion fell short.
- toomuchtodo 9y agoI can appreciate that living in tech hot spots can tilt the understanding of certain domains. “Unknown unknowns” if you will. Most people I know who build datacenters don’t expect a tech R&D budget or outrageous compensation, as it’s general contracting, racking/stacking equipment, and infrastructure engineering (pxe booting, remote consoles, reimaging for OS and/over hypervisor, etc). Here’s a non-comprehensive list I generated on the fly yesterday of tech orgs who run their own datacenters or equipment in colos: https://news.ycombinator.com/item?id=16449973 https://news.ycombinator.com/item?id=16449973
- kesor 9y agoI'd rather see a figure of how much more money they MADE than how much they saved by doing something that is not directly related to making more money.
- z1mm32m4n 9y agoDo they still use AWS for compute, with their own data center just for storage? Or do they have their own infrastructure for compute? If I remember correctly, they noticed they could dust costs significantly in storage compared to S3 because the vast majority of files in your Dropbox folder just sit there and hardly need to be read or written again (photos, etc.). If they’re not using AWS for compute, it’d be interesting to see what sort of similar reasoning they have for why the costs are cheaper in house.
- maccam94 9y agoMoving data out of AWS and using EC2 for baseline load are both more expensive than using physical servers co-located with the storage servers.
- vidarh 9y ago> If I remember correctly, they noticed they could dust costs significantly in storage compared to S3 because the vast majority of files in your Dropbox folder just sit there and hardly need to be read or written again (photos, etc.). Bandwidth costs alone for S3 are high enough that I've set up storage systems for clients that used S3 for durability but put it behind huge caching proxies that held a fairly substantial proportion of their total data, where the reduction in S3 storage paid for the cache setup several times over. Storage costs are harder to cut, but yes, S3 is typically expensive there too, but it's also riskier to do your own because of the risk of data loss. At Dropbox's scale it'll be worth it, but letting Amazon worry about durability is one of the few cases where I'll happily recommend AWS despite the high costs.
- anubhavmishra 9y agoI feel after getting to a certain scale moving to your own data center is a very smart choice. Its all about how well you can run your data center there after. So, good for them for scoping it out and saving money.
- romanovcode 9y agoIf it becomes cheaper to pay Sysadmins then it is always a good choice to move to your own.
- ChuckMcM 9y agoThis isn't surprising. At Blekko where I ran operations we did the math not once but twice (once when we got our series C to show that it made sense, and once when we got acquired by IBM to show why moving it into Softlayer was going to be too expensive). If you can build reasonable management tools and you need more than about 300 servers and a bunch of storage, cost wise its a win to do it on your own. The best win is to build out in a data center with a 10G pipe into Amazon's network so that you can spin up AWS only on peaks or while you are waiting to bring your own stuff up. That gives you the best of both worlds.
- fizwhiz 9y agoSo, rent the spike a la hybrid cloud?
- vidarh 9y agoYes. The beauty of it is that building in the capability makes hosting your own even more cost competitive. Because while you might have to plan for 2x or even 3x your average to handle normal peaks if you don't have anywhere else to send traffic, if you can spin up cloud instances to handle spikes you can plan for far higher utilization rates for your own equipment.
- danmaz74 9y agoMakes a lot of sense for the customer. I just wonder what Amazon could try to do to prevent this model becoming prevalent.
- sah2ed 9y agoContrary to what you think, they activately encourage this style of usage. AWS has a product called Direct Connect [0] to reduce bandwidth costs between AWS and your infrastructure. [0] https://aws.amazon.com/directconnect/ https://aws.amazon.com/directconnect/
- 9y ago
- mkagenius 9y agoIt’s understandable that lot of companies stick with AWS only because of inertia (read laziness) when they will be better off it.
- z3t4 9y agoThe difference between self hosting and running in "the cloud" is that when there's an error in "the cloud", you can't do anything about it ... But full stack debugging can even make a greybeard cry.
- biocomputation 9y agoFull stack debugging + shaders has made this greybeard cry.
- draw_down 9y agoThey’re just computers, you can do all the same stuff. Such as... debugging.
- closeparen 9y ago“The cloud” makes it easier to build an architecture with multiple failure domains. You don’t want to be debugging in an emergency, you want to be hitting the failover button.
- z3t4 9y agoI think hiring dedicated servers or VPS is a nice middle ground between self hosting and all out "cloud" cough lock-in. The cloud services do the failover and auto scaling for you, but often when something goes down it turns out they didn't even have any backups and all your data will be lost unless you actually backed it up yourself.
- encoderer 9y agoLast year I wrote about the AWS spend of my SaaS business, Cronitor [1]. I couldn't imagine building a service like this without modern cloud providers, but it is no wonder to me why AWS generates all of Amazon's profit. Essentially our migration over the years looks like: 1) Moving to EC2 from a VPS, The Ec2 instance with same specs is notably slower and you need to add a second instance where one worked before. 2) Moving to a managed service like RDS after running the service on Ec2, managed service with same specs is notably slower and you need a second instance where one worked before. In the end, it's worth it, in the RDS example you're getting millisecond replication times, point-in-time database recovery, hot failover, etc. But still, it would be great just once if you got the same performance from an RDS 2xl as you'd get running your own DB on a 2xl of your own. [1] https://blog.cronitor.io/the-aws-spend-of-a-saas-side-business-30bd5dbd91b https://blog.cronitor.io/the-aws-spend-of-a-saas-side-busine...
- betterworldb 9y ago> it is no wonder to me why AWS generates all of Amazon's profit. Going out on a limb here but I'm guessing that Amazon makes a good amount of profit from their online retail store.
- NightlyDev 9y agoUsing the cloud is a waste of money as long as you don't need to scale up and down, or change the infrastructure frequently. Bandwidth has always been insanely(!!) overpriced with most cloud providers.
- styfle 9y agoWhat kind of business doesn’t need to scale up or down? Maybe if there will only ever be one client connected 24/7 and that never changes but I can’t think of a real product that would avoid taking on more that one client.
- dmitriid 9y agoAWS (and other cloud provider) costs are non-uniform as you grow. It goes something like this (from when you small to when you're huge): - very/quite cheap - prohibitively expensive - much cheaper than building and maintaining own infra - prohibitively expensive/more expensive than building and maintaining own infra So various companies will end up at various stages of this cycle. Dropbox is big enough to be in the latest stage.
- nolite 9y agoSorry, what do you mean non-uniform? The costs are digressive with volume. If you have non-linearly increasing costs with linear workload, then you have a fundamental math problem with your architecture / business model
- _pdp_ 9y agoWell we removed all our servers from AWS and replaced them with lambda functions and dynamodb tables which resulted in 4.5 reduction in cost and increased performance by multiple factors. I suppose it all depends on what you are building and how you are building it. If you run servers I think it is no secret that AWS is not the cheapest option around.
- smt88 9y agoDid the switch to DynamoDB make a big cost difference? I've never really thought about the cost of Dynamo vs. RDS as being huge, but honestly I don't know. For most people (including every commercial project I've ever worked on), the time-saving and safety benefits of relational schemas was far greater than any theoretical infrastructure savings.
- _pdp_ 9y agoThe biggest part of the bill are the dynamodb indexes. The lambdas are a tiny fraction in comparison. That being said, you can avoid using as many index as we do. We did it because we wanted our lambda functions to be as pure and as micro as possible. If you are going to do joins and that sort of things, forget about dynamodb. It cannot do that and for a good reason. That being said, our architecture is mostly SPA so the lack of joins is solved at the client - there are just more calls to services client-side but the affect of that is still cheaper and faster product to run and maintain.
- bonesss 9y agoI think looking at something like DynamoDb as a wholesale replacement for anything that needs a relational schema is... well begging the question a little. DynamoDB as a hugely scalable limited-scope data-source in your app are likely where you'll find the optimal cost/scalability point. By using Dynamo for scalable read-heavy activity your let the rest of the app be barebones and 'KISS', preserve competencies & legacy code, and retain the benefits of relational schemas. Dynamos scaling then becomes your cloud-cost tuning mechanism. By way of example(s): if your editors all use RDS and you publish articles to DynamoDb you could be serving tens of millions of articles a day off a highly non-scalable CMS. If all your reporting functions pull from DynamoDB you could be serving a huge Enterprise post-merger while using the same payroll system as pre-merger. Shipping tracking posted/grabbed from Dynamo, purchasing logic on the best Perl code you could by in 1999 ;)
- bluedino 9y agoSo it only takes how many exabytes of storage for it to be cheaper than S3, by only < $40M? For all this talk about how expensive S3 would be for a filesystem, and how poorly suited AWS is for this kind of thing, Dropbox has seemed to make it work just fine.
- oneplane 9y agoI wonder what Dropbox is using. Of course they mention 'custom' software and possibly hardware, but I'm pretty sure it'll be some sort of Intel-based setup, maybe OCP, and Linux. The might be some hypervisory things going on, possibly Xen, maybe KVM. Then there will be some sort of firmware/OOB management as well, and perhaps some non-standard connectivity. Those are mostly given. What runs on top of it, that's what interests me. Some orchestration/ private cloud software, some container platform, and perhaps a storage platform, which/what are those for Dropbox? Would it be standard OpenStack, Kubernetes and the true in-hous stuff: dropbox object storage?
- maccam94 9y agoI don't think there's much public info about the compute infrastructure, but here's a blog post about the object storage system: https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/ https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pock...
- dzdt 9y agoAWS sells optionality. If you build your own data center, you are vulnerable to uncertain needs in a lot of ways. (1) your business scales at a different rate than you planned -- either faster or slower are problems! (2) you have traffic spikes, so you to over-provision. There is then a tradeoff doing it yourself: do you pay for infrastructure you barely ever use, or do you have reliability problems at peak traffic? (3) your business plans shift or pivot A big chunk of the Amazon price should be considered as providing flexibility in the future. It isn't fair to compare prices backwards-looking: where you know what were your actual needs and can compare what it would have cost to meet those by AWS vs in house. The valid comparison is forward looking: what will it cost to meet needs over an uncertain variety of scenarios by AWS compared to in-house. The corallary of this is, for a well-established business with predictable needs, going in-house will probably be cheaper. But for a growing or changing or inherently unpredictable business, the flexibility AWS sells makes more sense!
- chiefalchemist 9y agoPut another way: Would Dropbox have gotten off the ground and progressed as it did without AWS? Probably not. Outgrowing AWS is a great problem to have.
- hashkb 9y agoNot to mention they basically resold a single AWS feature. All they had to do was rebuild S3 (and ec2).
- deleted 9y ago[deleted]
- coherentpony 9y agoI think that's a little disingenuous. Dropbox provides: 1. a nice web interface; 2. synchronisation; 3. an easy-to-understand pricing model; 4. tightly integrated desktop support (at least for OS X).
- patsplat 9y agoDidn't Zynga report the same savings prior to collapsing? Is reporting massive savings on an infrastructure project a signal of softening demand and a weak product pipeline?
- late2part 9y agoWot Mate? You think when a company announces they are saving money on costs, lowering COGS and increasing margin by cutting costs by investing in a lower cost, higher performance solution, it's a signal of softening demand and weak product pipeline? There's a kernel of truth in what you're saying, perhaps. Any growing company has to choose between devoting resources (engineers, cash, executive overhead) between different projects. Alot of the time companies expand by capturing market share through adding features and expanding their TAM. But it's foolish to suggest that lowering COGS and increasing margin means a company is in trouble. Let's say your company is now making $1B/year (Dropbox) and you can choose between adding 3% more revenue or lowering your costs by 7%. In most cases, for your IPO, you'll get a higher aggregate shareholder value by cutting the costs and increasing your margin.
- greenyoda 9y agoAlso, once a company is no longer being funded by venture capital and has to finance its own growth out of its profits, those millions it saves by cutting infrastructure costs can be used to hire more staff, rent more office space, and all the other things required for growth.
- brudgers 9y agoFor Dropbox, it probably doesn't even matter that there was a cost savings to moving from AWS. There is strategic value in not taking a critical dependency on a potential competitor, and Amazon is an obvious potential competitor in the for consumer and small business online file storage. Using AWS meant that Amazon had IP logs for Dropbox's users. It had detailed information about Dropbox's business velocity. It had the ability to shape the customer experience of Dropbox's customers. In short, it made sense for Dropbox to move off of AWS for the same reasons that it makes sense for obvious Amazon competitors like Home Depot and Target not to build on AWS. However, unlike Home Depot or Target, the other major cloud providers, Google & Microsoft are also potential competitors in the file storage service market.
- guitarbill 9y ago> Using AWS meant that Amazon had IP logs for Dropbox's users. It had detailed information about Dropbox's business velocity. That doesn't seem fair to say, without proof. Why would they risk all the other business they do, plus GovCloud, etc, to spy on a company? Are AWS and Dropbox even competitors? I don't think that assertion holds at all.
- m3kw9 9y agoThe cost of switching including? What’s the percentage of saving?
- zenonu 9y agoWhat would be the cost difference moving to Google Cloud or other cloud provider? The reason why I'm asking is that the move itself requires a re-consideration of all production configuration. The indicated savings assumes that Dropbox was using Amazon 100% efficiently to begin with.
- sah2ed 9y agoFrom reviewing the thread's discussion, it looks like businesses turn to cloud-based infrastructure for a number of reasons: 1. To outsource non-core activities to experts and reduce risk, for firms that see IT as a cost center. A cost-cutting measure. 2. To provide dynamic capacity for mature businesses that experience anticipated workloads that are short-lived (seasonal or computational needs). A cost saving measure. 3. To provide dynamic capacity for new ventures growing in popularity i.e. fluctuating capacity requirements. This saves on large upfront infrastructure costs when long-term viability hasn't been established. A risk management measure. 4. To be described as "innovative" because peers are doing it, for firms that see IT as a revenue center (in industries that view such investments as a source of differentiation). A form of virtue signalling. 5. To make the accounts look good to investors by accounting for it as OPEX instead of CAPEX [0]. A seemingly irrational but valid reason. High OPEX numbers are easier to justify and more importantly, can be pared down with less friction than CAPEX, if things go south from intense competition for instance. Another risk management measure. [0] https://news.ycombinator.com/item?id=16458863 https://news.ycombinator.com/item?id=16458863
- tedivm 9y agoThis doesn't surprise me at all, considering storage is what Dropbox does and AWS has not really been competing on storage costs lately (as much as it's great when instances get cheaper, EBS has been 10 cents a gb/month for awhile now which is really problematic considering most of the new instance types are EBS backed).
- njwi332 9y agoNotably, the cost of doing so was over 50 million. So it's perhaps not realistic for majority of companies using AWS that are much smaller than Dropbox.
- justicezyx 9y agoThis is a success story for both aws and Dropbox.
- mankash666 9y agoThis is actually great P.R for AWS. Dropbox relied on AWS primarily for a decade, focusing on features and growth. Only when cost cutting became relevant to valuation, i.e. pre-IPO, they decided to invest in their own infrastructure. Put another way, use the cloud unless your valuation depends on cutting infrastructure operation costs
- pankajdoharey 9y agoThe article lacks a detailed analysis of how it saved so much money, there must be other stories like this when Cloud hosting doesnt make sense for really high volume data operations. The cost savings come purely out of large volume or is there something else to it?
- WheelsAtLarge 9y agoYes, it's cheaper but that's because Dropbox needs lots and lots of infrastructure and they can hire people to run it. At some point, if you have the capacity, it cheaper to build than to outsource. Dropbox found that point. Smaller companies don't have the option to hire a whole IT team.