10 ms·
Why Dropbox decided to build its own infrastructure and network
- swampthinker 9y agoI'll be 100% honest, I didn't realize Dropbox was still on AWS. You figure at a certain scale it makes more sense to run your own solution.
- rizjoj 9y agoand Netflix?
- eikenberry 9y agoThey have a completely different use case that benefits from being able to scale up and down quickly. I don't see them moving away from AWS anytime soon.
- kyledrake 9y agoThey run BGP and do an enormous amount of IX peering. https://www.peeringdb.com/asn/2906 https://www.peeringdb.com/asn/2906 AWS doesn't even let you do BGP. If you want to use them for a CDN, you're monopolized into their network and their blend of ridiculously overpriced bandwidth. They might be using AWS for the canonical store of data, but the sanity of using "the cloud" goes out the window the second you need to ship a lot of traffic to the public internet.
- pravinva 9y agoWhat are you on about? AWS practically requires BGP. Static routing is rarely the best choice when you set up DX. Netflix rolls its own CDNs because Cloudfront can't meet all their demands, yet.
- jimktrains2 9y ago> heir blend of ridiculously overpriced bandwidth. For someone not at netflix's level, can you recommend somewhere with significantly cheaper bandwidth.
- lsjdfkljdfwkwdf 9y agoAt that traffic volume Netflix has more negotiation power with ISPs
- toomuchtodo 9y agoAny colo or dedicated server provider.
- jimktrains2 9y agoAt a CoLo I have the rack, managment, and hardware overhead too, though? Also fewer PoP. EDIT: I guess I was talking about CloudFront specifically, not EC2. EC2 is pretty pricey.
- toomuchtodo 9y agoPrice compare various CDNs. It's a commodity business. Fastly comes to mind, but we had quite a few outages with them at my last org.
- thezilch 9y agoPretty much anywhere.
- corford 9y agoPretty much anyone... Hetzner, Linode, Choopa
- jimktrains2 9y agoFor those I need to run a VPS, which also costs money and has fewer PoP and more mangment overhead. EDIT: I guess I was talking about CloudFront specifically, not EC2. EC2 is pretty pricey.
- lsjdfkljdfwkwdf 9y agoName a CDN (excluding DDoS mitigation) that offers useful BGP service to customers? Even some CDNs are not quite good at running BGP
- _pmf_ 9y ago> you're monopolized into their network and their blend of ridiculously overpriced bandwidth. Wouldn't a customer the size of Netflix be granted more liberty in this regard?
- andyfleming 9y agoTo be fair, Netflix only runs some of their infrastructure on AWS. Basically, if I understand correctly, their applications run on AWS and their delivery is through their own CDN built on top of various ISPs[1]. [1] https://media.netflix.com/en/company-blog/how-netflix-works-with-isps-around-the-globe-to-deliver-a-great-viewing-experience https://media.netflix.com/en/company-blog/how-netflix-works-...
- yeukhon 9y agoCorrect, but some is probably an understatment. They are pretty much run everyting on AWS except what you cited (e.g video processing and content delivery). Netflix is the biggest AWS user.
- scurvy 9y agoEverything on AWS except their largest, most critical piece of infrastructure that needs to be the highest performing. That says it all right there.
- erikbye 9y agohttps://www.networkworld.com/article/3037428/cloud-computing/netflix-is-not-really-all-in-on-amazon-s-cloud.html https://www.networkworld.com/article/3037428/cloud-computing... And, > We rely on the cloud for all of our scalable computing and storage needs — our business logic, distributed databases and big data processing/analytics, recommendations, transcoding, and hundreds of other functions that make up the Netflix application. Video is delivered through Netflix Open Connect, our content delivery network that is distributed globally to efficiently deliver our bits to members’ devices. Source: https://media.netflix.com/en/company-blog/completing-the-netflix-cloud-migration https://media.netflix.com/en/company-blog/completing-the-net... Netflix still operates its own data centers and CDN to deliver video. This is not done on AWS.
- yeukhon 9y agoIsn't that what I said above?
- dandr01d 9y agoThis article isn't an announcement - it's a "Why we did it". They started transitioning years ago.
- praneshp 9y ago(Dropbox employee) I don't think this article is news. The editorialized title in HN is unfortunate and I wish a mod can change it. The article talks about the reasons Dropbox decided to move 500PB storage into our own data centers. Also like the article mentions, "They still use AWS for some workloads". Edit: Here is another HN submission of the same piece with the correct headline. https://news.ycombinator.com/item?id=15257331 https://news.ycombinator.com/item?id=15257331
- xbmcuser 9y agoYeah and that's why it amazes me that Netflix uses Amazon a straight competitor that uses the money it makes from Netflix to compete with Amazon prime.
- mulmen 9y agoI hear this line of thinking a lot but I don't really buy it. You could just as easily say it's amazing that Amazon allows a competitor like Netflix to run on it's platform. The reality is that relationship is more complex than that.
- kinkrtyavimoodh 9y agoNetflix is going to exist regardless of whether Amazon lets them run on AWS. Amazon letting them run on AWS is brilliant as they get a piece of Netflix's pie. So even if Netflix beats Amazon Prime, Amazon gets to dip into their pot via revenues from AWS. It's a great hedging of bets.
- peanutgallery27 9y agoThey're totally separate businesses. The people that treat Netflix like their customer aren't the same people making strategic decisions about Amazon Prime. Amazon is a ridiculously large company.
- siculars 9y agoThey are not totally separate when the P&L roll up on the same spreadsheet.
- user5994461 9y agoThat depends on how the Amazon business is organized. It's possible that the P&L never join on the same spreadsheet.
- ryacko 9y ago
- bsenftner 9y agoPlease, how fucking stupid. Of course they moved as much as possible off AWS, and only use AWS for new and experimental services. The markup of cloud servers is immoral, and the brainwashing that cloud services are a value at all is a testimony to how many people simply do not do the math.
- StreamBright 9y agoI am not sure what you are talking about. They moved out the 500PB data storage because ______at that scale______ it is cheaper to run it on your own hardware. This implies few things like having engineers with skills that are really not easy to find for starting and a bunch of other challenges that you need to solve.
- epa 9y agoThis means their EBITDA now probably shows profitability (servers they own are amortizable [the A in EBITDA], where are AWS is expensed.) Edit: Amortization/Depreciation can <i>generally</i> be used interchangeably.
- gordon_freeman 9y agoGood point. I always wonder how such accounting changes truly reflect the financial performance of the company. On one hand it's a lot of upfront cost to Dropbox with high risk too but all that won't show up completely if this cost is amortizable over the long run.
- scirocco 9y agoWith HPE Flexible Capacity they only pay for the HW they consume. Cloud-model on premise. https://www.hpe.com/emea_europe/en/services/flexible-capacity.html https://www.hpe.com/emea_europe/en/services/flexible-capacit...
- lobsterloga 9y agoMaybe but OpEx is tax deductible. CapEx is not.
- CodeWriter23 9y agoCapEx is deductible over the amortization period.
- qaq 9y agoI am pretty sure if they desire they can convert CapEx into OpEx.
- toomuchtodo 9y agoSure, if they sell the storage system to someone and arrange a leaseback.
- mr_tristan 9y agoUm, what is this crap? When did they actually move? How long did it take? The title of this HN link leads you to believe this is recent, but the article makes it seem like a multi-year effort, that in fact, could have finished a long time ago.
- dandr01d 9y agoI agree. The title is misleading and should be changed.
- mr_tristan 9y agoBut the blog itself lacks a lot of information. The timeline is really, really significant here. Did they initiate this back in 2013? 2015? My company has seen a remarkable shift in trust for online services change the last two years. Was Dropbox ahead of this curve? Were they someone who brought this change in mentality around? This blog article is crap because without a basic sense of _when_ it could be that they were riding the coattails of other industry leaders. Or they were leading the charge.
- manigandham 9y agoIt took several years and they finished early last year, much better article on Wired: https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama... TechCrunch is only useful for headlines today, go elsewhere for actual content.
- mr_tristan 9y agoThis article sounds like a PR puff piece. If you downvote me, please explain why this blog article actually means anything to you.
- GuiA 9y agoI am the CEO of a 3 person startup, and this piece has made me realize how critical it is to own your infrastructure if you truly want to disrupt and dominate the industry. I am making our engineers build our own data center at once, and am now hiring for Lead Data Center Engineers (15+ years AWS experience required). :)
- ChicagoDave 9y agoI had a chance to compare all of the file services a few years ago in a large corporation and I thought out of all of them, the DropBox engineers were the strongest. Microsoft second.
- qaq 9y agoPeople making sane choice very refreshing. I wish my company would make same decision.
- mrb 9y agoMake the sane choice of applying to work for Dropbox =:)
- uiri 9y agoCan we remove /amp/ from the end of the URL? Some people still use HN on a laptop or a desktop. Also, as mentioned elsewhere in the comments here, it should be retitled to "Why Dropbox decided to drop AWS"
- driverdan 9y agoI'm against Google hosted amp but what's the issue with self hosted? This page is so much nicer than the overly busy normal TC site.
- mulmen 9y agoThe problem is you making that decision for me.
- rev_bird 9y agoWait, but using the non-AMP version is a decision too. Why do you get to make that one instead? Shouldn't the person posting the link get some discretion?
- mulmen 9y agoPrinciple of least surprise. techcrunch.com has a default experience, amp is secondary. Since we can't all read each others minds the default behavior should be to use the ...default.
- shevy 9y agoBut we did not have AMP until recently. So OTHERS made a new decision for YOU - as the visitor.
- interfixus 9y agoSeparate URLs for separate devices? Talk about regression!
- SilverSlash 9y ago
- tcptraceroute 9y agoThe title here isn't quite accurate. Dropbox has moved user data from S3 to its own colocated data centers over the past few years, and is also doing compute in those data centers too. The compute actually existed for quite a long time - in the past you'd be talking to a Dropbox run server which would connect back to S3 to retrieve data. Dropbox is definitely still an AWS customer, just not a major S3 or EC2 customer anymore. For example, all transactional email uses SES, and DNS is hosted on Route 53.
- paxy 9y agoThey did this a while ago. Here's an article from last year with a lot more detail - https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...
- kibwen 9y agoThe article is dated today, but isn't this news from a few years ago? https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/ https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...
- antoncohen 9y agoDropbox hasn't dropped AWS, they moved things off AWS as it made sense to. The article is talking about two things, the move of file storage and a network backbone. Neither of which were done recently. The file storage move from S3 to Magic Pocket is detailed in these blog posts: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure/ https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr... https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/ https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pock... https://blogs.dropbox.com/tech/2016/07/pocket-watch/ https://blogs.dropbox.com/tech/2016/07/pocket-watch/ The network backbone is talked about here: https://blogs.dropbox.com/tech/2017/09/infrastructure-update-evolution-of-the-dropbox-backbone-network/ https://blogs.dropbox.com/tech/2017/09/infrastructure-update...
- codehusker 9y agoFrom the last link, in 2016 "Proxy-Stack deployed at our edge PoPs to terminate SSL connections closer to our users". Reminds me of the infamous "SSL added and removed here! :^)". Does anyone know if there are any additional protections from snooping on their internal network? edit: phrasing
- antoncohen 9y agoI believe they mean bringing the SSL/TLS handshake closer to the user is beneficial, it doesn't mean the proxy doesn't use SSL/TLS to its upstreams. In this post they say specifically it is about the handshake: https://blogs.dropbox.com/tech/2017/06/evolution-of-dropboxs-edge-network/ https://blogs.dropbox.com/tech/2017/06/evolution-of-dropboxs... I don't know if they use SSL/TLS to their upstreams, I'm just saying terminating in at the edge doesn't mean that is the end of all SSL/TLS. It is totally normal to terminate SSL/TLS at the edge, pretty much anyone using an HTTPS load balancer or CDN does it, but the LB or CDN can still use SSL/TLS to the upstreams and verify certificates of upstreams.
- lwf 9y agoEarly SSL/TLS termination is to reduce latency; the longer-lived connections from PoPs to Dropbox datacenters is over a TLS 1.2 connection with PFS. See an earlier blog post[1]: > We use TLS 1.2 and a PFS cipher suite at both our origin data centers and proxies. Additionally, we’ve enabled upstream certificate validation and certificate pinning on our proxy servers. This helps ensure that the edge proxy server knows it’s talking to our upstream server, and not someone attempting a man-in-the-middle attack. (N.B.: I work on security at Dropbox, and consulted on this design) [1]: https://blogs.dropbox.com/tech/2016/11/infrastructure-update-pushing-the-edges-of-our-global-performance/ https://blogs.dropbox.com/tech/2016/11/infrastructure-update...
- justonepost 9y agoIs 500 petabytes really that epic? 1500 tapes? I could probably stuff that in the back of my minivan.
- jsemrau 9y agoGreat comment for 2017. I would like to see the 1987 version and the 2037 version of it.
- jerf 9y agoI recently misplaced one of my 32GB SD Micro cards I was using to have an alternate image for my Raspberry Pi. So I just bought another one for the price of a half-decent lunch. And I did have a flashback to my first hard drive, all 220 glorious megabytes of it, and a great deal more expensive than "a decent lunch", and get a sweet hit of that yeah... I am in the future, aren't I? feeling.
- Johnny555 9y agoTo fit 500 petabytes on 1500 tapes, you'd need to put 333TB on a tape. Do any currently available tape formats support that much data on a single tape? Granted, I stopped paying attention to tape drive capacities a while ago, but the upcoming LTO-8 standard will "only" support 13TB/tape, so you'd need 38,000 of them to hold 500 PB. I've seen some higher density announcements in the 200 - 300TB/tape range, but couldn't find any products available. AWS does offer a "Snowmobile" [1] product, that can hold 100PB on disks in a tractor trailer. [1] https://aws.amazon.com/blogs/aws/aws-snowmobile-move-exabytes-of-data-to-the-cloud-in-weeks/ https://aws.amazon.com/blogs/aws/aws-snowmobile-move-exabyte...
- zuzun 9y ago"I have a few qualms with building storage data centers. For a Linux user, you can already build such a system yourself quite trivially by putting 1500 tapes into the back of your minivan and then mounting them locally via curlminivantapefs."
- jason_pomerleau 9y agoWhat will you tell your customers, exactly, while you go on a road trip with their data? And what happened to the data between the time you load Tape 0001 and Tape 1500? Moving 500PB on a live application is not a trivial task.
- goptimize 9y ago"We’re talking about a company that had 1500 employees, with just around a dozen on the infrastructure team" - what the rest 99% of the company is doing, marketing?
- vacri 9y agoI was also thinking that those numbers are really weird, for a company which does infrastructure(fileservers)-in-the-cloud. Obviously some devs doing software for various platforms, business-side folks and so forth. But 1500:12 is a really weird ratio, given the core function of the business. If they're not spending on infra staff... what staff are they spending on?
- siddboots 9y agoYes, marketing. Also software development, web development, design, customer services, corporate services, support, sales, PR/comms, human resources, finance & accounting, legal, and so on. 1500 sounds like a lot, but keep in mind how many customers they have, and in how many countries they sell their services.
- maccam94 9y agoThe wording got mixed up a bit. "Infrastructure" was a much larger organization, but there was an infrastructure team of about 12 people who worked on Magic Pocket at the time. https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/ https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pock...
- joelbondurant 9y agoLack of IT incompetence is exceedingly rare in the antihardware industry.
- shevy 9y ago> While cost is always something that we consider, our > goal with our network expansion was to improve > performance, reliability, flexibility and control > for our users — which we have succeeded in doing,” > a company spokesperson told TechCrunch. Who believes this? It was, easily noticable by everyone, wanting to reduce the cost. Which is fine, everyone does so, so why not admit that it was the primary impetus? I would not want to outsource control over any larger company that I were to run to other, even bigger companies.
- maccam94 9y agoPerformance and reliability definitely affect subscription rates. Revenue growth is actually more important than decreasing costs (as long as your costs don't start increasing disproportionately).
- late2part 9y agoI can see how you would think it is only about cost. However, then you've had Amazon tell you "no" when you try to get more resources, and when you've had your business go offline for hours while you wait for news from Amazon, you may want more control.
- mc32 9y agoWhat does AWS do when someone moves 500PB of storage off of their systems? Do they sit idle till some other big customer comes along or is their growth so phenomenal that a 500PB move off doesn't even slow new (storage) deployments at all and just keep up their pace?
- late2part 9y agowith no actual facts, i'll speculate. let's assume they have 12 buildings at each region with 500 racks in each. 6k racks Let's assume that S3 is 1/4 of their infrastructure. Let's assume you can put 60 * 12TB disks in 4U and they use 52 racks. That's 136012TB per cabinet or 9.4PB per rack. 1500 racks at 9.4PB raw capacity with RF=2.1 so best case that's that's 1500 * 9.4 / 2.1 = 6700PB at a site. So 500PB would be 7% of the space. They're probably growing 5-15% per month like most hot companies now, so that's 1-2 month's growth. Of course these numbers are certainly off. But probably in the magnitudinal ballpark.
- mc32 9y agoMy guess (and it's a terrible guess) is this may be about 4mo growth... and they might take the opp to retire older lower capacity drives and just keeping up the build-out. But who knows...
- chis 9y agoIf you're ordering 500 petabytes you probably sign a contract with Amazon, and have to let them know in advance if you want out.
- mc32 9y agoThat's a good point --but they didn't start with 500PB, when they first went to AWS they probably only used a few PB to begin with and grew "organically" over the years. But likely still had some contractual stuff for this kind of scenario.
- ramshanker 9y ago
- iUsedToCode 9y agoDropbox is cool and all, but i hate their pricing model. I just want to keep about 50-100 GB there (i don't hoard stuff). I don't wanna pay $10 / month for that. At backblaze i pay less than $0.5 / month. I could pay quadruple, since Dropbox is a lot better service (and quite a different one, too). But not 20 times as much. I know that Dropbox doesn't care about my tiny dollars and all. But why not let customers pay for what they use? This "constant growth" bullshit is probably the reason they don't care.
- mgkimsal 9y agowhat bugged me about their pricing model (perhaps it's changed?) is that things people share with me count to my cap. I had a client share stuff with me via dropbox. So I signed up. I get "2 gig free!" Yay. Another client shares stuff with me. Then another. Then another. Then I'm at my 2gig cap with just other peoples' stuff. They all think it's great, because none of them are paying for it. I needed to pay money to support their free use of the service, and I declined. Just one more $x/month service I didn't need to get hooked to. Maybe the pricing model has changed what they count now?
- kungito 9y agoHow do you have "clients" and you can't afford 10$ a month to not have to think about it?
- mgkimsal 9y agoIt was largely the principal of the thing. It's yet another $x/month, on top of other services being paid for to help service/manage those clients. Example - PM tools. I'm already paying for project management tools - put the files there, damnit. "oh, but I like dragging to my dropbox!". Oh, and someone else uses box.com, and so on. Again, what gets my goat is partially the payment, but mostly the "it's free for them, but I end up needing to pay to accommodate their use of the service" angle. If they - other parties - were paying for it first, it would bother me less. But raving about how 'free' a tool is which I have to pay for to work with them (and they're using it precisely because it's free) bugs me. Getting sucked in to a few more $x/month things here and there 1) dilutes where stuff is supposed to live (wrt files) and 2) just becomes a drag on finances. A handful of services can end up going from several hundred dollars in to the thousands if you don't keep tabs on things. Maybe I'm an HN failure because saving several hundred dollars per year matters to me? I guess my skills must suck - most people can apparently rustle up $300/hr Rust/Go/Elixir project work simply by starting to formulate the idea that they're considering taking project work. I'm not that skilled.
- sandworm101 9y agoThree data centers "biult" by only the dozen people on the infrastructure team? Not possible. I wish articles like this wouldnt hype small teams where it is obvious that most all of the work was outsourced. It would seem that dropbox here was still operating as customer: ordering rather than physically biulding much of anything. Those dozen dropbox people were not running cables.
- iscoelho 9y agoThe article is unfortunately not very specific and I can not find much information with a quick search, but: It's very possible that they did this all on their own provided they rented cabinets/cages in an existing facility like Equinix, which is extremely common. Then they do not need to manage power/generators, fiber into the building, or any other data center necessities. It does not take very many people to do the wiring inside a cage, especially considering Dropbox has been doing this over the span of a few years. If you've ever been physically in a data center (personally visited a few Equinix facilities myself), it's mostly one person from a company working in a cage that is wiring everything. I've rarely seen multiple people do wiring, and that same person will come back every day thereafter to continue working until the job is done. What they are doing sounds entirely feasible. If money is no object in regards to equipment, with 3 data centers, I'd honestly say you only need 4 competent people to get the job done. 3 on-site. 1 remote/office. From there, the more the merrier. A guy to lift the equipment up as well is nice sometimes - can be pretty heavy!
- notyourday 9y ago(a) CAF ran out (b) Without CAF, GC and AWS pricing is ridiculous - somewhat akin to paying $9.99/lb for a Purdue chicken a day before the expiration (c) Without the scale of AWS and GC one can get vendor prices at Google/Amazon MFN + 10% any day or +5% after schmoozing.
- knodi 9y agoI'll save you a click. Cost savings.
- raides 9y agoI am curious on the hardware build out used for the storage nodes. Some of the major issues I have seen with all the appliances out there for storage are the following: 1. Network throughput on the appliance is fast but for an Enterprise level the 10gigE cards used become a bottleneck for transactions because of how the software hypervisor scales the data. 2. Power consumption of the appliances in a rack mount environment are too high and leave needed space that has to stay empty because of the facility power per rack limitations. 3. The software hypervisor scales the stack vertically and relies on the software to load balance horizontally. The performance in a high transactional environment becomes dependent on the software to scale instead of the natural horizontal distribution that can be setup on the hardware out of the box. Standard multi-purpose storage arrays scale horizontal with very little over head from traditional software storage management. I only found one company whose software does not force the stack to be vertical but they fail to meet a reasonable performance in network/power. Streaming petabytes of data to keep a dynamic constant (static overall storage requirement that changes it's data life cycle via retention rules) becomes very hard with premade hardware. Does anyone have any recommendations or has attempted a similar exodus from S3 that they can share?
- flamedoge 9y agodid Dropbox really only have 12 people in the infra team at the time?