7 ms·
How Did Dropbox Scale To 175M Users? A Former Engineer Details The Early Days
- cldr 13y agoI think "replace the hard drive" is a terrible way to say it, since a hard drive (or SSD or SD or whatever) is exactly what you need to be able to use Dropbox.
- hobs 13y agoYeah it's definitely "replace the thumb drive"
- pedalpete 13y agoIn the long-term, I think you may look back at the 'replace the hard drive' statement as on par with Bill Gates 'a computer in every home'. If all of your apps save directly to Dropbox your 'hard-drive' and thumb-drive are no longer important, and in that way they are 'replacing' the hard-drive, even if not in technical terms, the sentiment is correct.
- wslh 13y agoIn the long-term, I think you may look back at the 'replace the hard drive' statement as on par with Bill Gates 'a computer in every home'. That's not possible, since your local storage will always be faster than a networked one.
- Dylan16807 13y agoAlways? Hard drives don't transfer at multi-gigabit speeds. Networks do.
- cldr 13y agoFor most users, their network speed is way way less than multi-gigabit. May or may not change soon.
- Dylan16807 13y agoI guess I should have just said 'gigabit', which is standard Ethernet speed and is roughly on par with hard drives. The point is that the network is plenty fast. Backbone and edge both. The only problem is a few isps.
- cityguy 13y agoErr, no. The SATA 3.0 spec is 6.0 Gb/s. Most NIC's might run 1Gb/s locally, but once you hit the WAN you're probably talking about 10 or 100Mbit. Network storage has always been slower than local, and until we get 10Gb to the home, will continue to be so.
- Dylan16807 13y agoHard drives can't saturate SATA. And many millions of people already have gigabit service. That's roughly as fast as most hard drives.
- wpietri 13y agoIt's totally possible. It doesn't have to be faster, just fast enough. For everybody's most important app, the browser, local storage is already basically irrelevant. That's trained people to be perfectly fine with little network delays. Things like Google Apps show that data and apps can easily live elsewhere and it's basically fine. In the long term, I think the only question is whether data+apps will live together with the app provider, like Google Docs, or whether data storage will stay generally separate, as DropBox would surely prefer. There will still be some of us who keep big local drives because we really want to know where our files are and control their movements. But for everybody else, local storage will just be cache. Which means whether it's spinning rust vs flash vs RAM is an optimization detail. And that decision will mainly be made by hardware product designers, not users. An interesting thing to think about in this context is how the long-term costs of storage are falling. In the last 20 years, RAM is ~4 orders of magnitude cheaper: http://www.jcmit.com/mem2013.htm http://www.jcmit.com/mem2013.htm Flash is an order of magnitude still cheaper for now. But if networks keep getting better and services like Dropbox get more solid, that may not matter. If you assume trends continue, then putting half a terabyte of RAM in a device may be cheap enough that the added complexity of flash just isn't worth it.
- minimaxir 13y agoThat assume you're "always-on" the Internet. Microsoft tried that assumption with the Xbox One, and we know how that went. :P
- cldr 13y agoOne of the points of the article was that storing data completely on the net, instead of synchronising local and remote versions, is not in their vision of the future because "users shouldn't have to worry whether they're online or not", so what you proposed is specifically what they're avoiding.
- bigiain 13y agoTurns out, Dropbox is _most_ useful to me on my devices with insufficient local storage - having access to all my (Dropboxed) files on my phone and iPad is _spectacularly_ useful. I have no doubt there's some very smart people at Dropbox (and Google and Microsoft amd JottaCloud and BTLabs and any number of established companies and startups I've never heard of) are working out how to cater to a new category of "cloud storage user" - people who _don't_ store cannonical versions of all their files/data on their own laptop/desktop, people with a smartphone and a tablet - but with no "main machine" as such. People who want to access all their files from any random shared, borrowed, or rented machine whenever they need a full sized screen and keyboard, but don't need that often enough to own such a machine.
- bdon 13y agoFrom the comments, a link to Rajiv's technical writeup: http://eranki.tumblr.com/post/27076431887/scaling-lessons-learned-at-dropbox-part-1 http://eranki.tumblr.com/post/27076431887/scaling-lessons-le... which covers some of the points in the TC article.
- eksith 13y agoMy only suggestion for choosing technology would be to pick lightweight things that are known to work and see a lot of use outside your company, or else be prepared to become the “primary contributor" to the project. This should be put on a header for anyone considering the sexy new thing to come out, be it Node.js, MongoDB or the bazillion JS frameworks etc... Not saying you should avoid these altogether; just be prepared to iron your own wrinkles if you ever intend to become popular.
- finkin1 13y agoHere's a link to the actual talk: http://www.ustream.tv/recorded/35654239 http://www.ustream.tv/recorded/35654239
- betageek 13y agoFrom https://www.dropbox.com/help/14/en https://www.dropbox.com/help/14/en "Dropbox's performance may start to decline when you store above 300,000 files." From personal experience replace "start to decline" with "not work at all." I don't have much faith in Dropbox's growth if they can't fix this.
- jamesaguilar 13y agoI think you are mistaken. Dropbox's growth will not be materially affected by an edge case like this. I doubt even one in a hundred people would want to store hundreds of thousands of files in their Dropbox.
- wikiburner 13y agoIf their long-term goal is to become the filesystem for all your devices, then I think you're mistaken. However, I'm sure Dropbox will be able to resolve this issue before it becomes more commonplace.
- bigiain 13y agobigiain% ls -R ~/ | wc -l 422365 So much for even storing my homedir in Dropbox... And they want to "replace the hard drive"? They've got a bit of work to do yet...
- brown9-2 13y agoCan anyone explain what this means? they always put lefts on values (like failed log-ins etc)
- oh_sigh 13y agoleft = y axis
- notJim 13y agoThis article is horribly written, seemingly by someone with no actual programming experience or knowledge, or even a basic command of English grammar. I mean, this passage reads like timecube (up until the very end where it becomes somewhat more cogent): Out of this came some learning such as, in terms of app specific metrics such as ease and ‘fooplroofness’. Plus, it emerged that “most graphs are useless”. Instead they built dashboards to analyze the performance; they always put lefts on values (like failed log-ins etc); and they kept some slack such as: extra queries were memcached, and delayed optimization of SQL queries.
- wikiburner 13y agoEither that or TechCrunch is closing in on the Turing Test.
- general_failure 13y agoI don't know what the big deal is scaling up. I have worked in a couple of "big scale" apps and it's really simple principles. At the end of the day, it's all about caching, powerful machines, caching, normal coding skillz, caching. And oh did I mention caching? I honestly don't think it's that big a deal to 'scale'. Programs like early wolf3d impress me more, much much more. And of course, things like the recent quine relay leave me demoralized.
- amix 13y agoDropbox does about 1 billion file updates pr. day across millions and millions of active users, petabytes of data and millions of devices. I think it's very ignorant to think that their problem is just solved by using powerful machines and caching...
- general_failure 13y agoOk. So can you give me a few pointers on topics to read up on how they might scale up? Everything I have read in this field is all about - more machines, split data into several machines aka sharing for nice map reduce queries etc. Genuine question not snarky. I have even read yahoo's fast website tricks - it's all really hacks and tricks, nothing mind blowing. What I mean is given some extra time most people can figure things out by some googling and trial and error. This is unlike wolf 3d. Unless you spend considerable time understaning math and graphics you cannot do it.
- smartwater 13y agoDropBox can't just easily Google an answer like you would with a Wordpress blog. Their bottleneck wouldn't be web based, so how they load their javascript or if they use image spriting is basically irrelevant. Scaling a website with exponential growth is very difficult. Fix one problem, two more pop up. It never ends. Oops just spent 50k on SSD's to find out it was a load balancer issue.
- general_failure 13y ago