43 ms·

Dropbox’s Exodus from the Amazon Cloud

671 points by moviuro 11y ago

jamwt 11y ago
Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.
- dikaiosune 11y ago
  The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? If the latter, I've heard recent versions have had some significant improvements -- has your team looked at that and thought that you would have been OK if you just waited, or was there a fundamental gap that GC improvements couldn't close? Could you comment more generally on what advantages Rust offered and where your team would like to see improvement? Are there portions of the Dropbox codebase where you'd love to use Rust but can't until <feature> RFCs/lands/hits stable? Are there portions where the decision to use Rust caused complications or problems?
  - jamwt 11y ago
    Good questions, let me try to tackle them one by one. > The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? The reasons for using rust were many, but memory was one of them. Primarily, for this particular project, the heap size is the issue. One of the games in this project is optimizing how little memory and compute you can use to manage 1GB (or 1PB) of data. We utilize lots of tricks like perfect hash tables, extensive bit-packing, etc. Lots of odd, custom, inline and cache-friendly data structures. We also keeps lots of things on the stack when we can to take pressure off the VM system. We do some lockfree object pooling stuff for big byte vectors, which are common allocations in a block storage system. It's much easier to do these particular kinds of optimizations using C++ or Rust. In addition to basic memory reasons, saving a bit of CPU was a useful secondary goal, and that goal has been achieved. The project also has a fair amount of FFI work with various C libraries, and a kernel component. Rust makes it very easy and zero-cost to work closely with those libraries/environments. For this project, pause times were not an issue. This isn't a particularly latency-sensitive service. We do have some other services where latency does matter, though, and we're considering Rust for those in the future. > Could you comment more generally on what advantages Rust offered and where your team would like to see improvement? The advantages of Rust are many. Really powerful abstractions, no null, no segfaults, no leaks, yet C-like performance and control over memory and you can use that whole C/C++ bag of optimization tricks. On the improvements side, we're in close contact with the Rust core team--they visit the office regularly and keep tabs on what we're doing. So no, we don't have a ton of things we need. They've been really great about helping us out when those things have sprung up. Our big ask right now is the same as everyone else's--improve compile times! > Are there portions where the decision to use Rust caused complications or problems? Well, Dropbox is mostly a golang shop on the backend, so Rust is a pretty different animal than everyone was used to. We also have a huge number of good libraries in golang that our small team had to create minimal equivalents for in Rust. So, the biggest challenge in using Rust at Dropbox has been that we were the first project! So we had a lot to do just to get started... The other complication is that there is a ton of good stuff that we want to use that's still being debated by the Rust team, and therefore marked unstable. As each release goes on, they stabilize these APIs, but it's sometimes a pain working around useful APIs that are marked unstable just because the dust hasn't settled yet within the core team. Having said that, we totally understand that they're being thoughtful about all this, because backwards compatibility implies a very serious long-term commitment to these decisions.
    - steveklabnik 11y ago
      > On the improvements side, we're in close contact with the Rust core team One small note here: this is something that we (Rust core team) are interested in doing generally, not just for Dropbox. If you use Rust in production, we want to hear from you! We're very interested in supporting production users.
    - anp 11y ago
      Thanks very much for the detailed and thoughtful answers! I've read before (somewhere, I think) that Dropbox effectively maintains a large internal "standard library" rather than relying on external open source efforts. How much does Magic Pocket rely on Rust's standard library and the crates.io ecosystem? Could you elaborate on how you ended up going in whichever direction you chose with regards to third-party open source code?
      - jamwt 11y ago
        We use 3rd parties for the "obvious" stuff. Like, we're not going to reinvent json serialization. But we typically don't use any 3rd party frameworks on the backend. So things like service management/discovery, rpc, error handling, monitoring, metadata storage, etc etc, are a big in-house stack. So, we use quite a few crates for the things it makes no sense to specialize in Dropbox-specific ways.
        dikaiosune 11y ago
        Cool. This might be getting into the weeds a bit, but are you still on rustc-serialize for json or are you trying to keep up with serde/serde_json? If you're using serde, are you on nightly? From your comment above I got the impression that only using stable features was very important, so I'm curious how your codebase implements/derives the serde traits.
        jamwt 11y ago
        We're on rustc-serialize. JSON is not really a part of our data pipeline, just our metrics pipeline. So the performance of the library is not especially critical.
- ignoramous 11y ago
  Please do a technical blog post on some of these: 1. Go vs Rust at Dropbox's scale and requirements. 2. Maintaining availability whilst moving from AWS to Diskotech and Magic Pocket. 3. Internals of Magic Pocket (file-system, storage engine, availability guarantees, scaling up and scaling out, compression details, load balancing, cloning etc) 4. Improvements in perf, stability, security, and cost. Thanks.
  - james_cowling 11y ago
    Yep we're going to get at least a few actual technical blog posts online in the coming month. We haven't got around to writing them yet tho so feel free to surface any requests :) Will most likely start with the following: 1. Overall architecture and internals. 2. Verification and projection mechanisms, how we keep data safe. 3. How we manage a large amount of hardware (hundreds of thousands of disks) with a small team, how we automatically deal with alerts etc. 4. A deep-dive into the Diskotech architecture and how we changed our storage layer to support SMR disks. Hopefully these will be of a sufficient level of detail. We certainly won't be getting into any details on cost but we're pretty open from a technical perspective. (Lemme just take a moment to say how great Backblaze's tech blog posts are btw.)
    - atYevP 11y ago
      Yev here -> Let me just say you're welcome and thanks for joining the party :D
    - wpietri 11y ago
      I would be especially interested to hear how a focus on actual customer experience was maintained during all of this deep technology work. One of the ways tech companies wrong is by putting the tech first and the users second. I think Dropbox's early competitive advantage was putting the user experience first. Everybody else had weird, complex systems; you folks just had a magic folder that was always correct. User focus like that is easier to pull off when it's just a few people. But once you break the customer problem up into enough pieces that hundreds of engineers can get involved, sometimes the tail starts wagging the dog and technically-oriented decisions start harming the user experience. How did you folks avoid that?
- tedmielczarek 11y ago
  I'm curious what your inter-language interop situation looks like. Do you have any Go code calling directly into Rust code (or vice versa)? Are they segregated into standalone programs? If so, are they doing communication via some form of IPC? Cool stuff, really interesting read!
  - jamwt 11y ago
    Not yet, but we will very soon. We'll be using Rust as a library with a C ABI, and calling it via CGO.
- djcapelis 11y ago
  Who's part of your security team over there? Most of the big cloud companies have people I trust working for them and I've been pretty comfortable telling most of my clients that cloud services are often going to have a better security team than they will. :) But now that you've moved away from AWS, have you expanded your security team to help make up for the fact that you need folks to cover all your infrastructure security needs now too? Have you made the hires you needed to deal with the low level security issues in hw and kernel? How have you all been dealing with this so far? I ask, because I help people make sense of their security issues and my clients explicitly ask me about your company. So keeping track of what you all are doing there is pretty relevant!
  - jamwt 11y ago
    We have several dozen very, very good engineers working on a few discrete teams--infrastructure security, product security, etc. I don't feel comfortable getting any more specific than that; we keep security information pretty close to the vest.
    - djcapelis 11y ago
      To be honest, that response is less than comforting. But I understand how that'd be your answer given your position in the organization. Hopefully someone else at Dropbox feels like sharing more at some point and being more forthright about your company's practices in protecting data others entrust to them. Feel free to refer others to this thread if you think it'd be helpful.
      - toyg 11y ago
        That's disingenious. The awesome sec teams at AWS or Google are not going to review and maintain your internet-facing VM network config, which is where the real threats are. Sure, their firewalls might be a bit more robust (you don't really know, you can't see them), but anything behind them is just as insecure as you configure it yourself, and that's where penetration happens in real life. Threats at hw or kernel level in a hypervisor are extremely hard to exploit, by the time you get close to that stack you can usually see much easier and juicier targets to harvest.
- abhv 11y ago
  Does magic pocket use any non-standard error-correction algorithms, or just parity-style RAID5 or 6?
  - jamwt 11y ago
    We use Reed Solomon variants with local reconstruction codes.
  - james_cowling 11y ago
    We use a variant on Reed-Solomon coding that's optimized for lower reconstruction cost. This is similar to Local Reconstruction Codes but a design/implementation of our own. The data placement isn't RAID. We encode aggregated extents of data in "volumes" that are placed on a random set of storage nodes (with sufficient physical diversity and various other constraints). Each storage node might hold a few thousand volumes, but the placement for each volume is independent of the others on the disk. If one disk fails we can thus reconstruct those volumes from hundreds of other disks simultaneously, unlike in RAID where you'd be limited in IOPS and network bandwidth to a fixed set of disks in the RAID array. That probably sounded confusing but it's a good topic for a blog post once we get around to it.
    - pnathan 11y ago
      That is extremely cool. Please tell the author(s) of that system that a stranger on the internet has appreciation for that feat!
      - willglynn 11y ago
        You might be interested in a paper Microsoft published a few years ago, entitled "Erasure Coding in Windows Azure Storage", which describes some similar concepts in greater detail: http://msr-waypoint.com/en-us/um/people/yekhanin/Papers/UsenixATC_2012.pdf http://msr-waypoint.com/en-us/um/people/yekhanin/Papers/Usen...
  - H4Y3s 11y ago
    Hey that is cool. I worked on the first commercial disk array at Thinking Machines Corporation in 1988. We used Reed-Solomon encoding because we didn't trust that the drives would catch all the errors (they did). Good work solving some of those bandwidth constraints in rebuilding the data. John (Smitty) Hayes
- bjz_ 11y ago
  Do you use much in the way of formal specifications? I've heard that Amazon currently uses TLA+ to verify the semantics of their systems.
  - dividuum 11y ago
    Interesting. Here's the paper amazon published: http://research.microsoft.com/en-us/um/people/lamport/tla/formal-methods-amazon.pdf http://research.microsoft.com/en-us/um/people/lamport/tla/fo... It's only covering a few use cases and doesn't go into too much detail and shows example code but it seems like using TLA+ has been beneficial for them.
- gvb 11y ago
  Do you know what Amazon is going to do with their excess 500PB of capacity? Are they scaling up fast enough that it isn't a big deal? Is Glacier selling your vacated storage? You must have left a large hole in AWS' revenue stream. I assume their fees for transferring your data out of their system helped cover that short-term, but 500PB of data is a lot of storage capacity to sell. Your other answers indicate your separation with AWS was very amicable, much more amicable than I would have expected. Either "its just business" or they have plans to backfill the hole you left.
  - discodave 11y ago
    One of the AWS people at Re:Invent mentioned that Amazon is storing 8EB across all storage services (S3, Glacier etc.) So 500PB is ~6% of that... Although I'm sure Amazon would have liked to keep Dropbox as a customer it's probably a pretty small percentage of their revenue. Also, if you assume that AWS is growing at ~50% year over year then 6% of S3 is only about one or two months worth of growth. I think it was this talk but I'm not sure: https://www.youtube.com/watch?v=3HDQsW_r1DM https://www.youtube.com/watch?v=3HDQsW_r1DM
  - toyg 11y ago
    With storage you're constantly mothballing and replacing old iron behind the scenes, so Amazon might just ditch a bunch of old depreciated drives without hard feelings. AWS being on the up and up for SMEs, they are probably not crying over losing one large but very demanding customer -- and DB are still on AWS for European customers anyway.
- luhn 11y ago
  > If a bunch of people shared some files via Dropbox, the company stored the files on Amazon’s Simple Storage Service, or S3, while housing all the metadata related to those files—who they belonged to, who was allowed to download them, and more—on its own machines inside its own data center space. I'm surprised at this. I would have expected the opposite: using AWS for the application and metadata but managing your own storage infrastructure. Much like how Netflix runs on AWS but streams video through its self-managed CDN. Can you shed some light on why it was designed this way?
  - jamwt 11y ago
    Building a massive, super-reliable storage system was not really feasible given Dropbox's resources back when it first started up.
- pm90 11y ago
  Not related to Magic Pocket, but when the article talks about Dropbox in house cloud, are they saying that you have your own instance of some minimal cloud-computing services? If you're permitted to disclose, I would love to know whether its based off OpenStack.
- crudbug 11y ago
  Which OS and FS the rust code is running on ?
  - james_cowling 11y ago
    No filesystem - the rust codebase directly handles the disk layout and scheduling. Our previous version of the storage layer ran on top of XFS which was probably a bad choice since we ran into a few XFS bugs along the way, but nothing serious.
    - jaysoncena 11y ago
      this is really cool! Can you give more details on how blocks, replications, metadata, etc. are being handled?
      - james_cowling 11y ago
        Lazy answer but we'll blog about this in the next month. On a high level it's variable-sized blocks packed into 1GB extents, which are then aggregated into volumes and erasure coded across a set of disks on different machines/racks/rows/etc. We also replicate cross-country in addition to this. Live writes are written into non-erasure-coded volumes and encoded in the background. The volume metadata on the disks contains enough information to be self-describing (as a safety precaution), but we also have a two-level index that maps blocks to volumes and volumes to disks. More info to come later.
        kdkeyser 11y ago
        Are live writes replicated in real time, or are they locally staged (which is what Facebook does, I believe) Also, do you mimic the eventually consistent behaviour of AWS, or do you offer a stronger form of consistency?
        jamwt 11y ago
        Live writes are written out with 4x redundancy in the local zone, then asynchronously replicated out to the remote zone. Some time later, it is erasure coded into a more space-efficient format, independently in each zone.
- tikue_ 11y ago
  What's it like working with Rust full-time? Is it similar to using any other language for a job? Do you find yourself enjoying coding more? Do you get unnecessarily caught up in micro-optimizations for things you wouldn't blink at in Go/Java?
steveklabnik 11y ago
A short mention of their use of Rust appears!
bluedino 11y ago
>> Transferring four petabytes of data, it turned out, took about a day. 46GB/s, is my math right?
- james_cowling 11y ago
  We were pushing over a terabit of data transfer at peak, so 4-5PB per day.
  - atYevP 11y ago
    Yev from Backblaze here -> we'd be curious to hear how! A lot of folks are looking for quick migration assistants for our B2 service, that's pretty fast!
    - james_cowling 11y ago
      We have a big fleet of what we call "backfill nodes" which handle the transfer but the most difficult component of the transfer is just the network capacity. We worked pretty closely with Amazon here since we have a long-standing relationship and a lot of data transfer between the two companies.
      - nilayp 11y ago
        Any plans on blogging about this and open sourcing your tools? From our vantage point, this would be hugely popular.
        jamwt 11y ago
        We might blog about it, these particular tools probably aren't very amenable to open sourcing. They've heavily dependent on in-house Dropbox infrastructure.
  - virtuallynathan 11y ago
    i.e. 1Tbps? Jeez! That is crazy.
planetjones 11y ago
Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. From my perspective the future of Dropbox looks bleak. Mass storage with Amazon is much cheaper [edit: from a consumer perspective]. I know Dropbox has superior software that works (as opposed to the poor apps by Amazon and Google) but I imagine a lot of people are like me i.e. Store most of the stuff at the cheapest location and use Dropbox just for docs that you want to sync on multiple devices. Total income from consumers like me equals 0. Dropbox also had that email app didn't they that they recently announced was closing down - mailbox if I recall correctly. Can anyone convince me that this move by Dropbox isn't going to end very badly for them? EDIT: downvoters - is HN unable to have a debate about whether this was a smart move or not.
- jamwt 11y ago
  > Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. We've had our own datacenters for a very long time; this project just represents moving "blob storage" into our own datacenter rather than just metadata. > Mass storage with Amazon is much cheaper As the article says, in Dropbox's particular case this is very untrue. Although I agree with your general point that, absent having something like Dropbox's scale/resources, just using S3 is probably your best bet. > Can anyone convince me that this move by Dropbox isn't going to end very badly for them? So, wrt "ending badly", the project has already happened. It's more efficient, faster, more flexible, etc. So I can say with some confidence it won't end badly. :-)
  - planetjones 11y ago
    Sorry I think you misunderstand me. Technically it sounds like a great achievement. Well done! Ending badly I mean that in 5 years time Dropbox will be left with a load of servers and a business model in tatters. It's the economics I think will end badly Also I am sure the in house data centre looks cheaper to Dropbox - but now they have their own hardware on the balance sheet. But I meant cheaper from a consumer point of view - I get unlimited storage at Amazon for $60 a year.
hakcermani 11y ago
Its an amazing remarkable feat indeed to move all those bits live to another location ! But seems contrary to what Netflix did (move everything to AWS ! - http://www.eweek.com/cloud/netflix-now-running-its-movies-exclusively-on-aws.html http://www.eweek.com/cloud/netflix-now-running-its-movies-ex...) I guess they have their specific usage and reasons ?
- jamwt 11y ago
  Both companies control the technology that most impacts their business. Dropbox stores quite a bit more data than Netflix. Data storage is our business. Ergo, we control storage hardware. On the other hand, Netflix pushes quite a bit more bandwidth than almost everyone, including us. Ergo, Netflix tightly manages their CDN. That's where they really focus their technical work. Their storage footprint is smaller and less important to optimize to the extreme.
- deleted 11y ago
  [deleted]
anp 11y ago
steveklabnik also pointed out on the Rust subreddit that there's a Dropbox blog post covering a few more technical items: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure/ https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr...
razster 11y ago
My reasons for leaving DB was due to their director of board choice. They were great but since then I have found equal if not better services.
asendra 11y ago
So, just for fun. The video quotes "over 1PB" of storage per box. I count 6 columns of 15 rows/drives. Might be 7 columns even, there's some panels that aren't fully open on the video. So, 90-105 drives. I'm guessing they are using 10TB drives, although maybe they can get bigger unannounced drives? Roughly, the math seems to check out. Quite impressive. Guess the Backblaze guys need a Storage Pod 6.0 soon :P (I know, I know, different requirements/constraints)
- atYevP 11y ago
  Yev from Backblaze here -> Yea :D Though if we used 10TB drives we could get about 450TB in one pod, it would just be hilariously expensive for us to do so. Granted that's not 1PB per pod, but we'd imagine our costs are lower. But yes, different use-cases, and we're working on 6.0 ;-)
- simonlc 11y ago
  Someone said above that they support 10 and 14TB drives.
ignoramous 11y ago
Key points: 1. Dropbox moved from AWS to its own datacenters after 8 months of rigourous testing. They didn't exactly build a S3 clone, but something tailored to their needs, they named it Magic Pocket. 2. Dropbox still uses AWS for its European customers. 3. Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally) viz. Diskotech. Some 8 Diskotech servers can store everything that humanity has ever written down. 4. Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. 5. No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. (Edit: Dropbox has a blog post up: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr... ) 6. Reminds people of Zynga... They did the same, and when the business plummeted, they went back to AWS. 7. Not a political move (in response to AWS' WorkDocs or CloudDrive), but purely an engineering one: Google and then Facebook succeeded by building their own data centers.
- jamwt 11y ago
  > Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performance is 3-5x better at tail latencies. Cost savings is.. dramatic. I can't be more specific there. Stability? S3 is very reliable, Magic Pocket is very reliable. I don't know if we can claim to have exceeded anything there yet, just because the project is so young, and S3s track record is long. But so far so good. Size? Exabytes of raw storage. Migration? Moving the data online was very tricky! Maybe we'll write a tech blog post at some point in the future about the migration.
  - deleted 11y ago
    [deleted]
  - toomuchtodo 11y ago
    I'd imagine that if Backblaze is realizing great cost savings building their own storage pods from commodity off the shelf components, Dropbox's costs we're even lower per GB. Well done.
riobard 11y ago
> Measuring only one-and-half-feet by three-and-half-feet by six inches, each Diskotech box holds as much as a petabyte of data This number is very interesting. Basically Diskotech stores 1PB in 18" × 6" × 42" = 4,536 cubic inch volume, which is 10% bigger than standard 7U (17" × 12.2" × 19.8" = 4,107 cubic inch). 124 days ago Dropbox Storage Engineer jamwt posted here (https://news.ycombinator.com/item?id=10541052 https://news.ycombinator.com/item?id=10541052) stating that Dropbox is "packing up to an order of magnitude more storage into 4U" compared to Backblaze Storage Pod 5.0, which is 180TB in 4U (assuming it's the deeper 4U at 19" × 7" × 26.4" = 3,511 cubic inch). Many doubted what jamwt claimed is physically possible, but doing the math reveals that Dropbox is basically packing 793TB in 4U if we naively scale linearly (of coz it's not that simple in practice). Not exactly an order of magnitude more but still. Put it another way, Diskotech is about 30% bigger in volume than Storage Pod 5.0 but with 470% more storage capacity. That was indeed some amazing engineering.
- jamwt 11y ago
  The difference to get even higher is our system supports host-managed SMR disks that come in 10T and 14T sizes.
  - riobard 11y ago
    Is that in production system now? W-O-W! :O
  - jl6 11y ago
    I've never heard of a 14TB disk drive. Is that what you mean? Who makes these?
    - benologist 11y ago
      http://www.theregister.co.uk/2016/03/03/samsung_2_5_inch_15tb_ssd/ http://www.theregister.co.uk/2016/03/03/samsung_2_5_inch_15t... 2.5" 15TB ... and SSD, and expected to get a lot bigger too!
      - 11y ago
myth_drannon 11y ago
Now that Golang is making headways at Dropbox, I guess Python codebase will diminish in its importance... I wonder how Guido Van Rossum feels about it. First he was at Google but they created Go and he left to Dropbox which was a large Python shop, now they are moving to Go too.
- james_cowling 11y ago
  I probably wasn't sufficiently clear in my other message re Golang: - Python is our primary development language for most stuff at Dropbox. - Mobile development and UI code use whatever is appropriate for each platform. - Go is the primary language for Infrastructure, meaning fairly deep-backend stuff: databases, storage systems, message pipelines, search indices etc. - Rust is used on Magic Pocket (which is within Infrastructure). - We still have a bit of C++ code floating around but there's not much new development in C++ within Infrastructure. C/C++ are still used for common libraries for mobile development however.
barkingcat 11y ago
The word "cloud" loses all meaning in this article. "The irony is that in fleeing the cloud, Dropbox is showing why the cloud is so powerful. It too is building infrastructure so that others don’t have to. It too is, well, a cloud company." Wait ... so using AWS is "cloud", having your own servers is "cloud" too. Everything is cloudy!
- fivesigma 11y ago
  Let us hope that the IoT buzzword doesn't get abused as much as "cloud" has in the past 4-5 years.
  - azdle 11y ago
    As someone who works for a company whose main business is IoT/M2M/Connected Devices/Embedded Stuff on the Internet, it already has been and it almost meaningless.
  - jaysoncena 11y ago
    There's already a Cloud over IP! My brain melt down trying to understand what that means. I hope there will be no "IoT over IP"
- scott_s 11y ago
  Dropbox is clearly a cloud company to its users. Whether or not they use a generic cloud, or maintain their own systems, is an implementation detail to users of the service.
- newman314 11y ago
  Remember, there is no cloud, there is just someone else's computer... =)
- jimbokun 11y ago
  Was discussing "in the cloud" with my wife yesterday, and I said "It just means on some company's servers." Which suddenly de-mystified the whole concept for her.
jamwt 11y ago
I want to point out one other thing about this project, if only to assuage my guilt. And this community of hackers and entrepreneurs seems as good a place as any to clear the air. There is a necessary abridgement that happens in media like this, wherein a few individuals in lead roles act as a vignette for the entire effort. In particular, on the software team, James and I are highlighted here, and it would be easy to assume we did this whole thing ourselves. Especially given very generous phrasing at times like "Turner rebuilt Magic Pocket in an entirely different programming language". The full picture is James and I were very fortunate to work with a team of more than a dozen amazing engineers on this project, that lead designs and implementations of key parts of it, and that stayed just as late at the office as we did during crunch time. In particular, the contributions of the SREs didn't make it into the article. And managing more than half a million disks without an incredible SRE team is basically impossible.
- skuhn 11y ago
  Nice of you to call this out. I think everyone understands that not all of the people involved will get mentioned by name in a Wired article. A project like this clearly isn't a four person operation, and hopefully most people realize there are always a bunch of people working behind the scenes. My only disappointment is that by trying to set a PR friendly narrative, Dropbox greatly compressed the time spent on the project from inception to completion, and that erases a lot of people's involvement. As someone who worked on MP when it actually was a four person project, it's a little frustrating to finally be able to talk about it after all this time while at the same time the public perception is that the project began after I left the company.
  - deleted 11y ago
    [deleted]
- james_cowling 11y ago
  Couldn't agree with this more. One of the things that's not immediately apparent in an initiative like this is the huge amount of effort required to provision, deploy and manage a system of this scale, across a whole bunch of different teams. We couldn't have done any of this without a tremendous group of SREs with the wisdom to invest heavily in automation, in addition to all the hard work. We'll follow up in a week or two with a blog post highlighting some of the work here. I think the framework for disk remediation (automatically detecting failures, re-provisioning, etc) is particularly interesting.
the_watcher 11y ago
I worked at a company that moved off AWS and the changing a tire on a moving car is exactly how the devops team described it.
iskander 11y ago
Q: Did the Dropbox devs working on the second version of Magic Pocket encounter any language stability issues with Rust? Did updates to the core language ever break existing code?
- jamwt 11y ago
  We probably "pulled forward" every two months or so. Back in early 2015, sometimes we'd have breakages that took a bit of work to remedy. But post 1.0, the language has been very stable. As usual, we actually have more challenges pulling the libraries forward than the language.
photonwins 11y ago
I am curious to know, with so many disks densely packed in a 4U configuration, I am guessing there is definitely increased heat generation and not to mention vibration. How do you handle these? Also, does it have any effect on MTBF?
- jamwt 11y ago
  Yeah, our hardware teams put a lot of qualification time into hardware profiles before we buy in bulk. And part of what we test is power utilization, heat, etc, under normal software loads vs extreme loads, to make sure we don't exceed power budget, don't spin everything up at the same time, etc. This process can take months. A lot of the system design is about talking closely to vendors about MTBF for specific loads, and we use a lot of metric data to characterize our loads accurately in those calculations. Then, of course, we measure once we're in the field to make sure we were on target, so the next round of estimations is better.
  - scurvy 11y ago
    It is indeed scary to watch a storage node pull 1.5-2 kW at power-on. I think my eyes popped out the first time I watched the PDU load dance around like that.
Spooky23 11y ago
I'm a recent Dropbox Pro subscriber -- I've been a user for many years. Really glad to see what's been going on from behind the scenes, and I'm looking forward to seeing what the future will bring to the front end of Dropbox. Thanks guys!
- jamwt 11y ago
  Awesome! Thanks for being a customer. We'll continue to work hard to make Dropbox better and better.
deleted 11y ago
[deleted]
sre_ops 11y ago
CAF deal ran out?
bane 11y ago
From a business sense, this is a great move. Amazon's cloud storage offerings are pretty expensive, even with various deduping strategies in place, Dropbox needed both store and move in and out massive amounts of traffic. If DB had reached the point where they could staff the extra over head of doing it themselves, and figure out how to spread the hardware cost out well, it will give them more pricing flexibility in the near future.
matt_wulfeck 11y ago
The open-source alternatives to S3 have a long way to go, especially since AWS S3 is so simple and reliable. I ran Openstack swift in production once upon a time and I remember the entire system coming down because the the rsyslog hostname did not resolve. Ouch. I'll let somebody else work out those bugs.
jessegreathouse 11y ago
Is there a link without the anti-adblock crap?
- bartvk 11y ago
  You can also save it with Pocket http://www.getpocket.com/ http://www.getpocket.com/
echelon 11y ago
Unrelated to the content of the article--I've never seen the oft talked about "anti-adblocker interstitial" before. I was surprised to find that the website blocks viewing of the article entirely because I'm running Adblock. It's an interesting subject. I'll never click on or be persuaded by ads, so they're not really gaining anything by showing me ads. I'm essentially worthless traffic to them no matter how they cut it. I do understand the problem they're trying to solve, and I would like to pay for content that I find worthwhile. A convenient microtransaction protocol where I _don't have to sign in or interact in any way_ would be nice. (I don't want to waste time interacting with their bespoke account system / sign-on, even if it is "frictionless".)
- darkarmani 11y ago
  I ended up refreshing, select all, copy and pasting it into my editor. That resulted in me spending no time on their page.
- ec109685 11y ago
  How do you know you are never influenced by the ads you see? Also, Google Contributor network let's you block any DoubleClick ad, while still paying the publisher for the page view, so that might get what you want.
- roel_v 11y ago
  "I'll never click on" Doesn't matter, impressions are valuable to marketers as well. "or be persuaded by ads" Yes you will, you just think you don't/won't.
  - echelon 11y ago
    > Yes you will, you just think you don't/won't. That's a really blanket statement. I'm rather minimalist, and I typically form negative opinions of the things I see in ads.
    - deanCommie 11y ago
      You are one person. Statistics show most people are not like you.
tschellenbach 11y ago
Why would you reinvent a piece of technology offered by various cloud providers? It doesn't make any sense, what a waste of engineering resources (fun exercise though). They should have been able to reach some middle ground with AWS about pricing. I really doubt their in-house storage backend will be cheaper compared to the lowest rate they can get from one of the cloud providers.
- johnloeber 11y ago
  Because having your entire business model depend on the pricing whims of another business (which in the grand scheme of things is a competitor, btw) is unsustainable. Getting out of AWS is exactly the right move for Dropbox's long term health.
  - tschellenbach 11y ago
    There are many cloud hosting providers and they are competing intensely. AWS marginal cost increase for supporting Dropbox is far below the cost of Dropbox building and maintaining their own solution. Given these facts they should be able to negotiate a far better price/cost than their current solution of building it in-house. There is nothing unique about their requirements which justify this move. It's a really weird decision. Probably we're missing something here. This must be part of some larger strategy they are executing.
    - toyg 11y ago
      As jamwt mentioned above, storage is simply the most critical element of their business, so it makes sense to control it as tightly as possible , similar to what Netflix does with their CDN. The cloud is for stuff you don't care too much about.
  - deleted 11y ago
    [deleted]
turingbook 11y ago
I hated the writing style of Wired guys. The blog from Dropbox is much more clear: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure/ https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr...
known 11y ago
TL;DR Amazon Cloud is expensive;
JackPoach 11y ago
I wonder how capital intensive the move is. Isn't Dropbox overall still unprofitable as a company?
ldom66 11y ago
Unrelated to the article but these portraits are really beautiful! Props to the photograph and studio.
virtuallynathan 11y ago
It looks like the servers they make use of are purchasable, part of the Dell DSS series - the DSS7000: http://downloads.dell.com/manuals/common/dss%207000_dss%207500_owner%27s%20manual_en.pdf http://downloads.dell.com/manuals/common/dss%207000_dss%2075... 90x 3.5in disks, 2x compute nodes with 2x E5-2600v3 CPUs.
- jamwt 11y ago
  We actually use several different servers from different vendors, and some of them are more customized than others.
  - virtuallynathan 11y ago
    Ah, yes I spot the 78 bay Quanta one: http://www.qct.io/Product/Storage/Storage-Server/4U/New-QuantaPlex-T21P-4U-p291c77c71c150c222 http://www.qct.io/Product/Storage/Storage-Server/4U/New-Quan... Can't identify the 3rd one yet... EDIT: Found it... HPE Cloudline CL5200, 80 drive bays. http://www.nextplatform.com/wp-content/uploads/2016/03/hpe-cl5200-server.jpg http://www.nextplatform.com/wp-content/uploads/2016/03/hpe-c...
max_ 11y ago
whats the "brand new programming language" the writer meant ?
- yberreby 11y ago
  It is Rust[1]: > So, in the middle of this two-and-half-year project, they switched to Rust on the Diskotech machines. And that’s what Dropbox is now pushing into its data centers. See https://www.reddit.com/r/rust/comments/4adabk/the_epic_story_of_dropboxs_exodus_from_the_amazon/ https://www.reddit.com/r/rust/comments/4adabk/the_epic_story... for more information. [1]: https://www.rust-lang.org/ https://www.rust-lang.org/
ccannon 11y ago
I applaud your efforts at answering all these questions and providing very detailed responses.
dmitrifedorov 11y ago
I'm done with Wired: cancelled its subscription a couple months ago (after many years), now they demand I login to read their web site because I use ad blockers. Done!
cj0 11y ago
Some requests for the DropBox Diskotech tech blog series: For which datacenter cooling system are the Diskotech storage pods designed? Hot aisle cold aisle? How evenly can the disks inside the Diskotech be cooled? With this question I try to ask, how good is it in keeping all drives cool? For example for the theoretical case where all disks in the Diskotech are dissipating identical heat, how much variation is there in drive temperature at roughly 25ºC intake temperature? How much power is consumed for cooling (despite that it will be fractionally low compared to what the disks will consume)? Is there any drive idle optimization done or is the expectance that not any disk will reach its idle (non-rotating) state ever? Is there any optimization done inside a Diskotech to reduce the (inrush) current surge when connecting the machine to power?
wchrisn 11y ago
Can you share your development practices in rust like TDD, Automated Builds etc
wchrisn 11y ago
Can you share your development practices in rust like TDD, Automated Builds etc
monomaniar 11y ago
Awosome

671 points by moviuro 11y ago

jamwt 11y ago

Hi HN! A couple of us from the Magic Pocket software team are around to answer questions if anyone has some.

dikaiosune 11y ago
The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? If the latter, I've heard recent versions have had some significant improvements -- has your team looked at that and thought that you would have been OK if you just waited, or was there a fundamental gap that GC improvements couldn't close? Could you comment more generally on what advantages Rust offered and where your team would like to see improvement? Are there portions of the Dropbox codebase where you'd love to use Rust but can't until <feature> RFCs/lands/hits stable? Are there portions where the decision to use Rust caused complications or problems?
- jamwt 11y ago
  Good questions, let me try to tackle them one by one. > The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go? The reasons for using rust were many, but memory was one of them. Primarily, for this particular project, the heap size is the issue. One of the games in this project is optimizing how little memory and compute you can use to manage 1GB (or 1PB) of data. We utilize lots of tricks like perfect hash tables, extensive bit-packing, etc. Lots of odd, custom, inline and cache-friendly data structures. We also keeps lots of things on the stack when we can to take pressure off the VM system. We do some lockfree object pooling stuff for big byte vectors, which are common allocations in a block storage system. It's much easier to do these particular kinds of optimizations using C++ or Rust. In addition to basic memory reasons, saving a bit of CPU was a useful secondary goal, and that goal has been achieved. The project also has a fair amount of FFI work with various C libraries, and a kernel component. Rust makes it very easy and zero-cost to work closely with those libraries/environments. For this project, pause times were not an issue. This isn't a particularly latency-sensitive service. We do have some other services where latency does matter, though, and we're considering Rust for those in the future. > Could you comment more generally on what advantages Rust offered and where your team would like to see improvement? The advantages of Rust are many. Really powerful abstractions, no null, no segfaults, no leaks, yet C-like performance and control over memory and you can use that whole C/C++ bag of optimization tricks. On the improvements side, we're in close contact with the Rust core team--they visit the office regularly and keep tabs on what we're doing. So no, we don't have a ton of things we need. They've been really great about helping us out when those things have sprung up. Our big ask right now is the same as everyone else's--improve compile times! > Are there portions where the decision to use Rust caused complications or problems? Well, Dropbox is mostly a golang shop on the backend, so Rust is a pretty different animal than everyone was used to. We also have a huge number of good libraries in golang that our small team had to create minimal equivalents for in Rust. So, the biggest challenge in using Rust at Dropbox has been that we were the first project! So we had a lot to do just to get started... The other complication is that there is a ton of good stuff that we want to use that's still being debated by the Rust team, and therefore marked unstable. As each release goes on, they stabilize these APIs, but it's sometimes a pain working around useful APIs that are marked unstable just because the dust hasn't settled yet within the core team. Having said that, we totally understand that they're being thoughtful about all this, because backwards compatibility implies a very serious long-term commitment to these decisions.
  - steveklabnik 11y ago
    > On the improvements side, we're in close contact with the Rust core team One small note here: this is something that we (Rust core team) are interested in doing generally, not just for Dropbox. If you use Rust in production, we want to hear from you! We're very interested in supporting production users.
  - anp 11y ago
    Thanks very much for the detailed and thoughtful answers! I've read before (somewhere, I think) that Dropbox effectively maintains a large internal "standard library" rather than relying on external open source efforts. How much does Magic Pocket rely on Rust's standard library and the crates.io ecosystem? Could you elaborate on how you ended up going in whichever direction you chose with regards to third-party open source code?
    - jamwt 11y ago
      We use 3rd parties for the "obvious" stuff. Like, we're not going to reinvent json serialization. But we typically don't use any 3rd party frameworks on the backend. So things like service management/discovery, rpc, error handling, monitoring, metadata storage, etc etc, are a big in-house stack. So, we use quite a few crates for the things it makes no sense to specialize in Dropbox-specific ways.
      - dikaiosune 11y ago
        Cool. This might be getting into the weeds a bit, but are you still on rustc-serialize for json or are you trying to keep up with serde/serde_json? If you're using serde, are you on nightly? From your comment above I got the impression that only using stable features was very important, so I'm curious how your codebase implements/derives the serde traits.
        jamwt 11y ago
        We're on rustc-serialize. JSON is not really a part of our data pipeline, just our metrics pipeline. So the performance of the library is not especially critical.
ignoramous 11y ago
Please do a technical blog post on some of these: 1. Go vs Rust at Dropbox's scale and requirements. 2. Maintaining availability whilst moving from AWS to Diskotech and Magic Pocket. 3. Internals of Magic Pocket (file-system, storage engine, availability guarantees, scaling up and scaling out, compression details, load balancing, cloning etc) 4. Improvements in perf, stability, security, and cost. Thanks.
- james_cowling 11y ago
  Yep we're going to get at least a few actual technical blog posts online in the coming month. We haven't got around to writing them yet tho so feel free to surface any requests :) Will most likely start with the following: 1. Overall architecture and internals. 2. Verification and projection mechanisms, how we keep data safe. 3. How we manage a large amount of hardware (hundreds of thousands of disks) with a small team, how we automatically deal with alerts etc. 4. A deep-dive into the Diskotech architecture and how we changed our storage layer to support SMR disks. Hopefully these will be of a sufficient level of detail. We certainly won't be getting into any details on cost but we're pretty open from a technical perspective. (Lemme just take a moment to say how great Backblaze's tech blog posts are btw.)
  - atYevP 11y ago
    Yev here -> Let me just say you're welcome and thanks for joining the party :D
  - wpietri 11y ago
    I would be especially interested to hear how a focus on actual customer experience was maintained during all of this deep technology work. One of the ways tech companies wrong is by putting the tech first and the users second. I think Dropbox's early competitive advantage was putting the user experience first. Everybody else had weird, complex systems; you folks just had a magic folder that was always correct. User focus like that is easier to pull off when it's just a few people. But once you break the customer problem up into enough pieces that hundreds of engineers can get involved, sometimes the tail starts wagging the dog and technically-oriented decisions start harming the user experience. How did you folks avoid that?
tedmielczarek 11y ago
I'm curious what your inter-language interop situation looks like. Do you have any Go code calling directly into Rust code (or vice versa)? Are they segregated into standalone programs? If so, are they doing communication via some form of IPC? Cool stuff, really interesting read!
- jamwt 11y ago
  Not yet, but we will very soon. We'll be using Rust as a library with a C ABI, and calling it via CGO.
djcapelis 11y ago
Who's part of your security team over there? Most of the big cloud companies have people I trust working for them and I've been pretty comfortable telling most of my clients that cloud services are often going to have a better security team than they will. :) But now that you've moved away from AWS, have you expanded your security team to help make up for the fact that you need folks to cover all your infrastructure security needs now too? Have you made the hires you needed to deal with the low level security issues in hw and kernel? How have you all been dealing with this so far? I ask, because I help people make sense of their security issues and my clients explicitly ask me about your company. So keeping track of what you all are doing there is pretty relevant!
- jamwt 11y ago
  We have several dozen very, very good engineers working on a few discrete teams--infrastructure security, product security, etc. I don't feel comfortable getting any more specific than that; we keep security information pretty close to the vest.
  - djcapelis 11y ago
    To be honest, that response is less than comforting. But I understand how that'd be your answer given your position in the organization. Hopefully someone else at Dropbox feels like sharing more at some point and being more forthright about your company's practices in protecting data others entrust to them. Feel free to refer others to this thread if you think it'd be helpful.
    - toyg 11y ago
      That's disingenious. The awesome sec teams at AWS or Google are not going to review and maintain your internet-facing VM network config, which is where the real threats are. Sure, their firewalls might be a bit more robust (you don't really know, you can't see them), but anything behind them is just as insecure as you configure it yourself, and that's where penetration happens in real life. Threats at hw or kernel level in a hypervisor are extremely hard to exploit, by the time you get close to that stack you can usually see much easier and juicier targets to harvest.
abhv 11y ago
Does magic pocket use any non-standard error-correction algorithms, or just parity-style RAID5 or 6?
- jamwt 11y ago
  We use Reed Solomon variants with local reconstruction codes.
- james_cowling 11y ago
  We use a variant on Reed-Solomon coding that's optimized for lower reconstruction cost. This is similar to Local Reconstruction Codes but a design/implementation of our own. The data placement isn't RAID. We encode aggregated extents of data in "volumes" that are placed on a random set of storage nodes (with sufficient physical diversity and various other constraints). Each storage node might hold a few thousand volumes, but the placement for each volume is independent of the others on the disk. If one disk fails we can thus reconstruct those volumes from hundreds of other disks simultaneously, unlike in RAID where you'd be limited in IOPS and network bandwidth to a fixed set of disks in the RAID array. That probably sounded confusing but it's a good topic for a blog post once we get around to it.
  - pnathan 11y ago
    That is extremely cool. Please tell the author(s) of that system that a stranger on the internet has appreciation for that feat!
    - willglynn 11y ago
      You might be interested in a paper Microsoft published a few years ago, entitled "Erasure Coding in Windows Azure Storage", which describes some similar concepts in greater detail: http://msr-waypoint.com/en-us/um/people/yekhanin/Papers/UsenixATC_2012.pdf http://msr-waypoint.com/en-us/um/people/yekhanin/Papers/Usen...
- H4Y3s 11y ago
  Hey that is cool. I worked on the first commercial disk array at Thinking Machines Corporation in 1988. We used Reed-Solomon encoding because we didn't trust that the drives would catch all the errors (they did). Good work solving some of those bandwidth constraints in rebuilding the data. John (Smitty) Hayes
bjz_ 11y ago
Do you use much in the way of formal specifications? I've heard that Amazon currently uses TLA+ to verify the semantics of their systems.
- dividuum 11y ago
  Interesting. Here's the paper amazon published: http://research.microsoft.com/en-us/um/people/lamport/tla/formal-methods-amazon.pdf http://research.microsoft.com/en-us/um/people/lamport/tla/fo... It's only covering a few use cases and doesn't go into too much detail and shows example code but it seems like using TLA+ has been beneficial for them.
gvb 11y ago
Do you know what Amazon is going to do with their excess 500PB of capacity? Are they scaling up fast enough that it isn't a big deal? Is Glacier selling your vacated storage? You must have left a large hole in AWS' revenue stream. I assume their fees for transferring your data out of their system helped cover that short-term, but 500PB of data is a lot of storage capacity to sell. Your other answers indicate your separation with AWS was very amicable, much more amicable than I would have expected. Either "its just business" or they have plans to backfill the hole you left.
- discodave 11y ago
  One of the AWS people at Re:Invent mentioned that Amazon is storing 8EB across all storage services (S3, Glacier etc.) So 500PB is ~6% of that... Although I'm sure Amazon would have liked to keep Dropbox as a customer it's probably a pretty small percentage of their revenue. Also, if you assume that AWS is growing at ~50% year over year then 6% of S3 is only about one or two months worth of growth. I think it was this talk but I'm not sure: https://www.youtube.com/watch?v=3HDQsW_r1DM https://www.youtube.com/watch?v=3HDQsW_r1DM
- toyg 11y ago
  With storage you're constantly mothballing and replacing old iron behind the scenes, so Amazon might just ditch a bunch of old depreciated drives without hard feelings. AWS being on the up and up for SMEs, they are probably not crying over losing one large but very demanding customer -- and DB are still on AWS for European customers anyway.
luhn 11y ago
> If a bunch of people shared some files via Dropbox, the company stored the files on Amazon’s Simple Storage Service, or S3, while housing all the metadata related to those files—who they belonged to, who was allowed to download them, and more—on its own machines inside its own data center space. I'm surprised at this. I would have expected the opposite: using AWS for the application and metadata but managing your own storage infrastructure. Much like how Netflix runs on AWS but streams video through its self-managed CDN. Can you shed some light on why it was designed this way?
- jamwt 11y ago
  Building a massive, super-reliable storage system was not really feasible given Dropbox's resources back when it first started up.
pm90 11y ago
Not related to Magic Pocket, but when the article talks about Dropbox in house cloud, are they saying that you have your own instance of some minimal cloud-computing services? If you're permitted to disclose, I would love to know whether its based off OpenStack.
crudbug 11y ago
Which OS and FS the rust code is running on ?
- james_cowling 11y ago
  No filesystem - the rust codebase directly handles the disk layout and scheduling. Our previous version of the storage layer ran on top of XFS which was probably a bad choice since we ran into a few XFS bugs along the way, but nothing serious.
  - jaysoncena 11y ago
    this is really cool! Can you give more details on how blocks, replications, metadata, etc. are being handled?
    - james_cowling 11y ago
      Lazy answer but we'll blog about this in the next month. On a high level it's variable-sized blocks packed into 1GB extents, which are then aggregated into volumes and erasure coded across a set of disks on different machines/racks/rows/etc. We also replicate cross-country in addition to this. Live writes are written into non-erasure-coded volumes and encoded in the background. The volume metadata on the disks contains enough information to be self-describing (as a safety precaution), but we also have a two-level index that maps blocks to volumes and volumes to disks. More info to come later.
      - kdkeyser 11y ago
        Are live writes replicated in real time, or are they locally staged (which is what Facebook does, I believe) Also, do you mimic the eventually consistent behaviour of AWS, or do you offer a stronger form of consistency?
        jamwt 11y ago
        Live writes are written out with 4x redundancy in the local zone, then asynchronously replicated out to the remote zone. Some time later, it is erasure coded into a more space-efficient format, independently in each zone.
tikue_ 11y ago
What's it like working with Rust full-time? Is it similar to using any other language for a job? Do you find yourself enjoying coding more? Do you get unnecessarily caught up in micro-optimizations for things you wouldn't blink at in Go/Java?

steveklabnik 11y ago

A short mention of their use of Rust appears!

bluedino 11y ago

>> Transferring four petabytes of data, it turned out, took about a day. 46GB/s, is my math right?

james_cowling 11y ago
We were pushing over a terabit of data transfer at peak, so 4-5PB per day.
- atYevP 11y ago
  Yev from Backblaze here -> we'd be curious to hear how! A lot of folks are looking for quick migration assistants for our B2 service, that's pretty fast!
  - james_cowling 11y ago
    We have a big fleet of what we call "backfill nodes" which handle the transfer but the most difficult component of the transfer is just the network capacity. We worked pretty closely with Amazon here since we have a long-standing relationship and a lot of data transfer between the two companies.
    - nilayp 11y ago
      Any plans on blogging about this and open sourcing your tools? From our vantage point, this would be hugely popular.
      - jamwt 11y ago
        We might blog about it, these particular tools probably aren't very amenable to open sourcing. They've heavily dependent on in-house Dropbox infrastructure.
- virtuallynathan 11y ago
  i.e. 1Tbps? Jeez! That is crazy.

planetjones 11y ago

Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. From my perspective the future of Dropbox looks bleak. Mass storage with Amazon is much cheaper [edit: from a consumer perspective]. I know Dropbox has superior software that works (as opposed to the poor apps by Amazon and Google) but I imagine a lot of people are like me i.e. Store most of the stuff at the cheapest location and use Dropbox just for docs that you want to sync on multiple devices. Total income from consumers like me equals 0. Dropbox also had that email app didn't they that they recently announced was closing down - mailbox if I recall correctly. Can anyone convince me that this move by Dropbox isn't going to end very badly for them? EDIT: downvoters - is HN unable to have a debate about whether this was a smart move or not.

jamwt 11y ago
> Oh dear I didn't realise Dropbox had invested all of that time and money moving into their own data centre. We've had our own datacenters for a very long time; this project just represents moving "blob storage" into our own datacenter rather than just metadata. > Mass storage with Amazon is much cheaper As the article says, in Dropbox's particular case this is very untrue. Although I agree with your general point that, absent having something like Dropbox's scale/resources, just using S3 is probably your best bet. > Can anyone convince me that this move by Dropbox isn't going to end very badly for them? So, wrt "ending badly", the project has already happened. It's more efficient, faster, more flexible, etc. So I can say with some confidence it won't end badly. :-)
- planetjones 11y ago
  Sorry I think you misunderstand me. Technically it sounds like a great achievement. Well done! Ending badly I mean that in 5 years time Dropbox will be left with a load of servers and a business model in tatters. It's the economics I think will end badly Also I am sure the in house data centre looks cheaper to Dropbox - but now they have their own hardware on the balance sheet. But I meant cheaper from a consumer point of view - I get unlimited storage at Amazon for $60 a year.

hakcermani 11y ago

Its an amazing remarkable feat indeed to move all those bits live to another location ! But seems contrary to what Netflix did (move everything to AWS ! - http://www.eweek.com/cloud/netflix-now-running-its-movies-exclusively-on-aws.html http://www.eweek.com/cloud/netflix-now-running-its-movies-ex...) I guess they have their specific usage and reasons ?

jamwt 11y ago
Both companies control the technology that most impacts their business. Dropbox stores quite a bit more data than Netflix. Data storage is our business. Ergo, we control storage hardware. On the other hand, Netflix pushes quite a bit more bandwidth than almost everyone, including us. Ergo, Netflix tightly manages their CDN. That's where they really focus their technical work. Their storage footprint is smaller and less important to optimize to the extreme.
deleted 11y ago
[deleted]

anp 11y ago

steveklabnik also pointed out on the Rust subreddit that there's a Dropbox blog post covering a few more technical items: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure/ https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr...

razster 11y ago

My reasons for leaving DB was due to their director of board choice. They were great but since then I have found equal if not better services.

asendra 11y ago

So, just for fun. The video quotes "over 1PB" of storage per box. I count 6 columns of 15 rows/drives. Might be 7 columns even, there's some panels that aren't fully open on the video. So, 90-105 drives. I'm guessing they are using 10TB drives, although maybe they can get bigger unannounced drives? Roughly, the math seems to check out. Quite impressive. Guess the Backblaze guys need a Storage Pod 6.0 soon :P (I know, I know, different requirements/constraints)

atYevP 11y ago
Yev from Backblaze here -> Yea :D Though if we used 10TB drives we could get about 450TB in one pod, it would just be hilariously expensive for us to do so. Granted that's not 1PB per pod, but we'd imagine our costs are lower. But yes, different use-cases, and we're working on 6.0 ;-)
simonlc 11y ago
Someone said above that they support 10 and 14TB drives.

ignoramous 11y ago

Key points: 1. Dropbox moved from AWS to its own datacenters after 8 months of rigourous testing. They didn't exactly build a S3 clone, but something tailored to their needs, they named it Magic Pocket. 2. Dropbox still uses AWS for its European customers. 3. Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally) viz. Diskotech. Some 8 Diskotech servers can store everything that humanity has ever written down. 4. Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. 5. No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. (Edit: Dropbox has a blog post up: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr... ) 6. Reminds people of Zynga... They did the same, and when the business plummeted, they went back to AWS. 7. Not a political move (in response to AWS' WorkDocs or CloudDrive), but purely an engineering one: Google and then Facebook succeeded by building their own data centers.

jamwt 11y ago
> Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines. Actually, full disclosure, we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang. > No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. Performance is 3-5x better at tail latencies. Cost savings is.. dramatic. I can't be more specific there. Stability? S3 is very reliable, Magic Pocket is very reliable. I don't know if we can claim to have exceeded anything there yet, just because the project is so young, and S3s track record is long. But so far so good. Size? Exabytes of raw storage. Migration? Moving the data online was very tricky! Maybe we'll write a tech blog post at some point in the future about the migration.
- deleted 11y ago
  [deleted]
- toomuchtodo 11y ago
  I'd imagine that if Backblaze is realizing great cost savings building their own storage pods from commodity off the shelf components, Dropbox's costs we're even lower per GB. Well done.

riobard 11y ago

> Measuring only one-and-half-feet by three-and-half-feet by six inches, each Diskotech box holds as much as a petabyte of data This number is very interesting. Basically Diskotech stores 1PB in 18" × 6" × 42" = 4,536 cubic inch volume, which is 10% bigger than standard 7U (17" × 12.2" × 19.8" = 4,107 cubic inch). 124 days ago Dropbox Storage Engineer jamwt posted here (https://news.ycombinator.com/item?id=10541052 https://news.ycombinator.com/item?id=10541052) stating that Dropbox is "packing up to an order of magnitude more storage into 4U" compared to Backblaze Storage Pod 5.0, which is 180TB in 4U (assuming it's the deeper 4U at 19" × 7" × 26.4" = 3,511 cubic inch). Many doubted what jamwt claimed is physically possible, but doing the math reveals that Dropbox is basically packing 793TB in 4U if we naively scale linearly (of coz it's not that simple in practice). Not exactly an order of magnitude more but still. Put it another way, Diskotech is about 30% bigger in volume than Storage Pod 5.0 but with 470% more storage capacity. That was indeed some amazing engineering.

jamwt 11y ago
The difference to get even higher is our system supports host-managed SMR disks that come in 10T and 14T sizes.
- riobard 11y ago
  Is that in production system now? W-O-W! :O
- jl6 11y ago
  I've never heard of a 14TB disk drive. Is that what you mean? Who makes these?
  - benologist 11y ago
    http://www.theregister.co.uk/2016/03/03/samsung_2_5_inch_15tb_ssd/ http://www.theregister.co.uk/2016/03/03/samsung_2_5_inch_15t... 2.5" 15TB ... and SSD, and expected to get a lot bigger too!
    - 11y ago

myth_drannon 11y ago

Now that Golang is making headways at Dropbox, I guess Python codebase will diminish in its importance... I wonder how Guido Van Rossum feels about it. First he was at Google but they created Go and he left to Dropbox which was a large Python shop, now they are moving to Go too.

james_cowling 11y ago
I probably wasn't sufficiently clear in my other message re Golang: - Python is our primary development language for most stuff at Dropbox. - Mobile development and UI code use whatever is appropriate for each platform. - Go is the primary language for Infrastructure, meaning fairly deep-backend stuff: databases, storage systems, message pipelines, search indices etc. - Rust is used on Magic Pocket (which is within Infrastructure). - We still have a bit of C++ code floating around but there's not much new development in C++ within Infrastructure. C/C++ are still used for common libraries for mobile development however.

barkingcat 11y ago

The word "cloud" loses all meaning in this article. "The irony is that in fleeing the cloud, Dropbox is showing why the cloud is so powerful. It too is building infrastructure so that others don’t have to. It too is, well, a cloud company." Wait ... so using AWS is "cloud", having your own servers is "cloud" too. Everything is cloudy!

fivesigma 11y ago
Let us hope that the IoT buzzword doesn't get abused as much as "cloud" has in the past 4-5 years.
- azdle 11y ago
  As someone who works for a company whose main business is IoT/M2M/Connected Devices/Embedded Stuff on the Internet, it already has been and it almost meaningless.
- jaysoncena 11y ago
  There's already a Cloud over IP! My brain melt down trying to understand what that means. I hope there will be no "IoT over IP"
scott_s 11y ago
Dropbox is clearly a cloud company to its users. Whether or not they use a generic cloud, or maintain their own systems, is an implementation detail to users of the service.
newman314 11y ago
Remember, there is no cloud, there is just someone else's computer... =)
jimbokun 11y ago
Was discussing "in the cloud" with my wife yesterday, and I said "It just means on some company's servers." Which suddenly de-mystified the whole concept for her.

jamwt 11y ago

I want to point out one other thing about this project, if only to assuage my guilt. And this community of hackers and entrepreneurs seems as good a place as any to clear the air. There is a necessary abridgement that happens in media like this, wherein a few individuals in lead roles act as a vignette for the entire effort. In particular, on the software team, James and I are highlighted here, and it would be easy to assume we did this whole thing ourselves. Especially given very generous phrasing at times like "Turner rebuilt Magic Pocket in an entirely different programming language". The full picture is James and I were very fortunate to work with a team of more than a dozen amazing engineers on this project, that lead designs and implementations of key parts of it, and that stayed just as late at the office as we did during crunch time. In particular, the contributions of the SREs didn't make it into the article. And managing more than half a million disks without an incredible SRE team is basically impossible.

skuhn 11y ago
Nice of you to call this out. I think everyone understands that not all of the people involved will get mentioned by name in a Wired article. A project like this clearly isn't a four person operation, and hopefully most people realize there are always a bunch of people working behind the scenes. My only disappointment is that by trying to set a PR friendly narrative, Dropbox greatly compressed the time spent on the project from inception to completion, and that erases a lot of people's involvement. As someone who worked on MP when it actually was a four person project, it's a little frustrating to finally be able to talk about it after all this time while at the same time the public perception is that the project began after I left the company.
- deleted 11y ago
  [deleted]
james_cowling 11y ago
Couldn't agree with this more. One of the things that's not immediately apparent in an initiative like this is the huge amount of effort required to provision, deploy and manage a system of this scale, across a whole bunch of different teams. We couldn't have done any of this without a tremendous group of SREs with the wisdom to invest heavily in automation, in addition to all the hard work. We'll follow up in a week or two with a blog post highlighting some of the work here. I think the framework for disk remediation (automatically detecting failures, re-provisioning, etc) is particularly interesting.

the_watcher 11y ago

I worked at a company that moved off AWS and the changing a tire on a moving car is exactly how the devops team described it.

iskander 11y ago

Q: Did the Dropbox devs working on the second version of Magic Pocket encounter any language stability issues with Rust? Did updates to the core language ever break existing code?

jamwt 11y ago
We probably "pulled forward" every two months or so. Back in early 2015, sometimes we'd have breakages that took a bit of work to remedy. But post 1.0, the language has been very stable. As usual, we actually have more challenges pulling the libraries forward than the language.

photonwins 11y ago

I am curious to know, with so many disks densely packed in a 4U configuration, I am guessing there is definitely increased heat generation and not to mention vibration. How do you handle these? Also, does it have any effect on MTBF?

jamwt 11y ago
Yeah, our hardware teams put a lot of qualification time into hardware profiles before we buy in bulk. And part of what we test is power utilization, heat, etc, under normal software loads vs extreme loads, to make sure we don't exceed power budget, don't spin everything up at the same time, etc. This process can take months. A lot of the system design is about talking closely to vendors about MTBF for specific loads, and we use a lot of metric data to characterize our loads accurately in those calculations. Then, of course, we measure once we're in the field to make sure we were on target, so the next round of estimations is better.
- scurvy 11y ago
  It is indeed scary to watch a storage node pull 1.5-2 kW at power-on. I think my eyes popped out the first time I watched the PDU load dance around like that.

Spooky23 11y ago

I'm a recent Dropbox Pro subscriber -- I've been a user for many years. Really glad to see what's been going on from behind the scenes, and I'm looking forward to seeing what the future will bring to the front end of Dropbox. Thanks guys!

jamwt 11y ago
Awesome! Thanks for being a customer. We'll continue to work hard to make Dropbox better and better.

deleted 11y ago

[deleted]

sre_ops 11y ago

CAF deal ran out?

bane 11y ago

From a business sense, this is a great move. Amazon's cloud storage offerings are pretty expensive, even with various deduping strategies in place, Dropbox needed both store and move in and out massive amounts of traffic. If DB had reached the point where they could staff the extra over head of doing it themselves, and figure out how to spread the hardware cost out well, it will give them more pricing flexibility in the near future.

matt_wulfeck 11y ago

The open-source alternatives to S3 have a long way to go, especially since AWS S3 is so simple and reliable. I ran Openstack swift in production once upon a time and I remember the entire system coming down because the the rsyslog hostname did not resolve. Ouch. I'll let somebody else work out those bugs.

jessegreathouse 11y ago

Is there a link without the anti-adblock crap?

bartvk 11y ago
You can also save it with Pocket http://www.getpocket.com/ http://www.getpocket.com/

echelon 11y ago

Unrelated to the content of the article--I've never seen the oft talked about "anti-adblocker interstitial" before. I was surprised to find that the website blocks viewing of the article entirely because I'm running Adblock. It's an interesting subject. I'll never click on or be persuaded by ads, so they're not really gaining anything by showing me ads. I'm essentially worthless traffic to them no matter how they cut it. I do understand the problem they're trying to solve, and I would like to pay for content that I find worthwhile. A convenient microtransaction protocol where I _don't have to sign in or interact in any way_ would be nice. (I don't want to waste time interacting with their bespoke account system / sign-on, even if it is "frictionless".)

darkarmani 11y ago
I ended up refreshing, select all, copy and pasting it into my editor. That resulted in me spending no time on their page.
ec109685 11y ago
How do you know you are never influenced by the ads you see? Also, Google Contributor network let's you block any DoubleClick ad, while still paying the publisher for the page view, so that might get what you want.
roel_v 11y ago
"I'll never click on" Doesn't matter, impressions are valuable to marketers as well. "or be persuaded by ads" Yes you will, you just think you don't/won't.
- echelon 11y ago
  > Yes you will, you just think you don't/won't. That's a really blanket statement. I'm rather minimalist, and I typically form negative opinions of the things I see in ads.
  - deanCommie 11y ago
    You are one person. Statistics show most people are not like you.

tschellenbach 11y ago

Why would you reinvent a piece of technology offered by various cloud providers? It doesn't make any sense, what a waste of engineering resources (fun exercise though). They should have been able to reach some middle ground with AWS about pricing. I really doubt their in-house storage backend will be cheaper compared to the lowest rate they can get from one of the cloud providers.

johnloeber 11y ago
Because having your entire business model depend on the pricing whims of another business (which in the grand scheme of things is a competitor, btw) is unsustainable. Getting out of AWS is exactly the right move for Dropbox's long term health.
- tschellenbach 11y ago
  There are many cloud hosting providers and they are competing intensely. AWS marginal cost increase for supporting Dropbox is far below the cost of Dropbox building and maintaining their own solution. Given these facts they should be able to negotiate a far better price/cost than their current solution of building it in-house. There is nothing unique about their requirements which justify this move. It's a really weird decision. Probably we're missing something here. This must be part of some larger strategy they are executing.
  - toyg 11y ago
    As jamwt mentioned above, storage is simply the most critical element of their business, so it makes sense to control it as tightly as possible , similar to what Netflix does with their CDN. The cloud is for stuff you don't care too much about.
- deleted 11y ago
  [deleted]

turingbook 11y ago

I hated the writing style of Wired guys. The blog from Dropbox is much more clear: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure/ https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr...

known 11y ago

TL;DR Amazon Cloud is expensive;

JackPoach 11y ago

I wonder how capital intensive the move is. Isn't Dropbox overall still unprofitable as a company?

ldom66 11y ago

Unrelated to the article but these portraits are really beautiful! Props to the photograph and studio.

virtuallynathan 11y ago

It looks like the servers they make use of are purchasable, part of the Dell DSS series - the DSS7000: http://downloads.dell.com/manuals/common/dss%207000_dss%207500_owner%27s%20manual_en.pdf http://downloads.dell.com/manuals/common/dss%207000_dss%2075... 90x 3.5in disks, 2x compute nodes with 2x E5-2600v3 CPUs.

jamwt 11y ago
We actually use several different servers from different vendors, and some of them are more customized than others.
- virtuallynathan 11y ago
  Ah, yes I spot the 78 bay Quanta one: http://www.qct.io/Product/Storage/Storage-Server/4U/New-QuantaPlex-T21P-4U-p291c77c71c150c222 http://www.qct.io/Product/Storage/Storage-Server/4U/New-Quan... Can't identify the 3rd one yet... EDIT: Found it... HPE Cloudline CL5200, 80 drive bays. http://www.nextplatform.com/wp-content/uploads/2016/03/hpe-cl5200-server.jpg http://www.nextplatform.com/wp-content/uploads/2016/03/hpe-c...

max_ 11y ago

whats the "brand new programming language" the writer meant ?

yberreby 11y ago
It is Rust[1]: > So, in the middle of this two-and-half-year project, they switched to Rust on the Diskotech machines. And that’s what Dropbox is now pushing into its data centers. See https://www.reddit.com/r/rust/comments/4adabk/the_epic_story_of_dropboxs_exodus_from_the_amazon/ https://www.reddit.com/r/rust/comments/4adabk/the_epic_story... for more information. [1]: https://www.rust-lang.org/ https://www.rust-lang.org/

ccannon 11y ago

I applaud your efforts at answering all these questions and providing very detailed responses.

dmitrifedorov 11y ago

I'm done with Wired: cancelled its subscription a couple months ago (after many years), now they demand I login to read their web site because I use ad blockers. Done!

cj0 11y ago

Some requests for the DropBox Diskotech tech blog series: For which datacenter cooling system are the Diskotech storage pods designed? Hot aisle cold aisle? How evenly can the disks inside the Diskotech be cooled? With this question I try to ask, how good is it in keeping all drives cool? For example for the theoretical case where all disks in the Diskotech are dissipating identical heat, how much variation is there in drive temperature at roughly 25ºC intake temperature? How much power is consumed for cooling (despite that it will be fractionally low compared to what the disks will consume)? Is there any drive idle optimization done or is the expectance that not any disk will reach its idle (non-rotating) state ever? Is there any optimization done inside a Diskotech to reduce the (inrush) current surge when connecting the machine to power?

wchrisn 11y ago

Can you share your development practices in rust like TDD, Automated Builds etc

wchrisn 11y ago

Can you share your development practices in rust like TDD, Automated Builds etc

monomaniar 11y ago

Awosome