4 ms·
(ML engineer from Dropbox here) For deep learning oriented projects, we train on EC2 GPU instances, generally p2.xlarge instances these days to get the Nvidia
by bradneuberg 10y ago
(ML engineer from Dropbox here)
For deep learning oriented projects, we train on EC2 GPU instances, generally p2.xlarge instances these days to get the Nvidia K80 GPUs. We can spin up many of these in parallel if we are doing model architecture searches or hyperparameter exploration.
We have an in-house data turking setup where we can efficiently roll new UIs to get ground-truth data for given problems, generally in the thousands to tens of thousands of real data. We also use data augmentation where possible, synthetically generating millions of example data points, combining it with the real turked data for fine tuning. Note that we never look at or train with real user data, unless explicitly donated, so data efficiency is important to us.
We've standardized on TensorFlow these days, doing inference on CPUs currently on Dropbox's compute infrastructure. We have a jail setup based off of LXC and Provost that allows us to safely execute these trained models in a contained way, responding to user requests as they come in. We use standard distributed systems plumbing for RPCs, queues, etc. to respond to user requests and route them to trained models. We version our trained models, and have an in-house experiments framework that we use to deploy new models and test them against subsets of user traffic to see how they are doing as we ramp up a new model.
Most of our day-to-day work is in Python, with occasional use of C++; other parts of Dropbox sometimes use Go and Rust, though we haven't had need for that on the ML team. Note that Dropbox is one of the largest users of Python in the world (Guido van Rossum actually works here).
BTW, the Machine Learning team at Dropbox is hiring. Come join us! Details: https://www.dropbox.com/jobs/listing/533100 https://www.dropbox.com/jobs/listing/533100
- locuscoeruleus 10y agoI'm curious, how (if at all) do you verify how well your models are performing on the user's data?
- soheil 10y ago> we never look at or train with real user data, unless explicitly donated What is the rationale behind this? Why not train?
- perlgeek 10y agoCouldn't user data make it into the model, and be reverse-engineered from it, if you used it to train the model? I mean, machine learning models aren't designed as one-way cryptographic hash functions, so you can't be sure that some of the training data can be inferred through the model's behavior.
- zek 10y agoYes! there is an interesting paper on this subject: https://arxiv.org/abs/1609.02943 https://arxiv.org/abs/1609.02943
- j_s 10y agoCounterpoint: https://arxiv.org/abs/1701.04082 https://arxiv.org/abs/1701.04082 Source: https://news.ycombinator.com/item?id=13824047 https://news.ycombinator.com/item?id=13824047
- gcr 10y agoPreserving users' privacy first and foremost is absolutely the right thing to do here. I'm really glad Dropbox is willing to trade away a bit of model accuracy for better user privacy.
- amelius 10y agoBecause the ML models might start reproducing snippets of user data.
- SmellTheGlove 10y agoI can answer this. I'm a paying Dropbox customer and I didn't donate my data. The information is mine, and we can discuss using it, but my expectation of privacy includes Dropbox not using it for any purpose that isn't storing and serving it to me unless I've expressly stated otherwise. While I'm not that interesting of a person, I do have an interest in things like my tax returns, financials, and other PII to not be in the hands of Dropbox employees. Short answer: Dropbox is doing the right thing, and I really appreciate that.
- fnbr 10y agoAre there any public posts discussing how you augment data? Also, what do you use you spin up multiple jobs in parallel using EC2?
- bradneuberg 10y agoNo public posts yet unfortunately. For spinning up jobs we have an internal tool that can spin up machines and provision them, but in my own experiments I just usually end up spinning up machines and rsyncing experiments over.
- fnbr 10y agoAh, fair enough. Too bad, I'd be really interested in hearing how you do that. I work for a consulting firm as a data scientist, and we have a lot of issues with data size. I'm aware of the tricks for image data (rotating & flipping, playing with different gamma levels), but I haven't seen much for standard, tabular style data.
- tokenizerrr 10y agoWhat does Dropbox use machine learning for?
- deleted 10y ago[deleted]
- bradneuberg 10y agoSee my reply to dnt404 for two examples. We have nearly an exabyte of data at Dropbox ( https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure/ https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr... ); there's lots of interesting ML things we can do with that, whether its searching, suggestions, etc.
- mattmanser 10y agoYou have an exabyte of other peoples private data that you're mining and essentially exporting to other files? Am i misunderstanding or are you admitting to severely abusing your users trust?
- bradneuberg 10y agoSee my original parent comment about not touching user data for privacy reasons :)
- wbeckler 10y agoHe says they don't use user data without explicit permission and they generate the training data using "turking," which means paying people to classify things manually.
- dnt404-1 10y agoI am studying ML this semester at university. I would like to get some ideas on where does a company like DropBox use ML. Can you state some problems that the team is working to solve or use cases?
- bradneuberg 10y agoTwo examples of where we use machine learning at Dropbox is corner and edge detection in our mobile docscanner (https://blogs.dropbox.com/tech/2016/08/fast-and-accurate-document-detection-for-scanning/ https://blogs.dropbox.com/tech/2016/08/fast-and-accurate-doc...) and computer vision for OCRing.
- nojvek 10y agoDoes Dropbox do OCR-ing? I couldn't find how to do that in the mobile app. I love the scan upload feature. I really hope you are employing world class security folks and not gonna have another password leak. I have found after the scan feature, I'm uploading more of my paper life.
- bradneuberg 10y agoWhen using the Docscanner, we do OCRing of the images for Dropbox for Business users only right now. The OCRed Docscanned PDF gets copy/pasteable text, and its search tokens go into the search index for later searching. We only offer full text search for Dropbox for Business customers as well right now, not Pro.
- jordancampbell 10y agoI'd also be really interested to learn how you generate synthetic data. Is it a 'simple' case of manually generating as much as possible and then using the statistics of this to bulk it out? Or are you using something more complex to augment it?
- tnecniv 10y agoThe general class of model that you can use to create synthetic data is called a generative model [1]. The style of learning that pits one of these generative models against a discriminative one (the class you can't generate data from, including traditional neural networks) is called adversarial learning [2]. It's worth noting that normally you can use generative models for classification on their own as well. [1] https://en.wikipedia.org/wiki/Generative_model https://en.wikipedia.org/wiki/Generative_model [2] https://en.wikipedia.org/wiki/Adversarial_machine_learning https://en.wikipedia.org/wiki/Adversarial_machine_learning
- bradneuberg 10y agoOne example of synthetic data generation was for our OCR project. We took a corpi of word choices (Project Gutenberg, modern books, the UPC database for receipts, etc.), took several thousand fonts, and combined it with geometric transformations that mimic distortions like shadows, creases, etc. to bootstrap millions of fake OCR like scannable documents. We aren't using GANs yet, but are definitely keeping an eye on them. Work like InfoGANs which has the GAN learn a ground-truth like label are very promising, but GANs don't yet work at the image sizes necessary to really make this promising. I do think in the next year or two we will see these problems solved and GANs will become an integral part of synthetic data generation.
- jacobzweig 10y agoOoh, neat! I've used GANs to generate synthetic temporal sequence data for training electrophysiologic (i.e., EEG, EMG) signal decoders. In fact, I wrote up the section of my dissertation on this topic today! In my experience it worked quite a bit better than other generative techniques (I've used convolutional variational autoencoders in the past for this and had so-so results). Looking forward to seeing what you guys do with this!
- plaguuuuuu 10y agoWhat the heck is data turking?
- clay_to_n 10y agoI think they mean manually labeling data en-masse. Maybe tagging photos with a description of what's in it, as a common example. "Turking" is probably a reference to Amazon's Mechanical Turk service, where you can to get lots of low-effort work done by people.
- bradneuberg 10y agoUsing something like Mechanical Turk to manually add ground truth data to something, though we don't use Mechanical Turk but rather other providers.
- prats226 10y agoWhat are other providers that you use? Since you are using it as input to your training set, it should have fairly high accuracy (even more accurate than manual errors in tagging)
- rtkaleta 10y agoDitto, are there any providers you are allowed to mention that you have had good experience with?
- dharma1 10y agoI guess crowd sourcing ground truthing/generating more validated data with mechanical turk?
- jxramos 10y agoThe crowd sourcing aspect is accurate, but I think the turk half is used more generically past the actual Amazon browser GUI experience tool to whatever annotation tool one has available or has invested development in. Stuff like https://www.crowdflower.com/ https://www.crowdflower.com/ One other thing worth pointing out is why you can't always lean on the Amazon Turk per se because some of the tasks you need to train from require specialist knowledge that takes years to hone. It is not a task for the common layperson who can spot a cat in an image or anything else that exploits "common knowledge". In the case of the company I'm at we had to pay contract work for these specialists to come in house and operate our annotation tools to build up our ground truth data.
- AndrewKemendo 10y agoI am surprised that you are using AWS. It's generally cheaper and faster when you are at massive scale to build your own. Why do you do it this way? in-house data turking setup Can you clarify: Are you are turking with customers inputs or with internal team members? synthetically generating millions of example data points Are you implementing this via GAN or hand tuned?
- lstyls 10y agoYou're right about scale, but I think forgetting that the model training is on very different hardware than what production web and storage servers use. Unless the have the need for dedicated GPU hardware at scale paying by compute unit still makes the best business sense.
- AndrewKemendo 10y agoDisagree. That's the whole point of me asking. It's cheaper and will be better tailored to build your own GPU stack if you need it at scale than it is to "rent one" from the cloud providers. I think it comes down to comfort and how critical it is to business processes.