12 ms·

How image search works at Dropbox

189 points by hyzyla 5y ago

blntechie 5y ago
One thing which annoys me with lot of these tech blogs are that they don’t support RSS. I would like to subscribe and read them as lot of these information are generally a good read and learning. To be fair, they have a neat feature to subscribe for their new posts by email but somehow RSS is harder?
- jperry 5y ago
  RSS is essentially dead, sadly. Can't push ads or gather analytics through it so it's been "optimised out" of the web. Even those rare sites that DO support RSS quite often only show the first paragraph or even just the title of the page, which I suppose is an acceptable compromise.
  - kenniskrag 5y ago
    They can for example: * only provide lead text * use html with image ads * use html with an ads-js
  - wfriesen 5y ago
    I use rssbridge for sites without a feed, and the feediron plugin for ttrss for sites whose feed is incomplete. https://github.com/RSS-Bridge/rss-bridge https://github.com/RSS-Bridge/rss-bridge https://github.com/feediron/ttrss_plugin-feediron https://github.com/feediron/ttrss_plugin-feediron https://git.tt-rss.org/fox/tt-rss https://git.tt-rss.org/fox/tt-rss
  - corobo 5y ago
    For folks on iOS the app Unread does an instapaper-like reader view of the feed text if it detects (or if you set manually) an excerpt feed
  - cblconfederate 5y ago
    As a protocol it hasnt been replaced by something better. So it isn't dead, just pushed out of the market by grabby walled gardens
- karlding 5y ago
  The Dropbox blog has an RSS feed [0] though. [0] https://tech.dropbox.com/feed/ https://tech.dropbox.com/feed/
  - blntechie 5y ago
    Thanks!! I tried to find and Feedly also for some reason couldn’t find GR it.
programLyrique 5y ago
Image search is one of the pain points when you want to deploy your own cloud, and what I'd miss compared to Google Photo or that image search in Dropbox. Is there something like that in Nextcloud for instance? I am aware of a basic face recognition app (https://github.com/matiasdelellis/facerecognition https://github.com/matiasdelellis/facerecognition)
- geekuillaume 5y ago
  PhotoPrism (https://github.com/photoprism/photoprism https://github.com/photoprism/photoprism) includes a TensorFlow model to tag pictures and search them.
rock_artist 5y ago
Image search is great. But in terms of photo backups Dropbox is inferior to Apple's iPhoto/iCloud (I'm aware of Apple's ability for better integration) but still Google photos is much more functional. I'm not asking for much but: - better caching. Their thumbnails slows up badly trying to go back few years ago. - better organization, search is great but I also like basic ability to visually search and organize content myself. - live photos support... And bugfixing... I've actually had very bad communication with them after they've broke HEIF copy paste on older iPhones... https://www.talaviram.com/uncategorized/when-did-dropbox-customer-service-went-bad/ https://www.talaviram.com/uncategorized/when-did-dropbox-cus...
- andrewshadura 5y ago
  What I can't understand is that they had a super fast and innovative application for photos, Carousel, and they killed it.
  - rock_artist 5y ago
    Yeah. I've actually started with carousel on Android and indeed it was with great potential. Too bad they've axed it
- Isn0gud 5y ago
  Google Photos also takes quite a bit with loading older thumbnails, especially a few years ago.
- iscrewyou 5y ago
  I really think this is a missed opportunity in the photo space. There are plenty of editors and tools for photos out there. They can provide something that helps with organization. Dropbox will do wonders if they had a Picasa like client that simply looked into the Dropbox folders and created albums based on folders and sub-folders structure.
ant6n 5y ago
I have a dropbox with Directories for all my projects. Let’s assume I also want to upload all my personal photos. Then I wonder - how do I keep the photos separate from the projects (which also includes images) - how will my photos go from my familys camera phones to the Dropbox? - how do i go look at photos scrolling through dec 2018 on my phone for the next 2 min? To me Google photos and drodpbox are conceptually different - photo albums vs files in Directories, and i can’t wrap my head around how albums could work in Dropbox.
vishnumohandas 5y ago
If you have concerns about unencrypted storage of personal photos, I’d request you to check out my project: ente.io[1]. It is essentially Google Photos, minus search capabilities, plus end-to-end encryption. [1]: https://ente.io https://ente.io
- leokennis 5y ago
  Looks good, but unfortunately my first thought is: this looks like a 1 or 2 person shop, how big is the risk of them calling it quits within 5 years from now? For that reason, I use https://www.photosync-app.com/ https://www.photosync-app.com/ and back up my photo's to my webhost and to B2: both the app and the storage locations are interchangeable in case any of them stops working.
  - vishnumohandas 5y ago
    Thanks for the feedback! > how big is the risk of them calling it quits In our specific case, the business is setup such that it is self-sustaining. There's no free plan, so for as long as you are paying for your storage, we'll be profitable. Outside that, I would say trust snowballs in the long run. Also, I don't know if size of a company is a metric that should warrant additional trust. The mission could be diluted in a larger organization, and hard-pivots could hurt them lesser.
    - john2010 5y ago
      What a strange ToS: We can also change the fees for our services (other than those you have already contracted and paid for) at any time if we give you notice. 10.2 make you pay, on demand, default interest on any amount you owe us at 10% per annum calculated on a daily basis, from the date when payment was due until the date when payment is actually made by you. You will also need to pay all expenses and costs (including our full legal costs) in connection with us trying to recover any unpaid amount from you. In circumstances where we cease providing our services for other reasons, we will, if we consider it appropriate, it is reasonably practicable and we are not prevented by law or likely to incur any liability in doing so, give you 30 days' notice to retrieve your data. Not sure 10.2 is legal in EU?
johnwalkr 5y ago
Between this type of searching and GPS tags, image libraries are such a joy to maintain and use compared to 10+ years ago.
- AlexDanger 5y ago
  And yet using music libraries (specifically Youtube Music in my case) is a source of both pain and frustration. Image libraries have done well, yet I feel music management has gone backwards.
  - johnwalkr 5y ago
    I’m happy with iTunes match, which gives you access to anything you own on Apple’s servers for streaming, plus uploads whatever you have and they don’t have.
phenkdo 5y ago
Searching with text tags seems a bit odd, couldn't you search-by-example with image embeddings? or is the scale too large for an ANN to handle?
- quonn 5y ago
  It seems like the obvious way to do it, to be honest, and clearly is much more scalable and flexible. So if the results are good enough, why not?
- londons_explore 5y ago
  > is the scale too large for an ANN to handle? Assuming you are searching just a single account, there are unlikely to be more than 1 million images in a typical account. A simple linear scan of all of those feature vectors should be a simple matter of milliseconds.
- gk1 5y ago
  That would work for sure if they wanted to add a "more images like this" feature. Here's an example using the SqueezeNet model and Pinecone (where scale isn't an issue): https://www.pinecone.io/docs/examples/image-similarity-search/ https://www.pinecone.io/docs/examples/image-similarity-searc... Encoding both word and image embeddings into the same index then doing ANN on that index might also work. See this example of text-to-image retrieval: https://paperswithcode.com/task/texture-image-retrieval https://paperswithcode.com/task/texture-image-retrieval Maybe someone from Dropbox can add more color to this and explain what other options they considered. As it stands, I still can't find what I need in Dropbox. And never could. From reading this article I'd think searching for a basic keyword like "dog" or "ship" or "runner" would yield some results from my tens of thousands of photos, yet I get nothing (nothing relevant, at least). Edit: On second reading, this is only available to Dropbox Pro and Business users. I hope they roll this out to other paying users soon.
  - tb99 5y ago
    Search by image is conceptually easier because you don't have to map between text and images, but it's a very different product. It is something we've considered. Encoding words and images into the same space and doing ANN is kind of what the current system is, if you look at it right. The ANN is framed in terms of similarity rather than distance -- and is approximate because of the sparseness approximation. But the big difference from the papers you linked is what we use as the encodings: not the traditional penultimate layer of a network, but classifier scores for images and projected word vectors for text. This gives us a space with semantically meaningful dimensions, which lets us build the system without a large multimodal training set; our text and image models are independently trained on different datasets.
golergka 5y ago
I didn't know that they do that, and now that I know, I'm not sure if I ever wanted to use that feature and feel a bit uneasy about Dropbox scanning all my private photos. It really feels like "your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should" situation.
- mxuribe 5y ago
  Just started a Jurassic Park marathon the other day, and can recognize this quote from that film! So relevant!
Arathorn 5y ago
Didn't Dropbox used to be E2EE? Is the indexing clientside (like Apple) or serverside?
- feross 5y ago
  Dropbox has never offered E2EE as far I’m aware. I specifically call this out in the Wormhole about page: https://wormhole.app/about https://wormhole.app/about
  - londons_explore 5y ago
    It would be good to see this indexing and searching done client side. Since there is no cross-user information used, I wonder if perhaps the algorithm was specifically designed for implementation client side...
gatestone 5y ago
I have Dropbox Plus. Image search only works on Professional? I am pretty happy with iPhone/iCloud image search. I sync to Dropbox and Google Photos as a backup. But my older photos are only in Dropbox. I am considering copying it all to iCloud, to have better management of it all.
- gk1 5y ago
  I'm with you. I get don't get the reasoning for giving it to paying Pro users but not paying personal (Plus and Family) users. I deal with far more photos in my personal account than in any work-related account.
  - Kye 5y ago
    They've fully committed to being a B2B company, and that apparently means putting ordinary B2C features at B2B price tiers even when it makes no sense.
londons_explore 5y ago
This is pretty basic keyword-category-vector stuff, and the results sadly match. Try searching your dropbox for "the day I took a bunch of photos on the subway on the way to work", and you'll see none... Yet dropbox probably has all the info to answer that search - they can parse the natural language query, they can detect when multiple photos were uploaded on the same day, they can look at location and time tags and see which ones might be 'on the way to work'. They can see which photos might visually look like they were taken on a subway. How about "Me on dress-silly day". Again, no matches. Or "My broken arm". No matches. But I totally have that image. Dropbox need to take a step back, and consider that for each query there probably is a correct answer. And they need to track what the user types, and which image they eventually view, as training data to refine their algorithm.
- Isn0gud 5y ago
  Do these complex queries work on google photos, apple photos or whatever the state of the art image search is? With search, performance is key which makes anything O(n^2) unusable.
  - londons_explore 5y ago
    For personal data search, the number of images that need to be searched is pretty small. Typically rather than an indexing system, it's best to just do as much precomputation as possible so that a linear scan is fast. That scales up to 1M+ images/user. Normally the approach taken is to preprocess all the images with a neural net (putting as input the image, metadata, some info from other images in the same location, same day, text from the web looked up from location coordinates, any other input that might answer a users query). Output an embedding vector of say 8192 elements. Then when the query comes in, put it as input to some big pretrained language model with a fine tuned embedding layer to give another vector. Then, for each image in the users account (1 million plus), run a tiny neural net to see if an image might be relevant. Such a network might only have a few thousand weights, and may only operate on part of the image and query vector. You'll probably want to use a GPU for this step, but it should work on a CPU too just about. Take the top scoring few thousand images, and run a bigger comparison net for the final ranking. You might want an extra input to the comparison net to give result diversity - ie. to try to avoid 50 very similar images all being returned at the top of the rankings. Then all networks should be end-to-end trained on user behaviour - ie. the image users actually found that answered their query.
rainboiboi 5y ago
Took them long enough to apply EfficientNet on images
h0nd 5y ago
I rather not want Dropbox to know about the contents of my images.
Graffur 5y ago
Unfortunately for Dropbox, Google Photos is by far the best at this.
eplanit 5y ago
I'm more and more motivated to just cobble together my own dropbox (I know that several folks here have done just that) for my simple needs, to be what dropbox originally was. Their ui has become ridiculous, and 'features' like image spy serve them more than me.
- mettamage 5y ago
  I'd call it: cloudwrap Logo: a cloud wrapping a file, or something more creative with the words cloud and wrap
  - mxuribe 5y ago
    I love this! Honestly, if i had heard the name "cloudwrap" on its own without your description, i would have thought: the uber of sandwich delivery...but i very much like your idea better! :-)
- cblconfederate 5y ago
  There's syncthing.net
- andrewzah 5y ago
  I use syncthing for that, and it's been great so far.
- rakoo 5y ago
  Take a server with sftp access, so you can mount it locally and browse with winscp if you need to. Put filestash on top and you have a nice ui. That's probably 90% of the original features => https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash
  - jaywalk 5y ago
    For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
slowhand09 5y ago
Nice try Dropbox. Dollar to donuts they are using Mechanical Turk....
- plasticsoprano 5y ago
  So the large blog post is just an elaborate lie?
heipei 5y ago
I haven't used the feature, but the way it's implemented feels overly complicated, especially for something like keyword search (and not similar-image-search). If they only use the Top10 categories in their feature vector for the documents, why don't they store these categories as tags on each documented and use standard inverted-index searching and scoring. I know the vector will express how much "beach" a certain image is, but your user-supplied query doesn't have a notion of how "much" beach the user expects, so the output can be a simple list ranked using standard term search mechanisms. What am I missing?
- ripper1138 5y ago
  Maybe the engineers were up for promo and needed something cool to stand out and inspire a blog post.
- tb99 5y ago
  Hi, I wrote this post. The retrieval stage amounts to storing the top 50 categories in the inverted index and searching for the top 10 from the query. The harder part is the ranking. There are approaches to the ranking that are more akin to the techniques you'd use for text document search -- think of the classifier scores as TF-IDF values, treat query-to-category matching as something like synonyms -- but to my mind they're more complicated than our approach, not less. We did experiment a bit with some of these ideas, but the results were worse.
emehrkay 5y ago
Cosine similarity is a great little tool to have in your box. When you have a map[dict|obj] with key => val(float) it is an easy way to compare a bunch of them. Years ago I threw together an example in JS https://jsfiddle.net/25o78yzt/3/ https://jsfiddle.net/25o78yzt/3/
basseq 5y ago
The overwhelming sentiments of comments on this is negative, but as a fairly non-technical audience far removed from ML, I find this pretty fascinating.
- yannoninator 5y ago
  HN is negative about all tech except Rust and Haskell.
  - willis936 5y ago
    Why strawman for user hostile policies? Users don't even know that they are giving up privacy. How is that remotely acceptable? How is criticizing such a glaring issue being "negative about tech". At some point it's just tiring to be a user that is continuously jerked around. I love tech, but everyday I understand the libre hermit POV more.
    - yannoninator 5y ago
      > Why strawman for user hostile policies? I'm stating a fact of this tech forum. > Users don't even know that they are giving up privacy. lol, Do you think the users care about privacy on Dropbox. The tech boffins might, but not the average user. How have you been spied on as a Dropbox user? Can you do a better service than them? What alternative do you suggest that is in the same league as Dropbox?
      - willis936 5y ago
        >I'm stating a fact of this tech forum. And I'm stating facts about Dropbox. Users are not told that their data will be looked at by others. Having the keys to do it and actually doing it are two very different things. Why are you asking all of these questions about the current market? Are you just trying to point out how bad the current situation is? If so, then I agree. That's why I and other users here paying attention are doling out criticism.
tb99 5y ago
Hi everyone, I wrote this post. I didn't notice it on hn until just now, but I'm happy to answer any questions about it.
- unknown_error 5y ago
  It works well! I remember being pleasantly surprised by this feature when I was searching for an unrelated document last time I logged in, and it found a bunch of images. I was like, wait, how did it know that...? It's a cool stealth feature. Any chance Dropbox might want to add more DAMS-like features in the future, so that Dropbox can be used as a central image repository not just for photo originals but public websites, etc.? Kinda like Imgix or Cloudinary? I know our org already uses Dropbox to store asset originals, and it would be lovely to be able to hook in to a powerful API to serve it at different responsive sizes, formats, cropping, etc.
  - tb99 5y ago
    Thanks! I can't talk about potential future releases, but we do have some nice thumbnailing capabilities that could be used for this sort of thing if we decided to go that way. Some of these capabilities are exposed in our public API: https://dropbox.tech/developers/scaling-down-large-image-files-using-the-get-thumbnail-api https://dropbox.tech/developers/scaling-down-large-image-fil...
- deleted 5y ago
  [deleted]
- duckburg2079 5y ago
  Very cool writeup! There is a lot of progress in multi-modal AI/image-text/video matching. Are you guys hiring researchers (who like to build things) in this area?
  - tb99 5y ago
    Yeah, there's been a lot of good stuff recently! We are definitely a build-things team more than a research team. I'm actually not sure about our hiring stance right now, but email me (address in profile) if you're interested and I'll look into it.
stuaxo 5y ago
TIL there is image search in Dropbox.

189 points by hyzyla 5y ago

One thing which annoys me with lot of these tech blogs are that they don’t support RSS. I would like to subscribe and read them as lot of these information are generally a good read and learning. To be fair, they have a neat feature to subscribe for their new posts by email but somehow RSS is harder?

jperry 5y ago
RSS is essentially dead, sadly. Can't push ads or gather analytics through it so it's been "optimised out" of the web. Even those rare sites that DO support RSS quite often only show the first paragraph or even just the title of the page, which I suppose is an acceptable compromise.
- kenniskrag 5y ago
  They can for example: * only provide lead text * use html with image ads * use html with an ads-js
- wfriesen 5y ago
  I use rssbridge for sites without a feed, and the feediron plugin for ttrss for sites whose feed is incomplete. https://github.com/RSS-Bridge/rss-bridge https://github.com/RSS-Bridge/rss-bridge https://github.com/feediron/ttrss_plugin-feediron https://github.com/feediron/ttrss_plugin-feediron https://git.tt-rss.org/fox/tt-rss https://git.tt-rss.org/fox/tt-rss
- corobo 5y ago
  For folks on iOS the app Unread does an instapaper-like reader view of the feed text if it detects (or if you set manually) an excerpt feed
- cblconfederate 5y ago
  As a protocol it hasnt been replaced by something better. So it isn't dead, just pushed out of the market by grabby walled gardens
karlding 5y ago
The Dropbox blog has an RSS feed [0] though. [0] https://tech.dropbox.com/feed/ https://tech.dropbox.com/feed/
- blntechie 5y ago
  Thanks!! I tried to find and Feedly also for some reason couldn’t find GR it.

programLyrique 5y ago

Image search is one of the pain points when you want to deploy your own cloud, and what I'd miss compared to Google Photo or that image search in Dropbox. Is there something like that in Nextcloud for instance? I am aware of a basic face recognition app (https://github.com/matiasdelellis/facerecognition https://github.com/matiasdelellis/facerecognition)

geekuillaume 5y ago
PhotoPrism (https://github.com/photoprism/photoprism https://github.com/photoprism/photoprism) includes a TensorFlow model to tag pictures and search them.

rock_artist 5y ago

Image search is great. But in terms of photo backups Dropbox is inferior to Apple's iPhoto/iCloud (I'm aware of Apple's ability for better integration) but still Google photos is much more functional. I'm not asking for much but: - better caching. Their thumbnails slows up badly trying to go back few years ago. - better organization, search is great but I also like basic ability to visually search and organize content myself. - live photos support... And bugfixing... I've actually had very bad communication with them after they've broke HEIF copy paste on older iPhones... https://www.talaviram.com/uncategorized/when-did-dropbox-customer-service-went-bad/ https://www.talaviram.com/uncategorized/when-did-dropbox-cus...

andrewshadura 5y ago
What I can't understand is that they had a super fast and innovative application for photos, Carousel, and they killed it.
- rock_artist 5y ago
  Yeah. I've actually started with carousel on Android and indeed it was with great potential. Too bad they've axed it
Isn0gud 5y ago
Google Photos also takes quite a bit with loading older thumbnails, especially a few years ago.
iscrewyou 5y ago
I really think this is a missed opportunity in the photo space. There are plenty of editors and tools for photos out there. They can provide something that helps with organization. Dropbox will do wonders if they had a Picasa like client that simply looked into the Dropbox folders and created albums based on folders and sub-folders structure.

ant6n 5y ago

I have a dropbox with Directories for all my projects. Let’s assume I also want to upload all my personal photos. Then I wonder - how do I keep the photos separate from the projects (which also includes images) - how will my photos go from my familys camera phones to the Dropbox? - how do i go look at photos scrolling through dec 2018 on my phone for the next 2 min? To me Google photos and drodpbox are conceptually different - photo albums vs files in Directories, and i can’t wrap my head around how albums could work in Dropbox.

vishnumohandas 5y ago

If you have concerns about unencrypted storage of personal photos, I’d request you to check out my project: ente.io[1]. It is essentially Google Photos, minus search capabilities, plus end-to-end encryption. [1]: https://ente.io https://ente.io

leokennis 5y ago
Looks good, but unfortunately my first thought is: this looks like a 1 or 2 person shop, how big is the risk of them calling it quits within 5 years from now? For that reason, I use https://www.photosync-app.com/ https://www.photosync-app.com/ and back up my photo's to my webhost and to B2: both the app and the storage locations are interchangeable in case any of them stops working.
- vishnumohandas 5y ago
  Thanks for the feedback! > how big is the risk of them calling it quits In our specific case, the business is setup such that it is self-sustaining. There's no free plan, so for as long as you are paying for your storage, we'll be profitable. Outside that, I would say trust snowballs in the long run. Also, I don't know if size of a company is a metric that should warrant additional trust. The mission could be diluted in a larger organization, and hard-pivots could hurt them lesser.
  - john2010 5y ago
    What a strange ToS: We can also change the fees for our services (other than those you have already contracted and paid for) at any time if we give you notice. 10.2 make you pay, on demand, default interest on any amount you owe us at 10% per annum calculated on a daily basis, from the date when payment was due until the date when payment is actually made by you. You will also need to pay all expenses and costs (including our full legal costs) in connection with us trying to recover any unpaid amount from you. In circumstances where we cease providing our services for other reasons, we will, if we consider it appropriate, it is reasonably practicable and we are not prevented by law or likely to incur any liability in doing so, give you 30 days' notice to retrieve your data. Not sure 10.2 is legal in EU?

johnwalkr 5y ago

Between this type of searching and GPS tags, image libraries are such a joy to maintain and use compared to 10+ years ago.

AlexDanger 5y ago
And yet using music libraries (specifically Youtube Music in my case) is a source of both pain and frustration. Image libraries have done well, yet I feel music management has gone backwards.
- johnwalkr 5y ago
  I’m happy with iTunes match, which gives you access to anything you own on Apple’s servers for streaming, plus uploads whatever you have and they don’t have.

phenkdo 5y ago

Searching with text tags seems a bit odd, couldn't you search-by-example with image embeddings? or is the scale too large for an ANN to handle?

quonn 5y ago
It seems like the obvious way to do it, to be honest, and clearly is much more scalable and flexible. So if the results are good enough, why not?
londons_explore 5y ago
> is the scale too large for an ANN to handle? Assuming you are searching just a single account, there are unlikely to be more than 1 million images in a typical account. A simple linear scan of all of those feature vectors should be a simple matter of milliseconds.
gk1 5y ago
That would work for sure if they wanted to add a "more images like this" feature. Here's an example using the SqueezeNet model and Pinecone (where scale isn't an issue): https://www.pinecone.io/docs/examples/image-similarity-search/ https://www.pinecone.io/docs/examples/image-similarity-searc... Encoding both word and image embeddings into the same index then doing ANN on that index might also work. See this example of text-to-image retrieval: https://paperswithcode.com/task/texture-image-retrieval https://paperswithcode.com/task/texture-image-retrieval Maybe someone from Dropbox can add more color to this and explain what other options they considered. As it stands, I still can't find what I need in Dropbox. And never could. From reading this article I'd think searching for a basic keyword like "dog" or "ship" or "runner" would yield some results from my tens of thousands of photos, yet I get nothing (nothing relevant, at least). Edit: On second reading, this is only available to Dropbox Pro and Business users. I hope they roll this out to other paying users soon.
- tb99 5y ago
  Search by image is conceptually easier because you don't have to map between text and images, but it's a very different product. It is something we've considered. Encoding words and images into the same space and doing ANN is kind of what the current system is, if you look at it right. The ANN is framed in terms of similarity rather than distance -- and is approximate because of the sparseness approximation. But the big difference from the papers you linked is what we use as the encodings: not the traditional penultimate layer of a network, but classifier scores for images and projected word vectors for text. This gives us a space with semantically meaningful dimensions, which lets us build the system without a large multimodal training set; our text and image models are independently trained on different datasets.

golergka 5y ago

I didn't know that they do that, and now that I know, I'm not sure if I ever wanted to use that feature and feel a bit uneasy about Dropbox scanning all my private photos. It really feels like "your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should" situation.

mxuribe 5y ago
Just started a Jurassic Park marathon the other day, and can recognize this quote from that film! So relevant!

Arathorn 5y ago

Didn't Dropbox used to be E2EE? Is the indexing clientside (like Apple) or serverside?

feross 5y ago
Dropbox has never offered E2EE as far I’m aware. I specifically call this out in the Wormhole about page: https://wormhole.app/about https://wormhole.app/about
- londons_explore 5y ago
  It would be good to see this indexing and searching done client side. Since there is no cross-user information used, I wonder if perhaps the algorithm was specifically designed for implementation client side...

gatestone 5y ago

I have Dropbox Plus. Image search only works on Professional? I am pretty happy with iPhone/iCloud image search. I sync to Dropbox and Google Photos as a backup. But my older photos are only in Dropbox. I am considering copying it all to iCloud, to have better management of it all.

gk1 5y ago
I'm with you. I get don't get the reasoning for giving it to paying Pro users but not paying personal (Plus and Family) users. I deal with far more photos in my personal account than in any work-related account.
- Kye 5y ago
  They've fully committed to being a B2B company, and that apparently means putting ordinary B2C features at B2B price tiers even when it makes no sense.

londons_explore 5y ago

This is pretty basic keyword-category-vector stuff, and the results sadly match. Try searching your dropbox for "the day I took a bunch of photos on the subway on the way to work", and you'll see none... Yet dropbox probably has all the info to answer that search - they can parse the natural language query, they can detect when multiple photos were uploaded on the same day, they can look at location and time tags and see which ones might be 'on the way to work'. They can see which photos might visually look like they were taken on a subway. How about "Me on dress-silly day". Again, no matches. Or "My broken arm". No matches. But I totally have that image. Dropbox need to take a step back, and consider that for each query there probably is a correct answer. And they need to track what the user types, and which image they eventually view, as training data to refine their algorithm.

Isn0gud 5y ago
Do these complex queries work on google photos, apple photos or whatever the state of the art image search is? With search, performance is key which makes anything O(n^2) unusable.
- londons_explore 5y ago
  For personal data search, the number of images that need to be searched is pretty small. Typically rather than an indexing system, it's best to just do as much precomputation as possible so that a linear scan is fast. That scales up to 1M+ images/user. Normally the approach taken is to preprocess all the images with a neural net (putting as input the image, metadata, some info from other images in the same location, same day, text from the web looked up from location coordinates, any other input that might answer a users query). Output an embedding vector of say 8192 elements. Then when the query comes in, put it as input to some big pretrained language model with a fine tuned embedding layer to give another vector. Then, for each image in the users account (1 million plus), run a tiny neural net to see if an image might be relevant. Such a network might only have a few thousand weights, and may only operate on part of the image and query vector. You'll probably want to use a GPU for this step, but it should work on a CPU too just about. Take the top scoring few thousand images, and run a bigger comparison net for the final ranking. You might want an extra input to the comparison net to give result diversity - ie. to try to avoid 50 very similar images all being returned at the top of the rankings. Then all networks should be end-to-end trained on user behaviour - ie. the image users actually found that answered their query.

rainboiboi 5y ago

Took them long enough to apply EfficientNet on images

h0nd 5y ago

I rather not want Dropbox to know about the contents of my images.

Graffur 5y ago

Unfortunately for Dropbox, Google Photos is by far the best at this.

eplanit 5y ago

I'm more and more motivated to just cobble together my own dropbox (I know that several folks here have done just that) for my simple needs, to be what dropbox originally was. Their ui has become ridiculous, and 'features' like image spy serve them more than me.

mettamage 5y ago
I'd call it: cloudwrap Logo: a cloud wrapping a file, or something more creative with the words cloud and wrap
- mxuribe 5y ago
  I love this! Honestly, if i had heard the name "cloudwrap" on its own without your description, i would have thought: the uber of sandwich delivery...but i very much like your idea better! :-)
cblconfederate 5y ago
There's syncthing.net
andrewzah 5y ago
I use syncthing for that, and it's been great so far.
rakoo 5y ago
Take a server with sftp access, so you can mount it locally and browse with winscp if you need to. Put filestash on top and you have a nice ui. That's probably 90% of the original features => https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash
- jaywalk 5y ago
  For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.

slowhand09 5y ago

Nice try Dropbox. Dollar to donuts they are using Mechanical Turk....

plasticsoprano 5y ago
So the large blog post is just an elaborate lie?

heipei 5y ago

I haven't used the feature, but the way it's implemented feels overly complicated, especially for something like keyword search (and not similar-image-search). If they only use the Top10 categories in their feature vector for the documents, why don't they store these categories as tags on each documented and use standard inverted-index searching and scoring. I know the vector will express how much "beach" a certain image is, but your user-supplied query doesn't have a notion of how "much" beach the user expects, so the output can be a simple list ranked using standard term search mechanisms. What am I missing?

ripper1138 5y ago
Maybe the engineers were up for promo and needed something cool to stand out and inspire a blog post.
tb99 5y ago
Hi, I wrote this post. The retrieval stage amounts to storing the top 50 categories in the inverted index and searching for the top 10 from the query. The harder part is the ranking. There are approaches to the ranking that are more akin to the techniques you'd use for text document search -- think of the classifier scores as TF-IDF values, treat query-to-category matching as something like synonyms -- but to my mind they're more complicated than our approach, not less. We did experiment a bit with some of these ideas, but the results were worse.

emehrkay 5y ago

Cosine similarity is a great little tool to have in your box. When you have a map[dict|obj] with key => val(float) it is an easy way to compare a bunch of them. Years ago I threw together an example in JS https://jsfiddle.net/25o78yzt/3/ https://jsfiddle.net/25o78yzt/3/

basseq 5y ago

The overwhelming sentiments of comments on this is negative, but as a fairly non-technical audience far removed from ML, I find this pretty fascinating.

yannoninator 5y ago
HN is negative about all tech except Rust and Haskell.
- willis936 5y ago
  Why strawman for user hostile policies? Users don't even know that they are giving up privacy. How is that remotely acceptable? How is criticizing such a glaring issue being "negative about tech". At some point it's just tiring to be a user that is continuously jerked around. I love tech, but everyday I understand the libre hermit POV more.
  - yannoninator 5y ago
    > Why strawman for user hostile policies? I'm stating a fact of this tech forum. > Users don't even know that they are giving up privacy. lol, Do you think the users care about privacy on Dropbox. The tech boffins might, but not the average user. How have you been spied on as a Dropbox user? Can you do a better service than them? What alternative do you suggest that is in the same league as Dropbox?
    - willis936 5y ago
      >I'm stating a fact of this tech forum. And I'm stating facts about Dropbox. Users are not told that their data will be looked at by others. Having the keys to do it and actually doing it are two very different things. Why are you asking all of these questions about the current market? Are you just trying to point out how bad the current situation is? If so, then I agree. That's why I and other users here paying attention are doling out criticism.

tb99 5y ago

Hi everyone, I wrote this post. I didn't notice it on hn until just now, but I'm happy to answer any questions about it.

unknown_error 5y ago
It works well! I remember being pleasantly surprised by this feature when I was searching for an unrelated document last time I logged in, and it found a bunch of images. I was like, wait, how did it know that...? It's a cool stealth feature. Any chance Dropbox might want to add more DAMS-like features in the future, so that Dropbox can be used as a central image repository not just for photo originals but public websites, etc.? Kinda like Imgix or Cloudinary? I know our org already uses Dropbox to store asset originals, and it would be lovely to be able to hook in to a powerful API to serve it at different responsive sizes, formats, cropping, etc.
- tb99 5y ago
  Thanks! I can't talk about potential future releases, but we do have some nice thumbnailing capabilities that could be used for this sort of thing if we decided to go that way. Some of these capabilities are exposed in our public API: https://dropbox.tech/developers/scaling-down-large-image-files-using-the-get-thumbnail-api https://dropbox.tech/developers/scaling-down-large-image-fil...
deleted 5y ago
[deleted]
duckburg2079 5y ago
Very cool writeup! There is a lot of progress in multi-modal AI/image-text/video matching. Are you guys hiring researchers (who like to build things) in this area?
- tb99 5y ago
  Yeah, there's been a lot of good stuff recently! We are definitely a build-things team more than a research team. I'm actually not sure about our hiring stance right now, but email me (address in profile) if you're interested and I'll look into it.

stuaxo 5y ago

TIL there is image search in Dropbox.