4 ms·

Searching with text tags seems a bit odd, couldn't you search-by-example with image embeddings? or is the scale too large for an ANN to handle?

quonn 5y ago
It seems like the obvious way to do it, to be honest, and clearly is much more scalable and flexible. So if the results are good enough, why not?
londons_explore 5y ago
> is the scale too large for an ANN to handle? Assuming you are searching just a single account, there are unlikely to be more than 1 million images in a typical account. A simple linear scan of all of those feature vectors should be a simple matter of milliseconds.
gk1 5y ago
That would work for sure if they wanted to add a "more images like this" feature. Here's an example using the SqueezeNet model and Pinecone (where scale isn't an issue): https://www.pinecone.io/docs/examples/image-similarity-search/ https://www.pinecone.io/docs/examples/image-similarity-searc... Encoding both word and image embeddings into the same index then doing ANN on that index might also work. See this example of text-to-image retrieval: https://paperswithcode.com/task/texture-image-retrieval https://paperswithcode.com/task/texture-image-retrieval Maybe someone from Dropbox can add more color to this and explain what other options they considered. As it stands, I still can't find what I need in Dropbox. And never could. From reading this article I'd think searching for a basic keyword like "dog" or "ship" or "runner" would yield some results from my tens of thousands of photos, yet I get nothing (nothing relevant, at least). Edit: On second reading, this is only available to Dropbox Pro and Business users. I hope they roll this out to other paying users soon.
- tb99 5y ago
  Search by image is conceptually easier because you don't have to map between text and images, but it's a very different product. It is something we've considered. Encoding words and images into the same space and doing ANN is kind of what the current system is, if you look at it right. The ANN is framed in terms of similarity rather than distance -- and is approximate because of the sparseness approximation. But the big difference from the papers you linked is what we use as the encodings: not the traditional penultimate layer of a network, but classifier scores for images and projected word vectors for text. This gives us a space with semantically meaningful dimensions, which lets us build the system without a large multimodal training set; our text and image models are independently trained on different datasets.
  - phenkdo 5y ago
    Interesting. Thanks for responding. 1. Did you look at CLIP? it provides a common (to images & text) embedding. 2. Do your models need specialized training (vs. open models)?
marcinzm 5y ago
If you're searching for a photo do you always have a similar photo to use as a reference? Text allows for people to simply search their photos based on what they think is in them. As a search feature that's going to have a lot more use for customers than image similarity. Image similarity has other uses but this is for search specifically.

quonn 5y ago

It seems like the obvious way to do it, to be honest, and clearly is much more scalable and flexible. So if the results are good enough, why not?

londons_explore 5y ago

> is the scale too large for an ANN to handle? Assuming you are searching just a single account, there are unlikely to be more than 1 million images in a typical account. A simple linear scan of all of those feature vectors should be a simple matter of milliseconds.

gk1 5y ago

That would work for sure if they wanted to add a "more images like this" feature. Here's an example using the SqueezeNet model and Pinecone (where scale isn't an issue): https://www.pinecone.io/docs/examples/image-similarity-search/ https://www.pinecone.io/docs/examples/image-similarity-searc... Encoding both word and image embeddings into the same index then doing ANN on that index might also work. See this example of text-to-image retrieval: https://paperswithcode.com/task/texture-image-retrieval https://paperswithcode.com/task/texture-image-retrieval Maybe someone from Dropbox can add more color to this and explain what other options they considered. As it stands, I still can't find what I need in Dropbox. And never could. From reading this article I'd think searching for a basic keyword like "dog" or "ship" or "runner" would yield some results from my tens of thousands of photos, yet I get nothing (nothing relevant, at least). Edit: On second reading, this is only available to Dropbox Pro and Business users. I hope they roll this out to other paying users soon.

tb99 5y ago
Search by image is conceptually easier because you don't have to map between text and images, but it's a very different product. It is something we've considered. Encoding words and images into the same space and doing ANN is kind of what the current system is, if you look at it right. The ANN is framed in terms of similarity rather than distance -- and is approximate because of the sparseness approximation. But the big difference from the papers you linked is what we use as the encodings: not the traditional penultimate layer of a network, but classifier scores for images and projected word vectors for text. This gives us a space with semantically meaningful dimensions, which lets us build the system without a large multimodal training set; our text and image models are independently trained on different datasets.
- phenkdo 5y ago
  Interesting. Thanks for responding. 1. Did you look at CLIP? it provides a common (to images & text) embedding. 2. Do your models need specialized training (vs. open models)?

marcinzm 5y ago

If you're searching for a photo do you always have a similar photo to use as a reference? Text allows for people to simply search their photos based on what they think is in them. As a search feature that's going to have a lot more use for customers than image similarity. Image similarity has other uses but this is for search specifically.