4 ms·
This is great and echoes a recent fascination for me. One application of compact word embeddings is that if they're small enough you can ship a whole model to
by dougb5 9y ago
This is great and echoes a recent fascination for me. One application of compact word embeddings is that if they're small enough you can ship a whole model to the user in a web app so that semantic computations can be done entirely client-side, which is useful for privacy.
I did a naive 1-bit quantization a few months ago in order to fit a large-vocabulary word embedding into a smallish (< 3 MB) JSON file so that synonyms could be generated by the user without hitting a server. (See https://docs.google.com/presentation/d/1SfbfnuNSlDRtUOpN1KNc2RMtw030defCDME6p_268fQ/edit#slide=id.g28366446a3_0_110 https://docs.google.com/presentation/d/1SfbfnuNSlDRtUOpN1KNc... for a description and https://prototypes.cortico.ai/search-history/ https://prototypes.cortico.ai/search-history/ for the application -- happy to send you more details/code if you want) I never got as far as formally evaluating the quality of these vectors, though.
- visarga 9y agoYou can also replace rare words with a linear combination of 1..3 of their closest neighbours, especially for topic classification tasks. This would allow higher precision for more frequent words and still handle the rare words, reducing the number of vectors you need to send to the thin client.
- narrator 9y agoFinally a use for 5g bandwidth: sending big ML models to the client.
- rpedela 9y agoCan you expand on the privacy use case? I don't understand how applying a word vector model on the client improves privacy.
- darawk 9y agoYou presumably do not have to ship the content of the user's query up to the server.
- rpedela 9y agoHow does that improve privacy in this case? The sort of queries that can be made against a word vector model, and only a word vector model, are pretty trivial.
- dougb5 9y agoThe tool I made works with local data -- it analyzes your Google search history from a file on your disk. You can filter it by specifying a topic, which is then matched semantically against the search queries in your history using the word vectors. Given the sensitive nature of this data, I felt it important not to send these topic queries (nor any of the raw search history queries) to a remote server. I can also imagine using the word vectors to do a topic clustering of the search history.