3 ms·

Thanks for pointing the memory optimizations out. The Google model trained on 100 billion words unzips as a 3.4 GB binary, which often is a non-starter on small

by legel 10y ago

Thanks for pointing the memory optimizations out. The Google model trained on 100 billion words unzips as a 3.4 GB binary, which often is a non-starter on smaller servers, while these vectors indeed come in at less than 100 MB. Mainly this is because only the most common 100k of the 3m vectors distributed are in here, but also because 32 bit floats for each of 300 elements per vector are converted to 16 bit floats, with virtually no loss in precision. (Research suggests we may be able to compress even further, perhaps to 5 bits of entropy per vector element.) In practice, across a wide variety of words and phrases, this pipeline can quickly derive new vectors by web searching, and thanks to the OP's HDBSCAN, can cluster them quite well. Typically it's the quality of keywords found online that ends up most limiting the quality of maps derived here, while there's probably room for breakthroughs in the keyword extraction algorithm put together here (which relies on word2vec indexes as a proxy for the idf in tf-idf). In any case if greater precision is needed - like when I wanted to map ~50 really great scientists who deserve all the credit here - one trick that can help is to increase the number of websites to scan per unknown word (e.g. from 10 to 50) in the research_keywords function.

mdda 10y ago
Embeddings can be compressed even further than that : https://arxiv.org/abs/1511.06397 https://arxiv.org/abs/1511.06397 (disclaimer: I'm the author)
- legel 10y ago
  Wow - 3 bits per element. Wonderful work with the quantization, I'd love to implement this. How would you recommend I proceed?
  - mdda 10y ago
    The paper (with fewer typos) was actually accepted into 'ICONIP' in Japan in October - so I'll definitely have code on GitHub by the end of the summer. Currently my Theano implementation is buried in typical exploratory kind of code, which just needs to be stripped away to make something functional from GloVE->Sparse in one command. The NNSE paper has associated code already, but I found setting the sparseness preference parameter was very hit-and-miss, which is why I preferred the explicit sparse-by-percentage measure in my work.
    - legel 10y ago
      ~6% sparsity + the way that you think about data representation is very interesting. I will have to have a shot at running the autoencoder over the Mikolov 3m word corpus. My goal is to get the first 1m words compressed at under 100 MB zipped, including all indexes, which currently allows us to distribute the vectors for free at github without paying for data transfer. (To date about 8000 words have been mapped through the words2map Google API, while I haven't really begun to do anything interesting here yet...)

by legel 10y ago

mdda 10y ago

Embeddings can be compressed even further than that : https://arxiv.org/abs/1511.06397 https://arxiv.org/abs/1511.06397 (disclaimer: I'm the author)

legel 10y ago
Wow - 3 bits per element. Wonderful work with the quantization, I'd love to implement this. How would you recommend I proceed?
- mdda 10y ago
  The paper (with fewer typos) was actually accepted into 'ICONIP' in Japan in October - so I'll definitely have code on GitHub by the end of the summer. Currently my Theano implementation is buried in typical exploratory kind of code, which just needs to be stripped away to make something functional from GloVE->Sparse in one command. The NNSE paper has associated code already, but I found setting the sparseness preference parameter was very hit-and-miss, which is why I preferred the explicit sparse-by-percentage measure in my work.
  - legel 10y ago
    ~6% sparsity + the way that you think about data representation is very interesting. I will have to have a shot at running the autoencoder over the Mikolov 3m word corpus. My goal is to get the first 1m words compressed at under 100 MB zipped, including all indexes, which currently allows us to distribute the vectors for free at github without paying for data transfer. (To date about 8000 words have been mapped through the words2map Google API, while I haven't really begun to do anything interesting here yet...)