4 ms·

WaveNet, if I remember correctly, has 1-from-256 encoding of input features. And 1-from-256 encoding of output features. It is extremely sparse. If you look a

• stephenroller 7y ago
  Embeddings tables aren't hard on the GPU (being only a lookup table), and the output softmax still requires you do the full matrix-multiply. The label may be sparse, but the computation is far from sparse.
  • yvdriess 7y ago
    The reverse is true, embeddings are both the performance and memory-footprint bottleneck of modern NN models. Check figure 6. of : https://arxiv.org/pdf/1906.00091.pdf https://arxiv.org/pdf/1906.00091.pdf Embeddings are used to lookup sparse features, so you have those pesky data-dependent lookups.
    • zeroxfe 7y ago
      > The reverse is true, embeddings are both the performance and memory-footprint bottleneck of modern NN models. They may be a bottleneck, but the alternative is worse -- you can't fit complex models with large vocabularies into GPU memory using sparse one-hot encodings.
      • thesz 7y ago
        No, you can successfully fit complex models if you use byte-pair or similar encodings (morphessor [1] comes to mind). [1] https://morfessor.readthedocs.io/en/latest/ https://morfessor.readthedocs.io/en/latest/ You also will get much more meaningful embeddings from summing embeddings of part of the word.
      • yvdriess 7y ago
        Surely you mean dense one-hot? Technically, the sparse one-hot encoding is the most efficient in terms of memory footprint. You simply store the non-zero coordinates. The problem in practice for GPUs is that sparse vector/matrix operations are too inefficient. The whole point of something like this paper is to skip the entire 'densification' step and to directly deal with the sparse matrix input as a sparse matrix. The LSH is used in this paper improves on directly using SpMSpV, as that is also inefficient on CPUs, although to a lesser extent than GPUs.
      • deleted 7y ago
        [deleted]
    • Der_Einzige 7y ago
      Only a bad bottleneck because proper database techniques aren't being used widely for embeddings yet within ML pipelines See libraries like magnitude for proper embedding lookup implementations
• zeroxfe 7y ago
  > If you look at language modeling, then things there are even sparsier - typical neural language model has 1-from-several-hundredths-of-thousands for full language Most real-world models don't use one-hot encodings of words -- they use embeddings instead, which are very dense vector representations of words. Outside of the fact that embeddings don't blow out GPU memory, they're also semantically encoded, so similar words cluster together.
  • thesz 7y ago
    First, you need to compute these embeddings at least once - sparsity, here you are! Second, these embeddings may be different between tasks and accuracy from their use may differ too. For example, the embeddings produced from CBOW and skipgram word2vec models are strikingly different in cosine similarity sense - different classes of words are similar in CBOW and skipgram.
  • yvdriess 7y ago
    So you agree that the problem is fundamentally sparse? Embeddings are used to make sparse (e.g. categorical) data possible on GPUs, and real-world models are limited by how large they can make the embeddings to fit in GPU memory. Embedding lookups is also a compute bottleneck: An example is facebook DLRM: https://arxiv.org/pdf/1906.00091.pdf https://arxiv.org/pdf/1906.00091.pdf