Interesting 'cause once you think of it as a lossy compression task, lots of possible tricks come to mind. One is that you could "cheat" to correct the most imp

Interesting 'cause once you think of it as a lossy compression task, lots of possible tricks come to mind. One is that you could "cheat" to correct the most important errors: look at which input words have high (distance from original point * frequency of use), and save the (in their example) 300-element error vector (original embedding - estimate) for those. Since we use the most common words a lot, you might be able to recover a noticeable chunk of accuracy without using much additional space. Really, you want to correct the errors that are important, and importance to the task is a different thing from error magnitude * word frequency. You might run the network on real tasks with the uncompressed table and the compressed table, and add correction vectors for the words to which you can attribute the most mistakes made by the compressed net. And you don't necessarily have to correct the whole vector or correct it perfectly; you could round values in the correction vector down to zero when they're small enough, saving space and computation, or save your corrections with low precision, or try another round of SVD on some error vectors and only store 30 more weights for correcting important errors (and one 30x300 matrix) not 300, or some combo of tricks like that. (I think this paper is saying that after you get the SVD-compressed lookup table, you do more NN training that tweaks the embedding and possibly the rest of the net. The links and weights for correcting important errors could be tweaked in the same process.) It's tempting to treat neural networks as their own black box in a world apart from other CS sometimes, when in fact there can be lots of interesting connections. Now where all those connections ought to be and their ideal weights I don't know :P