I think the new things are integrating the SVD output into the NN as a couple of layers and then training it more, and adjusting the learning rate differently f

I think the new things are integrating the SVD output into the NN as a couple of layers and then training it more, and adjusting the learning rate differently from how it was done before. The training step will tend to tune the compression to work well for the specific task. Getting good results on the task is different from making the compressed representation of the embedding better on all-purpose metrics (like average magnitude of difference from the original embedding). For example, you might end up with changes that improve representation of common words, or words where errors actually mess up results, at the cost of making the decompression worse by simple metrics. (It wasn't clear to me if the training adjusts the whole net's weights or just the two layers they add up front. If it adjusts everything, you could also get joint changes to the original net + decompression machinery that only work well together.)