I implemented a hierarchical model that pooled utf8 encoded sequences to word vectors and trained it with a decoder on text denoising. I think the future is a

I implemented a hierarchical model that pooled utf8 encoded sequences to word vectors and trained it with a decoder on text denoising. I think the future is a small word encoder model that replaces the token embedding codebook. And here’s the reason: you can still create a codebook after training and then use the encoder model only for OOV. I’m not sure there’s an excuse not to be doing this, but open to suggestions.