3 ms·
They probably won't since DM doesn't open-source most of its work. The authors claimed way back in November that they'd at least open-source the code (https://o
by gwern 7y ago
They probably won't since DM doesn't open-source most of its work. The authors claimed way back in November that they'd at least open-source the code (https://openreview.net/forum?id=SylKikSYDH https://openreview.net/forum?id=SylKikSYDH) but nothing yet. (The model isn't so big that open-sourcing it is all that important. It's no Turing-NLG https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1... that's for sure!)
In the mean time, there's always Reformer, which has Trax and PyTorch implementations.
- hooande 7y agoI believe the transformer-xl pre-trained model can also be downloaded, to provide a similar long term memory functionality as the compression transformer. I don't have a direct link, but it's available via huggingface https://huggingface.co/transformers/pretrained_models.html https://huggingface.co/transformers/pretrained_models.html
- ColanR 7y agoWow, that's a lot of models. Thanks for pointing that out.
- gwern 7y agoYeah. I didn't mention Transformer-XL because I'm not sure how much of a long-range dependency it actually learns to handle. The only papers I've seen on recurrency indicate that they tend to learn very short-range dependencies, while something like Reformer with direct access to thousands of timesteps seems more likely to actually be making use of them.