3 ms·
yup, training on DCT representation in general is an untapped research direction. can also help with larger images, longer audio, etc
by lucidrains 4y ago
yup, training on DCT representation in general is an untapped research direction. can also help with larger images, longer audio, etc
- astrange 4y agoDCT is a linear transformation so the model ought to be able to learn it (and typically already does). If you're going to feed in some non-learned transformation it seems like it should be nonlinear to add value.
- lucidrains 4y agoTrue, it goes against the spirit of deep learning Still the Transframer results are interesting and has encouraged me to look into it