4 ms·
DCT is a linear transformation so the model ought to be able to learn it (and typically already does). If you're going to feed in some non-learned transformatio
by astrange 4y ago
DCT is a linear transformation so the model ought to be able to learn it (and typically already does). If you're going to feed in some non-learned transformation it seems like it should be nonlinear to add value.
- lucidrains 4y agoTrue, it goes against the spirit of deep learning Still the Transframer results are interesting and has encouraged me to look into it