3 ms·

Not close in short term but it's inevitable https://sites.google.com/view/transframer https://sites.google.com/view/transframer

• lucidrains 4y ago
  yup, training on DCT representation in general is an untapped research direction. can also help with larger images, longer audio, etc
  • astrange 4y ago
    DCT is a linear transformation so the model ought to be able to learn it (and typically already does). If you're going to feed in some non-learned transformation it seems like it should be nonlinear to add value.
    • lucidrains 4y ago
      True, it goes against the spirit of deep learning Still the Transframer results are interesting and has encouraged me to look into it