3 ms·

Code for Scaling Transformer to 1M tokens and beyond with RMT (arxiv.org)

• mysterybox 3y ago
  I found this interesting link in paper, so this can be used as ready code for RMT Transformer https://github.com/booydar/t5-experiments/tree/scaling-report/rmt_utils https://github.com/booydar/t5-experiments/tree/scaling-repor...
• deleted 3y ago
  [deleted]
• mysterybox 3y ago
  Finetuning: https://github.com/booydar/t5-experiments/blob/scaling-report/run_finetuning_babilong_rmt.py https://github.com/booydar/t5-experiments/blob/scaling-repor...
• mysterybox 3y ago
  [flagged]