3 ms·
This reminds me a lot of the tricks to turn BERT into a generative model. I guess the causal masking that keeps it to essentially be autoregressive is an import
by radarsat1 6mo ago
This reminds me a lot of the tricks to turn BERT into a generative model. I guess the causal masking that keeps it to essentially be autoregressive is an important difference though. Kind of best of both worlds.
- deleted 6mo ago[deleted]
- krackers 6mo agoMasked language modeling has been compared loosely to text diffusion [1], so the paper's title claim may be loosely true in some sense even if it's misleading. [1] https://nathan.rs/posts/roberta-diffusion/ https://nathan.rs/posts/roberta-diffusion/