Interestingly, I did see this tweet [0] mentioning a phase shift that occurs in transformers at exactly the scale RetNet stopped at. Probably simply coincident

Interestingly, I did see this tweet [0] mentioning a phase shift that occurs in transformers at exactly the scale RetNet stopped at. Probably simply coincidental but I was previously unaware of this phenomenon at such a scale in transformers. [0] https://twitter.com/gordic_aleksa/status/1682479676910870529 https://twitter.com/gordic_aleksa/status/1682479676910870529