27 ms·
Recurrent Looped Transformer
- jal278 15d agonot sure w/o context why this is important -- no results/implementation & i believe there are prior combinations of transformers/RNNs. but perhaps I'm missing the relevance/insight
- dankai 15d agoIs this just a theory about an architecture or are there actually some benchmarks/results to substantiate it?
- jal278 15d agoappears it's just a theory, sort of surprised this lifted up so far on hn
- fc417fc802 15d agoA fairly obvious theory at that, unless I've critically misunderstood what's being described. As with so many obvious ideas I've always assumed that the reason I haven't come across it in the wild is because it doesn't work (or is comparatively inefficient, or tends to blow up during the training run, or etc). > Realized reasoning gains, hardware efficiency, and RL scaling remain to be established. Yeah so the first entry in that list is - if I may be so bold - typically what you'd start with at a small scale _before_ writing up and publishing your "genius" idea. This is the usual crank with delusions of grandeur presenting something straightforward that he hasn't tested as though it were a working breakthrough. Ironically the cost of testing such theories has fallen to an all time low given the capabilities of coding models. I wouldn't be surprised if a frontier model could one shot a test of this.
- lstodd 15d agoI thought it's about https://en.wikipedia.org/wiki/Autotransformer https://en.wikipedia.org/wiki/Autotransformer
- fc417fc802 15d agoI can't believe that actually works? This is far more interesting than the half baked ML idea. Edit: Thanks to your link I've now learned about the zigzag transformer which is yet another design that somehow feels like cheating reality. https://en.wikipedia.org/wiki/Zigzag_transformer https://en.wikipedia.org/wiki/Zigzag_transformer
- lstodd 13d agothere's also more fun stuff like https://www.electricalvolt.com/cvt-constant-voltage-transformer-its-working-and-application/ https://www.electricalvolt.com/cvt-constant-voltage-transfor...
- vatsachak 15d agoPaper written by AI