6 ms·
Do you think there could be a kind of universality principle at work here, where once you make a good enough architecture then the details don't matter so much
by ftxbro 3y ago
Do you think there could be a kind of universality principle at work here, where once you make a good enough architecture then the details don't matter so much compared to the model size and training flops and dataset size? In other words, maybe it wasn't a coincidence that your architecture worked about as well as the transformer architecture?
- pico_creator 3y agoThere is a reasonable argument for that (heard the idea go around between multiple AI engineers, that once you go past a certain scale, it does not matter for its evals) One of the biggest issue for testing all of this, is it takes a crap ton of GPUs to prove all the alternatives to transformers beyond 1B param. For example I’m waiting for someone to do a 1B-14B text based diffusion network Finally, if this is truely the case (and all that really matter is size+dataset) We really should use an architecture that is cheaper to train and run. And that’s what RWKV represents here You can even run the 7B quantized model reasonably on most laptops (try the rwkv-cpp / rwkv-cpp-node project)