3 ms·
transformers don’t show any sign of convergence yet. The more data and compute we throw at them, weather it be chess, go, image classification, text prediction
by naveen99 2y ago
transformers don’t show any sign of convergence yet. The more data and compute we throw at them, weather it be chess, go, image classification, text prediction, the better they get. Tunnel vision is simply focus on finding a horizon, or a limit of some kind.
- jiggawatts 2y agoHow can you be so confident when nobody has managed to convincingly beat GPT-4 yet, including Open AI themselves? All evidence is the throwing even textbook qualify data at a model of almost any size just approaches an asymptote just a tiny bit above GPT 4. A better model of some data starts to look increasingly like then data, not like something else beyond the data.
- naveen99 2y agoi don't know about the future. I am only saying, we haven't seen diminishing returns from training transformers on more data / compute yet.