2 ms·
No, it’s a valid point, which I didn’t interpret as literally “we’re working on eyeballs today” but rather “we’re scaling up these imperfect methods to a trilli
by janalsncm 2y ago
No, it’s a valid point, which I didn’t interpret as literally “we’re working on eyeballs today” but rather “we’re scaling up these imperfect methods to a trillion dollar GPU cluster”, the latter of which is genuinely something people talk about. The models will learn to mimic more and more of the long tail of the distribution of training data, which to us looks like an emergent understanding. So there’s a theoretical amount of data you could provide for them to memorize physical laws.
The issue is practical. There isn’t enough data out there to learn the long tail. If neural nets genuinely understood the world they would be getting 100% on ARC.
- bastawhiz 2y agoYou don't need a trillion dollar GPU to accomplish this stuff. Models from two years ago look incredibly different than the ones today, not just because they're bigger but because they're more sophisticated. And the data from two years ago looks a lot like the data today: messy and often poorly annotated. Even if we added no new compute and capped the resources used for inference and training, models would produce higher fidelity results over time due to improvements in the architecture.