4 ms·
Eh, it's only a matter of time until new nvidia clusters are in production and we start seeing what hardware designed for training massive models (roughly 100x
by kachapopopow 2y ago
Eh, it's only a matter of time until new nvidia clusters are in production and we start seeing what hardware designed for training massive models (roughly 100x bigger than anything we know of at least) can do.
With some additional architectural changes we might start seeing some sparks of real human stupidity.
- null_investor 2y agoThat's incorrect, it isn't that better than h100s
- kachapopopow 2y agoThey chain a lot better and have much lower software complexity while also being able to share memory between two different nodes. Also the latency is much lower. (Also I'm sort of mocking LLM's here)
- bearjaws 2y agoMaybe 4x, 100x is many years away. The exponential graph people are posting is just to pump the stock price.
- HarHarVeryFunny 2y agoThe exponential graph, to the extent it is measuring anything, is measuring progress on LLM benchmarks, which are based on tasks that LLMs can tackle and show progress on. The set of benchmarks used to measure LLMs do not include things that require significant run-time learning or reasoning (e.g. Chollet's ARC test, although frankly that's a pretty simple test). This exponential progress will flatten once LLMs can no longer improve along that LLM-benchmark axis (maybe they score 100% on all LLM benchmarks), but this "100% achieved on all tests" won't mean "mission accomplished - AGI achieved!", it'll just mean it's time for a new set of more challenging benchmarks, one step closer to what humans are capable of.