4 ms·
The current crop of LLMs is like Paleozoic megafauna, or like Egyptian Pyramids. It takes a few relatively simple approaches and stretches them wildly, using co
by nine_k 3y ago
The current crop of LLMs is like Paleozoic megafauna, or like Egyptian Pyramids. It takes a few relatively simple approaches and stretches them wildly, using colossal computing resources.
Live systems in nature seem to solve similar problems with way less compute available. There should be better architectures.
Also, as somebody said, every exponential growth curve is a lower part of a sigmoid. LLMs will plateau at some level. That level may be impressively high though.
- __loam 3y agoI'm not convinced we're going to get much further than this, at least without a see change in how this stuff is done. We saw a pretty big amount of diminishing returns with GPT-4, which cost 2 orders of magnitude more money to train than 3.5. Is the performance more than 2 orders of magnitude better than 3.5? How much compute is too much to be spending on this bullshit. 30% of humanity's total computing resources? 70%? GPT was a big innovation in parallelizing the training process. I think the optimism that we're on the beginning of a sigmoid curve here ought to be questioned.
- danenania 3y agoThe performance doesn't have to scale with the training cost. It's ok if we have diminishing returns. The only question is if we can keep getting improvements. If we could get a 10x improvement over GPT-4 (while maintaining safety/alignment), the potential benefits to productivity across the entire world economy are so great that it justifies nearly unlimited investment.
- __loam 3y agoThis is delusional.
- deleted 3y ago[deleted]
- the8472 3y ago> There should be better architectures. The drumbeat of progress has been quite steady. On log charts. https://files.catbox.moe/w93was.png https://files.catbox.moe/w93was.png
- garblegarble 3y ago>Live systems in nature seem to solve similar problems with way less compute available Do they really? They're certainly more energy-efficient in business-as-usual mode, but a human brain has 86 billion neurons, 600+ trillion synapses(!), and each instance takes 15-20+ years to train to do complex logical tasks. Even if the per-cell work is tiny (and, is it? cells are amazingly complex), 86 billion (or 600+ trillion) times 20 years is a lot of computation.
- pdimitar 3y agoYou are ignoring the fact that a toddler, once their musculature develops, is able to learn to walk after several tries. Show me a humanoid robot that can do that.
- Drakim 3y agoTo be fair a toddler's brain isn't "offline" while it's musculature develops, they kick and move their limbs constantly, and then later crawl and and even stand, before they finally take the first walking steps. It's just objectively wrong to say a toddler goes from zero to walking in "several tries".
- pdimitar 3y agoSure, not several. Not several trillions either though. As mentioned in a sibling comment, HN anthropomorphizes AI too much. And is too optimistic about it. I just don't see the results and the value, people trip up all over themselves to congratulate themselves and the researches, yet 99.99% of the problems in the world persist. I am one of these a-holes that wants to see results when money are invested. It still comes as a shock to some apparently.
- flangola7 3y agoBlank checks are flying like it is the dot com era again. Money is most definitely being invested in it.
- 3y ago
- FrustratedMonky 3y agoEveryone has tunnel vision on LLM's like they are the only thing just because they are the latest big shiny thing. "oh, they are at their limit". People were saying they were at their limit 5 years ago too. There are a lot of AI techniques, some more advanced than LLM, just with other limitations so they don't look 'amazing'. Seems like engineering at this point to tie them together for the next 'big thing'.
- StackOverlord 3y agohttps://thequantuminsider.com/2023/08/22/tqi-exclusive-bob-coecke-on-exploring-human-like-ai-and-brain-like-quantum-computers/ https://thequantuminsider.com/2023/08/22/tqi-exclusive-bob-c...