4 ms·

> It is a strong assertion... I think you may be right if you consider humanity as a whole, but when evaluating LLMs I think we should perhaps hold them to sli

> It is a strong assertion... I think you may be right if you consider humanity as a whole, but when evaluating LLMs I think we should perhaps hold them to slightly higher standards. It seems silly to compare a system that clearly has an incredible level of knowledge and writing ability to toddlers or completely uneducated remote tribespeople (or even just bigoted people living in otherwise advanced societies). Since LLMs can operate on vastly different timescales to humans, we should probably evaluate them against reasonably knowledgeable humans given several hours and a chance to look back over their work as many times as they like before final submission. Since we’re trying to abstractly compare some sort of computational ability, we should probably give them equal amounts of computational resources/number of steps/clock cycles/etc. Given this setup, I think some of the ‘human mistakes’ we talk about would no longer apply. Comparing systems like this is not an easy (or objective) matter anyway… I’m not claiming this is the only way of thinking about it and that all others are wrong. > it is interesting to see which problems are uniquely LLM (or human) I certainly agree with that. It’s fascinating to be able to observe and study something in real time that appears on the surface to approximate human thought so shockingly well whilst also being so different.

stared 3y ago
It is hard to define a fair comparison between humans and LLMs. Artificial "neurons" (i.e., vector entries + ReLU) have little in common with biological neurons (each one is a computing unit on its own). Data efficiency is crucial, but humans have a lot of pre-trained multi-modal data. For energy efficiency, ML is orders of magnitude more efficient. Regarding the general powers (and weaknesses) of LLMs, I am shocked they work in an Artificial General(ish) Intelligence way. Text generation with GPT2 and GPT3 - sure, it still felt like a very advanced text autocomplete. GPT4 - here, I could have a bet (and lose money) that this level requires some form of reinforcement learning and a two-way interaction with the environment. Sure, there is some RL there, but I assumed something closer to a bot learning to talk with people, running and getting results from code, and looking at data online (for the training!), or maybe even - literally walking with a camera attached. Yes, there is still a lot to do. There is a difference between a Go model beating novices, advanced players, and everyone, including world champions.

stared 3y ago

It is hard to define a fair comparison between humans and LLMs. Artificial "neurons" (i.e., vector entries + ReLU) have little in common with biological neurons (each one is a computing unit on its own). Data efficiency is crucial, but humans have a lot of pre-trained multi-modal data. For energy efficiency, ML is orders of magnitude more efficient. Regarding the general powers (and weaknesses) of LLMs, I am shocked they work in an Artificial General(ish) Intelligence way. Text generation with GPT2 and GPT3 - sure, it still felt like a very advanced text autocomplete. GPT4 - here, I could have a bet (and lose money) that this level requires some form of reinforcement learning and a two-way interaction with the environment. Sure, there is some RL there, but I assumed something closer to a bot learning to talk with people, running and getting results from code, and looking at data online (for the training!), or maybe even - literally walking with a camera attached. Yes, there is still a lot to do. There is a difference between a Go model beating novices, advanced players, and everyone, including world champions.