3 ms·
A system that could perfectly predict what I would do in response to any particular stimuli, as a continuing sequence, would be exactly as intelligent as me. T
by lewhoo 3y ago
A system that could perfectly predict what I would do in response to any particular stimuli, as a continuing sequence, would be exactly as intelligent as me.
That's certainly interesting but it's not a depiction of a LLM is it ? LLM's are not deterministic, and (perhaps) so are we so two non-deterministic systems can only occasionally align (or so I assume). Intuition says they may get "close enough", whatever that might be, and close enough is good enough in this case but I think you are making a giant assumption to the likes of since we can speed up matter to 1000km/h then IF we sped it up to light speed then ...[something]...
- stormfather 3y agoLLMs are deterministic if the temperature parameter is set to 0. Randomness is artificially injected into their outputs otherwise in order to make them more interesting, but they're just a series of math operations.
- Philpax 3y agoTo elucidate on this: the LLM can be viewed as a function that takes the context as an attachment and produces a probability distribution for the next token over all known tokens. Most inference samples from that distribution using a composition of sampling rules or such, but there's nothing stopping you from just always taking the most probable token (temperature = 0) and being fully deterministic. The results are quite bland, but it's perfect for extraction tasks. (note: GPT-4 is not fully deterministic; there's no details on this but the running theory is it is a mixture of experts model and that their expert routing algorithm is not deterministic/is dependent on the resources available)
- lewhoo 3y agoRandomness is artificially injected I'd argue everything about a LLM is artificial, there is no natural process involved is there ? Since its design is to mimic us (at face value, though I don't know how fair of a description this is) then randomness is essential I think.
- ted_bunny 3y agoWhy do you assume the human brain is non-deterministic? It might still be, but at a level of complexity we don't yet grasp.
- IanCal 3y ago> LLM's are not deterministic, They definitely can be, but it doesn't matter. > but I think you are making a giant assumption to the likes of since we can speed up matter to 1000km/h then IF we sped it up to light speed then ...[something]... This is an odd comparison. The point here is that a sequence prediction system can be as intelligent as the system it's predicting unless you invoke woo. That doesn't make llms intelligent but it means the argument that they just predict the next thing isn't enough to say the can't be.