4 ms·
> What model training is doing is building up a semantic space of vectors from which astronomically large numbers of true facts and ideas can be derived during
by tbrownaw 2y ago
> What model training is doing is building up a semantic space of vectors from which astronomically large numbers of true facts and ideas can be derived during inference. I mean like a number of facts larger than the number of molecules in the known universe. A googolplex more facts than the sum of all of humanity has ever "thought".
If you've ever downloaded a model to play with locally, you may have noticed that it does not in fact contain more bits than there are atoms in the universe.
- error_logic 2y agoIf you've ever shuffled cards, you may have noticed that you can generate a deck sequence no one has ever seen before rather easily. None of the cards are unique, but the sequence can be. The GP specifically said that accurate predictions can be derived from the patterns in the model, not that it contains a list of all the facts one by one.
- tbrownaw 2y agoA limited number of bits means a limited amount of information.
- quantadev 2y agoI said googolplex facts can be "derived". I didn't say they're stored. Also a single perceptron can answer an infinite number of questions. Here's how... For example the knowledge of how to answer the linear equation "Y=mX+b" contains an infinite number of "facts". For any X prompt, you put in, you can get out a "fact" Y. That's a question and an answer. This is actually a perfect analogy too, because all Perceptrons are really doing is this exact linear math (aside from things like a tanh activation functions, etc).