3 ms·
> The pretraining is the knowledge, not the intelligence. I thought the knowledge is the training set and the intelligence is the emergent/side effect of repro
by raducu 2y ago
> The pretraining is the knowledge, not the intelligence.
I thought the knowledge is the training set and the intelligence is the emergent/side effect of reproducing that knowledge by making sure the reproduction is not rote memorisation?
- ta8645 2y agoI'd say that it takes intelligence to encode knowledge, and the more knowledge you have, the more intelligently you can encode further knowledge, in a virtuous cycle. But once you have a data set of knowledge, there's nothing to emerge, there are no side effects. It just sits there doing nothing. The intelligence is in the algorithms that access that encoded knowledge to produce something else.
- esafak 2y agoThe data set is flawed, noisy, and its pieces are disconnected. It takes intelligence to correct its flaws and connect them parsimoniously.
- ta8645 2y agoIt takes knowledge to even know they're flawed, noisy, and disconnected. There's no reason to "correct" anything, unless you have knowledge that applying previously "understood" data has in fact produced deficient results in some application. That's reinforcement learning -- an algorithm, which requires accurate knowledge acquisition, to be effective.
- esafak 2y agoEvery statistical machine learning algorithm, including RL, deals with noisy data. The process of fitting aims to remove the sampling noise, revealing the population distribution, thereby compressing it into a model. The argument being advanced is that intelligence is the proposal of more parsimonious models, aka compression.
- ta8645 2y agoI've lost track of what we're disagreeing about.