2 ms·
We do not know whether or not that is what humans do or how qualitatively different it is to what humans do, because we don't know all that much about human rea
by vidarh 4y ago
We do not know whether or not that is what humans do or how qualitatively different it is to what humans do, because we don't know all that much about human reasoning works.
Put another way: Some Markov models are Turing complete, so "merely" computing probabilities can be Turing complete with only minor steps, so trying to downplay the potential capabilities of models like this by handwaving about things we don't know is foolish.
We don't need that scale of input, but we also don't know if LLMs need that much input to do well, or if our current training protocols are simply poor. With ongoing work on reducing the training cost, it is at a minimum clear that current training methods are far from optimal.