4 ms·
But the output of the LLM is trained on a synthesis of multiple inputs. Even if every input has information loss, new and novel combinations can be created. I i
by cornercasechase 3y ago
But the output of the LLM is trained on a synthesis of multiple inputs. Even if every input has information loss, new and novel combinations can be created. I imagine it’s the same for humans since we have lossy memory.
- bhouston 3y agoIt doesn't matter if it is multiple inputs if they are just approximations of approximations of approximations. You need to engage in something that isn't a rough approximation, but something richer than your representation. Humans have lossy memory but we constantly engage with the real world so it doesn't matter. The key here is we engage with the real world on a regular basis. The key idea is that you can improve a low fidelity representation by training on a higher fidelity representation. But you can not get a higher fidelity representation from a low fidelity representation. Any inference is just making things up and likely to be wrong, so it is more like to just have less details.