3 ms·

I know almost nothing about this space but “lossy compression of millions of past tokens” feels a lot more like how actual human memory works than “perfect acce

I know almost nothing about this space but “lossy compression of millions of past tokens” feels a lot more like how actual human memory works than “perfect access to a small number of recent tokens”.

throwup238 3y ago
> feels a lot more like how actual human memory works Beware of feel-good traps like this. If you were to map the human connectome to a computational neural network down to the ion channel, it'd be at least 500 quadrillion parameters*. That's at least 5-6 orders of magnitude beyond what is currently possible with SOTA ML which means that even if the human brain was 99% devoted to compressing those tokens, that 1% that could actually do work with them is still a thousand times bigger than GPT4. There be emergent dragons. * This is a fascile argument to begin with since biological neuron signals aren't quantized and ion channels are far too complex to map to a single static parameter
- fwip 3y ago
  > biological neuron signals aren't quantized Perhaps, but they inherently have imprecision due the nature of being biological. Expose the same neuron to the same input N times, and you'll get a range of output. The effect of noisy analog data is, broadly speaking, similar to the effect of low-resolution digital data.
  - deleted 3y ago
    [deleted]
  - throwup238 3y ago
    Expose the same neuron to the same input N times and you won't get a randomly distributed range of outputs, you'll get a signal that is either attenuated or potentiated. That's not noise, that's the simplest form of neuroplasticity at work (which is why static parameters can never represent ion channels).
- bbor 3y ago
  Maybe against HN rules, but: thank you. Very helpful comment!
thentherewere2 3y ago
RNNs and LSTMs from the past did this as well (but cannot be trained in parallel as each token has to be compressed sequentially). Transformers ate their cake. Newer methods are going back to similar concepts but trying to get past previous bottlenecks given what we've learned since then about transformers.

throwup238 3y ago

> feels a lot more like how actual human memory works Beware of feel-good traps like this. If you were to map the human connectome to a computational neural network down to the ion channel, it'd be at least 500 quadrillion parameters*. That's at least 5-6 orders of magnitude beyond what is currently possible with SOTA ML which means that even if the human brain was 99% devoted to compressing those tokens, that 1% that could actually do work with them is still a thousand times bigger than GPT4. There be emergent dragons. * This is a fascile argument to begin with since biological neuron signals aren't quantized and ion channels are far too complex to map to a single static parameter

fwip 3y ago
> biological neuron signals aren't quantized Perhaps, but they inherently have imprecision due the nature of being biological. Expose the same neuron to the same input N times, and you'll get a range of output. The effect of noisy analog data is, broadly speaking, similar to the effect of low-resolution digital data.
- deleted 3y ago
  [deleted]
- throwup238 3y ago
  Expose the same neuron to the same input N times and you won't get a randomly distributed range of outputs, you'll get a signal that is either attenuated or potentiated. That's not noise, that's the simplest form of neuroplasticity at work (which is why static parameters can never represent ion channels).
bbor 3y ago
Maybe against HN rules, but: thank you. Very helpful comment!

thentherewere2 3y ago

RNNs and LSTMs from the past did this as well (but cannot be trained in parallel as each token has to be compressed sequentially). Transformers ate their cake. Newer methods are going back to similar concepts but trying to get past previous bottlenecks given what we've learned since then about transformers.