4 ms·
One of three things has to be true. Either: a) this is false b) perfect recall is false (ie. as the internal state is overwritten, you lose information about
by wokwokwok 3y ago
One of three things has to be true. Either:
a) this is false
b) perfect recall is false (ie. as the internal state is overwritten, you lose information about previous entries in the context)
c) the inference time scales by the context length.
It’s not possible to have perfect recall over an arbitrary length in fixed time.
Not hard. Totally not possible at all
That would mean you can scan an infinite amount of data perfectly in fixed time.
So… Hrm… this kind of claim rings some kind of alarm bells, when it’s combined with this kind of sweeping announcement.
It seems to good to true; either it’s not that good, or the laws of the universe no longer hold true.
- Harrisonv 3y agoperfect recall is often a function of the architecture allowing for data to bleed through linkages. you can increase the perfect token recall through dialated wavenet structures, or, in the case of v5, the use of multi-head linear attention creates multiple pathways where information can skip forward in time
- sebzim4500 3y ago(b) is the sacrifice made in these linear attention type architectures. As a mitigation, you can leave a few normal attention layers in the model but replace the rest.
- deleted 3y ago[deleted]