4 ms·
Even with an incredibly long context window (say, 1M tokens), attention still suffers from a problem with long-term dependencies. This is probably why OpenAI ha
by fzliu 4y ago
Even with an incredibly long context window (say, 1M tokens), attention still suffers from a problem with long-term dependencies. This is probably why OpenAI hasn't publicly released their 32k token length model just yet.
- Der_Einzige 4y agoI think they haven't released it because the capabilities it has are simply too powerful when combined with a vectorDB.
- sparker72678 4y agoMaybe. It's also probably staggeringly expensive to run.
- nullsense 4y agoProbably all of the above
- ChatGTP 4y agoCurious what you mean by “too powerful”?