3 ms·
Sure we just need next level super computers for these large models and the patience of multiple days to wait for output
by jhoelzel 4y ago
Sure we just need next level super computers for these large models and the patience of multiple days to wait for output
- WithinReason 4y agoIncreasing the token limit without needing more resources to run the network is a solvable problem
- jhoelzel 4y agobut you do sure see the problem with a codebase right?
- WithinReason 4y agoThe current token limit comes from a O(N^2) memory requirements for N tokens, there is research that's trying to reduce this towards O(N), for example as the (downvoted) sibling comment suggests. This is not exactly straightforward but not impossible either. It's not a fundamental limitation of language models going forward.
- jhoelzel 4y agothat still will only be enough to hold an extended cli application
- throwaway3672 4y agoNot necessarily - you just need hierarchical abstraction memory. I reckon my "token" limit when analysing code is around 7.