5 ms·
It seems to me like these long context lengths are going to make a huge difference in the capabilities of things people are able to produce. Agents can use LLMs
by aqme28 4y ago
It seems to me like these long context lengths are going to make a huge difference in the capabilities of things people are able to produce. Agents can use LLMs to subdivide a large corpus into manageable chunks. More context just makes them much much better at that.
Regardless, for most programming tasks, I doubt my equivalent human context length is any better than 32k tokens.
- jerpint 4y agoThe problem is not the short term context length, but long term. If you want to do things like long term goal planning, keeping track of distant past events can be of high value
- PaulHoule 4y agoI’d argue for planning that the answer is to couple the LLM to some other system rather than try to improve the LLM. Combinatorial optimization is a well-understood problem that is NP-complete in theory but practical to solve in practice. For doing math we might as well give the LLM a pocket calculator, why not couple it to a planner, theorem prover and similar tools?
- pixl97 4y agoThis is pretty much what people are testing now with Auto-GPT
- pmoriarty 4y ago"For doing math we might as well give the LLM a pocket calculator, why not couple it to a planner, theorem prover and similar tools?" ChatGPT has already been hooked up to Wolfram Alpha.[1] For hooking up other things, see HuggingGPT and TaskMatrix.AI.[2][3] [1] - https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/ https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its... [2] - https://arxiv.org/pdf/2303.17580.pdf https://arxiv.org/pdf/2303.17580.pdf [3] - https://arxiv.org/pdf/2303.16434.pdf https://arxiv.org/pdf/2303.16434.pdf
- btbuildem 4y agoPrecisely. LLMs are the Perl of the future
- sharemywin 4y agoThat's the interesting thing. if LLM start to just become interfaces for other systems you could probably just use them to train smaller systems like in the style of alpaca and other recent LLMs.
- taneq 4y agoMaybe that’s what our inner monologue is for… to cycle relevant context through regularly so it stays in scope. I mean, that’s not actually why we have one but it’d be cute.
- PartiallyTyped 4y agoOur context is hierarchical, at multiple different resolutions, so I don't think it's comparable.
- btbuildem 4y agoI've had good results using that approach with LLMs for domain-specific problem solving.
- letitgo12345 4y agoGood chance so is OAI's https://arxiv.org/abs/2110.13711 https://arxiv.org/abs/2110.13711 (paper has the long context lead at OAI as a co-author)
- XorNot 4y agoI'd say when you're actively trying to remember an event and "narrowing it down" mentally is a process that feels suspiciously like a dialogue with an LLM where you keep asking for the answer to be improved.
- aqme28 4y agoI was trying to describe a hierarchy of LLMs as an agent. I don't think that's a uniquely human ability.
- NhanH 4y agoHuman expert relies in the “chunking” effect for expertise, which is mostly not part of the context length (working memory). For a pseudo-analogy, each human is fine-tuned from the raw intelligence (by training and education). In that way generic LLM probably can’t beat us yet so don’t despair!
- onos 4y agoAny job focused on fact retrieval is at risk.
- RandomLensman 4y agoNeed to know that the facts exist in the first place, though.
- orbifold 4y agoGPT-4 is already at a level that it has graduate level math and physics knowledge, with a level of recall that most students can only dream of.
- RandomLensman 4y agoDo those actually matter in jobs? Most valuable facts I encounter are far more niche, often in someone's head and not written down, only stored privately somewhere etc.
- awesomeMilou 4y agoWhat kind of job is only focused on fact retrieval?
- fnordpiglet 4y agoJeopardy contestant
- istjohn 4y ago
- jacquesm 4y ago'Superhuman' has many dimensions. It can be speed, it can be ability to retain a lot of information, it can be the ability to deal with a large quantity of information at once and many other dimensions besides. For the longest time Chess was considered a domain where computers could be dilettantes but could never dominate. Chess is now in 'superhuman' territory and likely we will never see a reversal because any insight that benefits humans also benefits computers but not the other way around. The fact that this is such a multi-dimensional problem is frequently overlooked in the debate about AI/AGI etc, it may not matter all that much if the non AGI AI is already superhuman on enough dimensions other than the ones that the 'but it isn't AGI' crowd cling to. The consequences are what matters, not the fine print or the implementation details, and those consequences are directly tied to the number of dimensions along which a computer can beat humanity. To give an example: if a chess program was 3000 Elo before but so slow that it would lose under competition rules then humans still dominated chess. Likewise if it would be only 2500 Elo but fast enough, it would still lose from the best humans. But for a large fraction of society it would have already moved out into 'superhuman' territory. And a couple of technological leaps of progress later and we're all looking at that AI as if it has moved into superhuman regions. This sort of thing will happen on many fronts, and all of those fronts are moving, if enough of them go past the threshold then whether it is AGI or not is irrelevant and for every person that threshold is at different points. Maybe a computer will be able to calculate faster and better than you can, maybe it will be able to translate text faster and better than you, maybe it will be able to organize information faster and better than you. At which point we cross the line into saying that it can think faster and better than you is hard, but we can see that we are getting close to that line without even knowing exactly where that line is.
- pmoriarty 4y ago"Chess is now in 'superhuman' territory and likely we will never see a reversal because any insight that benefits humans also benefits computers but not the other way around." What is considered human is malleable. It is conceivable that humans will be enhanced in various biological and non-biological ways to a point that they can once again compete with computers.
- 4y ago
- kmeisthax 4y agoThe problem with comparing LLM and human context length is that LLMs don't update their weights based on their input. The 32k of context that they do have is their only memory. Humans have multiple layers of memory and can recall things and concepts from years in the past - akin to millions of tokens' worth of recall in an LLM. Yes, that memory is extremely lossy, but it's there.