4 ms·

It learns through the context. The context is state. There's a bit of a cheat that's going on here though in that the model is being given the fundamental int

It learns through the context. The context is state. There's a bit of a cheat that's going on here though in that the model is being given the fundamental integration operations as part of the problem. That means the model hasn't had to learn what they are. It might not have needed to be given them, but it does feel like that's giving the model a leg up in the benchmarks that it wouldn't otherwise have, and when there's a direct comparison to (e.g.) DeepSeek, that's an unfair advantage.

viking123 2y ago
Does the massive context still needs to be dragged around? Until get a neural network that adjusts weights in real time without relying on a huge clump of context being cycled there, I don't think there will be an AGI or even an impressive "agent". Current agents are just LLM looping lmao sold to people with no knowledge how they work at all.
- regularfry 2y ago
  I've been wondering about context compression, actually. I remember a bunch of prompt compression tricks from a couple of years ago that must be usable. If you were to say "anything past half context gets compressed, including the previously-compressed context" would mean you'd still have reasonable workspace and potentially infinite recollection of the most important things. Then yes, you'd be dragging a massive context around, but you're maximising the return on using it. I presume somebody's got a toolkit for this that I don't know the right terms to google for.
  - mirekrusin 2y ago
    Compress first half (even recursively) is a nice idea, but you'd have to sort it by importance first – ie. you don't know if the most important sentence didn't exist at the beginning (ie. the whole goal could have been stated at the beginning, you don't want to loose it – you want memory to fade in terms of importance instead).
    - regularfry 2y ago
      I don't know that, but the LLM might be able to make that judgement. If it's involved in the compression process, it can choose to drop cruft from the middle. Maybe "summarise then compress" is a better description.
      - bloomingkales 2y ago
        Yep, let the LLM operate like functional programming functions. Explode(context) … map, reduce, flatten, explode, filter, find, filter, map, flatten, explode … reduce(context). This is the imagination loop. OpenAI sold the prior reasoning loop. But that’s all this is. Quite frankly, I wouldn't come to HN for anything less than this quality of content: https://www.youtube.com/watch?v=DX3qLIwHoUo#t=1m29s https://www.youtube.com/watch?v=DX3qLIwHoUo#t=1m29s The machine reforming visually (explode, .., reduce) is what I'm describing.
- mhmmmmmm 2y ago
  There is https://www.rwkv.com/ https://www.rwkv.com/ which is an LLM based on RNN's, thus having "infinite" context length, it comes with its own tradebacks though. (Notably that its impossible to actually store infinite information in the network, so it prunes based on which information it finds more important.)
- esafak 2y ago
  The "massive" context is still tiny compared with the size of the model whose weights you propose to update.
  - deleted 2y ago
    [deleted]
- InvidFlower 2y ago
  There's been some work on memory lately like Transformer² and Titans. But that may not be necessary for decent agents. Even the "context in a loop" is getting better as general reliability of tasks increases, and as people find better ways to manage context. Like Cline has the Memory Bank prompt where there's a folder of markdown files at diff levels of abstractions of info on the project, current feature, current task progress, etc. And so it can update those files and read from them as a consistent memory.

viking123 2y ago

Does the massive context still needs to be dragged around? Until get a neural network that adjusts weights in real time without relying on a huge clump of context being cycled there, I don't think there will be an AGI or even an impressive "agent". Current agents are just LLM looping lmao sold to people with no knowledge how they work at all.

regularfry 2y ago
I've been wondering about context compression, actually. I remember a bunch of prompt compression tricks from a couple of years ago that must be usable. If you were to say "anything past half context gets compressed, including the previously-compressed context" would mean you'd still have reasonable workspace and potentially infinite recollection of the most important things. Then yes, you'd be dragging a massive context around, but you're maximising the return on using it. I presume somebody's got a toolkit for this that I don't know the right terms to google for.
- mirekrusin 2y ago
  Compress first half (even recursively) is a nice idea, but you'd have to sort it by importance first – ie. you don't know if the most important sentence didn't exist at the beginning (ie. the whole goal could have been stated at the beginning, you don't want to loose it – you want memory to fade in terms of importance instead).
  - regularfry 2y ago
    I don't know that, but the LLM might be able to make that judgement. If it's involved in the compression process, it can choose to drop cruft from the middle. Maybe "summarise then compress" is a better description.
    - bloomingkales 2y ago
      Yep, let the LLM operate like functional programming functions. Explode(context) … map, reduce, flatten, explode, filter, find, filter, map, flatten, explode … reduce(context). This is the imagination loop. OpenAI sold the prior reasoning loop. But that’s all this is. Quite frankly, I wouldn't come to HN for anything less than this quality of content: https://www.youtube.com/watch?v=DX3qLIwHoUo#t=1m29s https://www.youtube.com/watch?v=DX3qLIwHoUo#t=1m29s The machine reforming visually (explode, .., reduce) is what I'm describing.
mhmmmmmm 2y ago
There is https://www.rwkv.com/ https://www.rwkv.com/ which is an LLM based on RNN's, thus having "infinite" context length, it comes with its own tradebacks though. (Notably that its impossible to actually store infinite information in the network, so it prunes based on which information it finds more important.)
esafak 2y ago
The "massive" context is still tiny compared with the size of the model whose weights you propose to update.
- deleted 2y ago
  [deleted]
InvidFlower 2y ago
There's been some work on memory lately like Transformer² and Titans. But that may not be necessary for decent agents. Even the "context in a loop" is getting better as general reliability of tasks increases, and as people find better ways to manage context. Like Cline has the Memory Bank prompt where there's a folder of markdown files at diff levels of abstractions of info on the project, current feature, current task progress, etc. And so it can update those files and read from them as a consistent memory.