3 ms·
The same reason why you don't let Mr Musk do all the work. He can't. One LLM is limited, one obvious limitation is its context window. Using a swarm of LLMs th
by SanderNL 4y ago
The same reason why you don't let Mr Musk do all the work. He can't.
One LLM is limited, one obvious limitation is its context window. Using a swarm of LLMs that each do a little task can alleviate that.
We do it too and it's called delegation.
Edit: BTW, "swarm" is meaningless with LLMs. It can be the same instance, but prompted differently each time.
- ParetoOptimal 4y ago> The same reason why you don't let Mr Musk do all the work. He can't. Better to limit his incompetence to one position.
- akiselev 4y agoI beg to differ. Imagine him taking down Twitter, Facebook, Instagram, and all the others in one fell swoop!
- int_19h 4y agoContext window is a limitation, but have we actually hit the ceiling wrt scaling that? For GPT, you need O(N^2) VRAM to handle larger context sizes, but that is a "I need more hardware" problem ultimately; as I understand, the reason why they don't go higher is because of economic viability of it, not because it couldn't be done in principle. And there are many interesting hardware developments in the pipeline now that the engineers know exactly what kind of compute they can narrowly optimize for. So, perhaps, there aren't swarms yet just because there are easier ways to scale for now?
- SanderNL 4y agoI am sure the context window can go up, maybe into the MB range. But I still see delegation as a necessary part of the solution. For the same reason one genius human does not suddenly need less support staff, they actually need more. Edit: and why it isn’t here yet is because it’s new and hard.