5 ms·
I think this is a classic case of us overestimating the immediate impact and underestimating the long term impact. Right now, they are definitely useful time s
by jrumbut 4y ago
I think this is a classic case of us overestimating the immediate impact and underestimating the long term impact.
Right now, they are definitely useful time savers, but they need a lot of handholding. Eventually, someone will figure out how to get hundreds of LLMs supervising teams of millions of LLMs to do some really wild stuff that is currently completely impossible.
You could spin up a giant staff the way we do servers now. There has to be a world changing application of that.
- mach1ne 4y ago>Eventually, someone will figure out how to get hundreds of LLMs supervising teams of millions of LLMs to do some really wild stuff that is currently completely impossible. This is an intuitive direction. In fact, it’s so intuitive that it’s a little bit odd that nobody seems to have made proper progress with LLM swarm computation.
- contemplatter 4y agoIn particular, it's odd that the greatest software developer in the world (ChatGPT) hasn't made progress with LLM swarm computation.
- loandbehold 4y agoHow is "LLM swarm computation" different that single bigger LLM?
- staunton 4y agoIt's easy to distribute across many computers which communicate with high latency
- alexvoda 4y agoLLMs are already running distributed on swarms of computers. A swarm of swarms is just a bigger swarm. So again, what is the actual difference you are imagining? Or is it just that distributed X is fashionable?
- staunton 4y agoSignificantly higher latency than you have within a single datacenter. Think "my GPU working with your GPU".
- alexvoda 4y agoThere are already LLMs hosted across the internet (Folding@Home style) instead of in a single data center. Just because the swarm infrastructure hosting an LLM has higher latency across certain paths does not make it a swarm of LLMs.
- staunton 4y ago> There are already LLMs hosted across the internet (Folding@Home style) Interesting, I haven't heard of that. Can you name examples?
- alexvoda 4y agoI read about Petals (1) some time ago here on HN. There are surely others too, but I don't remember the names. 1. https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals
- pixl97 4y agoRather large parts of your brain are more generalized, but in particular places we have more specialized areas. Now, you looking at it would consider it all the same brain most likely, but if you're looking at it in systems thinking view, it's a small separate brain with a slightly different task than the rest of the brain. If 80% of the processors in a cluster are running 'general LLM' and 20% are running 'math LLM' are they the same cluster? Could you host the cluster in a different data center? What if you want to test different math LLM modules out with the general intelligence?
- SanderNL 4y agoThe same reason why you don't let Mr Musk do all the work. He can't. One LLM is limited, one obvious limitation is its context window. Using a swarm of LLMs that each do a little task can alleviate that. We do it too and it's called delegation. Edit: BTW, "swarm" is meaningless with LLMs. It can be the same instance, but prompted differently each time.
- ParetoOptimal 4y ago> The same reason why you don't let Mr Musk do all the work. He can't. Better to limit his incompetence to one position.
- akiselev 4y agoI beg to differ. Imagine him taking down Twitter, Facebook, Instagram, and all the others in one fell swoop!
- int_19h 4y agoContext window is a limitation, but have we actually hit the ceiling wrt scaling that? For GPT, you need O(N^2) VRAM to handle larger context sizes, but that is a "I need more hardware" problem ultimately; as I understand, the reason why they don't go higher is because of economic viability of it, not because it couldn't be done in principle. And there are many interesting hardware developments in the pipeline now that the engineers know exactly what kind of compute they can narrowly optimize for. So, perhaps, there aren't swarms yet just because there are easier ways to scale for now?
- SanderNL 4y agoI am sure the context window can go up, maybe into the MB range. But I still see delegation as a necessary part of the solution. For the same reason one genius human does not suddenly need less support staff, they actually need more. Edit: and why it isn’t here yet is because it’s new and hard.
- jrumbut 4y agoI've read about people doing it, I haven't read about people achieving anything particularly interesting with it. It's early days. There will be a GPT 5 I'm sure, maybe that one will be better at teamwork.
- Bjartr 4y agoThis sounds like that old economics joke that says it's impossible to find $20 on the ground, because if it had been there, someone would have already picked it up.
- uxcolumbo 4y agoI”m not in ML, so excuse this maybe naive question: > get hundreds of LLMs supervising teams of millions of LLMs What does this mean or what can you do with this setup… do you mean running LLMs in parallel?
- jacquesm 4y agoYes, that's called 'ensembling'. There is a lot of work being done on this kind of solution. One way in which it could work is that you can use multiple models that have been fine tuned for various problems and then use the answer that returns the highest confidence.
- jazzyjackson 4y agoYou can also have adverserial generation where models given different expertises and attitude can go back and forth criticizing each others work
- namaria 4y agoSounds like the 'dead internet' is just around the corner!
- sdenton4 4y agoAsk the LLM to perform a complex task by splitting it into sub tasks to be performed by other LLM instances, then integrate the results...
- shmoogy 4y agoIs this something like langchain is working towards?
- arthurcolle 4y agoAutoGPT, BabyAGI
- SanderNL 4y agoWe call that “companies”. We just need to apply what we learned in business school to a different set of workers, slightly deficient workers.
- Eisenstein 4y ago> We just need to apply what we learned in business school Please don't. You've already ruined enough industries. Let the MBAs do finance and Wall Street and leave them out of the chain of command in organizations that make things.
- SanderNL 4y agoTake shelter under my protective wings, O, sweet summer children.
- brookst 4y agoEvery time you go to the store and find that the store is still in business and there is food on the shelf, it is because someone went to business school and knows how to optimize demand estimation, pricing, and logistics. Yes, some MBAs fuck things up. Just like some CS grads fuck things up. But advocating against the study of business is just as naive as advocating against the study of computer science just because there are some bad CS grads.
- ResearchCode 4y agoStores existed before the MBA, but MBAs could be why food prices are up 30% since last year.
- Eisenstein 4y ago> Every time you go to the store and find that the store is still in business and there is food on the shelf, it is because someone went to business school Are you contending that business were not successful before Wharton started pumping out MBAs? > But advocating against the study of business is just as naive as advocating against the study of computer science I didn't say 'don't study business', I said 'stick to finance'. MBAs tend to end up destroying innovation and productivity for short term growth and stats. Jack Welch showed what a successfully motivated 'business oriented' leader can do to an innovative and productive legacy organization when given complete control over it. The MBAs happen to just do it on a smaller scale.