5 ms·
What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that
by Havoc 3y ago
What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away.
Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.
- gct 3y agoBasically Ray Kurzweil's argument, he's been saying $1000 worth of compute will be able to match human performance around 2029 for decades now.
- KerrAvon 3y agoFirst, there has to be something capable of matching human performance at a much higher cost. This is still just spicy autocomplete.
- Waterluvian 3y agoHumans just do spicy autocomplete too.
- goatlover 3y agoNo they don't. You're "just" doing what everyone else in the past has done with the brain/human intelligence and using the latest technology as a metaphor without realizing it.
- Waterluvian 3y agoWe want to think we’re exceptional but all we can do is say “human consciousness is special” without having any way of measuring it or disproving the assertion that we’re just really fancy pattern matchers. Take any metaphor you want, it’s the same outcome: we may all be philosophical zombies.
- roarcher 3y agoWe may "just" be neural networks that run on meat instead of silicon, but that does not mean that we're LLMs.
- Waterluvian 3y agoWhy doesn’t it?
- marginalia_nu 3y agoIt's a formal logical error. One does not follow from the other without affirming the consequent.
- roarcher 3y agoBecause not all neural networks are LLMs. A GAN is a neural network, does that make it an LLM?
- hkt 3y agoWe have inputs other than words, for a start
- goatlover 3y agoI’m conscious, maybe you’re not? Not that I really believe that. I think you probably see colors and hear sounds, even in your dreams! But engineering types tend to be persuaded by a particular view of the world, failing to understand that it is a view and not nature itself.
- not2b 3y agoNo, a human isn't born with a set of knowledge like a freshly trained LLM, keeping the model fixed and responding to input. The analog to the model changes based on the human's experience. Just making bigger and bigger LLMs won't give you this.
- Waterluvian 3y agoSo a human is different because it keeps training its neural network?
- eurekin 3y agoWhoa, imagine you get a good base LLM model and save all conversations with it. Run a batch process every night to fine tune a LORA on convo dataset. If I ever came across such a chat bot I'd probably freak out as to why it remembers things outside of the context window, without summarisation
- MichaelZuo 3y agoThat's a pretty neat idea, I would be surprised if no one is already working on that.
- catchnear4321 3y agomonkeys make monkeys accidentally. monkeys make meseeks on purpose. there is a difference, but will it be fun?
- Waterluvian 3y agoI am 100% invested in how much ridiculous fun this era is going to be. Right up until the moment when it becomes a horror.
- gerdesj 3y agoReally? I can guess at what spicy autocomplete might actually mean but I doubt a LLM ... OK ChatGPT did a pretty good job of it (I've just asked it), whilst sidestepping the definition of spicy in this context. It is after all a very good next word guesser, given a context, and its not ... me! I am capable of hallucinating but it was 30 odd years ago since I hunted out certain fungi on Dartmoor, or smoked hemp. To be fair, we humans do often interrupt each other to second guess a sentence completion. Done correctly it is a brief satisfying collaboration. Done wrong ... I've been married for 18 years and know when to bite my tongue, but I still get it wrong from time to time - sometimes deliberately. Despite that, me and the wiff can autocomplete each other's sentences with uncanny accuracy and end up with perfect harmony or a cough slight disagreement as a result. We are getting some phenomenal slide rules these days but the darleks are not going to be flying up the stairwell just yet, nor will SkyNet be taking over tomorrow. That said, you just know that some noddy is trying to sell a nuclear "deterrent" LLM AI thingie somewhere. Thankfully, production military equipment takes quite a while to get to deployment. There is a good chance that we will get to grips with all this stuff before SkyNet is let loose for real 8)
- enterprise_cog 3y agoMaybe you do. Luckily, not everyone is quite so simple.
- ggm 3y agoThe irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the deep semantic meaning of the language presented. There's no intelligence.
- coolspot 3y ago> the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the deep semantic meaning of the language presented. There's no intelligence. Can say the same about a half of population, tbh
- binary132 3y agoSo what you’re saying is.... we’re going to have human-level AI, and it’s going to be incredibly stupid
- bugglebeetle 3y agoBAAGI (Below Average Artificial General Intelligence).
- djmips 3y agoOr more simply not AGI but AGS
- glimshe 3y agoWhy do you say they do nothing to advance AGI? Do you know what it takes to advance AGI? It's hard to state that without knowing how AGI would work yourself. LLMs would be considered magic just a couple years ago. Sure, not AGI but behaves just like one for certain workloads. I find hard to believe we're not a bit closer now - or maybe even a lot closer.
- csjh 3y agoIMO the direction we're going seems more like having a few small models in a MoE that are equivalent to a current 200bn model
- Aerroon 3y agoA 200b 4-bit quantized model could potentially fit into 128 GB of RAM. The inference would just be really slow. Ie you could technically run something like that today. I think more VRAM on GPUs isn't necessarily a technical limitation either. I think GPU manufacturers could add a lot more VRAM to their cards if they wanted to. The question is whether it would be worth the price increase.
- rootusrootus 3y ago> Ie you could technically run something like that today. Yep, on higher end machines it should already be feasible. I can do 2.5-3 tok/sec on a 70B model quantized at 4 bit today with my MacBook Pro M2 MAX w/96GB. It's a little slower than a 30B, but the difference is less than I had guessed it would be. That's not super fast, but it's usable. And that's on a machine that isn't designed for this workload. Over the next few years things should improve quite a bit. 200B does not seem like a reach.
- eurekin 3y agoAbout the RAM. I doubt they wanted to do that, since basic gpu function is to render a frame in as little ms as possible. Currently VRAM is latency optimized on consumer gpus and all memory chips are an inch away from the gpu. Light only travels as far in the gigahertz realm. Thats why they started mounting vram chips on both sides of the board, cause there was no more place left on the first side. Just checked: light travels 30cm in one nanosecond. So if the gpu is running at 4GHz it goes only 7.5 cm.
- Mertax 3y agoAnd then things like neural implants and BCIs -- seems like your dog could have language capabilities sooner than you'd think ;)