47 ms·
>Look i'm not going to say the transformer is as efficient as the brain but you are not starting from Zero. Still, we can rewrite the parent's argument as: If
by coldtea 4y ago
>Look i'm not going to say the transformer is as efficient as the brain but you are not starting from Zero.
Still, we can rewrite the parent's argument as:
If we train an AI on the amount of non-code-related dataset (writing read/speech heard) I've consumed, and then add to it all the amount of code-related writing/speech (coding book, coding lessons taken, code and manual pages read, man pages, etc.) I've consumed, would it even remotely as good as coding as me? Or even as good as itself is now?
I'd guess no. It's less effiecient, and thus needs way more coding dataset to get the point of coding than a human. Which brings us to:
>Your brain is baked with millions of years of evolution with specific areas already predisposed to certain types of processing before you ever utter a word.
Isn't that the whole point the parent is making?
That our advantage isn't about dataset-volume, but architecture.
- famouswaffles 4y agoThe closest biological equivalent to a parameter in an ann is a synapse. Well humans have about 100 trillion synapses. We already know that the higher the parameter count, the lower the training data required. a 50 billion parameter model will far outperform a 5 billion one trained on the same data. and a 500b one would far outperform that 50 billion one. Current LLMs are actually nowhere near the scale of the human brain, either in parameters/neurons or training data (all the text we've ever trained an LLM on would be dwarfed by all the sense data humans perceive), as well as not having the headstart the human brain has. It's a bogus comparison when you really think about it. You could easily make the case that LLMs are far more efficient.
- chaxor 4y agoIt is a bogus comparison, because typical language models used already have textual representations, and output textual representations; which is a very small fraction of 'the brain'. An astounding portion of the brain's neurons really go towards proprioception, smell, taste, motor function, etc. Which are not at all even slightly part of most models today. Wernicke's area, a small sliver of frontal cortex, and a dash of dopaminergic circuitry is maybe the best 'coverage' made in these models if you want to be exceedingly facile/ham-handed about the analogies here. That's a very small portion of cortex, and much closer than what you may think in terms of capability/ TEPS and unit count.
- reaperman 4y agoIt's not clear how many classical calculations a single human neuron is equivalent to. There's a strong analog component in multiple domains (strength, frequency, timing) and each neuron can connect to up to 15,000 other neurons. Assuming the brain's neurons are (probably unrealistically) fairly 'digital' we get an estimation of the human brain being equivalent to 1 exaflop (this is the currently accepted lower bound, and rather disputed as being too low). Current TPUv4 pods currently provide approximately 9 exaflops. I don't think we're currently reaching human-level learning rates. There’s currently no accepted “upper bound” on estimates of FLOP equivalency to a human brain. https://www.youtube.com/watch?v=HB5TrK7A4pI https://www.youtube.com/watch?v=HB5TrK7A4pI is a recently posted video to HN Frontpage which was summarized as such: > Though we have been building and programming computing machines for about 60 years and have learned a great deal about composition and abstraction, we have just begun to scratch the surface. > A mammalian neuron takes about ten milliseconds to respond to a stimulus. A driver can respond to a visual stimulus in a few hundred milliseconds, and decide an action, such as making a turn. So the computational depth of this behavior is only a few tens of steps. We don't know how to make such a machine, and we wouldn't know how to program it. > The human genome -- the information required to build a human from a single, undifferentiated eukariotic cell -- is about 1GB. The instructions to build a mammal are written in very dense code, and the program is extremely flexible. Only small patches to the human genome are required to build a cow or a dog rather than a human. Bigger patches result in a frog or a snake. We don't have any idea how to make a description of such a complex machine that is both dense and flexible. > New design principles and new linguistic support are needed. I will address this issue and show some ideas that can perhaps get us to the next phase of engineering design. > Gerald Sussman Massachusetts Institute of Technology
- chaxor 4y agoMy understanding is that TEPS were used to determine computing for these types operations, rather than FLOPS, as they were more useful specifically for that comparison. There metrics put them in the same order of magnitude; however, as stated before, these miss the point by quite a bit, since much of the 'computations' humans do are quite irrelevant (taste, smell, etc) to producing language or solving algorithmic problems, etc. For example, the cerebellum is 50-80% of what people keep quoting here (Number of neurons in the brain) and is not activated much in language processing. Wernicke's area spans just a few percent of the cortical neurons. The amount of pre processing we do by providing text is actually quite enormous, so that already removes a remarkable amount of complexity from the model. So, despite the differences between biology and ANNs, it's not unreasonable what were seeing right now.
- coldtea 4y ago>a 50 billion parameter model will far outperform a 5 billion one trained on the same data. and a 500b one would far outperform that 50 billion one. I'm not so sure. I'm pretty sure there are diminishing returns at play after some point. Plus haven't we already seen models with much less billions of parameters perform the same or very close to ChatGPT with had a much higher count (Llama and its siblings)?
- famouswaffles 4y ago>a 50 billion parameter model will far outperform a 5 billion one trained on the same data. and a 500b one would far outperform that 50 billion one. I'm not so sure. I'm pretty sure there are diminishing returns at play after some point. We can speculate about just how far this scaling can go or how far is even necessary but all i've said there is true. We have models trained and evaluated on all those sizes. >Plus haven't we already seen models with much less billions of parameters perform the same or very close to ChatGPT with had a much higher count (Llama and its siblings)? Only by training on far more data. Llama 13b has to be trained on over 3x more data just to reach the original GPT-3 model from 2020 (not 3.5).
- deleted 4y ago[deleted]
- coldtea 4y ago>We can speculate about just how far this scaling can go or how far is even necessary but all i've said there is true. We have models trained and evaluated on all those sizes. The part about "far outperforming", which is the main claim, is wrong though. We saw models much smaller being developed that fare quite well, and are even competitive, with the larger ones. You already said "only by training on far more data", which is different than "more parameters" being the only option.
- famouswaffles 4y ago>You already said "only by training on far more data", which is different than "more parameters" being the only option. I never said more parameters was the only way to increase performance. I said the training data required to reach any arbitrary performance x reduces with parameter size. It's literally right there in what I wrote. >a 50 billion parameter model will far outperform a 5 billion one TRAINED ON THE SAME DATA.
- somenameforme 4y agoYou run into the typical neural net problem with this logic. OpenAI (or at least Sam Altman) have already publicly acknowledged that the diminishing returns they're seeing in terms of model size are sufficient to effectively declare that 'the age of giant models is already over.' [1] It seems many people were unaware of his comments on this topic. Neural networks in literally every other field always repeat the exact same pattern. You can get from 0-80 without breaking a sweat. 80-90 is dramatically harder but you finally get there. So everybody imagines getting from 90-100 will be little more than a matter of a bit more compute and a bit more massaging of the model. But it turns out that each fraction of a percent progress you make starts becoming exponentially more difficult - and you eventually run into an asymptote that's nowhere near what you are aiming for. A prediction based on the typical history of neural nets would be that OpenAI will be able to continue to make progress on extremely specific metrics, like scoring well on some test or another, largely by hardcoding case-specific workarounds and tweaks. But in terms of general model usage, we're unlikely to see any real revolutionary leaps in the foreseeable future. If we see model accuracy increase I'd expect it to be thanks not to model improvement, but instead by doing something like adding a second layer where the software cross references the generated output against a 'fact database' and regenerates its answer when some correlation factor is insufficiently high. Of course that'd completely cripple the model's ability to ever move 'beyond' its training. It'd be like if mankind was forced to double check that any response on astronomy we made confirmed that the Earth is indeed the center of the universe, with no ability to ever change that ourselves. [1] - https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...
- oezi 4y agoThere is an argument that Altman's statement is just trying to distract competitors from outspending OpenAI. Prior to GPT-4 there was no indications that there are diminishing returns (at least on a log scale). The tremendous progress over the last year makes me vary of your statement that progress will stop coming from model size improvements.
- coldtea 4y ago
- littlestymaar 4y ago> Well humans have about 100 trillion synapses. We already know that the higher the parameter count, the lower the training data required. Do you have any reference to back this claim, because it sounds is very curious to me. My understanding was pretty much the opposite, that current LLM technology require a bigger training set as you increase the parameter count. I'm no NN expert in any way though.
- famouswaffles 4y agoYou can look at any training run comparing multiple parameter sizes on the dataset. https://arxiv.org/abs/2204.02311 https://arxiv.org/abs/2204.02311 If you're increasing parameter size, it's a no brainer to increase data too as that will still also increase performance. The point is that for any arbitrary performance x, the data required to reach it reduces with size.
- littlestymaar 4y ago> If you're increasing parameter size, it's a no brainer to increase data too as that will still also increase performance. It also increases the cost by a lot, so it's not a no-brainer at all. If they could beat the state of the art with only a fraction of the training cost, I suspect that they'd do so… > The point is that for any arbitrary performance x, the data required to reach it reduces with size. This is the claim you're making, but it's not substantiated.
- famouswaffles 4y ago>It also increases the cost by a lot, so it's not a no-brainer at all. Okay?.. Parameter size increases also increase cost a lot. Far more than more training data. Costs that stay well beyond training. Training on 1T tokens vs 500b won't change how resources it takes to run. Not the cases with parameter sizes. >If they could beat the state of the art with only a fraction of the training cost, I suspect that they'd do so… Not sure what this has to do with anything lol >This is the claim you're making, but it's not substantiated. I'm sorry but can you perhaps just read the paper sent ? Google trained 3 differently sized models of the same architecture (8b, 62b, 540b) on the same dataset of 780b tokens and evaluated all 3 on various tasks.
- bionhoward 4y agoGpt4 is 1% of the brain’s param count, which means we’re basically already there. It’s exponential growth!
- nerpderp82 4y agoThis is why all the Alpha%wildcard% projects use self play, that along with a fitness function, or score maximizer generates their training data rather than have to show it records of games. You just let it play the game.
- reaperman 4y agoNot all of them. AlphaStar for StarCraft was largely supervised learning, to mimic top human players. Once it was sufficiently mimicking existing replays, then it switched to Reinforcement Learning. That's why you see it do "nonsensical" things like destroying the "unbuildable plates" at the bottom of its natural ramp. This 100% wouldn't happen if it had only learned through self-play.
- dwallin 4y agoThat’s an arbitrary line to make though. The human brain starts with a general architecture that has been gone through billions and trillions of generations of evolutionary training, before being fine tuned in a single individual over decades, and then you did a little bit of fine tuning and few shot at the end and claim that is comparable to the entire training of a LLM from scratch? Not to mention the many more orders of magnitude of neurons that a human brain has. I could equally argue that an LLM takes zero training, since we have ALREADY trained models and I can just copy the model and run it and get a new “brain” performing a task, instead of taking decades of training to get there. Even your statement about programming skills is debatable, it depends on how you measure programming skill. They certainly are faster at it, and they know more computer languages than most people have even heard of. In fact, human programming strength seems to be more about general logic and planning skills over programming-specific skills, both things where the bulk of training happened evolutionarily and more generally over the course of a life. The truth is, the two are not directly comparable. They are completely different architectures, at completely different scales, with entirely different strengths and weaknesses.