5 ms·

Large Language Models: A New Moore's Law?

54 points by EvgeniyZh 5y ago

plutonorm 5y ago
"All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident." You can tell we are approaching the end of the first stage because articles like this are getting increasingly non nonsensical. What on earth is this guy on about? Spectre and meltdown? What? All exponential are bad? What?
MontyCarloHall 5y ago
Comparisons between the number of neurons/synapses in the human brain and the number of parameters in these language models are meaningless. The computational power of a single real neuron is immense, whereas an artificial neuron is trivial. For example, a real neuron performs complex signal processing on input action potentials (e.g. spike train decoding), and is capable of responding very differently to many different “encodings” of those potentials (i.e. neurotransmitters). On the other hand, an artificial neuron simply performs weighted sums of the inputs and passes them into trivial activation functions (e.g. sigmoid or ReLU). To me, these analogies are like comparing an ant colony to human civilization solely based on the number of individual organisms.
- lysecret 5y ago
  I mean yes, but you could say take a sufficiently large set of hidden units (say 10-100) with all their parameters and they could easily represent the kinds of functions you are talking about. So the comparrisonn is off (we probably need about 100 times more parameters than the brain has neurons if that is the right measure)
  - skohan 5y ago
    I'm not totally convinced that hidden units are enough to model every neuronal system. For instance what the post above is alluding to with highly specific temporal information of encoding is not so easily represented in current ANN's as far as I'm aware. And there's also a ton of metabolic processes in the neuron which affect information processing in different ways at different time scales. I would not be so confident a neuron can be effectively modeled by just scaling up node count. And even if it can, that would mean you probably need an ANN of orders-of-magnitude greater size than neuron count to simulate a human brain with any accuracy. I.e. we might be talking about multiple nodes per synapse.
- FeepingCreature 5y ago
  Artificial neurons aren't all that simple either- the more complex your learning algorithms, the more additional state has to be associated with each neuron during training. And it's not clear how much of the computational work of a biological neuron is housekeeping overhead.
  - robbedpeter 5y ago
    The output of a biological neuron doesn't appear to be graduated. When a neuron fires, it releases a consistent signal, and it is everything downstream from the activation where dynamic learning takes place. Some connections can lead to inhibitory or excitatory signaling, reducing or increasing the voltage differentials, suppressing or reinforcing contextual signals, but the output of a neuron is a binary event. The attenuation or reinforcement of output signals occurs discretely from the activation of neurons, so the myelin can get thicker in dendritic connections, or connections can be added and pruned, or chemistry changes can affect the signal in a dynamic way. Once we figure out the algorithms encoded in the connectome/physical architecture, we'll have the opportunity to uncover the algorithms encoded in the data being transported between regions of the brain.
- leecarraher 5y ago
  For reference this recent article describes research that tries to nail down the exact computational ratio https://www.quantamagazine.org/how-computationally-complex-is-a-single-neuron-20210902/ https://www.quantamagazine.org/how-computationally-complex-i... spoiler : 5-8 layers of artificial neurons per complexity of biological neuron
  - tgb 5y ago
    That's how many it takes to "represent" a neuron. But what does that mean and is it relevant? I haven't checked the original research, but if it means to model the results of a neuron, then it's not relevant (unless you're trying to do brain uploading). Two ANNs with different activation functions will need a lot of nodes to model a single node of the other network, but they have comparable computational power. We don't need to match the neurons to match the computational abilities.
    - dxuh 5y ago
      What's the theoretical basis of this though? I don't have a theoretical background in computability, but in physics and there the complexity of a system very much correlates with the number of degrees of freedom. When simulating something, you want a bijection of the state of the simulation to the state of the system you are simulating, so you need to match (or exceed) the number of degrees of freedom. So if you know how many parameters you need to simulate another system until the simulation works, you can estimate how many internal degrees of freedom it has (degrees of freedom that are relevant to whatever you are measuring of course).
- nmca 5y ago
  Meaningless feels like a stretch here. Your evidence supports a claim much more like "mistaken by almost two orders of magnitude" - which at a sufficient growth rate might be moot.
- svantana 5y ago
  You can just as easily turn the comparison around: how many neurons does it take to compute a single 32-bit float multiplication? Clearly a lot, since >99.99% of humans are enable to do it without any tools. Whereas artificial nets rely on trillions of such operations per second. Computers and brains are quite different, but I would say computers are nearer to simulating a brain than vice versa.
  - raducu 5y ago
    That is irrelevant because in all the neural nets that I'm aware of, the activation functions we are using are very simple non-linear functions. We only use fp multiplication because that' the way we add up the weights to activations, nature has analogue ways of doing that.
  - qualudeheart 5y ago
    I believe that to be conflating conscious thought and unconscious thought. The brain has to be able to do fairly complex physics simulations just predicting how fast you’re going somewhere or wether a projectile flying through air will hit you. Consciously doing that is extremely hard (most people can’t do that math) but thatms just because consciousness is computationally inefficient. Most brain computation is unconscious and doesn’t have that limit.
- raducu 5y ago
  I always thought the big advantage of human/biological neurons was that they each could have 10.000 axons/dendrites and each one of those could also interact with thousands and thousands of neurons; and new connections could be created on the fly. And I'm sure some neurons could perform complex computations themselves, but artificial neurons can also do that, but so far we haven't found that to be particularly useful and only use a bunch of simple non-linear functions.
niklasd 5y ago
> "training BERT on GPU is roughly equivalent to a trans-American flight" This gets cited often, but is a number that doesn't impress me very much. Given the fact that (in theory) a language model like BERT has to be pre-trained only once[1], and these weights then can be reused by anyone to fine-tune the model[2], the number doesn't seem impudently high. I think its good to examine the environmental impact, but there are so many unnecessary flights every day, that could be substituted by trains, that for me the fact that pre-training a state of the art ML model for the CO2 cost of a flight did not seem very high for me. Or do I overlook something there? [1] Of course, in reality it's not so easy – there is hyperparameter search, domain adpotion, further experiments, etc. [2] Also because of the great open source work by Hugging Face.
- jillesvangurp 5y ago
  Also, the likes of Google, Amazon, MS, etc. generally use renewable power for their data centers and are each very actively investing in making their entire operations carbon neutral. So, yes they produce and use a lot of energy but not a lot of carbon.
sharikous 5y ago
I find this opinion piece rather inane - Investing immense amounts of resources historically helped to find algorithms that could be run with fewer resources. It's just that the raw computing power helped to lower the cost for research and exploration - Who is he speaking to? To the big companies pulling out tens of millions to do this? To the small ones? I think everybody knows the costs of deep learning training. The big companies took that into account and still find it cost effective. The small ones already know they cannot afford that power. - Spectre and Meltdown are bad examples of Moore's law?? I wonder where the author's mind is when he ignores the revolution inherent in stuffing today's tens of billions of transistors in chips that some decade ago contained tens of thousands of them. What has that to do with security vulnerabilities? Granted - those vulnerabilities are in large part due to the increased parallelization needed in today's chips, but that's due to the end of Moore's Law - As the other commenter said, comparison between neurons and parameters is completely meaningless
- evrydayhustling 5y ago
  Yeah, lots of respect for HuggingFace but this argument is pretty reductive. A very thin slice of the ML community/industry is rushing to short-term commercialization of giant models, and there's nothing wrong with poking some fun at what's being ignored in that haste. But for most of the field, ever-inflating models are a probe for exploring the many types of unreasonable effectiveness of gradient descent. Knowing more about what's possible is incredibly valuable to engineering pragmatic applications of ML. Research and commercialization can happen in parallel, and both sides would be better off if they educated the public about the difference.
HarHarVeryFunny 5y ago
I very much doubt we'll see continued 10x parameter growth per year for too many more years, although in the short term GPT-4 will confirm that trend (100T params in a "few years"). Even if continued scaling of these models is deemed worthwhile, I doubt training hardware can keep up to keep the cost manageable. Even GPT-3's $6M training cost (& time) is significant. Does anyone have the stomach for $100M training costs, say? How many times are you willing to iterate that as you tweak the model? GPT-3 is a very interesting line of research though, given the non-intuitive capabilities as this type of model is scaled up. It'll certainly be interesting to see what GPT-4 is capable of. However, I doubt that these transformer architectures are capturing everything that the cortex is doing, and intelligence probably involves more than just the cortex anyway (hypothalamus, maybe cerebellum, etc). I expect in the next decade we'll see more research on elaborating these architectures, and making them more efficient, rather than just scaling them up much beyond the planned 100T GPT-4 type of size. Maybe by the time we've figured out the architecture for intelligence the hardware will have caught up to make training and runtime costs more practical.
- albertzeyer 5y ago
  > Does anyone have the stomach for $100M training costs, say? No, but the point of such law is that it will get exponentially cheaper as well (also count in the inflation of money over time). So, maybe next year we can train some 10x bigger model in just the same amount of money.
domenicrosati 5y ago
I think the author is not doing a good job of presenting the research program behind ever larger language models. Many researchers are interested in the effects of scale on natural language understanding tasks and generalization. One of the surprising results of large language models is their performance on GLUE or the like without needed specific training indicating they are extremely good at generalization. In the NLP community we do want to understand how far we can push this since generalization is an important aspect of modeling how we understand language. The fact is we dont know how much we can push this , and that very fact is very interesting to me and many others. I think many researchers involved in this would acknowledge that this research program may not provide direct and interpretal insight into many language tasks... but that is sort of missing the point. I am also skeptical that you would have initiatives like DistillBert and other attempts at smaller models that are as effective without the larger ones. Also the amount of money required for training these large scale models is available to researchers through public research funding. Researchers in other disciplines regularly have million dollar research programs. I do wish the author of this article had taken the time to present the research program of scaling large models clearer as it reads as a shameless ad for huggingface (use canned models instead of pushing for the limits of research) which doesn't really apply to research in NN design anyway..
DeathArrow 5y ago
It seems everybody and their mom is using some kind of neural network (preferable something with deep in it's name). What happened to other kinds of ML algorithms, linear regression, logistic regression, decision tree, SVM, naive Bayes, kNN, K-means, random forest, gradient boosting?

54 points by EvgeniyZh 5y ago

"All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident." You can tell we are approaching the end of the first stage because articles like this are getting increasingly non nonsensical. What on earth is this guy on about? Spectre and meltdown? What? All exponential are bad? What?

MontyCarloHall 5y ago

Comparisons between the number of neurons/synapses in the human brain and the number of parameters in these language models are meaningless. The computational power of a single real neuron is immense, whereas an artificial neuron is trivial. For example, a real neuron performs complex signal processing on input action potentials (e.g. spike train decoding), and is capable of responding very differently to many different “encodings” of those potentials (i.e. neurotransmitters). On the other hand, an artificial neuron simply performs weighted sums of the inputs and passes them into trivial activation functions (e.g. sigmoid or ReLU). To me, these analogies are like comparing an ant colony to human civilization solely based on the number of individual organisms.

lysecret 5y ago
I mean yes, but you could say take a sufficiently large set of hidden units (say 10-100) with all their parameters and they could easily represent the kinds of functions you are talking about. So the comparrisonn is off (we probably need about 100 times more parameters than the brain has neurons if that is the right measure)
- skohan 5y ago
  I'm not totally convinced that hidden units are enough to model every neuronal system. For instance what the post above is alluding to with highly specific temporal information of encoding is not so easily represented in current ANN's as far as I'm aware. And there's also a ton of metabolic processes in the neuron which affect information processing in different ways at different time scales. I would not be so confident a neuron can be effectively modeled by just scaling up node count. And even if it can, that would mean you probably need an ANN of orders-of-magnitude greater size than neuron count to simulate a human brain with any accuracy. I.e. we might be talking about multiple nodes per synapse.
FeepingCreature 5y ago
Artificial neurons aren't all that simple either- the more complex your learning algorithms, the more additional state has to be associated with each neuron during training. And it's not clear how much of the computational work of a biological neuron is housekeeping overhead.
- robbedpeter 5y ago
  The output of a biological neuron doesn't appear to be graduated. When a neuron fires, it releases a consistent signal, and it is everything downstream from the activation where dynamic learning takes place. Some connections can lead to inhibitory or excitatory signaling, reducing or increasing the voltage differentials, suppressing or reinforcing contextual signals, but the output of a neuron is a binary event. The attenuation or reinforcement of output signals occurs discretely from the activation of neurons, so the myelin can get thicker in dendritic connections, or connections can be added and pruned, or chemistry changes can affect the signal in a dynamic way. Once we figure out the algorithms encoded in the connectome/physical architecture, we'll have the opportunity to uncover the algorithms encoded in the data being transported between regions of the brain.
leecarraher 5y ago
For reference this recent article describes research that tries to nail down the exact computational ratio https://www.quantamagazine.org/how-computationally-complex-is-a-single-neuron-20210902/ https://www.quantamagazine.org/how-computationally-complex-i... spoiler : 5-8 layers of artificial neurons per complexity of biological neuron
- tgb 5y ago
  That's how many it takes to "represent" a neuron. But what does that mean and is it relevant? I haven't checked the original research, but if it means to model the results of a neuron, then it's not relevant (unless you're trying to do brain uploading). Two ANNs with different activation functions will need a lot of nodes to model a single node of the other network, but they have comparable computational power. We don't need to match the neurons to match the computational abilities.
  - dxuh 5y ago
    What's the theoretical basis of this though? I don't have a theoretical background in computability, but in physics and there the complexity of a system very much correlates with the number of degrees of freedom. When simulating something, you want a bijection of the state of the simulation to the state of the system you are simulating, so you need to match (or exceed) the number of degrees of freedom. So if you know how many parameters you need to simulate another system until the simulation works, you can estimate how many internal degrees of freedom it has (degrees of freedom that are relevant to whatever you are measuring of course).
nmca 5y ago
Meaningless feels like a stretch here. Your evidence supports a claim much more like "mistaken by almost two orders of magnitude" - which at a sufficient growth rate might be moot.
svantana 5y ago
You can just as easily turn the comparison around: how many neurons does it take to compute a single 32-bit float multiplication? Clearly a lot, since >99.99% of humans are enable to do it without any tools. Whereas artificial nets rely on trillions of such operations per second. Computers and brains are quite different, but I would say computers are nearer to simulating a brain than vice versa.
- raducu 5y ago
  That is irrelevant because in all the neural nets that I'm aware of, the activation functions we are using are very simple non-linear functions. We only use fp multiplication because that' the way we add up the weights to activations, nature has analogue ways of doing that.
- qualudeheart 5y ago
  I believe that to be conflating conscious thought and unconscious thought. The brain has to be able to do fairly complex physics simulations just predicting how fast you’re going somewhere or wether a projectile flying through air will hit you. Consciously doing that is extremely hard (most people can’t do that math) but thatms just because consciousness is computationally inefficient. Most brain computation is unconscious and doesn’t have that limit.
raducu 5y ago
I always thought the big advantage of human/biological neurons was that they each could have 10.000 axons/dendrites and each one of those could also interact with thousands and thousands of neurons; and new connections could be created on the fly. And I'm sure some neurons could perform complex computations themselves, but artificial neurons can also do that, but so far we haven't found that to be particularly useful and only use a bunch of simple non-linear functions.

niklasd 5y ago

> "training BERT on GPU is roughly equivalent to a trans-American flight" This gets cited often, but is a number that doesn't impress me very much. Given the fact that (in theory) a language model like BERT has to be pre-trained only once[1], and these weights then can be reused by anyone to fine-tune the model[2], the number doesn't seem impudently high. I think its good to examine the environmental impact, but there are so many unnecessary flights every day, that could be substituted by trains, that for me the fact that pre-training a state of the art ML model for the CO2 cost of a flight did not seem very high for me. Or do I overlook something there? [1] Of course, in reality it's not so easy – there is hyperparameter search, domain adpotion, further experiments, etc. [2] Also because of the great open source work by Hugging Face.

jillesvangurp 5y ago
Also, the likes of Google, Amazon, MS, etc. generally use renewable power for their data centers and are each very actively investing in making their entire operations carbon neutral. So, yes they produce and use a lot of energy but not a lot of carbon.

sharikous 5y ago

I find this opinion piece rather inane - Investing immense amounts of resources historically helped to find algorithms that could be run with fewer resources. It's just that the raw computing power helped to lower the cost for research and exploration - Who is he speaking to? To the big companies pulling out tens of millions to do this? To the small ones? I think everybody knows the costs of deep learning training. The big companies took that into account and still find it cost effective. The small ones already know they cannot afford that power. - Spectre and Meltdown are bad examples of Moore's law?? I wonder where the author's mind is when he ignores the revolution inherent in stuffing today's tens of billions of transistors in chips that some decade ago contained tens of thousands of them. What has that to do with security vulnerabilities? Granted - those vulnerabilities are in large part due to the increased parallelization needed in today's chips, but that's due to the end of Moore's Law - As the other commenter said, comparison between neurons and parameters is completely meaningless

evrydayhustling 5y ago
Yeah, lots of respect for HuggingFace but this argument is pretty reductive. A very thin slice of the ML community/industry is rushing to short-term commercialization of giant models, and there's nothing wrong with poking some fun at what's being ignored in that haste. But for most of the field, ever-inflating models are a probe for exploring the many types of unreasonable effectiveness of gradient descent. Knowing more about what's possible is incredibly valuable to engineering pragmatic applications of ML. Research and commercialization can happen in parallel, and both sides would be better off if they educated the public about the difference.

HarHarVeryFunny 5y ago

I very much doubt we'll see continued 10x parameter growth per year for too many more years, although in the short term GPT-4 will confirm that trend (100T params in a "few years"). Even if continued scaling of these models is deemed worthwhile, I doubt training hardware can keep up to keep the cost manageable. Even GPT-3's $6M training cost (& time) is significant. Does anyone have the stomach for $100M training costs, say? How many times are you willing to iterate that as you tweak the model? GPT-3 is a very interesting line of research though, given the non-intuitive capabilities as this type of model is scaled up. It'll certainly be interesting to see what GPT-4 is capable of. However, I doubt that these transformer architectures are capturing everything that the cortex is doing, and intelligence probably involves more than just the cortex anyway (hypothalamus, maybe cerebellum, etc). I expect in the next decade we'll see more research on elaborating these architectures, and making them more efficient, rather than just scaling them up much beyond the planned 100T GPT-4 type of size. Maybe by the time we've figured out the architecture for intelligence the hardware will have caught up to make training and runtime costs more practical.

albertzeyer 5y ago
> Does anyone have the stomach for $100M training costs, say? No, but the point of such law is that it will get exponentially cheaper as well (also count in the inflation of money over time). So, maybe next year we can train some 10x bigger model in just the same amount of money.

domenicrosati 5y ago

I think the author is not doing a good job of presenting the research program behind ever larger language models. Many researchers are interested in the effects of scale on natural language understanding tasks and generalization. One of the surprising results of large language models is their performance on GLUE or the like without needed specific training indicating they are extremely good at generalization. In the NLP community we do want to understand how far we can push this since generalization is an important aspect of modeling how we understand language. The fact is we dont know how much we can push this , and that very fact is very interesting to me and many others. I think many researchers involved in this would acknowledge that this research program may not provide direct and interpretal insight into many language tasks... but that is sort of missing the point. I am also skeptical that you would have initiatives like DistillBert and other attempts at smaller models that are as effective without the larger ones. Also the amount of money required for training these large scale models is available to researchers through public research funding. Researchers in other disciplines regularly have million dollar research programs. I do wish the author of this article had taken the time to present the research program of scaling large models clearer as it reads as a shameless ad for huggingface (use canned models instead of pushing for the limits of research) which doesn't really apply to research in NN design anyway..

DeathArrow 5y ago

It seems everybody and their mom is using some kind of neural network (preferable something with deep in it's name). What happened to other kinds of ML algorithms, linear regression, logistic regression, decision tree, SVM, naive Bayes, kNN, K-means, random forest, gradient boosting?