3 ms·

I have to admit ignorance. Is this just a parameters race? Like is 3T parameters 6 times better that 500b? Does the task / work scale linearly with parameters

I have to admit ignorance. Is this just a parameters race? Like is 3T parameters 6 times better that 500b? Does the task / work scale linearly with parameters? I know I could just ask chatgpt but wanted to support human content on the internet.

tambourine_man 4y ago
That last phrase hit me hard.
- didericis 4y ago
  ChatGPT is human content on the internet. It's just indexed and compressed and was made "generative" in a mind boggling way.
cafed00d 4y ago
Does it still count as human content if I blindly copy paste from ChatGPT? :))
vc9999 4y ago
For now increasing data size has been enough to show better results. We don't when diminishing returns will be significant.
tomohelix 4y ago
From what I know, parameter count correlates directly with capability to a certain degree. But Google has also shown there is a benefit in training that can enable lower parameter count models to match or even outperform those with bigger counts. The trade off is in the cost for training. But in return, you get a smaller and cheaper to run model that can be used by more people. That is not what PALM E is about though. This is the paper where they patch an LLM so that it can give command to a real, physical robot and basically let the LLM interact with meatspace.
- narrator 4y ago
  Suddenly those jailbreaks of the LLM become a lot more serious. What happens when bad people can get a robot to help them commit various crimes?
  - Animats 4y ago
    What happens when it decides that the human interfering in the example is the source of the problem and needs to be eliminated?
    - hackinthebochs 4y ago
      All I could think of watching that video was I would not want to be that guy.
  - flextheruler 4y ago
    As far as I’m aware the only “robots” we have at the moment that are capable of large scale physical damage, without direct human involvement while the damage is occurring, are military drones and missiles. And neither missiles nor drones are capable of functioning totally independent with both requiring large amounts of weapon systems infrastructure. You may be able to argue missiles could be deployed almost independently if failsafes are circumvented but that would require modernization of missile weapon systems which is about as likely as the colonization of space. At present I’d be more worried about network and social engineering attacks using LLM which I’m sure is already happening and will continuously increase.
    - narrator 4y ago
      The household robot that can cook will be able to use a knife to cut anything it arbitrarily thinks needs to be cut which could lead to it being directed to assist with serious criminal acts. Self-driving cars out on the streets have already killed people, but those were just bugs, not intentional behavior, as might occur if they were driven by LLMs.
  - ficklepickle 4y ago
warent 4y ago
I know I could just ask chatgpt but wanted to support human content on the internet. The irony of this sentence. As if the millionth repetitive copy of defeatist doomsday commentary on chatGPT is somehow a very different kind of content
lynndotpy 4y ago
Kind of. Speaking very generally, it's not just the number of weights ("parameters") but how you use them. The architecture of the neural network is where the most interesting new work is taking place. Stable Diffusion, which was beating out Dall-E2, was a big deal not just because the weights were public, but also because they were small enough to run on consumer hardware. This is because of the clever architectural choice (plus some other methods.) This was surprising to me, because most of the interesting image generation required huge models running on commercial GPUs. You'd generally expect diminishing returns from adding weights (as well as a lot more training time and maybe more data) but this is not a law.
dougmwne 4y ago
Yes and no. The performance does scale up with parameters, though it’s not linear. As discovered by Google in their work on their Chinchilla LLM, performance also scales up with the size of your training set. They were able to do work to define the optimal amount of training for a model of a given size to get the most out of your budget. So even if we don’t find any better model architectures, which we probably will, if we increase the size of our models, training corpus and budget, we should continue to get more performant models.
- freeqaz 4y ago
  How does that performance scale though? If you double the inputs you train on, do you get twice the performance? Less? More? I've heard conflicting things and I'm curious to hear your reasoning!
  - peteradio 4y ago
    I asked ChatGPT, it said: "Uhhh it depends."
  - ivalm 4y ago
    Chinchilla paper proposes the following empirical scaling power law Loss= (N0/N)^a+(D0/D)^b Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.
  - 152334H 4y ago
    Where A,B,C are empirical constants, Loss = A/parameters + B/dataset_size + C Just read the chinchilla paper.
  - stevenhuang 4y ago
    The scaling appears linear in relationship, though on some performance metrics it is super linear. See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728 https://arxiv.org/abs/2301.03728 All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).
marcusf 4y ago
Parameters and training corpus size matter. Last year a new paper (Google "Chinchilla optimality") focused on compute-optimal LLMs found that we'd been under-training models -- i.e. you could wring more performance out of smaller models by training on more data. But (as far as I understand it - interested layman here) - for a given amount of data, model performance seems to scale more or less linearly with parameters. Now, we could see another model architecture than the current reigning transformer architecture upend this (much work is ongoing on breaking the quadratic term in the transformer that computationally bounds its performance - an example is the Hyena paper that was published just the other day). Biggest computer and most data wins is still the paradigm here.
kerpotgh 4y ago
[dead]

tambourine_man 4y ago

That last phrase hit me hard.

didericis 4y ago
ChatGPT is human content on the internet. It's just indexed and compressed and was made "generative" in a mind boggling way.

cafed00d 4y ago

Does it still count as human content if I blindly copy paste from ChatGPT? :))

vc9999 4y ago

For now increasing data size has been enough to show better results. We don't when diminishing returns will be significant.

tomohelix 4y ago

From what I know, parameter count correlates directly with capability to a certain degree. But Google has also shown there is a benefit in training that can enable lower parameter count models to match or even outperform those with bigger counts. The trade off is in the cost for training. But in return, you get a smaller and cheaper to run model that can be used by more people. That is not what PALM E is about though. This is the paper where they patch an LLM so that it can give command to a real, physical robot and basically let the LLM interact with meatspace.

narrator 4y ago
Suddenly those jailbreaks of the LLM become a lot more serious. What happens when bad people can get a robot to help them commit various crimes?
- Animats 4y ago
  What happens when it decides that the human interfering in the example is the source of the problem and needs to be eliminated?
  - hackinthebochs 4y ago
    All I could think of watching that video was I would not want to be that guy.
- flextheruler 4y ago
  As far as I’m aware the only “robots” we have at the moment that are capable of large scale physical damage, without direct human involvement while the damage is occurring, are military drones and missiles. And neither missiles nor drones are capable of functioning totally independent with both requiring large amounts of weapon systems infrastructure. You may be able to argue missiles could be deployed almost independently if failsafes are circumvented but that would require modernization of missile weapon systems which is about as likely as the colonization of space. At present I’d be more worried about network and social engineering attacks using LLM which I’m sure is already happening and will continuously increase.
  - narrator 4y ago
    The household robot that can cook will be able to use a knife to cut anything it arbitrarily thinks needs to be cut which could lead to it being directed to assist with serious criminal acts. Self-driving cars out on the streets have already killed people, but those were just bugs, not intentional behavior, as might occur if they were driven by LLMs.
- ficklepickle 4y ago

warent 4y ago

I know I could just ask chatgpt but wanted to support human content on the internet. The irony of this sentence. As if the millionth repetitive copy of defeatist doomsday commentary on chatGPT is somehow a very different kind of content

lynndotpy 4y ago

Kind of. Speaking very generally, it's not just the number of weights ("parameters") but how you use them. The architecture of the neural network is where the most interesting new work is taking place. Stable Diffusion, which was beating out Dall-E2, was a big deal not just because the weights were public, but also because they were small enough to run on consumer hardware. This is because of the clever architectural choice (plus some other methods.) This was surprising to me, because most of the interesting image generation required huge models running on commercial GPUs. You'd generally expect diminishing returns from adding weights (as well as a lot more training time and maybe more data) but this is not a law.

dougmwne 4y ago

Yes and no. The performance does scale up with parameters, though it’s not linear. As discovered by Google in their work on their Chinchilla LLM, performance also scales up with the size of your training set. They were able to do work to define the optimal amount of training for a model of a given size to get the most out of your budget. So even if we don’t find any better model architectures, which we probably will, if we increase the size of our models, training corpus and budget, we should continue to get more performant models.

freeqaz 4y ago
How does that performance scale though? If you double the inputs you train on, do you get twice the performance? Less? More? I've heard conflicting things and I'm curious to hear your reasoning!
- peteradio 4y ago
  I asked ChatGPT, it said: "Uhhh it depends."
- ivalm 4y ago
  Chinchilla paper proposes the following empirical scaling power law Loss= (N0/N)^a+(D0/D)^b Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.
- 152334H 4y ago
  Where A,B,C are empirical constants, Loss = A/parameters + B/dataset_size + C Just read the chinchilla paper.
- stevenhuang 4y ago
  The scaling appears linear in relationship, though on some performance metrics it is super linear. See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728 https://arxiv.org/abs/2301.03728 All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).

marcusf 4y ago

Parameters and training corpus size matter. Last year a new paper (Google "Chinchilla optimality") focused on compute-optimal LLMs found that we'd been under-training models -- i.e. you could wring more performance out of smaller models by training on more data. But (as far as I understand it - interested layman here) - for a given amount of data, model performance seems to scale more or less linearly with parameters. Now, we could see another model architecture than the current reigning transformer architecture upend this (much work is ongoing on breaking the quadratic term in the transformer that computationally bounds its performance - an example is the Hyena paper that was published just the other day). Biggest computer and most data wins is still the paradigm here.

kerpotgh 4y ago

[dead]