4 ms·

Which goes to show you one of the best strategies in ML isn’t necessarily thinking about learning theory, it’s thinking about what our fastest computers are mos

Which goes to show you one of the best strategies in ML isn’t necessarily thinking about learning theory, it’s thinking about what our fastest computers are most capable of doing. But this does run out of steam (or rather…data) eventually. The largest LLM was ibm’s tangora model back in the 1980s. 8 trillion params. We’ll go through the cycle again with a new paradigm.

IIAOPSW 3y ago
"It just goes to show the best strategy in assembling wood together isn't learning to use a screw driver, its about pounding it in with the hardest and most capable hammer."
- simonh 3y ago
  Yep. Sometimes so. https://www.thesprucecrafts.com/wood-joinery-methods-use-no-fasteners-3536634 https://www.thesprucecrafts.com/wood-joinery-methods-use-no-...
  - IIAOPSW 3y ago
    I'll acknowledge Large Hammer Models have done surprisingly well, but true General Artificial Fastening is obviously not just Hammer models scaled up. There's things a screw driver can do which a hammer is hopeless on. Such as taking apart your eye glasses (and more importantly, putting them back together). This isn't something that is going to be solved once the tradesmen have a bit more training. Its fundamental to the way hammers work. We can only do so much by hammering in all our screws.
yarg 3y ago
There was a similar argument for not improving the efficiency of software in general - why bother to improve the design if upgrading the underlying hardware can double performance without changing the software? It's not entirely invalid, but it does encourage a rather blasé attitude towards actually understanding the problem that you're dealing with.
- balder1991 3y ago
  And at some point the cost of hardware starts to add up.
tornato7 3y ago
8 trillion params in 1980? I don't think so bud. There wasn't that much memory in the entire world at the time.
deleted 3y ago
[deleted]
creatonez 3y ago
> The largest LLM was ibm’s tangora model back in the 1980s. 8 trillion params. As far as I can tell, this comes from a typo in a 1992 paper? https://aclanthology.org/J92-4003.pdf https://aclanthology.org/J92-4003.pdf Does it mean the space of possible inputs is 8 trillion?
- gwern 3y ago
  It's an n-gram model so it can easily have as many parameters as you want to store. Skimming it, I think the '8 trillion parameters' here means '8 trillion trainable n-gram parameters but it's very sparse and most of them are simply defined by omission to be a small constant like epsilon due to not being represented in the training data at all and so suffering from the 0-count problem'. (Old problem - even Turing and Good spent a lot of time discussing how to handle 0s in contingency tables because that happens a lot in cryptography!)
- IshanMi 3y ago
  I think the 8 trillion parameters is accurate- Tangora is an N-gram model with a vocab size of 20,000 words and N = 3. Parameters for an N-gram model = V^(N-1) * (V-1) Plugging in V=20,000 words and N = 3 for Tangora, you'd get 7.9996E12. Most of the parameters are likely zero or close to it because many 3-grams are possible but not likely to occur. (However the aggregate probability of all 3-grams is substantial and thus they have to be included.)

IIAOPSW 3y ago

"It just goes to show the best strategy in assembling wood together isn't learning to use a screw driver, its about pounding it in with the hardest and most capable hammer."

simonh 3y ago
Yep. Sometimes so. https://www.thesprucecrafts.com/wood-joinery-methods-use-no-fasteners-3536634 https://www.thesprucecrafts.com/wood-joinery-methods-use-no-...
- IIAOPSW 3y ago
  I'll acknowledge Large Hammer Models have done surprisingly well, but true General Artificial Fastening is obviously not just Hammer models scaled up. There's things a screw driver can do which a hammer is hopeless on. Such as taking apart your eye glasses (and more importantly, putting them back together). This isn't something that is going to be solved once the tradesmen have a bit more training. Its fundamental to the way hammers work. We can only do so much by hammering in all our screws.

yarg 3y ago

There was a similar argument for not improving the efficiency of software in general - why bother to improve the design if upgrading the underlying hardware can double performance without changing the software? It's not entirely invalid, but it does encourage a rather blasé attitude towards actually understanding the problem that you're dealing with.

balder1991 3y ago
And at some point the cost of hardware starts to add up.

tornato7 3y ago

8 trillion params in 1980? I don't think so bud. There wasn't that much memory in the entire world at the time.

deleted 3y ago

[deleted]

creatonez 3y ago

> The largest LLM was ibm’s tangora model back in the 1980s. 8 trillion params. As far as I can tell, this comes from a typo in a 1992 paper? https://aclanthology.org/J92-4003.pdf https://aclanthology.org/J92-4003.pdf Does it mean the space of possible inputs is 8 trillion?

gwern 3y ago
It's an n-gram model so it can easily have as many parameters as you want to store. Skimming it, I think the '8 trillion parameters' here means '8 trillion trainable n-gram parameters but it's very sparse and most of them are simply defined by omission to be a small constant like epsilon due to not being represented in the training data at all and so suffering from the 0-count problem'. (Old problem - even Turing and Good spent a lot of time discussing how to handle 0s in contingency tables because that happens a lot in cryptography!)
IshanMi 3y ago
I think the 8 trillion parameters is accurate- Tangora is an N-gram model with a vocab size of 20,000 words and N = 3. Parameters for an N-gram model = V^(N-1) * (V-1) Plugging in V=20,000 words and N = 3 for Tangora, you'd get 7.9996E12. Most of the parameters are likely zero or close to it because many 3-grams are possible but not likely to occur. (However the aggregate probability of all 3-grams is substantial and thus they have to be included.)