3 ms·

I think that is a sort of a reverse scaling fallacy. Given the right resources and environments, many small models can function together in an emergent way. I’v

I think that is a sort of a reverse scaling fallacy. Given the right resources and environments, many small models can function together in an emergent way. I’ve been on the lookout for an SLM version of Conway’s game of life. SLM always reminds me of slime molds, which demonstrate a form of intelligence which is remarkable.

nomel 1mo ago
> Given the right resources and environments, many small models can function together in an emergent way. What is this based on? Every researcher I've heard talk about this says it's exactly not true, as an uncontested rule, because the larger models will more effectively contain the smaller models, and use them together in ways that the connections between the smaller models can't. Remember, even MOE is to save compute/memory, not to help performance/parameter.
- grumbelbart2 1mo ago
  That is my understanding as well. Thousands of monkeys do not equal or surpass a man, intellectually, even if working together. There is some intrinsic super linear scaling in intelligence.
- 0x20cowboy 1mo ago
  Does that mean an African elephant 2.57×10^11 (neurons) is smarter than a human 8.6×10^10 (neurons)?
  - fl4regun 1mo ago
    This is not a good comparison, because the brain doesn't only do "being smart" - it has to do things like innervate muscle and other tissue through the body, elephants will require more neurons given their larger size, just to be able to *walk*
  - nomel 1mo ago
    You're comparing completely different training data, harness overhead, and cost function. You're also comparing brains, which aren't really related to this discussion at all. But, it depends on what you're measuring. By spatial/navigational memory, yes, elephants are far far better. Reasoning, no. It would be interesting to see what an elephant or whale eugenics program could result in, since humans have that pesky (or maybe instrumental?) birth canal problem.
- porridgeraisin 1mo ago
  Yes, this is the understanding, specifically that very large models more effectively contain and combine smaller "models" inside them.

nomel 1mo ago

> Given the right resources and environments, many small models can function together in an emergent way. What is this based on? Every researcher I've heard talk about this says it's exactly not true, as an uncontested rule, because the larger models will more effectively contain the smaller models, and use them together in ways that the connections between the smaller models can't. Remember, even MOE is to save compute/memory, not to help performance/parameter.

grumbelbart2 1mo ago
That is my understanding as well. Thousands of monkeys do not equal or surpass a man, intellectually, even if working together. There is some intrinsic super linear scaling in intelligence.
0x20cowboy 1mo ago
Does that mean an African elephant 2.57×10^11 (neurons) is smarter than a human 8.6×10^10 (neurons)?
- fl4regun 1mo ago
  This is not a good comparison, because the brain doesn't only do "being smart" - it has to do things like innervate muscle and other tissue through the body, elephants will require more neurons given their larger size, just to be able to *walk*
- nomel 1mo ago
  You're comparing completely different training data, harness overhead, and cost function. You're also comparing brains, which aren't really related to this discussion at all. But, it depends on what you're measuring. By spatial/navigational memory, yes, elephants are far far better. Reasoning, no. It would be interesting to see what an elephant or whale eugenics program could result in, since humans have that pesky (or maybe instrumental?) birth canal problem.
porridgeraisin 1mo ago
Yes, this is the understanding, specifically that very large models more effectively contain and combine smaller "models" inside them.