3 ms·
Depends upon the intelligence vs compute scaling law— which I think no one really knows. Pretty likely to be some degree of diminishing returns, but how much?
by mlyle 4mo ago
Depends upon the intelligence vs compute scaling law— which I think no one really knows. Pretty likely to be some degree of diminishing returns, but how much? Is it logarithmic, inverse quadratic, …
If training models gets way cheaper, I would expect the diminishing returns to get steeper too.
- trhway 4mo ago>Pretty likely to be some degree of diminishing returns intelligence may be different. If we look at biological brains - do we get diminishing returns or completely opposite scaling law when we compare our brain against say gorilla's ?
- Vetch 4mo agoInteresting thought to consider in principle but fails because gorilla brains continued to evolve too, just along a different path. They're not snapshots of ancestral species locked in time.
- hedora 4mo agoAlso, it’s definitely diminishing returns, by weight, at least. Architecture / biological structure matters more. I’d expect weight and wattage to be proportional for animals, at least.
- pixl97 4mo agoAnd you're right, no one has any clue what the limits of intelligence are. Though to me it seems odd that humanity has reached the pinnacle of it in the last million years or so after a few billion years of lifes development. Just seems improbable we are close to the limits.
- mlyle 4mo agoI am not making an argument about limits. I just expect some degree of diminishing returns. A related argument is speed of intelligence vs capability at that speed. You can think of a three way trade off between latency, cost, and capability that is unlikely to be linear in any dimension and that changes in steps as technology or biology evolves. Ultimately relating to the properties of the computing substrate and almost certainly bounded by some kind of thermodynamic limits that present systems do not approach.