4 ms·

So we build self driving software by creating one big brain that does gradient descent. Would we have been more effective if we built other things like this? 4

by TekMol 5y ago

So we build self driving software by creating one big brain that does gradient descent. Would we have been more effective if we built other things like this? 400,000 people were involved in the Apollo program. Would it have been better if we had just one person with a really big brain? How about the Linux kernel and Wikipedia? And why should I figure out a better algorithm? Shouldn't that be the job of the one guy with the biggest brain who comes up with everything?

IshKebab 5y ago
Uhm it's a lot easier to make a single supercomputer a million times faster than it is to make a single brain a million times cleverer.
- TekMol 5y ago
  What is the difference? Couldn't evolution have settled on bigger brains if they are an advantage? Why all that slow interpersonal communication if it is more efficient to have the combined thought processes inside a single brain?
  - alpaca128 5y ago
    Evolution doesn't care about brains. They're just an accidental byproduct that happens to improve our chances of survival.
  - BenoitP 5y ago
    I think you might be interested in learning about Amdahl's law: https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law Some parts of an algorithm are parallelizable, but only up to a certain limit; and that's where the endgame bottleneck is: the last remaining 'in series' parts and the communication overhead for the parallelized parts. Note that their Dojo system has absolutely massive networking equipment: that's for having a better endgame at Amdahl's law. ---- On another note, one can modify the algorithm itself so that is has more inherent parallelism. For example DNA sequencing: you match short reads, them send them off to be reconciled together. The way to have more parallelism is to generate more matches -by lowering the threshold for matching-. Having more matches means more communication costs, but better parallelism. For gradient descent, one of the tricks is learning in batches of input data: the neural network doesn't learn from the freshest point of view, but it can be done in parallel (also the way to aggregate new knowledge is to do an average of the weights, which must limit the amount of things learned)
  - joshmarlow 5y ago
    > Couldn't evolution have settled on bigger brains if they are an advantage? Biological brains are limited by other factors. Human brain size in particular is limited by factors like needing to fit through a human birth canal without wrecking an upright walking gate ([0]) and being able to be powered using a hunter-gather diet. [0] - loosing the upright walking/running gate would have limited certain ecological options (like persistence hunting). Instead, human gestation is shorter than it should be given our body size when you compare it to other mammals.
ben_w 5y ago
Some things can be optimised in parallel, but other things can’t. The example my father gave when I was a teen was: “nine women can’t make one baby in one month”. > And why should I figure out a better algorithm? They said “if”; if you do, you could get very, very rich.

by TekMol 5y ago

IshKebab 5y ago

Uhm it's a lot easier to make a single supercomputer a million times faster than it is to make a single brain a million times cleverer.

TekMol 5y ago
What is the difference? Couldn't evolution have settled on bigger brains if they are an advantage? Why all that slow interpersonal communication if it is more efficient to have the combined thought processes inside a single brain?
- alpaca128 5y ago
  Evolution doesn't care about brains. They're just an accidental byproduct that happens to improve our chances of survival.
- BenoitP 5y ago
  I think you might be interested in learning about Amdahl's law: https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law Some parts of an algorithm are parallelizable, but only up to a certain limit; and that's where the endgame bottleneck is: the last remaining 'in series' parts and the communication overhead for the parallelized parts. Note that their Dojo system has absolutely massive networking equipment: that's for having a better endgame at Amdahl's law. ---- On another note, one can modify the algorithm itself so that is has more inherent parallelism. For example DNA sequencing: you match short reads, them send them off to be reconciled together. The way to have more parallelism is to generate more matches -by lowering the threshold for matching-. Having more matches means more communication costs, but better parallelism. For gradient descent, one of the tricks is learning in batches of input data: the neural network doesn't learn from the freshest point of view, but it can be done in parallel (also the way to aggregate new knowledge is to do an average of the weights, which must limit the amount of things learned)
- joshmarlow 5y ago
  > Couldn't evolution have settled on bigger brains if they are an advantage? Biological brains are limited by other factors. Human brain size in particular is limited by factors like needing to fit through a human birth canal without wrecking an upright walking gate ([0]) and being able to be powered using a hunter-gather diet. [0] - loosing the upright walking/running gate would have limited certain ecological options (like persistence hunting). Instead, human gestation is shorter than it should be given our body size when you compare it to other mammals.

ben_w 5y ago

Some things can be optimised in parallel, but other things can’t. The example my father gave when I was a teen was: “nine women can’t make one baby in one month”. > And why should I figure out a better algorithm? They said “if”; if you do, you could get very, very rich.