5 ms·

I think 0.2 watt is sufficently humiliating to prove that deep learning research inefficiency of large models is indicative of missing a fundamental parcimoniou

I think 0.2 watt is sufficently humiliating to prove that deep learning research inefficiency of large models is indicative of missing a fundamental parcimonious paradigm shift. What is it that the brain do that transcend our algorithms so much? Well the C.elegans research underfunding will not help us answer that question. But hey deepming & ci have their priorities and only the incoming transformers winter will force them to evolve.

TaylorAlexander 4y ago
I’m confident that everyone in the deep learning industry knows our algorithms are horribly inefficient. The canonical example is training inefficiency. They are constantly noting that humans can do one shot learning and our algorithms generally can’t. EDIT: That said there is valid criticism of their focus on very large models at the expense of more efficient algorithm research, so maybe that’s what you were getting at.
- deepsun 4y ago
  Well, most training happened before the brain is even born.
  - SemanticStrengh 4y ago
    DNA is less than a few megabytes
    - deepsun 4y ago
      No it's about 1.5 Gigabytes. Long answer: https://www.quora.com/How-many-bytes-memory-size-is-a-humans-DNA https://www.quora.com/How-many-bytes-memory-size-is-a-humans...
      - SemanticStrengh 4y ago
        that is fallacious considering e.g. non-coding DNA see e.g. https://www.quora.com/How-many-bytes-memory-size-is-a-humans-DNA/answer/Brett-Watters https://www.quora.com/How-many-bytes-memory-size-is-a-humans... the estimate is more like 50MB and the difference between humans of the dozen of MBs. Considering that the brain is just one component of our very complex bodies, that's just a couple of MBs at max allocated to Brain encoding, including data that is needed for constructing the brain structurally, and not encoding a memory or running the runtime. The brain runtime probably has a ~1MB max DNA budget, which is pathetically small. Alhough that is an underesearched topic (there is not even a dedicated wikipedia page), genetic memory is a thing (how do spiders learn to draw the web, some animals know how to walk at birth, to breath, birds innately know how to nest (and some maybe how to fly). Genetic memory is a nice topic and I wonder if we have findings how it manifest (via IRMs, etc) but it's unclear how/where it is encoded given that the DNA budget is very very tight.
    - TaylorAlexander 4y ago
      All the more reason to research algorithmic efficiency. It’s a hell of a compact encoding.
      - SemanticStrengh 4y ago
        and the majority of the encoding is probably built into designing the body of which the brain is only a small part.
paulryanrogers 4y ago
Human brains have hundreds of millions of years of evolution baked in. And each one still takes about 20 years of training and development to get to the point where it can apply logic rationally.
- vidanay 4y ago
  Many of them never can.
  - kaba0 4y ago
    None of us can. At most we can do some very virtualized logic interpretation, but we are all faulty, emotional-driven “machines”, and even behind the most logical people’s actions the reason is simply they felt that way.
    - SemanticStrengh 4y ago
      trained people that have no involvement/conflict of interest in something can perfectly analyse it in a cold and objective manner, with the given provided information. People's actions are necessarilly irational, that's not the same thing as an external analysis/decision making output.
PartiallyTyped 4y ago
For one, unlike DL models, we approximate things, and I do not mean that in the universal approximator sense. What I mean is that most of the things we do are essentially pulling from memory similar situations and stitching everything together. Unless you are John Von Neumann, you are either approximately correct or very slow in solving novel tasks. On the other hand, we have the capacity to generalize much easier than our DL models. I think that that relates to our memory. That is, we compress information in a way that allows us to search for the structure, map to the target domain, and then stitch it all together. Hofstadter argues about it a lot in his books, that is, that we use analogies to learn stuff. In the same vein and in a very handwave-y manner, we can think of analogies as functors across categories. That is, as mappings between one domain to another where the relationship (morphism) between the objects is preserved. For whatever reason we are capable of identifying those mappings quickly and applying them everywhere. This relates to symmetries where we have little to no trouble identifying objects that underwent certain transformations but our DL have trouble with. A user in HN suggested that with respect to the visual transformations, we can do that because our optic nerve travels a long distance, from our eyes to our visual cortex, and that allows our brains to perform all kinds of data cleaning and signal processing. The ideas about compression that I mentioned here is essentially what we saw in deepmind's GATO. More precisely, creating multimodal embeddings in that manner seems to enable good multi-task performance. Sorry if this comes across as rambling.

TaylorAlexander 4y ago

I’m confident that everyone in the deep learning industry knows our algorithms are horribly inefficient. The canonical example is training inefficiency. They are constantly noting that humans can do one shot learning and our algorithms generally can’t. EDIT: That said there is valid criticism of their focus on very large models at the expense of more efficient algorithm research, so maybe that’s what you were getting at.

deepsun 4y ago
Well, most training happened before the brain is even born.
- SemanticStrengh 4y ago
  DNA is less than a few megabytes
  - deepsun 4y ago
    No it's about 1.5 Gigabytes. Long answer: https://www.quora.com/How-many-bytes-memory-size-is-a-humans-DNA https://www.quora.com/How-many-bytes-memory-size-is-a-humans...
    - SemanticStrengh 4y ago
      that is fallacious considering e.g. non-coding DNA see e.g. https://www.quora.com/How-many-bytes-memory-size-is-a-humans-DNA/answer/Brett-Watters https://www.quora.com/How-many-bytes-memory-size-is-a-humans... the estimate is more like 50MB and the difference between humans of the dozen of MBs. Considering that the brain is just one component of our very complex bodies, that's just a couple of MBs at max allocated to Brain encoding, including data that is needed for constructing the brain structurally, and not encoding a memory or running the runtime. The brain runtime probably has a ~1MB max DNA budget, which is pathetically small. Alhough that is an underesearched topic (there is not even a dedicated wikipedia page), genetic memory is a thing (how do spiders learn to draw the web, some animals know how to walk at birth, to breath, birds innately know how to nest (and some maybe how to fly). Genetic memory is a nice topic and I wonder if we have findings how it manifest (via IRMs, etc) but it's unclear how/where it is encoded given that the DNA budget is very very tight.
  - TaylorAlexander 4y ago
    All the more reason to research algorithmic efficiency. It’s a hell of a compact encoding.
    - SemanticStrengh 4y ago
      and the majority of the encoding is probably built into designing the body of which the brain is only a small part.

paulryanrogers 4y ago

Human brains have hundreds of millions of years of evolution baked in. And each one still takes about 20 years of training and development to get to the point where it can apply logic rationally.

vidanay 4y ago
Many of them never can.
- kaba0 4y ago
  None of us can. At most we can do some very virtualized logic interpretation, but we are all faulty, emotional-driven “machines”, and even behind the most logical people’s actions the reason is simply they felt that way.
  - SemanticStrengh 4y ago
    trained people that have no involvement/conflict of interest in something can perfectly analyse it in a cold and objective manner, with the given provided information. People's actions are necessarilly irational, that's not the same thing as an external analysis/decision making output.

PartiallyTyped 4y ago

For one, unlike DL models, we approximate things, and I do not mean that in the universal approximator sense. What I mean is that most of the things we do are essentially pulling from memory similar situations and stitching everything together. Unless you are John Von Neumann, you are either approximately correct or very slow in solving novel tasks. On the other hand, we have the capacity to generalize much easier than our DL models. I think that that relates to our memory. That is, we compress information in a way that allows us to search for the structure, map to the target domain, and then stitch it all together. Hofstadter argues about it a lot in his books, that is, that we use analogies to learn stuff. In the same vein and in a very handwave-y manner, we can think of analogies as functors across categories. That is, as mappings between one domain to another where the relationship (morphism) between the objects is preserved. For whatever reason we are capable of identifying those mappings quickly and applying them everywhere. This relates to symmetries where we have little to no trouble identifying objects that underwent certain transformations but our DL have trouble with. A user in HN suggested that with respect to the visual transformations, we can do that because our optic nerve travels a long distance, from our eyes to our visual cortex, and that allows our brains to perform all kinds of data cleaning and signal processing. The ideas about compression that I mentioned here is essentially what we saw in deepmind's GATO. More precisely, creating multimodal embeddings in that manner seems to enable good multi-task performance. Sorry if this comes across as rambling.