3 ms·

I actually did a full write-up on this here fyi: https://photonlines.substack.com/p/intuitive-and-visual-guide-to-transformers https://photonlines.substack.com/

I actually did a full write-up on this here fyi: https://photonlines.substack.com/p/intuitive-and-visual-guide-to-transformers https://photonlines.substack.com/p/intuitive-and-visual-guid.... You can skip most of this and scroll down to the end-section called 'The Mental Model for Understanding LLMs' where I try to map how transformers are able to mimic human thinking. I think that comparing them to auto-associative / auto-regressive networks is actually a really good analogy FYI and I do believe a lot of human learning is based on this.

solidasparagus 2y ago
An article this long needs a table of contents IMO.
- photon_lines 2y ago
  Will note this and try to add a TOC next time I do this sort of write-up. This article wasn't supposed to be this long - I originally meant to make a more compact version but kept on finding more things which I needed to add in order to map to a correct 'mental model' of what ChatGPT is.
semi-extrinsic 2y ago
We don't have sufficient understanding of human thinking and mental processes to make any claims like the ones you are trying to make. If we did, psychology would be a quantitative hard science; it most definitely is not.
Workaccount2 2y ago
We have no idea how human thinking works and no idea what consciousness is. I'm getting so tired of listening to software engineers LARP pseudo neuroscientists with 6th grade level insights. >Of course, the [AI] brain isn’t ‘conscious.’ It doesn’t have any survival instincts which we humans do. Bruh...
- photon_lines 2y ago
  So instead of shouting insults, maybe you can shine some light on how anything stated there is wrong? Everything that I said there is true - we don't have a good mapping of what consciousness is - if you want to add some insights add them instead of making insulting and useless statements on HN.
  - Workaccount2 2y ago
    We have no idea if ChatGPT is conscious or not. We have no idea if it is "thinking" or not. What we know with a reasonably high level of certainty is that consciousness and "thought" are physical processes. That's about it. Pulling out the scalpel to start dividing up what physical process is and isn't conscious is a fools errand. And especially foolish when just making up arbitrary qualifications for it. Am I saying that ChatGPT is conscious? No. But I am saying is you shouldn't give much credence to people who are anything more than agnostic about it.
nyrikki 2y ago
Human neurons are continuous input, with active dendrites and dendritic compartmentalization. Spikey artificial NNs seem to hit problems with riddled basins so far. A riddled basin is a set with no open subsets. Feed forward networks are effectively DAGs, and circuit like, not TM like. Caution is warranted when comparing perceptrons with biological neurons. Dendrites can perform XOR operations before anything makes it to the soma for another lens. While there is much to learn, here is one highly cited paper on dendritic compartmentalization. https://mcgovern.mit.edu/wp-content/uploads/2019/01/1-s2.0-S0092867418311061-main-1.pdf https://mcgovern.mit.edu/wp-content/uploads/2019/01/1-s2.0-S... I think that the perceptron model of learning plasticity is on pretty shaky ground as being a primary learning model for humans.

solidasparagus 2y ago

An article this long needs a table of contents IMO.

photon_lines 2y ago
Will note this and try to add a TOC next time I do this sort of write-up. This article wasn't supposed to be this long - I originally meant to make a more compact version but kept on finding more things which I needed to add in order to map to a correct 'mental model' of what ChatGPT is.

semi-extrinsic 2y ago

We don't have sufficient understanding of human thinking and mental processes to make any claims like the ones you are trying to make. If we did, psychology would be a quantitative hard science; it most definitely is not.

Workaccount2 2y ago

We have no idea how human thinking works and no idea what consciousness is. I'm getting so tired of listening to software engineers LARP pseudo neuroscientists with 6th grade level insights. >Of course, the [AI] brain isn’t ‘conscious.’ It doesn’t have any survival instincts which we humans do. Bruh...

photon_lines 2y ago
So instead of shouting insults, maybe you can shine some light on how anything stated there is wrong? Everything that I said there is true - we don't have a good mapping of what consciousness is - if you want to add some insights add them instead of making insulting and useless statements on HN.
- Workaccount2 2y ago
  We have no idea if ChatGPT is conscious or not. We have no idea if it is "thinking" or not. What we know with a reasonably high level of certainty is that consciousness and "thought" are physical processes. That's about it. Pulling out the scalpel to start dividing up what physical process is and isn't conscious is a fools errand. And especially foolish when just making up arbitrary qualifications for it. Am I saying that ChatGPT is conscious? No. But I am saying is you shouldn't give much credence to people who are anything more than agnostic about it.

nyrikki 2y ago

Human neurons are continuous input, with active dendrites and dendritic compartmentalization. Spikey artificial NNs seem to hit problems with riddled basins so far. A riddled basin is a set with no open subsets. Feed forward networks are effectively DAGs, and circuit like, not TM like. Caution is warranted when comparing perceptrons with biological neurons. Dendrites can perform XOR operations before anything makes it to the soma for another lens. While there is much to learn, here is one highly cited paper on dendritic compartmentalization. https://mcgovern.mit.edu/wp-content/uploads/2019/01/1-s2.0-S0092867418311061-main-1.pdf https://mcgovern.mit.edu/wp-content/uploads/2019/01/1-s2.0-S... I think that the perceptron model of learning plasticity is on pretty shaky ground as being a primary learning model for humans.