2 ms·

I'll give a very simplistic answer to your question and skirt some detail. Essentially, Transformers can 'look at' everything within the context with perfect fi

by chaxor 3y ago

I'll give a very simplistic answer to your question and skirt some detail. Essentially, Transformers can 'look at' everything within the context with perfect fidelity (only one 'hop' between tokens to 'get the info'), while RNNs struggle with this because the information gets 'muddied' with all of the tokens in between. Like telephone between you and one friend (transformer) vs you and a sea of people (RNN). Hopefully this gets the point across without the mathematics and a precise description.

pico_creator 3y ago
Yea using the analogy of the eyes can see the entire document Vs I need to memorise everything as it’s spoken out, and then answer on it Is a good approximate on the difference. The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training) It’s also why it is recommended by the community to ask the question first, then put your context document. In a document QnA task
- soulofmischief 3y ago
  In the shower today I was thinking about Jungian archetypes as a visual programming language for the subconscious. For example, if someone says, "I should be a better mother", and hold this impression in their mind over time, the associations their brain makes with that image and various behaviors can be used as a guiding force during the physical reorganization which occurs when one creates a new habit, or adjusts a behavior or muscle memory. And these associations depend upon observations made by the individual, which they have either been taught or innately know can be categorized as mother-like. The brain is typically trained to make this association in the case of the mother by mirroring and attaching to the first female who exhibits nursing behavior. These archetypes are a visual representation of deeper mental models which we are primed to carry on between generations. And then I wondered how this might apply to neural models, if some system of archetypes naturally emerges, if they're useful for programming or memorization/compression, or how we might explore imbuing existing models with an archetypical system.

by chaxor 3y ago

pico_creator 3y ago

Yea using the analogy of the eyes can see the entire document Vs I need to memorise everything as it’s spoken out, and then answer on it Is a good approximate on the difference. The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training) It’s also why it is recommended by the community to ask the question first, then put your context document. In a document QnA task

soulofmischief 3y ago
In the shower today I was thinking about Jungian archetypes as a visual programming language for the subconscious. For example, if someone says, "I should be a better mother", and hold this impression in their mind over time, the associations their brain makes with that image and various behaviors can be used as a guiding force during the physical reorganization which occurs when one creates a new habit, or adjusts a behavior or muscle memory. And these associations depend upon observations made by the individual, which they have either been taught or innately know can be categorized as mother-like. The brain is typically trained to make this association in the case of the mother by mirroring and attaching to the first female who exhibits nursing behavior. These archetypes are a visual representation of deeper mental models which we are primed to carry on between generations. And then I wondered how this might apply to neural models, if some system of archetypes naturally emerges, if they're useful for programming or memorization/compression, or how we might explore imbuing existing models with an archetypical system.