2 ms·

> 4 year olds don't even need a prompt, they'll ramble on and on until they get tired; if you listen in, you may even notice they seem to have a context window

> 4 year olds don't even need a prompt, they'll ramble on and on until they get tired; if you listen in, you may even notice they seem to have a context window that's about 30 seconds worth of tokens wide :) As the parent of a 4 year old, this is spot-on, especially the 30-second context window. My kiddo will often embark upon a sentence with no idea of how to choose anything beyond the next few tokens, and then continue reaching for new stuff, merely to keep the sentence going. I say this as someone with a deep respect for Humans. However, I also think most of us tend to operate in "default mode" (in the sense that David Foster Wallace described in his "What is Water?" commencement speech). Overall, LLMs are a stark reminder to us that without careful and intentional living, we are in danger of being more like LLMs than we'd like.

TeMPOraL 2y ago
> My kiddo will often embark upon a sentence with no idea of how to choose anything beyond the next few tokens, and then continue reaching for new stuff, merely to keep the sentence going. Exactly right. Now, the "context window" observation hit me when I started to think not where the story is going, but where it's already been, and suddenly noticed that the repetitiveness in my daughter's rambling is meaningful. I.e. she'd go on and on saying things like, "and my brother turtle has this, and his brother hedgehog, hedgehog the brother of my brother the turtle wore an apple, and the brother of the turtle which is my brother, turtle is my brother and his brother hedgehog did this and that, etc." - and I realized that this is her way of carrying context forward; anything that wasn't repeated in the past ~30 seconds would never appear again in the story, or would in a changed form ("the hedgehog, the sister of giraffe Betty, ..."). Over time, I could see how she was learning to be more economical with limited working memory, and faster at accessing the long-term memory (initially, recalling some detail would take a lot of visible effort, and take so long it would wipe half the context, leading to a huge shift in the story) - and I can't help but think that having to push hard against the limits of a small context window is important in development of language and intelligence in general. Figuring out different associations, faster associations, more indirect associations. I wonder what the internal drive for this is, too - what made her improve her thinking to produce more interesting stories within the same memory limits, instead of getting forever stuck sounding like a Markov chain trained on the Book of Chronicles[0]. Some positive response to novelty, perhaps? -- [0] - https://www.biblegateway.com/passage/?search=1+Chronicles+1&version=NIV https://www.biblegateway.com/passage/?search=1+Chronicles+1&...
- ben_w 2y ago
  > Over time, I could see how she was learning to be more economical with limited working memory, and faster at accessing the long-term memory (initially, recalling some detail would take a lot of visible effort, and take so long it would wipe half the context, leading to a huge shift in the story) - and I can't help but think that having to push hard against the limits of a small context window is important in development of language and intelligence in general. Figuring out different associations, faster associations, more indirect associations. Interesting… I've wondered previously if the reason for childhood amnesia is that we need to learn how to form memories. I didn't consider the obvious implication of that idea, that if we learn how to form memories, we probably also need to learn how to access those memories.

TeMPOraL 2y ago

> My kiddo will often embark upon a sentence with no idea of how to choose anything beyond the next few tokens, and then continue reaching for new stuff, merely to keep the sentence going. Exactly right. Now, the "context window" observation hit me when I started to think not where the story is going, but where it's already been, and suddenly noticed that the repetitiveness in my daughter's rambling is meaningful. I.e. she'd go on and on saying things like, "and my brother turtle has this, and his brother hedgehog, hedgehog the brother of my brother the turtle wore an apple, and the brother of the turtle which is my brother, turtle is my brother and his brother hedgehog did this and that, etc." - and I realized that this is her way of carrying context forward; anything that wasn't repeated in the past ~30 seconds would never appear again in the story, or would in a changed form ("the hedgehog, the sister of giraffe Betty, ..."). Over time, I could see how she was learning to be more economical with limited working memory, and faster at accessing the long-term memory (initially, recalling some detail would take a lot of visible effort, and take so long it would wipe half the context, leading to a huge shift in the story) - and I can't help but think that having to push hard against the limits of a small context window is important in development of language and intelligence in general. Figuring out different associations, faster associations, more indirect associations. I wonder what the internal drive for this is, too - what made her improve her thinking to produce more interesting stories within the same memory limits, instead of getting forever stuck sounding like a Markov chain trained on the Book of Chronicles[0]. Some positive response to novelty, perhaps? -- [0] - https://www.biblegateway.com/passage/?search=1+Chronicles+1&version=NIV https://www.biblegateway.com/passage/?search=1+Chronicles+1&...

ben_w 2y ago
> Over time, I could see how she was learning to be more economical with limited working memory, and faster at accessing the long-term memory (initially, recalling some detail would take a lot of visible effort, and take so long it would wipe half the context, leading to a huge shift in the story) - and I can't help but think that having to push hard against the limits of a small context window is important in development of language and intelligence in general. Figuring out different associations, faster associations, more indirect associations. Interesting… I've wondered previously if the reason for childhood amnesia is that we need to learn how to form memories. I didn't consider the obvious implication of that idea, that if we learn how to form memories, we probably also need to learn how to access those memories.