4 ms·
Related: "Here's a phenomenon I was surprised to find: you'll go to talks, and hear various words, whose definitions you're not so sure about. At some point yo
by tikkun 3y ago
Related:
"Here's a phenomenon I was surprised to find: you'll go to talks, and hear various words, whose definitions you're not so sure about. At some point you'll be able to make a sentence using those words; you won't know what the words mean, but you'll know the sentence is correct. You'll also be able to ask a question using those words. You still won't know what the words mean, but you'll know the question is interesting, and you'll want to know the answer. Then later on, you'll learn what the words mean more precisely, and your sense of how they fit together will make that learning much easier. The reason for this phenomenon is that mathematics is so rich and infinite that it is impossible to learn it systematically, and if you wait to master one topic before moving on to the next, you'll never get anywhere. Instead, you'll have tendrils of knowledge extending far from your comfort zone. Then you can later backfill from these tendrils, and extend your comfort zone; this is much easier to do than learning "forwards". (Caution: this backfilling is necessary. There can be a temptation to learn lots of fancy words and to use them in fancy sentences without being able to say precisely what you mean. You should feel free to do that, but you should always feel a pang of guilt when you do.)"
Reminds me of the attention mechanism in transformers!
http://math.stanford.edu/~vakil/potentialstudents.html http://math.stanford.edu/~vakil/potentialstudents.html
And for any parents with toddler age children, seeing the way that toddlers relate to language, and that people relate to toddlers about language, leads to lots of fun observations that remind me of LLM related concepts.
- esafak 3y agohttps://en.wikipedia.org/wiki/Distributional_semantics https://en.wikipedia.org/wiki/Distributional_semantics
- tudorw 3y agofascinating, it's how I feel about multidimensional topological manifolds, I understand them, I cannot math them... interacting frequently with an AI is changing me, it's encouraging me to approach task in a more structured way, use concise and accurate language, consider the context and apparent vagueness of communication in a human way, with its inflections, facial expressions and demeanour. One cannot use technology without being affected, in the same way most of us no longer train a part of our brain to remember phone numbers, this will change us.
- _puk 3y agoThanks for this! I have made a point over the years of hanging out with people that are far more intelligent and talented than myself, many of whom are in completely different fields to myself.. and I realise that I've always done this! Whether it's art, music, or the future of power generation, I've been able to hold many conversations that have an aha moment halfway through, where some nugget clicks and backfills the conversation to that point. And yes, I feel a pang of guilt when entertaining these conversations, but I've made solid friends off of a number of these interactions, so I figure I can't be a completely unbearable bore! Or maybe I'm a bot.
- jameshart 3y agoThe important part is in the parens at the end of course: > There can be a temptation to learn lots of fancy words and to use them in fancy sentences without being able to say precisely what you mean. You should feel free to do that, but you should always feel a pang of guilt when you do. GPT - as far as we know - feels no guilt pangs whatsoever.
- jimmygrapes 3y agoSurely there is a confidence score (perhaps hidden) that could be used to emulate such a sense of guilt. Maybe next time.
- s0ulphire 3y ago'Temperature' would probably already be close, which controls how much GPT takes low probability (could reasonably be interpreted as confidence?) into account.