5 ms·
This was the highlight of the article for me on how Claude Shannon gained a deep intuition: Shannon took a youthful interest in cryptanalysis after reading a d
by robwwilliams 3y ago
This was the highlight of the article for me on how Claude Shannon gained a deep intuition:
Shannon took a youthful interest in cryptanalysis after reading a detective story and continued to reference detective stories in later years as he worked out his ideas about information. (In the early 1950s, for instance, he tested his intuitions about probability and communication with a noir story by Raymond Chandler, “Pickup on Noon Street.” He spelled out a sentence, letter by letter, to his wife and research assistant, Betty. After each letter, she guessed what the next would be. Eventually, demonstrating the predictability of communication if you have a large enough sample text, she was correctly guessing three letters at once. A S-M-A-L-L O-B-L-O-N-G R-E-A-D-I-N-G L-A-M-P O-N T-H-E D … E-S-K. Shannon figured that the letters we can predict aren’t information, because they don’t tell us anything new.)
- WalterBright 3y agoIsn't that how Chatgpt works? Picking the next word based on the previous words?
- agumonkey 3y agoIt's more high dimensional and has some attention function to optimize the valuation relationship between tokens.
- AlecSchueler 3y agoYou say it's moreso, but moreso than what? Can we actually make statements about the dimensionality of human based word prediction functions?
- agumonkey 3y agoit seems that Chuck was describing a simple markovian process, which is very linear in nature.
- Jensson 3y agoWho is Chuck?
- agumonkey 3y agoWoops, I meant walter
- AlecSchueler 3y agoBut the comparison was made to Shannon's wife predicting the next letter. The process is linear but the selection function in her mind is something else.
- ta988 3y agoHumans have mental states which are not related to anything that happened in the current discussion. Those include memories l, emotions, fatigue, excitation and so many more. Words are only a tiny part of what makes us humans, but they are great to communicate about our mental states with others.
- AlecSchueler 3y agoThat sounds like the human prediction function would also have a high enough dimensionality to consider all of those things, and more, then.
- canjobear 3y agoIt's exactly next-word prediction, at least before RLHF. The attention function and high-dimensional vectors inside are all implementation details which are optimized towards predicting the next word from the previous words.
- rf15 3y agoYeah, basically. Just with a lot of fancy math behind it. Making it all the funnier that it's so prompt/chat focused, since you can just write matter-of-factly to steer large language models.
- deleted 3y ago[deleted]
- corethree 3y agoIsn't this also part of how a human works?
- Jensson 3y agoHumans has extensive state separate from the previous words. If I ask you "what did you do during the weekend", you will not generate a response only based on those words like ChatGPT would. If you mean that humans writes one word at a time, yes that is true. Humans generate words sequentially. But that isn't a very interesting comparison.
- corethree 3y agoI would generate a response based off of an existing body of knowledge within me. Exactly like chatGPT would. In fact chatGPT is able to generate a accurate and self aware response to that query that is fitting and within context: Over the weekend, I was right here, ready to assist users like you with their queries and requests. As an AI, I don't have personal experiences or weekends in the traditional sense, but I'm always available to help with any questions or tasks you have! How can I assist you today?
- Jensson 3y agoIt generates that response every weekend, since that is a pre-programmed response that it has been taught to answer based on that query. If I ask that to a human they will respond with different things from weekend to weekend, or depending on their mood or if they trust me enough to talk about it or if they did something sensitive that they want to hide. It isn't the same thing.
- AlecSchueler 3y agoWhy would you expect its output to vary when it does the same thing every weekend? Why would they pre-program an answer that it should have no difficulty with out of the box and word it just like it's normally inclined to do?
- teleforce 3y agoYes, and Shannon published an early seminal paper for the random based NLP when deterministic NLP seems to be more intuitive at the time [1] [1]Shannon, Claude E. "Prediction and Entropy of Printed English." Bell System Technical Journal, vol. 30, no. 1, 1951, pp. 50–64. Edit: Thanks to ChatGPT I've managed to recall the paper
- ta988 3y agoBased on the previous words up to severeal tens of thousands of them. And that's the major difference with a system like a markov-chain where this would be impossible to do efficiently at that scale.
- gwern 3y agoStrikes me as possibly cribbed from a recent essay... But if you found that brief discussion interesting, you may be interested in https://www.lesswrong.com/posts/Y7WP47tL9zQwkLTqZ/a-conceptual-precursor-to-today-s-language-machines-shannon https://www.lesswrong.com/posts/Y7WP47tL9zQwkLTqZ/a-conceptu...