8 ms·
AI 'breakthrough': neural net has human-like ability to generalize language
- drcwpl 3y agoBy contrast, GPT-4 struggled with the same task, failing, on average, between 42 and 86% of the time, depending on how the researchers presented the task. “It’s not magic, it’s practice,”
- samfriedman 3y agoLink to the actual paper: https://www.nature.com/articles/s41586-023-06668-3 https://www.nature.com/articles/s41586-023-06668-3
- mistrial9 3y agosplendid sales headline - however, human communication is varied and multi-layered. Since there are too many angles to summarize -- suffice it to say that IMHO a healthy skepticism is warranted, along with evidence and test results. link to cited repo: https://github.com/brendenlake/MLC/tree/v1.0.0 https://github.com/brendenlake/MLC/tree/v1.0.0
- tomohelix 3y agoI will take this article with a bag of salt but I think we are very close to solving the human communication problem. Not because the machines are perfect but because the human, our benchmarks, are prone to mistakes and so a "close enough" machine would pass as a human just fine.
- marysnovirgin 3y agoThere is inherently no solution to the human communication problem. The mistake has always been in thinking technology would unite us and improve communication when clearly it's doing the opposite.
- deleted 3y ago[deleted]
- Beijinger 3y agoI agree. But "Nature" is quite a name.
- matt123456789 3y agoAnd desperate to retain their relevance in the face of the onslaught of advancements that open access papers has unleashed.
- schleck8 3y agoSo to retain their relevance you claim they abandon academic standards and sensationalize an openly accessible study that anyone can read and thus dispute the article. Okay.
- robertlagrant 3y agoAnyone can read but most press will read and believe the abstract, and write breathless articles on it.
- dotnet00 3y agoIt isn't exactly unheard of for them to do that. They'll just wipe their hands of it afterwards if it turns out to be bad science by saying that their goal isn't to publish research that's guaranteed correct.
- caddemon 3y agoI don't think it is quite that intentional, but just as human economic systems regularly create bubbles there are blind spots in the academic science collective. People desperately want to get in on LLMs right now.
- jltsiren 3y agoThat's a category error. You don't submit your manuscript to a prestige journal in order to publish it but to take advantage of their PR machinery. The services Nature provides are very good and cost-effective, but you have to meet their standards. As research papers, the papers published in Nature/Science/etc. are rarely that useful. Because the journals target the general scientifically literate audience, the interesting details are usually hidden in supplementary material. And the supplements are often little more than giant info dumps that nobody has paid sufficient attention to. It would be nice to live in a world where such PR services are unnecessary, but that's not the world we are living in. The academia is very competitive, and there is never enough funding and never enough good jobs.
- deleted 3y ago[deleted]
- valine 3y agoFor anyone wondering about the architecture this is a 1.4 million parameter transformer model with both an encoder and decoder. The vocabulary size is comically small, it only understands 8 words. It learns new ideas with very few examples, to the extent you can express new ideas with a vocabulary of eight words.
- Izkata 3y agoMorse code has a vocabulary of 3 words (dot, dash, pause between human words).
- yawnxyz 3y agothose are more like tokens, not words
- IMTDb 3y agoThen English has a vocabulary of 27 words: A -> Z + space
- nerdponx 3y agoIf you tokenize using character n-grams, this isn't totally wrong.
- Izkata 3y agoExactly. There was a post on here a few months ago about training using single characters as tokens instead of words, and it worked really well, being able to create new Shakespeare-like text despite not using human words as tokens. What a (human) word is can be learned by the model instead of encoded in the training set.
- sterlind 3y agoI think a fairer comparison would be Toki Pona, a micro-language with ~120 words. you can express lots of things if you have a great deal of patience, Up-Goer Five style. In logic there's also SKI combinator calculus, which is Turing-complete with three symbols, and unlike the Morse or A-Z examples, but like the Toki Pona example, each symbol has a semantic meaning. If you just want to describe ideas in an abstract realm like sequences of colors, like this paper, it's not surprising you don't need many words.
- deleted 3y ago[deleted]
- vessenes 3y agoI actually read the (first half) of the paper. I can't speak to whether or not Nature should publish this, but I don't find the outcomes in the paper spectacular. In brief, they create an ontology -- in the initial examples, four made up color words, each corresponding to a real color and then three made up 'function' words, representing 'before', 'in the middle of' and 'triple'. They then ask humans and their AI to generate sequences of colors based on short and progressively longer strings of color words and functions. The AI is about as good at this as humans, which is to say 85% successful for instructions longer than three or four words. The headline says that GPT4 is worse at this than MLC. I am doubtful about this claim. I feel a quality prompt engineer could beat 85% with GPT4. The claims in the document are that MLC shows similar error types to humans, and that this backs some sort of theory of mind I don't know anything about. That's as may be. I would be surprised if this becomes a radical new architecture based on what I'm reading. I couldn't find a lot of information as to size of the network used; I suppose if its very small that might be interesting. But this read to me very like an 'insider' paper, its target other academics who care about some of these theories of mind, not people who need to get sentences like 'cax kiki pup wif gep' turned into real color circles right away.
- SirMaster 3y agoIf I am evaluating how intelligent an AI is against a human then IMO it's only fair they both get the same prompt. It should be counted as a fault of the AI if it can't understand the prompt as well as the human, or another AI. That's all part of being a useful entity, how well you understand a prompt and how well you can infer the original meaning from what prompt you have been given.
- famouswaffles 3y ago>I feel a quality prompt engineer could beat 85% with GPT4. I generally don't pull the "you're not prompting it good enough" card unless i have direct experience with the task but this does raise a few flags - "between 42 and 86% of the time, depending on how the researchers presented the task." I would have liked them to show how they presented this task, at least something more than a single throw away line, given how integral it is to the main claim.
- MattPalmer1086 3y agoI am no expert in this field, but this seems an important result, even if on a limited model. I don't think that anyone was very sure if generalisation was actually happening with current architectures. Happy to be corrected by anyone who knows more.
- MattPalmer1086 3y agoInstead of downvoting, maybe someone can explain why this is not important?
- m3kw9 3y agoWhen an important word is quoted, don’t take it seriously
- danielmarkbruce 3y agoThis seems dumb. Sure, an LLM can't learn a new word. An LLM isn't an entire system. You could make a system (which extensively used an LLM) to do this fairly easily.
- valine 3y agoLLMs can absolutely learn new words. Try it with GPT4, you can make up words with definitions and it will start using them in conversation.
- danielmarkbruce 3y agoLearn implies change, but it's stateless. What you are seeing a nice trick from the chatGPT application where they are keeping the history of your conversation (not in it's entirety, but a sort of summarized version), and in a single turn feeding it all back in. In a single turn it can be told to use word x to mean y and generate output, but I don't think that's "learning".
- valine 3y agoIt’s called in-context learning. I wouldn’t call it a trick so much as the most exciting and useful feature of LLMs. A lot of research is being done to expand the context window which would allow you to prime the model with entire books or codebases. GPT4 has a max context window of 32k tokens up from 2k with GPT3.5. Whose to say iteratively shifting the output distribution isn’t a form of learning.
- danielmarkbruce 3y agoIf you want to call in-context learning learning, that's fine. I thought the presented paper was about the model being updated so that it could be used in "fresh contexts", but I might have misunderstood what they meant by "fresh context". I thought they meant next prompt. I think everyone is aware that if you say "I use the word jimbo to mean dog. Write a story about dogs using my way" it will work. https://chat.openai.com/share/617ef2c3-ad3e-4d1f-8e6b-3940f28b1d2c https://chat.openai.com/share/617ef2c3-ad3e-4d1f-8e6b-3940f2...
- matmulbro 3y agocheck out other articles by the same author: https://www.nature.com/search?author=Max+Kozlov https://www.nature.com/search?author=Max+Kozlov
- caddemon 3y agoThis guy is a science journalist, the authors of the source research paper are more relevant if you're trying to infer anything about the material from the authors: https://www.nature.com/articles/s41586-023-06668-3 https://www.nature.com/articles/s41586-023-06668-3 FWIW I think even the original paper is more of a master class in riding an academic bubble than anything of important substance. I'm just not sure what looking at other articles by this journalist is supposed to convey.
- CSMastermind 3y agoThe link is to a news article about a recent paper not the a link to the research itself. The editorialization of this being a breakthrough was done by the reporter not the authors of the paper. I see people here comparing this to GPT-4 but I can't find that section in the actual paper nor in the underlying data. Can someone point me in the right direction?
- pama 3y agoThe fourth sentence of the discussion points to the relevant work in their supplement. Here is the link to the supplement: https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-023-06668-3/MediaObjects/41586_2023_6668_MOESM1_ESM.pdf https://static-content.springer.com/esm/art%3A10.1038%2Fs415...
- jmsuth 3y agosounds lit
- kunley 3y agoThe question is, what kind of neural network was that. I mean, for a start, one of https://en.m.wikipedia.org/wiki/Types_of_artificial_neural_networks https://en.m.wikipedia.org/wiki/Types_of_artificial_neural_n... The article only says "To make the neural net human-like, the authors trained it to reproduce the patterns of errors they observed in humans’ test results.", this is not much. I'd especially like to know the deciding factor why it worked while LLM failed.
- deadly_syn 3y agoI have a theory we can get the same levels of output with a optimized pipeline of various specialozed chain. Output is forced into a ruleset to mimick english grammar. Hpt even seems to say this pattern has a lot of potential