3 ms·
That went on for a long time before getting to the point. "Trained on gigabytes of input, GPT-2 is uncannily good at producing sensible text by simply predicti
by Animats 7y ago
That went on for a long time before getting to the point.
"Trained on gigabytes of input, GPT-2 is uncannily good at producing sensible text by simply predicting the next word that should follow a given list of input words."
There are now a few systems like that - autocomplete with a big text database. They have no underlying model of what they're talking about. So, after about three sentences, you realize the output makes no sense. It's just words strung together with probabilities that match known text.
This seems to be where machine learning gets stuck. There needs to be some underlying model of the subject matter to have a useful dialog. Outside of well defined subject areas (sports, weather, travel planning, shopping, etc.) that barely exists. Frustrating to see this half a century after Eliza.
Chatbots have the same problem. Either they force you onto a specific track, like a phone tree, or they just natter endlessly without going anywhere. I've been playing around with Rasa lately. This is a chatbot which uses Tensorflow to match user questions with canned answers. That's about all they get from the machine learning part. Outside of that it's a phone tree engine. There's a file of "smalltalk" questions and answers, so it can natter better.
What do we want? Chatbots!
When do we want them? Sorry, I don't understand the question." - bag at chatbot conference.
- thomasahle 7y agoI suppose the NLP idea has been for first RNN and LSTM and now that Transformer and friends to act as a kind of memory. Mostly those memories have been pretty black box however, and it is interesting if ideas from data structures etc. could be useful.
- sombremesa 7y agoIdeas from "data structures" will keep you stuck in the problem GP is talking about. The problem being lamented isn't that we don't know clever tricks for representing the training that a machine has undergone (as well as various tricks for having 'memory'), it's that for there to be a meaningful output for certain domains, there needs to be a thoughtfully implemented model that codes what it is to be (for example) a DM. Said model is, of course, ridiculously complex, so we won't see anyone coding it by hand anytime soon. We'd sooner have ways to copy the relevant bits of a human brain.
- mysterydip 7y agoIsn't this just a Markov chain? I did the same thing with a chatbot years ago, trained on emails or chat logs to sound like a certain person. Are we really making any progress?
- pharke 7y agoI'd like to see someone use a procedurally generated world with a chat bot text description to add flavor to it. The author highlights the problem of procedural worlds in the article as being endless expanses of bland repetition but what if that just served as the armature onto which you add the bizarre imaginings of GPT-2. Use the procedural world to seed chat bot and then use the chat bot output to set the description and parameters of a simulation.
- omarhaneef 7y agoI know what you mean. You can get a decent sentence or three but coherent paragraphs, or sets of paragraphs are an issue. Here is one idea: Two sets of predictors. One trains on outlines. The other trains on translating outlines to full text. Then you generate an outline with one and full text with other. The thing that really excites me about this model is that it might be how humans work too.
- blackearl 7y agoI think even humans can still have the same problem. It's the difference between actually knowing subject matter and bullshitting. There are some incredible bullshitters out there and they can be very successful. That BS gap for chatbots is getting smaller, but the ability to actually learn something and apply that knowledge abstractly seems to be as wide as its ever been. Sure you have these bots winning Go and SC2 tournies, but good luck asking it something like "How would you improve those games that you're programmed to play?"