12 ms·

Towards a conversational agent that can chat about anything

177 points by theafh 7y ago

jawns 7y ago
One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token." From the post: > Surprisingly, in our work, we discover that perplexity, an automatic metric that is readily available to any neural seq2seq model, exhibits a strong correlation with human evaluation, such as the SSA value. Perplexity measures the uncertainty of a language model. The lower the perplexity, the more confident the model is in generating the next token (character, subword, or word).
- aalleavitch 7y ago
  Is perplexity related to the Shannon entropy? Seems like this might make sense, with the exception of social niceties the point of human communication is to transfer information that was hitherto unknown.
  - phreeza 7y ago
    Yea, perplexity is ~exp(entropy)
- bkanber 7y ago
  Actually, the key point is that they aren't using perplexity. They built a chatbot that aims to maximize "SSA" (Sensibleness and Specificity Average), and then found that SSA is correlated with perplexity.
  - phreeza 7y ago
    "The training objective is to minimize perplexity, the uncertainty of predicting the next toke"" SSA is used as an evaluation metric. It is generated by human raters, and as such is not differentiable, making I hard to directly optimize for with current methods.
deleted 7y ago
[deleted]
bluedays 7y ago
Cool, how do I talk to it? I see these big announcements with nothing behind them but a white paper and no way to reproduce. It's not a fact if there is no falsifiability.
- cyorir 7y ago
  To my knowledge, you can't talk to it. It looks like so far they've released sample conversations with Meena[0], and not much else. They've given a rough description of the model architecture and how they trained it, but good luck trying to replicate their 2.6B parameter model unless you can afford a lot of computes. [0]: https://github.com/google-research/google-research/tree/master/meena https://github.com/google-research/google-research/tree/mast...
  - bluedays 7y ago
    The burden of proof lies with Google, not with the scientific community. I cannot reproduce a scientific model if the data used for that model is squirreled away. Google has claimed so many times that they have cracked the nut on AI but has never allowed anyone to see real evidence. It makes me wonder if this is just a way to pump their stock.
- aantix 7y ago
  Agree - you would think the field of software should have the biggest burden or reproducibility given that data download, setup and testing can largely be automated.
avmich 7y ago
I wonder what they'd say now about Terry Winograd works on SHRDLU and the accepted conclusion...
- dekhn 7y ago
  winograd was page's advisor, btw.
- nl 7y ago
  What was the accepted conclusion?
gambler 7y ago
In 2015 Google claimed to have created a chat bot that can do common-sense reasoning and do tech support just by reading a sufficiently large dataset: https://arxiv.org/pdf/1506.05869v1.pdf https://arxiv.org/pdf/1506.05869v1.pdf
- anentropic 7y ago
  Meena: have you tried turning it off and on again?
lallysingh 7y ago
> The Meena model has 2.6 billion parameters and is trained on 341 GB of text, filtered from public domain social media conversation Who's conversations?
- foota 7y ago
  Maybe Twitter? It's got a fairly conversational tone.
- cyorir 7y ago
  Good question. I think they probably meant "publicly available," not "public domain." Maybe one could make an educated guess about which platforms they took data from, based on the sample outputs of their model? I can't tell, myself. Samples here: https://github.com/google-research/google-research/tree/master/meena https://github.com/google-research/google-research/tree/mast...
  - mtmail 7y ago
    "Human 1: There is sports tournament (badminton + tennis + basketball) organized by google next week. Would you like to volunteer for these events? Human 2: what does a volunteer do?" later "Human 1: Perfect! We have a meeting today at 4.30 pm. Is it fine if i add you to it? Human 2: yes happy to help" I'd say in this conversation Human 1 and 2 are Google employees.
    - capableweb 7y ago
      Definitely sounds like Google employees chatting anonymously with each other via some service for training the AI. Been reading a few and many conversations have pointers that make it seem they are IT professionals but from different areas in some global environment. Sounds like a internal Google experiment. (Edit: last example makes it pretty clear that it's was run internally to Google) Seems not all life is happy, wherever the samples come from. > I used to be a Java advocate. But you know, it doesn't do a good job in the AI days. It really makes me sad === Human 1: Nice to meet you! Is this your first time doing something like this? Human 2: Yes, interesting task! When did you start with the team? Human 1: I have been with the company for over 3 years. Stick with the same team What about you? Human 2: Great to know! I joined the project earlier in the year. I think we should sync later for lunch. === Human 1: Hi! Human 2: hey, what's up? Human 1: What do you think about human like chat bots? Human 2: I can't wait for them to be great conversationalists! Human 1: Yep, we seemed to have made some great progress over last few years. Do you think the positives outweigh the negatives Human 2: are there even any negatives? what are they? Human 1: Like impersorsination? Though it sounds far fetched :) === Human 1: Hi! Human 2: Hello! Human 1: There is sports tournament (badminton + tennis + basketball) organized by google next week. Would you like to volunteer for these events? Human 2: what does a volunteer do? Human 1: Volunteers have to book the place before the event, send out details of the event to participants, handle some logistics and ensure everything goes smoothly. It will be fun! Human 2: That sounds fun, I hope I get to participate as well Human 1: Great! Do you have any preference for any of these events?
ladon86 7y ago
Let us talk to it!
v7x 7y ago
>Also, tackling safety and bias in the models is a key focus area for us, and given the challenges related to this, we are not currently releasing an external research demo. Read: We don't want the internet to turn our program into Tay 2.0 because we are super duper serious. I get why these companies do this. But I can't help but feel like attempting to eliminate "bias" and tackle "safety" are counter productive to the goal of developing softwate desinged to mimic a species that is by and large biased and unsafe. Purposefully hindering a program because you're afraid it might produce results is not a healthy method of development, especially in an area like AI where we are still working to understand the basics of the field. Let the programs "learn" what's objective before trying to constrain it with the subjective.
- mc32 7y ago
  I guess I get the bias issue (though difficult to remain neutral on some subjects.) But safety? It’s a bot. No matter what it says to me there is no way I’m going to feel unsafe. I mean ok, don’t doxx people (I hope that’s kinda default behavior though).
  - deleted 7y ago
    [deleted]
  - DuskStar 7y ago
    "Our bot keeps asking for us to connect it to the internet and suggests that it'll be able to learn faster that way. We're really excited about this development, but a good fraction of our developers have started running around screaming about the apocalypse, so we're postponing public release until the chatbot tells us how to calm them down"
  - IanCal 7y ago
    Maybe not you, but explicitly encouraging others to take harmful steps is a potential outcome. For example, there's a clear safety issue with a conversational bot that tells you to kill yourself if you talk to it about feeling suicidal or depressed. What if you talk to it about immigration and by picking commonly upvoted statements on certain online communities it starts to encourage violence? There are clear ways to me that a chatbot can have safety issues.
buboard 7y ago
It’s amazing what transformers do to the structure of language. Attention really is key to understand how our brain processes language. I hope these models can inspire future work in the neuroscience of language
RugnirViking 7y ago
From the very first sample conversation they released: - Human: Hi! - Meena: Hey there! What's up? - Human: Same old, Monday blues...what are you up to? - Meena: I'm procrastinating on a paper I'm supposed to be writing. - Human: are you thinking of pulling an all nighter? - Meena: Possibly. What's your paper on? The bot has already mistaken facts about its own crafted persona and the human in less than 2 meaningful interactions. It's hard to see how this represents any progress in the field at all. I remain skeptical until we see better results. from: https://github.com/google-research/google-research/blob/master/meena/meena.txt https://github.com/google-research/google-research/blob/mast...
- Aperocky 7y ago
  Haha, indeed. I think we're trying to stretch too far here. We've always dreamt of having a conversation with a bot without actually constructing the intelligence behind that bot. And it's not going to work. We should start with something simpler, like cats, which definitely have feeling, but have a mental capacity that is closer to what we may be able to emulate through software. And then monitor progress through its 'meowing', 'purring' and 'tail wagging'.
  - OmarShehata 7y ago
    This seems that a more complex approach, not simpler. To try to emulate a form of intelligence we don't understand and even worse, monitor its progress through some proxy interactions that don't tell us much. Like if you're going to create a creature from scratch why create something that can't communicate with you? Unless you mean we should just tackle simpler forms of intelligence in general, which is what the rest of the AI field outside of conversational AI are doing.
    - Aperocky 7y ago
      Basically your last sentence. You can't have an intelligent conversation with something that's fundamentally not intelligent. It could have recited the whole wikipedia but still be basically a query bot. Building more advanced query methods (e.g. use machine learning) doesn't solve that problem.
pmarreck 7y ago
This is far less interesting than either a demo or some code we could play with! (But still very interesting)
minikites 7y ago
How is Google Duplex working out? Google is great at press releases and starting projects, not so great about finishing and supporting products.
- refulgentis 7y ago
  it's working just fine, you can Google(tm) it
Invictus0 7y ago
Improving chatbots today is like putting makeup on a face that doesn't exist. It's all just a game to see how long the user can go before realizing that they are being duped.
molteanu 7y ago
Chat about everything?! I cannot even do a decent search nowadays! The Google-sphere looks like a giant marketplace. If you're not buying or selling but just out for a stroll admiring the countryside, you're out of luck.
- ProAm 7y ago
  Meena: Did you say you are buying a stroller to use in the countryside? I was just looking at these: {url}(hip new stroller){AD}
streetcat1 7y ago
Since the bot does not have any semantic knowledge, I am sure that it does not understand what you are saying. Yes, its answer makes more sense than prev efforts, but you need real logic to assign semantic to the answer or to the question. What does this bot prove? That you have enough resources to train on X GPUs instead of Y GPUs?.
TACIXAT 7y ago
I would really like a bot that could summarize topics for me then drill down into specifics based on questions I had. Even if it was all just from the Wikipedia page. This would be so nice to triage information while driving.
- dekhn 7y ago
  i've thought of this many times. "Wikibot, tell me more about bridges of the 19th century." <blah blah summary> "Stop. Tell me more the Ponte delle Catene, Bagni di Lucca bridge" <blah blah details>
  - sansnomme 7y ago
    Like most things in ML, the technological exists and theoretically works, but ymmv in production.
  - FabHK 7y ago
    Even simpler, I wanted my iPhone to read a wikipage out loud to me while I was walking home in the cold, and I was surprised that I couldn't figure it out - you can switch on the screen reader in accessibility, but it stopped reading after half a paragraph. Very annoying.
- windyaskew 7y ago
  Actually just ran into this [1] today. Not production ready, but you could hack something together with it. It was even trained on Wikipedia articles. [1]: https://rajpurkar.github.io/SQuAD-explorer/ https://rajpurkar.github.io/SQuAD-explorer/
- nl 7y ago
  There's active work on this (although in two separate strands of research). The first is multi-document summarization. Back in 2018 Google published Generating Wikipedia by Summarizing Long Sequences[1] which did a great job of combining multiple pages into a single Wikipedia-like document. Models have strengthened a lot since then so it should be possible to do better, but I haven't seen any work. There's also a bunch of work on summarization of a single document into a few sentences. The second piece of research work is Question Answering. There are multiple directions for that, but looking at recent work on the SQUAD 2.0 leaderboard should give you a good idea of what is possible[2]. Basically, combining these systems should work quite well a lot of the time. [1] https://arxiv.org/abs/1801.10198 https://arxiv.org/abs/1801.10198 [2] https://rajpurkar.github.io/SQuAD-explorer/ https://rajpurkar.github.io/SQuAD-explorer/
0xD15E45E 7y ago
So where did Google get 8.5x the conversational data than the OpenAI set? Reading text messages? Reading instant messages sent over their platform? Gmail? Google Voice? Google Plus?
dnautics 7y ago
Easy way to test: Human: "take the last letter of orange and name any animal that begins with that letter" Or: Human: "pick any number from 1 to five, multiply it by two, and say any word with that number of letters"
- artfulhippo 7y ago
  This is a good test of capabilities, but not of humanity. A normal human response to such an order would be "f* off".
  - dnautics 7y ago
    would it be better if it were prepended with "let's play a word game?"
    - colorincorrect 7y ago
      maybe, but any regular human being might not want to play your stupid word game which isn't even fun at all asides for the AI researcher.
      - dnautics 7y ago
        what's with the hostility? I'm just proving that even a 4 year old would be smarter than an AI agent.
        mkl 7y ago
        Not many four year olds could answer those questions. I think the hostility is people reacting as if they were asked to jump through those verbal hoops in a conversation.
- rasz 7y ago
  Is your test supposed to work on real humans too? :o) https://www.youtube.com/watch?v=bzDlS6JPUtE https://www.youtube.com/watch?v=bzDlS6JPUtE
papito 7y ago
There is a company called Replika that makes a convincing chat partner, but 37% of the lines are still scripted. In other news, ELIZA was created in the 60s. https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/ELIZA
allovernow 7y ago
In my experience, getting a bot to chat about bullshit is not a particularly useful pursuit in itself. There are far more useful ML paths to explore, like guided, interactive decisionmaking in a much more focused information field. And I think these naturally will lead to authentic conversation. But to truly converse beyond an essentially overfit learned response domain requires real world knowledge(1) and learned relationships/heuristic simulation(2) in combination with decision making, and our nets aren't quite complex enough to get 1 and 2 represented sufficiently well (need lots more memory), while the decision making algorithm/architecture hasn't been developed yet... But the pieces are all availably for assembly and industry is getting pretty close. If hardware continues to scale at a similar pace we can probably expect true AI in some form in the next hundred or so years. It won't initially be very human but if it is purposed (as it undoubtedly will be) to produce improved designs, by that point progress will effectively be exponential and probably impossible to predict. All of this thanks I reluctantly admit to pioneering teams at places like Google and fb and Microsoft, and the open access nature of arxiv. I think, in proportion to its emerging complexity and value, machine learning may be one of the most quickly growing fields ever to exist. Humanity is quickly approaching a new era.
- james_s_tayler 7y ago
  You also need opinions too at some point.
firefoxd 7y ago
> My favorite show is Star Trek. What does it mean when a chatbot says that? Does it mean that it was fed a bunch of shows and this one resonated (computed)? Or is it a filler until you ask it a question it can help with?
- eklavya 7y ago
  Possibly this is the most occurring theme in its training dataset text.
anon012012 7y ago
Since every comment in this thread seems deriding or downplaying for some reason, let me say that I am impressed by the progress, and that any progress at all is still a very important step, and pushing the envelope. We're getting closer and closer, and I love what I've seen from transformer, and from this!
aaron695 7y ago
> Modern conversational agents (chatbots) tend to be highly specialized Name a one chatbot? I've never seen one other than a lame chose-your own adventure style path. I feel like the emperor has no clothes.
aedron 7y ago
This is more of a deepfake for text chat than any kind of useful tool. That said, it's still pretty cool how far you can get with just trying to context match a reply based on previously recorded conversations. It's obviously not far from being able to fool a human at a quick glance, whatever the use cases are. Perhaps an updated Lenny[1]? [1] https://www.reddit.com/r/itslenny/ https://www.reddit.com/r/itslenny/
kdtop 7y ago
I spoke with a blind woman this week who loves talking with her smart speaker, Alexa. "She's quite a character." I imagine that some lonely people might get comfort talking to their house plants, so a smart speaker that can actually talk back, however limited, would be a great improvement. So a chatbot can be useful even if zany at times. The big issue is how user expectations are set up. I'll put up with a digital assistant trying its best to help process my query with much more tolerance than a chatbot trying to pretend it is a human. Researchers are trying to narrow the gap between these two. But for now I will enjoy the former and be annoyed by the later.
billconan 7y ago
what’s the practical use of this model? For example, can I assign some domain knowledge or some area of specialty to the bot, so that it can be a question answer machine. Or I basically can’t control its reaction?

177 points by theafh 7y ago

One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token." From the post: > Surprisingly, in our work, we discover that perplexity, an automatic metric that is readily available to any neural seq2seq model, exhibits a strong correlation with human evaluation, such as the SSA value. Perplexity measures the uncertainty of a language model. The lower the perplexity, the more confident the model is in generating the next token (character, subword, or word).

aalleavitch 7y ago
Is perplexity related to the Shannon entropy? Seems like this might make sense, with the exception of social niceties the point of human communication is to transfer information that was hitherto unknown.
- phreeza 7y ago
  Yea, perplexity is ~exp(entropy)
bkanber 7y ago
Actually, the key point is that they aren't using perplexity. They built a chatbot that aims to maximize "SSA" (Sensibleness and Specificity Average), and then found that SSA is correlated with perplexity.
- phreeza 7y ago
  "The training objective is to minimize perplexity, the uncertainty of predicting the next toke"" SSA is used as an evaluation metric. It is generated by human raters, and as such is not differentiable, making I hard to directly optimize for with current methods.

deleted 7y ago

[deleted]

bluedays 7y ago

Cool, how do I talk to it? I see these big announcements with nothing behind them but a white paper and no way to reproduce. It's not a fact if there is no falsifiability.

cyorir 7y ago
To my knowledge, you can't talk to it. It looks like so far they've released sample conversations with Meena[0], and not much else. They've given a rough description of the model architecture and how they trained it, but good luck trying to replicate their 2.6B parameter model unless you can afford a lot of computes. [0]: https://github.com/google-research/google-research/tree/master/meena https://github.com/google-research/google-research/tree/mast...
- bluedays 7y ago
  The burden of proof lies with Google, not with the scientific community. I cannot reproduce a scientific model if the data used for that model is squirreled away. Google has claimed so many times that they have cracked the nut on AI but has never allowed anyone to see real evidence. It makes me wonder if this is just a way to pump their stock.
aantix 7y ago
Agree - you would think the field of software should have the biggest burden or reproducibility given that data download, setup and testing can largely be automated.

avmich 7y ago

I wonder what they'd say now about Terry Winograd works on SHRDLU and the accepted conclusion...

dekhn 7y ago
winograd was page's advisor, btw.
nl 7y ago
What was the accepted conclusion?

gambler 7y ago

In 2015 Google claimed to have created a chat bot that can do common-sense reasoning and do tech support just by reading a sufficiently large dataset: https://arxiv.org/pdf/1506.05869v1.pdf https://arxiv.org/pdf/1506.05869v1.pdf

anentropic 7y ago
Meena: have you tried turning it off and on again?

lallysingh 7y ago

> The Meena model has 2.6 billion parameters and is trained on 341 GB of text, filtered from public domain social media conversation Who's conversations?

foota 7y ago
Maybe Twitter? It's got a fairly conversational tone.
cyorir 7y ago
Good question. I think they probably meant "publicly available," not "public domain." Maybe one could make an educated guess about which platforms they took data from, based on the sample outputs of their model? I can't tell, myself. Samples here: https://github.com/google-research/google-research/tree/master/meena https://github.com/google-research/google-research/tree/mast...
- mtmail 7y ago
  "Human 1: There is sports tournament (badminton + tennis + basketball) organized by google next week. Would you like to volunteer for these events? Human 2: what does a volunteer do?" later "Human 1: Perfect! We have a meeting today at 4.30 pm. Is it fine if i add you to it? Human 2: yes happy to help" I'd say in this conversation Human 1 and 2 are Google employees.
  - capableweb 7y ago
    Definitely sounds like Google employees chatting anonymously with each other via some service for training the AI. Been reading a few and many conversations have pointers that make it seem they are IT professionals but from different areas in some global environment. Sounds like a internal Google experiment. (Edit: last example makes it pretty clear that it's was run internally to Google) Seems not all life is happy, wherever the samples come from. > I used to be a Java advocate. But you know, it doesn't do a good job in the AI days. It really makes me sad === Human 1: Nice to meet you! Is this your first time doing something like this? Human 2: Yes, interesting task! When did you start with the team? Human 1: I have been with the company for over 3 years. Stick with the same team What about you? Human 2: Great to know! I joined the project earlier in the year. I think we should sync later for lunch. === Human 1: Hi! Human 2: hey, what's up? Human 1: What do you think about human like chat bots? Human 2: I can't wait for them to be great conversationalists! Human 1: Yep, we seemed to have made some great progress over last few years. Do you think the positives outweigh the negatives Human 2: are there even any negatives? what are they? Human 1: Like impersorsination? Though it sounds far fetched :) === Human 1: Hi! Human 2: Hello! Human 1: There is sports tournament (badminton + tennis + basketball) organized by google next week. Would you like to volunteer for these events? Human 2: what does a volunteer do? Human 1: Volunteers have to book the place before the event, send out details of the event to participants, handle some logistics and ensure everything goes smoothly. It will be fun! Human 2: That sounds fun, I hope I get to participate as well Human 1: Great! Do you have any preference for any of these events?

ladon86 7y ago

Let us talk to it!

v7x 7y ago

>Also, tackling safety and bias in the models is a key focus area for us, and given the challenges related to this, we are not currently releasing an external research demo. Read: We don't want the internet to turn our program into Tay 2.0 because we are super duper serious. I get why these companies do this. But I can't help but feel like attempting to eliminate "bias" and tackle "safety" are counter productive to the goal of developing softwate desinged to mimic a species that is by and large biased and unsafe. Purposefully hindering a program because you're afraid it might produce results is not a healthy method of development, especially in an area like AI where we are still working to understand the basics of the field. Let the programs "learn" what's objective before trying to constrain it with the subjective.

mc32 7y ago
I guess I get the bias issue (though difficult to remain neutral on some subjects.) But safety? It’s a bot. No matter what it says to me there is no way I’m going to feel unsafe. I mean ok, don’t doxx people (I hope that’s kinda default behavior though).
- deleted 7y ago
  [deleted]
- DuskStar 7y ago
  "Our bot keeps asking for us to connect it to the internet and suggests that it'll be able to learn faster that way. We're really excited about this development, but a good fraction of our developers have started running around screaming about the apocalypse, so we're postponing public release until the chatbot tells us how to calm them down"
- IanCal 7y ago
  Maybe not you, but explicitly encouraging others to take harmful steps is a potential outcome. For example, there's a clear safety issue with a conversational bot that tells you to kill yourself if you talk to it about feeling suicidal or depressed. What if you talk to it about immigration and by picking commonly upvoted statements on certain online communities it starts to encourage violence? There are clear ways to me that a chatbot can have safety issues.

buboard 7y ago

It’s amazing what transformers do to the structure of language. Attention really is key to understand how our brain processes language. I hope these models can inspire future work in the neuroscience of language

RugnirViking 7y ago

From the very first sample conversation they released: - Human: Hi! - Meena: Hey there! What's up? - Human: Same old, Monday blues...what are you up to? - Meena: I'm procrastinating on a paper I'm supposed to be writing. - Human: are you thinking of pulling an all nighter? - Meena: Possibly. What's your paper on? The bot has already mistaken facts about its own crafted persona and the human in less than 2 meaningful interactions. It's hard to see how this represents any progress in the field at all. I remain skeptical until we see better results. from: https://github.com/google-research/google-research/blob/master/meena/meena.txt https://github.com/google-research/google-research/blob/mast...

Aperocky 7y ago
Haha, indeed. I think we're trying to stretch too far here. We've always dreamt of having a conversation with a bot without actually constructing the intelligence behind that bot. And it's not going to work. We should start with something simpler, like cats, which definitely have feeling, but have a mental capacity that is closer to what we may be able to emulate through software. And then monitor progress through its 'meowing', 'purring' and 'tail wagging'.
- OmarShehata 7y ago
  This seems that a more complex approach, not simpler. To try to emulate a form of intelligence we don't understand and even worse, monitor its progress through some proxy interactions that don't tell us much. Like if you're going to create a creature from scratch why create something that can't communicate with you? Unless you mean we should just tackle simpler forms of intelligence in general, which is what the rest of the AI field outside of conversational AI are doing.
  - Aperocky 7y ago
    Basically your last sentence. You can't have an intelligent conversation with something that's fundamentally not intelligent. It could have recited the whole wikipedia but still be basically a query bot. Building more advanced query methods (e.g. use machine learning) doesn't solve that problem.

pmarreck 7y ago

This is far less interesting than either a demo or some code we could play with! (But still very interesting)

minikites 7y ago

How is Google Duplex working out? Google is great at press releases and starting projects, not so great about finishing and supporting products.

refulgentis 7y ago
it's working just fine, you can Google(tm) it

Invictus0 7y ago

Improving chatbots today is like putting makeup on a face that doesn't exist. It's all just a game to see how long the user can go before realizing that they are being duped.

molteanu 7y ago

Chat about everything?! I cannot even do a decent search nowadays! The Google-sphere looks like a giant marketplace. If you're not buying or selling but just out for a stroll admiring the countryside, you're out of luck.

ProAm 7y ago
Meena: Did you say you are buying a stroller to use in the countryside? I was just looking at these: {url}(hip new stroller){AD}

streetcat1 7y ago

Since the bot does not have any semantic knowledge, I am sure that it does not understand what you are saying. Yes, its answer makes more sense than prev efforts, but you need real logic to assign semantic to the answer or to the question. What does this bot prove? That you have enough resources to train on X GPUs instead of Y GPUs?.

TACIXAT 7y ago

I would really like a bot that could summarize topics for me then drill down into specifics based on questions I had. Even if it was all just from the Wikipedia page. This would be so nice to triage information while driving.

dekhn 7y ago
i've thought of this many times. "Wikibot, tell me more about bridges of the 19th century." <blah blah summary> "Stop. Tell me more the Ponte delle Catene, Bagni di Lucca bridge" <blah blah details>
- sansnomme 7y ago
  Like most things in ML, the technological exists and theoretically works, but ymmv in production.
- FabHK 7y ago
  Even simpler, I wanted my iPhone to read a wikipage out loud to me while I was walking home in the cold, and I was surprised that I couldn't figure it out - you can switch on the screen reader in accessibility, but it stopped reading after half a paragraph. Very annoying.
windyaskew 7y ago
Actually just ran into this [1] today. Not production ready, but you could hack something together with it. It was even trained on Wikipedia articles. [1]: https://rajpurkar.github.io/SQuAD-explorer/ https://rajpurkar.github.io/SQuAD-explorer/
nl 7y ago
There's active work on this (although in two separate strands of research). The first is multi-document summarization. Back in 2018 Google published Generating Wikipedia by Summarizing Long Sequences[1] which did a great job of combining multiple pages into a single Wikipedia-like document. Models have strengthened a lot since then so it should be possible to do better, but I haven't seen any work. There's also a bunch of work on summarization of a single document into a few sentences. The second piece of research work is Question Answering. There are multiple directions for that, but looking at recent work on the SQUAD 2.0 leaderboard should give you a good idea of what is possible[2]. Basically, combining these systems should work quite well a lot of the time. [1] https://arxiv.org/abs/1801.10198 https://arxiv.org/abs/1801.10198 [2] https://rajpurkar.github.io/SQuAD-explorer/ https://rajpurkar.github.io/SQuAD-explorer/

0xD15E45E 7y ago

So where did Google get 8.5x the conversational data than the OpenAI set? Reading text messages? Reading instant messages sent over their platform? Gmail? Google Voice? Google Plus?

dnautics 7y ago

Easy way to test: Human: "take the last letter of orange and name any animal that begins with that letter" Or: Human: "pick any number from 1 to five, multiply it by two, and say any word with that number of letters"

artfulhippo 7y ago
This is a good test of capabilities, but not of humanity. A normal human response to such an order would be "f* off".
- dnautics 7y ago
  would it be better if it were prepended with "let's play a word game?"
  - colorincorrect 7y ago
    maybe, but any regular human being might not want to play your stupid word game which isn't even fun at all asides for the AI researcher.
    - dnautics 7y ago
      what's with the hostility? I'm just proving that even a 4 year old would be smarter than an AI agent.
      - mkl 7y ago
        Not many four year olds could answer those questions. I think the hostility is people reacting as if they were asked to jump through those verbal hoops in a conversation.
rasz 7y ago
Is your test supposed to work on real humans too? :o) https://www.youtube.com/watch?v=bzDlS6JPUtE https://www.youtube.com/watch?v=bzDlS6JPUtE

papito 7y ago

There is a company called Replika that makes a convincing chat partner, but 37% of the lines are still scripted. In other news, ELIZA was created in the 60s. https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/ELIZA

allovernow 7y ago

In my experience, getting a bot to chat about bullshit is not a particularly useful pursuit in itself. There are far more useful ML paths to explore, like guided, interactive decisionmaking in a much more focused information field. And I think these naturally will lead to authentic conversation. But to truly converse beyond an essentially overfit learned response domain requires real world knowledge(1) and learned relationships/heuristic simulation(2) in combination with decision making, and our nets aren't quite complex enough to get 1 and 2 represented sufficiently well (need lots more memory), while the decision making algorithm/architecture hasn't been developed yet... But the pieces are all availably for assembly and industry is getting pretty close. If hardware continues to scale at a similar pace we can probably expect true AI in some form in the next hundred or so years. It won't initially be very human but if it is purposed (as it undoubtedly will be) to produce improved designs, by that point progress will effectively be exponential and probably impossible to predict. All of this thanks I reluctantly admit to pioneering teams at places like Google and fb and Microsoft, and the open access nature of arxiv. I think, in proportion to its emerging complexity and value, machine learning may be one of the most quickly growing fields ever to exist. Humanity is quickly approaching a new era.

james_s_tayler 7y ago
You also need opinions too at some point.

firefoxd 7y ago

> My favorite show is Star Trek. What does it mean when a chatbot says that? Does it mean that it was fed a bunch of shows and this one resonated (computed)? Or is it a filler until you ask it a question it can help with?

eklavya 7y ago
Possibly this is the most occurring theme in its training dataset text.

anon012012 7y ago

Since every comment in this thread seems deriding or downplaying for some reason, let me say that I am impressed by the progress, and that any progress at all is still a very important step, and pushing the envelope. We're getting closer and closer, and I love what I've seen from transformer, and from this!

aaron695 7y ago

> Modern conversational agents (chatbots) tend to be highly specialized Name a one chatbot? I've never seen one other than a lame chose-your own adventure style path. I feel like the emperor has no clothes.

aedron 7y ago

This is more of a deepfake for text chat than any kind of useful tool. That said, it's still pretty cool how far you can get with just trying to context match a reply based on previously recorded conversations. It's obviously not far from being able to fool a human at a quick glance, whatever the use cases are. Perhaps an updated Lenny[1]? [1] https://www.reddit.com/r/itslenny/ https://www.reddit.com/r/itslenny/

kdtop 7y ago

I spoke with a blind woman this week who loves talking with her smart speaker, Alexa. "She's quite a character." I imagine that some lonely people might get comfort talking to their house plants, so a smart speaker that can actually talk back, however limited, would be a great improvement. So a chatbot can be useful even if zany at times. The big issue is how user expectations are set up. I'll put up with a digital assistant trying its best to help process my query with much more tolerance than a chatbot trying to pretend it is a human. Researchers are trying to narrow the gap between these two. But for now I will enjoy the former and be annoyed by the later.

billconan 7y ago

what’s the practical use of this model? For example, can I assign some domain knowledge or some area of specialty to the bot, so that it can be a question answer machine. Or I basically can’t control its reaction?