13 ms·

Blog post: https://ai.facebook.com/blog/state-of-the-art-open-source-chatbot/ https://ai.facebook.com/blog/state-of-the-art-open-source-ch... Paper: https://ar

Blog post: https://ai.facebook.com/blog/state-of-the-art-open-source-chatbot/ https://ai.facebook.com/blog/state-of-the-art-open-source-ch... Paper: https://arxiv.org/pdf/2004.13637.pdf https://arxiv.org/pdf/2004.13637.pdf Open Source: https://parl.ai/projects/recipes/ https://parl.ai/projects/recipes/ Ask us anything, the Facebook team behind it is happy to answer questions here.

mindfulmonkey 6y ago
How do you feel about this probably being most used by bad actors trying to screw with society and elections?
- pesenti 6y ago
  That's definitely a fair concern. We believe that open science and transparency are the right approach here. By releasing it, we ensure that everyone is on the same page with respect to abilities and defense.
  - robmccoll 6y ago
    > By releasing it, we ensure that everyone is on the same page with respect to abilities and defense. This seems like wishful thinking. Having knowledge and having the resources to do something with it are two very different things.
    - phreeza 6y ago
      Defending against such an "attack" is much easier if the technology is widely available and many people can play around with it and explore the limits.
      - ori_b 6y ago
        So is crafting such an attack. Given that the attacks are obvious, and I haven't seen much word on defenses, the result seems inevitable.
        colordrops 6y ago
        At some point you have to let the rest of society in on it. As technology advances, if you keep the power away from the public, eventually you will split into two civilizations a vast divide creating an asymmetry that will be exploited by those on the elite side of the divide. It must be an aspect of technological rollout to figure out how to keep its usage safe once it is widespread.
      - rumanator 6y ago
        Performing the attack is only a possibility because the tech was made available.
        ceres 6y ago
        You are assuming that such tech is not already being (or has been) developed covertly by malicious actors. Developing this and making it open source brings more awareness about the subject and will make it easier to develop defense models against such bots (whether already in existence or that will be developed in future).
        rumanator 6y ago
        > You are assuming that such tech is not already being (or has been) developed covertly by malicious actors. I fail to see where playing out the "but others are doing it too" card exempts the responsibility of those who either lower or eliminate the barrier to entry to these attacks.
  - dkural 6y ago
    So what's the defense?
    - phs318u 6y ago
      If it comes from Facebook, assign it zero credibility?
    - serf 6y ago
      that's the rub. every AI sound/word/picture editor i've ran into says something along the lines of "we're releasing this data set to help stay secure in this day and age of easy counterfeiting of X.", but they never really mention how you apply the data in an adversarial way against itself -- they just sort of hand-wave that part. Same with fake AI generated Obama video and sound, and earlier data-set generated chatbots; it's plastered all over the projects things like "Since these methods are available we think that it's important that this data is disseminated so that other's can use it to validate real world data sources", but again -- how? We have the real data, we have the fake data -- how is this diff done, exactly? I'm willing to bet it isn't as easy as all the AI researchers who release this stuff claim it may be.
      - jdsully 6y ago
        If its secret or not publicly available people will argue using Occam’s razor or that only “State actors” could use this. With the subtext being your not important enough. With the data public its more akin to driveby ssh login attempts. Not being important doesn’t mean your not under attack and people can take the necessary precautions.
        dralley 6y ago
        That's a bit like saying that nuclear secrets should be made public so that people can "take precautions" because "anyone can have a nuclear weapon, not just state actors". There are few reasonable ways to "take precautions" against nuclear weapons and there are few reasonable ways to "take precautions" against something like this short of swearing off of social media entirely. Without reasonable defences, all you really accomplish is ramping up proliferation.
  - m1sta_ 6y ago
    Everyone =/= everyone unfortunately. I agree with your approach though.
  - toper-centage 6y ago
    Sounds like the arguments made by pro gun people.
- jerkstate 6y ago
  On the other hand, what if this technology was used to influence society and elections in a good way?
  - selestify 6y ago
    One of the problem is, who gets to define what “good” is?
  - rumanator 6y ago
    Do you believe anyone is in a position to dictate the outcome of an election without the people having a say on the matter?
- meritt 6y ago
  It's already happening across social media, just with software a whole lot more advanced than this.
klmadfejno 6y ago
How consistent is the bot? Will it answer the same question phrased differently with the same opinion/fact/feeling?
- ekdinan 6y ago
  One of the tasks that we fine-tuned the model on is ConvAI2 (or "Persona-chat") which specifically aims to improve the model's consistency by conditioning its responses on a given persona. See here: https://arxiv.org/abs/1801.07243 https://arxiv.org/abs/1801.07243. In this research we found that conditioning on a persona improves consistency, but that the models still aren't always perfectly consistent. In particular, the model can only see a few turns back in the dialogue history, and may contradict itself if the conversation is long enough that the model simply does not see the full conversation. As far as facts go, we also fine-tuned the model on the Wizard of Wikipedia task (https://arxiv.org/abs/1811.01241 https://arxiv.org/abs/1811.01241), which helps improve its knowledgeability. Again, it still isn't perfect.
Cakez0r 6y ago
I was really blown away by the results you achieved. Amazing work! My jaw hit the floor when I saw the witty farewell "fun guy" quip, and I was in stitches when I read the song about baking. I look forward to the day I can take the model for a spin - unfortunately I don't have the requisite $18,000 hardware ;) I have a few questions: Could this be used as a tool to get a feel for public sentiment? For example, could you ask the bot what it thinks about gun control and have it spit out a policy that appeals to the common public? If you ask the bot what it thinks about how a company will perform, how accurately does it predict? I know that the model will contain the biases of the data set, but I'm curious if you've run these types of experiments. What do you think the results would be if you had an even bigger, more diverse corpus? (devil's advocate, for the sake of discussion: perhaps everyone's fb messenger and WhatsApp chat history) Finally, you have clearly gone to great lengths to make the bot pleasant to interact with. What sort of results to you get when you train such a huge model on an uncurated corpus and don't try to tweak its personality? I find myself wishing that you didn't try to do this as the bot seems to be hyper-agreeable. I. E Too many responses like "You like watching paint dry? That's super interesting! I love watching paint dry!".
- stephenroller 6y ago
  I would not encourage using the model for anything other than AI research -- we're still in the early days of dialogue, and there are a lot of unexplored avenues. There are still nuances around safety, controlling generation, consistency, and knowledge involvement. For instance, the bot cannot remember what you said even a few turns ago, due to limitations in memory size. In the paper, we did explore what happens when you do NOT fine tune it on the specialized tasks (knowledge, empathy and personality). The non-finetuned bot was both less engaging and more toxic. The special finetuning is really important to getting this bot to be as high quality as it is.
  - Cakez0r 6y ago
    But toxicity and quality is subjective. The technical achievement is undeniably brilliant, but the quality of the personality is subject to opinion - as I mentioned, I did not personally enjoy the agreeability of the bot. What's toxic today may not be toxic tomorrow and vice versa. It's just a matter of time before a model of this size can be run on commodity hardware and somebody will take the brakes off and/or attempt to run experiments that aren't just "can this thing pass the turing test?". I'd be really interested to know the thoughts of the team, given their expert knowledge and experience with the matter.
AndrewKemendo 6y ago
Would FB possibly put some instances up for people to chat with? It's way too heavy/expensive to host as an individual.
- pesenti 6y ago
  We are evaluating the possibility. But this is really for research only. To go beyond that we need further improvements to the safety layer.
  - aerovistae 6y ago
    What does safety mean in this context? (I'm not an AI researcher.)
    - hutzlibu 6y ago
      Maybe that the bot does not start denying the holocaust, like the one from microsoft. https://en.m.wikipedia.org/wiki/Tay_(bot) https://en.m.wikipedia.org/wiki/Tay_(bot) (but I am not sure if this bot does indeed learn from new conversations)
ma2rten 6y ago
For most the evaluations you reported at engagingness (expect Figure 16). Did you also look humanness? I would be especially interested how human your chatbot is compared to real humans (Figure 17). This would be similar to a turning test.
- pesenti 6y ago
  The bot is really optimized on engagingness rather than humanness - in particular this is how we chose the hyper parameters. We did evaluate Meena vs. Human (31 vs. 69) and Meena vs. BlenderBot (35 vs. 65). But didn't do BlenderBot vs. Human. Good suggestion though.
ebg13 6y ago
> Ask us anything, the Facebook team behind it is happy to answer questions here. Why do you think that Facebook paid for this research?
- hadtodoit 6y ago
  It's not obvious? Facebook has enough of your conversations to fingerprint your writing style. Now they can de-anonimize your reddit account and serve you even more personalized ads.
  - deleted 6y ago
    [deleted]
  - hutzlibu 6y ago
    That sounds like total overkill. To deanonymize a reddit account, it is easier to just use a add/cookie/script on reddit to link it to your facebook account you have open in another tab.
    - colordrops 6y ago
      How exactly would you do that? Reddit doesn't allow random scripts by 3rd parties.
      - hutzlibu 6y ago
        Ok, I don't know about any reddit details, but I just opened reddit.com without a adblocker and took a look in the network tab in the dev tools. I mean, I don't see a plain facebook connection, but I see googleads and amazon connection and don't know what else obscure things. I doubt it would be hard to sneak something in, that just checks whether this is the same browser where you just switched over from the facebook tab (where the onblur event just got captured). But again, I am not an expert in tracking ads, nor reddit, I just know the web and its various data transmitting technices quite good.
    - hadtodoit 6y ago
mx24 6y ago
Anything? Why do you choose to use your incredible talents for the benefit of such a disgusting company?
- pesenti 6y ago
  I have answered similar questions on HN before. See https://news.ycombinator.com/item?id=19036507 https://news.ycombinator.com/item?id=19036507 for example.
dwd 6y ago
Does it respond appropriately when presented with a potential "switcharoo"?
- pesenti 6y ago
  It does work pretty well with changes of topic (often integrating the broader context as long as it fits into a few turns).
tuananh 6y ago
> Ask us anything, the Facebook team behind it is happy to answer questions here. Just use the bot. Put it into action. Let the bot answer HN's questions
PlasticTank 6y ago
How long until you have to shutdown because it starts getting offensive like Tay?
- pesenti 6y ago
  The team put a lot of effort on dialog safety. You can find more info here: https://parl.ai/projects/dialogue_safety/ https://parl.ai/projects/dialogue_safety/ and the paper here: https://arxiv.org/abs/1908.06083 https://arxiv.org/abs/1908.06083. Still, unlike Tay, we purposely did not create a service for it and do not advise creating one. This is for research purpose only and more effort needs to be made on safety before it can me more broadly consumed.

mindfulmonkey 6y ago

How do you feel about this probably being most used by bad actors trying to screw with society and elections?

pesenti 6y ago
That's definitely a fair concern. We believe that open science and transparency are the right approach here. By releasing it, we ensure that everyone is on the same page with respect to abilities and defense.
- robmccoll 6y ago
  > By releasing it, we ensure that everyone is on the same page with respect to abilities and defense. This seems like wishful thinking. Having knowledge and having the resources to do something with it are two very different things.
  - phreeza 6y ago
    Defending against such an "attack" is much easier if the technology is widely available and many people can play around with it and explore the limits.
    - ori_b 6y ago
      So is crafting such an attack. Given that the attacks are obvious, and I haven't seen much word on defenses, the result seems inevitable.
      - colordrops 6y ago
        At some point you have to let the rest of society in on it. As technology advances, if you keep the power away from the public, eventually you will split into two civilizations a vast divide creating an asymmetry that will be exploited by those on the elite side of the divide. It must be an aspect of technological rollout to figure out how to keep its usage safe once it is widespread.
    - rumanator 6y ago
      Performing the attack is only a possibility because the tech was made available.
      - ceres 6y ago
        You are assuming that such tech is not already being (or has been) developed covertly by malicious actors. Developing this and making it open source brings more awareness about the subject and will make it easier to develop defense models against such bots (whether already in existence or that will be developed in future).
        rumanator 6y ago
        > You are assuming that such tech is not already being (or has been) developed covertly by malicious actors. I fail to see where playing out the "but others are doing it too" card exempts the responsibility of those who either lower or eliminate the barrier to entry to these attacks.
- dkural 6y ago
  So what's the defense?
  - phs318u 6y ago
    If it comes from Facebook, assign it zero credibility?
  - serf 6y ago
    that's the rub. every AI sound/word/picture editor i've ran into says something along the lines of "we're releasing this data set to help stay secure in this day and age of easy counterfeiting of X.", but they never really mention how you apply the data in an adversarial way against itself -- they just sort of hand-wave that part. Same with fake AI generated Obama video and sound, and earlier data-set generated chatbots; it's plastered all over the projects things like "Since these methods are available we think that it's important that this data is disseminated so that other's can use it to validate real world data sources", but again -- how? We have the real data, we have the fake data -- how is this diff done, exactly? I'm willing to bet it isn't as easy as all the AI researchers who release this stuff claim it may be.
    - jdsully 6y ago
      If its secret or not publicly available people will argue using Occam’s razor or that only “State actors” could use this. With the subtext being your not important enough. With the data public its more akin to driveby ssh login attempts. Not being important doesn’t mean your not under attack and people can take the necessary precautions.
      - dralley 6y ago
        That's a bit like saying that nuclear secrets should be made public so that people can "take precautions" because "anyone can have a nuclear weapon, not just state actors". There are few reasonable ways to "take precautions" against nuclear weapons and there are few reasonable ways to "take precautions" against something like this short of swearing off of social media entirely. Without reasonable defences, all you really accomplish is ramping up proliferation.
- m1sta_ 6y ago
  Everyone =/= everyone unfortunately. I agree with your approach though.
- toper-centage 6y ago
  Sounds like the arguments made by pro gun people.
jerkstate 6y ago
On the other hand, what if this technology was used to influence society and elections in a good way?
- selestify 6y ago
  One of the problem is, who gets to define what “good” is?
- rumanator 6y ago
  Do you believe anyone is in a position to dictate the outcome of an election without the people having a say on the matter?
meritt 6y ago
It's already happening across social media, just with software a whole lot more advanced than this.

klmadfejno 6y ago

How consistent is the bot? Will it answer the same question phrased differently with the same opinion/fact/feeling?

ekdinan 6y ago
One of the tasks that we fine-tuned the model on is ConvAI2 (or "Persona-chat") which specifically aims to improve the model's consistency by conditioning its responses on a given persona. See here: https://arxiv.org/abs/1801.07243 https://arxiv.org/abs/1801.07243. In this research we found that conditioning on a persona improves consistency, but that the models still aren't always perfectly consistent. In particular, the model can only see a few turns back in the dialogue history, and may contradict itself if the conversation is long enough that the model simply does not see the full conversation. As far as facts go, we also fine-tuned the model on the Wizard of Wikipedia task (https://arxiv.org/abs/1811.01241 https://arxiv.org/abs/1811.01241), which helps improve its knowledgeability. Again, it still isn't perfect.

Cakez0r 6y ago

I was really blown away by the results you achieved. Amazing work! My jaw hit the floor when I saw the witty farewell "fun guy" quip, and I was in stitches when I read the song about baking. I look forward to the day I can take the model for a spin - unfortunately I don't have the requisite $18,000 hardware ;) I have a few questions: Could this be used as a tool to get a feel for public sentiment? For example, could you ask the bot what it thinks about gun control and have it spit out a policy that appeals to the common public? If you ask the bot what it thinks about how a company will perform, how accurately does it predict? I know that the model will contain the biases of the data set, but I'm curious if you've run these types of experiments. What do you think the results would be if you had an even bigger, more diverse corpus? (devil's advocate, for the sake of discussion: perhaps everyone's fb messenger and WhatsApp chat history) Finally, you have clearly gone to great lengths to make the bot pleasant to interact with. What sort of results to you get when you train such a huge model on an uncurated corpus and don't try to tweak its personality? I find myself wishing that you didn't try to do this as the bot seems to be hyper-agreeable. I. E Too many responses like "You like watching paint dry? That's super interesting! I love watching paint dry!".

stephenroller 6y ago
I would not encourage using the model for anything other than AI research -- we're still in the early days of dialogue, and there are a lot of unexplored avenues. There are still nuances around safety, controlling generation, consistency, and knowledge involvement. For instance, the bot cannot remember what you said even a few turns ago, due to limitations in memory size. In the paper, we did explore what happens when you do NOT fine tune it on the specialized tasks (knowledge, empathy and personality). The non-finetuned bot was both less engaging and more toxic. The special finetuning is really important to getting this bot to be as high quality as it is.
- Cakez0r 6y ago
  But toxicity and quality is subjective. The technical achievement is undeniably brilliant, but the quality of the personality is subject to opinion - as I mentioned, I did not personally enjoy the agreeability of the bot. What's toxic today may not be toxic tomorrow and vice versa. It's just a matter of time before a model of this size can be run on commodity hardware and somebody will take the brakes off and/or attempt to run experiments that aren't just "can this thing pass the turing test?". I'd be really interested to know the thoughts of the team, given their expert knowledge and experience with the matter.

AndrewKemendo 6y ago

Would FB possibly put some instances up for people to chat with? It's way too heavy/expensive to host as an individual.

pesenti 6y ago
We are evaluating the possibility. But this is really for research only. To go beyond that we need further improvements to the safety layer.
- aerovistae 6y ago
  What does safety mean in this context? (I'm not an AI researcher.)
  - hutzlibu 6y ago
    Maybe that the bot does not start denying the holocaust, like the one from microsoft. https://en.m.wikipedia.org/wiki/Tay_(bot) https://en.m.wikipedia.org/wiki/Tay_(bot) (but I am not sure if this bot does indeed learn from new conversations)

ma2rten 6y ago

For most the evaluations you reported at engagingness (expect Figure 16). Did you also look humanness? I would be especially interested how human your chatbot is compared to real humans (Figure 17). This would be similar to a turning test.

pesenti 6y ago
The bot is really optimized on engagingness rather than humanness - in particular this is how we chose the hyper parameters. We did evaluate Meena vs. Human (31 vs. 69) and Meena vs. BlenderBot (35 vs. 65). But didn't do BlenderBot vs. Human. Good suggestion though.

ebg13 6y ago

> Ask us anything, the Facebook team behind it is happy to answer questions here. Why do you think that Facebook paid for this research?

hadtodoit 6y ago
It's not obvious? Facebook has enough of your conversations to fingerprint your writing style. Now they can de-anonimize your reddit account and serve you even more personalized ads.
- deleted 6y ago
  [deleted]
- hutzlibu 6y ago
  That sounds like total overkill. To deanonymize a reddit account, it is easier to just use a add/cookie/script on reddit to link it to your facebook account you have open in another tab.
  - colordrops 6y ago
    How exactly would you do that? Reddit doesn't allow random scripts by 3rd parties.
    - hutzlibu 6y ago
      Ok, I don't know about any reddit details, but I just opened reddit.com without a adblocker and took a look in the network tab in the dev tools. I mean, I don't see a plain facebook connection, but I see googleads and amazon connection and don't know what else obscure things. I doubt it would be hard to sneak something in, that just checks whether this is the same browser where you just switched over from the facebook tab (where the onblur event just got captured). But again, I am not an expert in tracking ads, nor reddit, I just know the web and its various data transmitting technices quite good.
  - hadtodoit 6y ago

mx24 6y ago

Anything? Why do you choose to use your incredible talents for the benefit of such a disgusting company?

pesenti 6y ago
I have answered similar questions on HN before. See https://news.ycombinator.com/item?id=19036507 https://news.ycombinator.com/item?id=19036507 for example.

dwd 6y ago

Does it respond appropriately when presented with a potential "switcharoo"?

pesenti 6y ago
It does work pretty well with changes of topic (often integrating the broader context as long as it fits into a few turns).

tuananh 6y ago

> Ask us anything, the Facebook team behind it is happy to answer questions here. Just use the bot. Put it into action. Let the bot answer HN's questions

PlasticTank 6y ago

How long until you have to shutdown because it starts getting offensive like Tay?

pesenti 6y ago
The team put a lot of effort on dialog safety. You can find more info here: https://parl.ai/projects/dialogue_safety/ https://parl.ai/projects/dialogue_safety/ and the paper here: https://arxiv.org/abs/1908.06083 https://arxiv.org/abs/1908.06083. Still, unlike Tay, we purposely did not create a service for it and do not advise creating one. This is for research purpose only and more effort needs to be made on safety before it can me more broadly consumed.