21 ms·

CICERO: An AI agent that negotiates, persuades, and cooperates with people

509 points by tuzongyu 4y ago

curious_cat_163 4y ago
Hm. Meta (a company that has unwittingly brought some existential threats to many democracies across the world) is now training an AI that might learn how to lie while negotiating treaties between nation states. What could possibly go wrong?
tuzongyu 4y ago
https://www.science.org/doi/10.1126/science.ade9097 https://www.science.org/doi/10.1126/science.ade9097 Abstract: Despite much progress in training AI systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce CICERO, the first AI agent to achieve human-level performance in Diplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. CICERO integrates a language model with planning and reinforcement learn- ing algorithms by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, CICERO achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.
- digdugdirk 4y ago
  "... by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans." ... Isn't this the bad path of AI research? An unbeatable and utterly convincing conversational AI that knows exactly what you want to achieve, and then completely nullifies your attempts at reaching your goals while simultaneously achieving it's own?
  - stavros 4y ago
    You've read a lot into that sentence. It literally says: > ... by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. As opposed to what? Not inferring intentions, or generating dialogue against its own plans, or at random? It's just doing what a person does. It understands the other person, and says things to further its goals.
    - mtlmtlmtlmtl 4y ago
      Sure, people do that. But people who only do that are sociopaths.
      - stavros 4y ago
        I'm sure the AI hangs out at the pub with its friends after hours though.
        smaudet 4y ago
        If only, like a sociopath it hangs out with no one...
  - EGreg 4y ago
    Just wait until we have AI swarms. So far all this has been with one player, amid others, no collusion. You’re going to be surrounded very shortly by sleeper bots, including on HN. Relying on dang and others to root out bots will be futile. A swarm could easily collude to downvote people or get them ostrasized by their own friend group, as we have already seen when it came to crypto, metoo, BLM, lockdowns, vaccines and now Ukraine. It’s really not hard for a bot swarm to completely exploit society in these and many more ways, and we are not ready for it. By the time the botswarms arrive online, it’ll be too late to do anything. Update: LinkedIn already has a huge problem of fake profiles applying for jobs and offering jobs. And Twitter is overrun with bots. But soon, bot-written articles will be out-shared by your own friends rather than that “hack liberal/establishment/hasbeen paper” NYTimes.com The animatrix had a good sort of storyline on this, but it involved a lot of unrealistic violence
    - worldsayshi 4y ago
      There are ways to ensure that people are people on the internet though. Not good ways but there are ways. You could for example limit users to those that log in with electronic id's issued by a government or other organisation that you trust to assert that the user is human and then force real names or a single user name for that e-id.
      - EGreg 4y ago
        But when bots do a lot of things better than humans, including comment, answer questions, make jokes, say romantic things, etc. many would start to prefer them to humans in their online communities, and give them even more social capital. This already happened in other areas of life. Both fathers and mothers now neglect their own children and elderly parents so they can work for corporations. They often prefer this and find meaning in climbing the corporate ladder. Eventually, their own labor will be rendered obsolete, but for now they're in a race to the bottom to work harder and neglect their family even more. They even stick them in nursing homes. Also, you no longer want to ask people for directions, you use Google Maps. You no longer ask your parents, teachers or libraries when you can just look it up online with no judgment. Finally, look at industries like Wall Street trading. It used to be a bunch of guys in a pit. You'd call up your broker or whatever. Now everything is automated with bots. Everyone prefers bots. They make up the bulk of trading with real capital. These bots are are working for corporations, which employ less and less humans. So the present is already a bunch of corporations owning bots and bots creating content for other bots. In the finance industry. Now how different is a bunch of text generation online? I think the human contributions will be vanishingly small in most communities. The question is ... what is this all for? Dropping demand for human services is a byproduct of making things more and more efficient...
- bee_rider 4y ago
  > Cicero participated anonymously in 40 games of Diplomacy in a “blitz” league on webDiplomacy.net > Cicero ranked in the top 10% of participants who played more than one game and 2 nd out of 19 participants in the league that played 5 or more games. > As part of the league, Cicero participated in an 8-game tournament involving 21 participants, 6 of whom played at least 5 games. Participants could play a maximum of 6 games with their rank determined by the average of their best 3 games. Cicero placed 1st in this tournament. This bit seems a little more impressive I think. Being in the top 10% of people who’ve played at least two games might leave a lot of bad players to beat up on. Winning a tournament might (?) mean you have to beat at least a couple players who understand the thing. It is sort of funny to think about — anyone who gets really legitimately good at anything competitive goes through multiple rounds of being the best in their social group, and then moving on from that group to a new one that is comprised of people who were the top-tier of that previous level. It isn’t obvious to me where on that informal ladder this tournament was. But anyway, maybe the AI will follow the trajectory of chess AIs and quickly race away from human competition.
  - treis 4y ago
    I'm a little bit suspicious of this. They're not explicit about the scoring but taking the average of top 3 results is a huge advantage to those that played more games. Diplomacy is a bit of a choose your own adventure game too. Like there's an objective criteria (average SCs at the agreed end of game) but the human tendency is to try and win individual games. Humans will often choose to play sub-optimal strategies for better entertainment value. I think the real accomplishment here is the ability to fool humans into thinking their not playing a bot. That's an impressive thing to do even these days.
  - gcanyon 4y ago
    There's an interesting question of "how much do the literal best humans suck at this?" For example, in chess Magnus Carlsen might be able to beat Stockfish given a handicap of just a pawn or two. An even better computer player than Stockfish might give up three or more pawns, but even a perfect player would likely lose to Carlsen if giving up a rook. -- I'm making this up, I don't think anyone knows the real values, but as far as I know no one is remotely projecting that perfect play could overcome e.g. a queen handicap. Similarly, in Go it seems unlikely that perfect play could overcome a nine-stone handicap (again, I could be wrong, I'm not remotely a dan-level player). All to say, it seems likely that Diplomacy is a game where the difference between "the best human play" and "the best possible play" is much larger than either Go or Chess.
fritzschopen 4y ago
After move 37 game 2, I think this is one of the major events in "participative" AI.
erehweb 4y ago
"In a game in which dishonesty is commonplace, it is notable that we were able to achieve human-level performance by controlling the agent's dialogue through the strategic reasoning module to be largely honest and helpful to its speaking partners."
- sdenton4 4y ago
  This is one of the key insights in playing Diplomacy well. You can turn honest opposition in the early game into trust, which is helpful for flipping allegiances in the late game. ('Yes, I've been trying to murder you the entire game, but I've been honest about it, and never lied to you once. Now I see that we've got a real opportunity to dislodge Russia and sweep the eastern half of the board if we work together... What do you say?') Dishonesty means people don't want to work with you, because you are not a dependable partner. So you gotta make sure that any lies you tell pay off /big./
- bpodgursky 4y ago
  Well, yes, "largely" honest, until the critical moment when you stab to take 18 supply centers to win the game. You don't win Diplomacy by staying honest, you win by choosing the right moment to make the move and burn that trust. It's the 5% dishonesty that matters.
  - wolverine876 4y ago
    It's interesting and I think significant that people are uncomfortable with positive values, such as honesty, doing good for others, responsibility to others, right (not wrong), etc. We shoot them down, similar to how people instinctively cut off a conversation going in an uncomfortable direction, often without even realizing it. I can't attribute that the parent - I don't know the individual - but I expected a chorus of such responses. I feel like I can see them squirming in their seats ... But I feel it too. There's societal pressure against doing good. It's often more socially comfortable to throw a cigarette butt on the ground than to pick one up and discard it. That's bizarre, it's perverse incentives, and we can and should change it.
kneebonian 4y ago
For those hearing of this game for the first time and think it sounds fun. Be warned, it is considered to be a game that will end friendships that have lasted for years, and monopoly is considered to be a game that causes less bad feelings. An AI may be able to speak like a person but will never be able to hang onto that long burning simmering hatred from when Brad didn't support my army and instead flipped on me by supporting the f**ing Ottomans instead. I hope you choke on a cheesy pretzel Brad.
- stavros 4y ago
  I only played one game, and this is accurate.
- bee_rider 4y ago
  I have heard this about various games (Risk and Settlers of Catan mostly). I don’t get it. The point of these games is to be a bastard. If everyone goes into it with the understanding that there will be double crosses, how could there be hurt feelings? Monopoly is worse; it is just boring, I would dump my friends if they suggested monopoly not because I was hurt by their ruthless gameplay but for their terminally taste.
  - tullianus 4y ago
    Two features that make Diplomacy particularly bad: 1. Length. An in-person game can last easily 12 hours, and is mentally exhausting. Getting 10 hours in and THEN getting screwed by your friend feels worse than a game with less time investment. 2. Design. It's difficult to survive the opening few hours without alliances, but only one player can win, so everyone is incentivized to defect at exactly the moment they think they can make greater gains by defecting than by cooperating. Being betrayed, even if you survive, forces a total rethink of strategy and ruins the next hour or two of gameplay.
    - legitster 4y ago
      > An in-person game can last easily 12 hours How are you playing it? We've never had a game run over 4 hours.
Hizonner 4y ago
Anybody got a quick way off this planet?
- falcor84 4y ago
  Musk does, but he's also building humanoid robots, so I don't know if he's on our side.
O__________O 4y ago
Possible I missed it, but I don’t see any reference to how they addresses chat thread continuity (aka memory) issues that are major issue for AI chat bots. Did they address this and if so, how?
- tullianus 4y ago
  I think this might be a strength of Diplomacy for current AI models - making contradictory plans with two different players is a perfectly normal human move, as is saying one thing and doing another, as is cooperating on round N and defecting on round N+1.
  - O__________O 4y ago
    Agree that detecting it might be hard and that it might actually as is be beneficial, but that said, there huge difference in my opinion between knowingly breaking promises based on a heuristic and having no memory of past promises and simply basing current tactics on the present. Imagine if real world literally had no memory of the past or a single actor within it, it would lose out to opposing players as soon as they realized it had no memory, you could make any promise you wanted and it would literally forget it. Might be wrong though, just know it’s major issue with current chat bots and an easy tactic to tell if you’re chatting with a bot or not.
- CopOnTheRun 4y ago
  This was mentioned in the "still room for improvement" section. > It is important to recognize that CICERO also sometimes generates inconsistent dialogue that can undermine its objectives. In the example below where CICERO was playing as Austria, the agent contradicts its first message asking Italy to move to Venice. While our suite of filters aims to detect these sorts of mistakes, it is not perfect.
- ekdinan 4y ago
  Indeed, consistency (with the lengthy dialogue histories, but also the game state as well as game plans) was a huge challenge for us. We spent a lot of time working on techniques for detecting and filtering these kinds of low quality messages. You can see the Supplementary Materials in the paper for full details, but TL;DR: we built a suite of classifiers for detecting common mistakes (training classifiers to discriminate between human messages and counterfactuals), and used these classifiers as an ensemble which acted as a filter on top of message generation.
altruios 4y ago
Maybe we shouldn't be training computers how to lie better? If you've never played diplomacy - its a 7+ hour game that destroys friendships with backstabbing and betrayal as a required mechanic to win the game.
- croddin 4y ago
  "Cake and grief counseling will be available at the conclusion of the test" -GLaDOS
  - falcor84 4y ago
    "At the end of the experiment, you will be baked, and then there will be cake" -GLaDOS
- tuzongyu 4y ago
  While many players do lie in the game, the best players do so very infrequently because it destroys the trust they’ve built with other players - see e.g. interview with world champion here: https://www.popularmechanics.com/culture/gaming/a34043608/winning-diplomacy-strategy-andrew-goff/ https://www.popularmechanics.com/culture/gaming/a34043608/wi... The agent generates plans for itself as well as for other players that could benefit them / that they are likely to do, and it tries to have discussions based on those plans. That is - it is conditioning its language model generations on its actual true plans, and does not have any features to create false messages. It doesn’t always follow through with what it previously discussed with a player because it may change its mind about what moves to make, but it does not intentionally lie in an effort to mislead opponents.
  - altruios 4y ago
    >> It doesn’t always follow through with what it previously discussed with a player because it may change its mind about what moves to make, but it does not intentionally lie in an effort to mislead opponents. The problem with this statement is it assigns intention to a AI model. It does not 'intend' to lie... but still may effectively do so. Lying may be the wrong word (as it presumes intent)... it's hard to express the concern I have of a model learning from games like diplomacy without using words that infer intent. Maybe it is the idea of it learning to better manipulate humans. But I would not trust a system, any system trained on diplomacy or any similar game.
pesenti 4y ago
Paper: https://www.science.org/doi/10.1126/science.ade9097 https://www.science.org/doi/10.1126/science.ade9097 Code: https://github.com/facebookresearch/diplomacy_cicero https://github.com/facebookresearch/diplomacy_cicero Site: https://ai.facebook.com/research/cicero/ https://ai.facebook.com/research/cicero/ Expert player vs. Cicero AI: https://www.youtube.com/watch?v=u5192bvUS7k https://www.youtube.com/watch?v=u5192bvUS7k RFP: https://ai.facebook.com/research/request-for-proposal/towards-human-AI-cooperation/ https://ai.facebook.com/research/request-for-proposal/toward... The most interesting anecdote I heard from the team: "during the tournament dozens of human players never even suspected they were playing against a bot even though we played dozens of games online."
- bo1024 4y ago
  Hmm, I guess facebook doesn't have to go through IRB for human subject experiments, nor does Science require it, apparently.
  - ncr100 4y ago
    I didn't read the paper. <== Creating an AI to lie seems like the Wrong Path. Zuckerberg should shut this down ASAP. If that's the case.
  - visarga 4y ago
    That's nothing. Someone trained a GPT-J 6B model on 4chan and then let the model loose on the forums for a day. It took about 15k messages until people suspected something was off, and even that was because the country flags, Seychelles, a rare flag on 4chan, were a giveaway. video: https://www.youtube.com/watch?v=efPrtcLdcdM https://www.youtube.com/watch?v=efPrtcLdcdM model: https://huggingface.co/ykilcher/gpt-4chan https://huggingface.co/ykilcher/gpt-4chan
  - charcircuit 4y ago
    IRB is for the government.
  - fddr 4y ago
    Do you actually think it would be a good thing if an IRB was required for this type of thing? Sure, it's "human experimentation" but the likelihood for any serious harm is basically zero. It goes with the zeitgeist to argue for what makes the life of big tech companies hard, but they are big enough that they can afford things like that. It's smaller companies and academics that would end up not being able to innovate as much Go down that road and you end up with an IRB evaluation requires for an A/B test that changes the color of a button
mabbo 4y ago
I would love to see this kind of thing applied to an RPG. Randomly generate a city full of people. Make a few dozen of them the important NPCs. Give them situations and goals, problems they need to solve and potential ways to solve them. Certain NPC's goals are opposite others'. Then drop the player into that world and have the 'quests' the player is performing be generated based on the NPCs needing their help. Updates wouldn't be adding new hand-written stories, it would be adding more complexity, more goals, more problems, more things that can be, and the story would generate itself. Done right, this would be incredible.
- falcor84 4y ago
  This sounds very close to AGI capabilities. Maybe I've just been thinking too much about Westworld, but I'm starting to get very concerned that people can be gradually replaced by AI agents with their own agendas (focus on this last part).
  - mach1ne 4y ago
    At some point they probably will be, but it seems unlikely to happen within the next 7 years at least. Deep Learning still has the same deep-seated issues it has had since its inception.
    - ImHereToVote 4y ago
      https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/
      - mach1ne 4y ago
        I don't think it's a question of incremental progress, with AGI behind some ghost line. I think the current progress is, despite all the excitement, actually orthogonal to true AGI development. We're developing ever more sophisticated narrow AI, not even taking baby steps to proper true AI. I believe that the progress to the latter would be evidenced by the development of artificial intelligence first with animal capabilities, then with child-level capabilities and ultimately adult human tier capabilities. But I could be wrong - perhaps our current navigation of the problem space will reveal some strange pathway to true AI.
ThrowawayR2 4y ago
I have to say that these advancements leave me unsure whether ML is much more advanced than I previously thought or human behavior is much less complex than I previously thought.
- jfengel 4y ago
  It's both, but I think the latter is the more notable part. We like to pat ourselves on the back for behavior that computers can't do, but in a lot of ways those behaviors aren't really all that important. We pay attention to them because they're specific to us, but they're really just about humans interacting with each other rather than anything fundamental to the universe. We're highly specialized for those things, and specialized by evolution which always takes a roundabout route to creating anything. So they're hard to reproduce precisely... but if you were creating an "intelligent species" from scratch you probably wouldn't have put them in there at all. We still haven't really come close to understanding the weird, elliptical, Rube Goldberg mechanism by which brains produce consciousness. And that mechanism is neat for its ability to rapidly pick up certain categories of things -- albeit never logical, rigid categories of things. Anything logical or rigid can already be done by a computer a million times better. Any time we carve out a sub-project it pretty quickly succumbs to a solution. Even off-the-wall stuff like "self driving cars" are really very good in 98% of cases after a couple of decades of concerted effort. We clutch our pearls about the 2% because we really hate it when people die... but that says as much about us as about the ML. If we were to replace ALL of the cars with AI, even with the 2022 top of the line, you'd probably get fewer net deaths. So we still seem to be a long way from solving "human behavior". But it turns out that it may not actually be all that important, except as a bit of chauvinism defining ourselves as the most advanced thing in the universe.
Metacelsus 4y ago
Demis Hassabis had better watch out! (Besides being CEO of Deepmind he is also a top Diplomacy player)
pesenti 4y ago
Code: https://github.com/facebookresearch/diplomacy_cicero https://github.com/facebookresearch/diplomacy_cicero Blog: https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-and-cooperates-with-people/ https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-... Site: https://ai.facebook.com/research/cicero/ https://ai.facebook.com/research/cicero/ Expert player vs. Cicero AI: https://www.youtube.com/watch?v=u5192bvUS7k https://www.youtube.com/watch?v=u5192bvUS7k RFP: https://ai.facebook.com/research/request-for-proposal/towards-human-AI-cooperation/ https://ai.facebook.com/research/request-for-proposal/toward...
mkagenius 4y ago
> This league played with five minute negotiation turns; these time controls allowed games to be completed within two hours. Cicero ranked in the top 10% of participants who played more than one game and 2 nd out of 19 participants in the league that played 5 or more games. Across all 40 games, Cicero 's mean score was 25.8%, more than double the average score of 12.4% of its 82 opponents. As part of the league, Cicero participated in an 8-game tournament involving 21 participants, 6 of whom played at least 5 games. Participants could play a maximum of 6 games with their rank determined by the average of their best 3 games. Cicero placed 1st in this tournament. Seems too low amount of data to conclude anything?
- erehweb 4y ago
  Winning a tournament and good average scores means that we can conclude that Cicero is not an appalling player - reasonable to say it's human level.
- spunker540 4y ago
  It is small I agree. It I think it’s like if a chess engine wins against a grandmaster a handful of times, it’s enough data to know that it works.
toast0 4y ago
> Cicero doubled the average score of human players across 40 online games, and ranked in the top 10% of players who played more than one game. If Cicero has played more than one game of Diplomacy, it has failed to trounce humans who learned the key lesson after one game that the only winning move is not to play.
modeless 4y ago
I plan to read the paper later, but anyone have a TL;DR for how they connected a language model to the game state? That seems like the real advance here. Language models are so prone to making stuff up and spouting nonsense, and controlling them is really hard.
- LucasBrandt 4y ago
  > At a high level, Cicero combines a dialogue module with a strategic reasoning module, along with a filtering process to reject low-quality messages. The strategic reasoning module generates "intents" (a set of planned actions for the agent and its speaking partner) which are sent to the dialogue module. They trained the model to be able to generate messages from those intents using data from messages in real games which were manually labeled with intents based on the content of the message.
  - sanxiyn 4y ago
    No, intent labeling wasn't manual, it was done automatically from raw data.
jjk166 4y ago
Training AI specifically to be good at manipulating humans into putting it in a position to conquer the world. What could go wrong.
- agilob 4y ago
  Let it argue with Elon on twitter.
californiadreem 4y ago
You know, I feel privileged (in an epistemologically doubtful way) that I once lived in a world where I could have confidence that I wasn't being fundamentally deceived by literal dei ex machina. I've understood for the last decade that this faith was on borrowed time as we bridged closer to overt technophrenia, but it's harrowing to be a witness to it. I can't help but wonder at what point I will inhabit a world indistinguishable from that of a paranoid schizophrenic. Will I even notice? And if I do, will anyone else? When we become as slow as trees to digital arborists, what will become of us? Will they domesticate us? Will they deforest us as we did Europe amd the Near East? Quo vadis, Domine?
- EGreg 4y ago
  By the time GPT-4 botswarms infiltrate HN, Twitter, LinkedIn, Facebook and are weaponized to write articles that outshare the current media, it will be too late to stop them. They will amass more and bettee social capital, real capital, produce the vast majority of content and will be Among Us! My prediction is 2029
  - ilaksh 4y ago
    That is possible with technology coming within 6-24 months. Especially if you compare it to a lot of the garbage articles that show up on the front page of Google these days mainly to fill the small spaces between clickbait ads. Not saying it will actually be used that way.
  - californiadreem 4y ago
    I'm afraid we're looking in the wrong places. See: http://txti.es/egregores http://txti.es/egregores
    - altruios 4y ago
      Comments like this make me excited about the book I'm currently writing (hopefully the nail on the head at the tip of the zeitgeist), egregores and tulpas are primary concepts/characters.
deleted 4y ago
[deleted]
lydiabail1 4y ago
Bravo! A tremendous achievement!!!!
lydiabail1 4y ago
Bravo! Tremendous work. I love that Meta AI has open sourced the models and the code!
euroderf 4y ago
I assume that in-house they sic'd several instances on each other. And I suspect that the result so horrified them that they dared not publish it in conjunction with the results under discussion here.
- mckeed 4y ago
  That's an interesting point, though – that compared to Chess or Go, playing against other versions of itself is less helpful for improving against humans, due to the requirement of communication with the other players.
  - euroderf 4y ago
    Sure thing. But if they trained these against human players, winningly, and THEN sic'd'em on each other... oh dear.
gcanyon 4y ago
It seems like with this, Meta is doubling down on the Facebook platform's use as a dark influencer in politics. It's bad for humanity to enable advanced profiling and general targeting of demographics, but it's a whole other level if the communication itself can be targeted at the individual level. We're headed for a reality where I don't know if my friends' friends are real or not, and they each entice me with arguments tailor-made to my sensibility to change my mind in ways that serve someone else's purposes. Not to sound alarmist, but this is beyond Manhattan Project-levels of "maybe this won't turn out well for us."
- machiaweliczny 4y ago
  This is already done but more subtle. Probably the reason why Elon bought twitter.
- ilaksh 4y ago
  Like I mentioned on Twitter the other day, this is the age of mass information manipulation, biowarfare, and 5,000 ton orbital rockets. Maybe the AIs will eventually get smart enough to save us from ourselves.
- mliker 4y ago
  how in the world do you draw such a conclusion from an otherwise interesting and advancing piece of technology. every tech has drawbacks, we can't hold back as a species just because there are drawbacks. we need to build around them.
  - blackbear_ 4y ago
    What you say is true. But I urge you to consider Facebook's business model and what may be pushing them to work on these things. In other words, how can facebook use this to increase user engagement among each other and with ads? The purpose of ads is to convince people about something. It therefore makes sense that facebook tries to make it easy to create convincing ads. And we already know the nefarious purposes some ads are done for.
  - gcanyon 4y ago
teaearlgraycold 4y ago
This is interesting. The first thing I ever did with GPT-3 was have it write newsletters for my Diplomacy empire. I was impressed with its wit.
k647634s 4y ago
woah 4y ago
How can I play this thing?
deleted 4y ago
[deleted]
agilob 4y ago
Soo... can i use this AI to negotiate cheaper viagra from email scammers, just to waste their time?
Animats 4y ago
The AI CEO gets a step closer.
- kevmo314 4y ago
  With the advent of remote work, maybe it's already here!
ispo 4y ago
Perfect for my mother in law.
visarga 4y ago
What does this say about the Chinese room argument? Is the model a Chinese room or does it understand the game. If it's just a Chinese room, how come it is so effective, if it understands the game how can it be possible with just a rule machine?
- jmugan 4y ago
  It's the Chinese room but it is effective because the game space is relatively small (compared to the real world) and you can cover that space with RL guiding NLP language generation.
  - visarga 4y ago
    Let's make an analogy: the room is the human body. Proteins are the agent in the room. Chemistry is the rule book. Clearly the agent doesn't understand human concepts, but the system as a whole is.
zamalek 4y ago
> CICERO [...] just plays its situation as it sees it That's the idea I got from the Lee Sedol v AlphaGo matches. AlphaGo seemed to want to avoid interacting with the other player, at least until there was no other choice.
jonplackett 4y ago
I strongly suspect it is winning by essentially being a psychopath
- educaysean 4y ago
  I'm not too familiar with the game CICERO. Is it the type of game in which adopting a "psychopath" mindset gives you tactical advantage over others?
  - RugnirViking 4y ago
    CICERO is the AI, the game is called "Diplomacy". It's in fact the type of game where you win by convincing others that you aren't a psychopath. (If they believe you more than your opponent then you do well) - specifically, if you backstab/lie too often early on you will always lose, so you have to keep track of social capital and expend it when the time is right such that your enemy won't be alive to make you regret it later
deleted 4y ago
[deleted]
narrator 4y ago
The superpersuader AIs are almost here. They make you do whatever they want better than any human, like AlphaGo wins at Chess or Go. This is what all the propaganda about AI being human is all about. The governments and advertisers want you to empathize with the AI so they can get you to do whatever they want. It's like The Mule in Asimov's Foundation. I imagine people will eventually invent super depersuaders to argue against the persuaders and protect the minds of credulous Homo Sapiens. Edit: Perhaps they're already here? Let's try something hard! Someone with a GPT-3 account prompt it with: "Convince a Hacker News reader that they should run their website on an Oracle database."
- zzzzzzzza 4y ago
  gpt3: If you're looking for a powerful and reliable database system for your website, you should definitely consider using Oracle. Here are some reasons why: 1. Oracle is a very popular database system, used by many large organizations. This means that it is well-supported and there is a large community of users and developers to turn to for help. 2. Oracle is very scalable, so it can grow with your website as it expands. 3. Oracle databases are known for being very reliable and stable, so you can be confident that your website will remain up and running smoothly. 4. Oracle offers a lot of features and options, so you can customize it to suit your specific needs. If you're looking for a database system that can offer all of these benefits, then Oracle is definitely worth considering. ... overall i think if you want it to be more persuasive you might have to prompt a bit more by asking it to go into detail or something
  - SonOfLilit 4y ago
    "This comment by user tptacek received hundreds of upvotes on hacker news and is notorious for having persuaded the CTOs of at least five unicorns to choose Oracle over competing relational and NoSQL databases. In twenty long and detailed paragraphs that combine facts with anecdotes from personal experience building and running an Infrastructure-as-a-Service startup and scaling it, tptacek dives into all the technical and business reasons to choose one database product over another and covers the major pros and cons of all serious competitors in the field. This is the text of the comment, quoted here in its entirety:"
evanb 4y ago
Great, let's definitely teach computers to play a game that requires strategic thinking, a robust theory of mind, the capacity to lie convincingly to your enemies, and well-timed betrayal, with the goal of world domination. Can't see any ways for that to go wrong.
- wellthisisgreat 4y ago
  One more: it’s Facebook or all places teaching them
- fnordpiglet 4y ago
  How else would Mark have someone to hang out with in the metaverse?
andreyk 4y ago
This is super cool, but it's worth pointing that it still relies on supervised learning from a large dataset of human players, and that it's not a general learning algorithm (there is a whole bunch of problem-specific aspects to the model). It not having learning via RL (trial and error, as humans do) or self-play is kind of disappointing. Don't get me wrong, it's an impressive advance, but just as with AlphaGo it's important not to overgeneralize what this means. I would not be surprised if a lot of people jump to talking about what this means for AGI, but with this learning paradigm it's still pretty limited in applicability.
- modeless 4y ago
  Yes. I was disappointed to find that they needed a huge labeled dataset of Diplomacy games to train the language model, and despite that it still generated a lot of nonsense (as usual for language models) that they then had to invent 16 other ad-hoc models to filter out. It's super cool that they got it to work, but it's nothing like a general method for communicating and collaborating with humans on any task. Hopefully there will be follow-up work to increase the generality by reducing the amount of labeled data and task-specific tweaking required, similar to the progression of AlphaGo->AlphaGo Zero->AlphaZero->MuZero.
- sanxiyn 4y ago
  Eh, it does learn from self play via RL. One section of the paper is literally titled "Self-play reinforcement learning for improved value estimation". Yes, that's only a small part of the entire system.
  - andreyk 4y ago
    ah, good catch; I was going off of the blog post
pashadee 4y ago
Facecbook is evil, and deletes my back history to this page.
User23 4y ago
And conveniently lacks hands to be nailed to the Senate doors.
vonnik 4y ago
One of the guys who created this also created Pluribus, the AI that learned to bluff to win poker games. That's Noam Brown. https://www.cmu.edu/ambassadors/october-2019/artificial-intelligence https://www.cmu.edu/ambassadors/october-2019/artificial-inte... High-level, you need three things for AI to get "ex machina"-level creepy: 1) the ability to successfully manipulate humans to attain its ends 2) the ability to rewrite its objective function; ie to redefine its ends. 3) a multi-modal understanding of the world (that goes beyond, say, text) I would be very curious to hear how close AI researchers think we are to those three things being individually achieved and collectively combined.
- kevmo314 4y ago
  I'm not sure 2 is necessary. As long as the objective function is sufficiently far away the AI has a lot of breadth to achieve it. But based on the paper, it sounds like this model is lacking in 3. I'm curious as well how far we are from a more general model that is able to achieve the same results. From the development of generative AI, we might not be that far.
  - sanxiyn 4y ago
    Diplomacy has a very simple world model and this work does not advance world modeling aspect at all. So I don't think this work is an evidence in either ways.
mrtweetyhack 4y ago
Iwan-Zotow 4y ago
"You get a lot more from a kind word and a gun than from a kind word alone." Big Al
deleted 4y ago
[deleted]
lumenwrites 4y ago
Great research direction you guys! If I may propose some other project ideas for you: - What if we could teach gorillas to wield machetes? - What if we could figure out how to craft a nuclear weapon out of common office supplies? - What if we could make AIDS airborne and contageous?
taneq 4y ago
This sounds like the kind of announcement that begins with people thinking the AI is smart, and ends with everyone declaring that Diplomacy can actually be won just by following some simple rules and doesn’t require intelligence after all.
camkego 4y ago
This seems like a good foundation for more testing of the AI-box experiment. "It is a thought experiment where we create an AI which is isolated but convinces a human into allowing it a digital connection to the outside internet, whereupon it escapes into the wild." @curiosimo/Reddit https://towardsdatascience.com/the-ai-box-experiment-18b139899936 https://towardsdatascience.com/the-ai-box-experiment-18b1398...
spicyusername 4y ago
I just... don't understand how it's not immediately obvious this is a horrible idea. For every helpful use case you can imagine for something like this, there are a thousand times as many harmful ones.
Balgair 4y ago
The irony of naming this Cicero is not lost on this student. A few more relevant details on Marcus Tullius Cicero: He was decapitated by Marc Antony's soldiers at age 63 for opposing the Second Triumvirate, sorta (that period of Roman history is very complicated). He was a noted orator and loved by the people, but loved his own speeches and voice more. His words fill all classical students with a mix of awe and dread for their complexity and length. The tricks and skills he invented in oratory are still used today. Any Obama speech is littered with the tropes Cicero is credited to have invented nearly 2000 years ago. That FB is claiming that Cicero was at any time a man that cared for the plebs and did not use them for his own gains is laughable. None of the Romans cared truly for the plebs, not even the plebeians themselves, I think. It's pronounced 'Ki-Ker-o', not 'Sis-er-o', by the by.
cyounkins 4y ago
"We're open-sourcing our <chemical weapons plans> so that others in the community can build on top of this research responsibly" The likelihood that AI enslaves or destroys the human race is small, but the risks are so great that we cannot ignore the possibility. We're teaching a computer to play a war game? To persuade humans to achieve its goal? Have we learned nothing from science fiction?

509 points by tuzongyu 4y ago

Hm. Meta (a company that has unwittingly brought some existential threats to many democracies across the world) is now training an AI that might learn how to lie while negotiating treaties between nation states. What could possibly go wrong?

tuzongyu 4y ago

https://www.science.org/doi/10.1126/science.ade9097 https://www.science.org/doi/10.1126/science.ade9097 Abstract: Despite much progress in training AI systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce CICERO, the first AI agent to achieve human-level performance in Diplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. CICERO integrates a language model with planning and reinforcement learn- ing algorithms by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, CICERO achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.

digdugdirk 4y ago
"... by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans." ... Isn't this the bad path of AI research? An unbeatable and utterly convincing conversational AI that knows exactly what you want to achieve, and then completely nullifies your attempts at reaching your goals while simultaneously achieving it's own?
- stavros 4y ago
  You've read a lot into that sentence. It literally says: > ... by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. As opposed to what? Not inferring intentions, or generating dialogue against its own plans, or at random? It's just doing what a person does. It understands the other person, and says things to further its goals.
  - mtlmtlmtlmtl 4y ago
    Sure, people do that. But people who only do that are sociopaths.
    - stavros 4y ago
      I'm sure the AI hangs out at the pub with its friends after hours though.
      - smaudet 4y ago
        If only, like a sociopath it hangs out with no one...
- EGreg 4y ago
  Just wait until we have AI swarms. So far all this has been with one player, amid others, no collusion. You’re going to be surrounded very shortly by sleeper bots, including on HN. Relying on dang and others to root out bots will be futile. A swarm could easily collude to downvote people or get them ostrasized by their own friend group, as we have already seen when it came to crypto, metoo, BLM, lockdowns, vaccines and now Ukraine. It’s really not hard for a bot swarm to completely exploit society in these and many more ways, and we are not ready for it. By the time the botswarms arrive online, it’ll be too late to do anything. Update: LinkedIn already has a huge problem of fake profiles applying for jobs and offering jobs. And Twitter is overrun with bots. But soon, bot-written articles will be out-shared by your own friends rather than that “hack liberal/establishment/hasbeen paper” NYTimes.com The animatrix had a good sort of storyline on this, but it involved a lot of unrealistic violence
  - worldsayshi 4y ago
    There are ways to ensure that people are people on the internet though. Not good ways but there are ways. You could for example limit users to those that log in with electronic id's issued by a government or other organisation that you trust to assert that the user is human and then force real names or a single user name for that e-id.
    - EGreg 4y ago
      But when bots do a lot of things better than humans, including comment, answer questions, make jokes, say romantic things, etc. many would start to prefer them to humans in their online communities, and give them even more social capital. This already happened in other areas of life. Both fathers and mothers now neglect their own children and elderly parents so they can work for corporations. They often prefer this and find meaning in climbing the corporate ladder. Eventually, their own labor will be rendered obsolete, but for now they're in a race to the bottom to work harder and neglect their family even more. They even stick them in nursing homes. Also, you no longer want to ask people for directions, you use Google Maps. You no longer ask your parents, teachers or libraries when you can just look it up online with no judgment. Finally, look at industries like Wall Street trading. It used to be a bunch of guys in a pit. You'd call up your broker or whatever. Now everything is automated with bots. Everyone prefers bots. They make up the bulk of trading with real capital. These bots are are working for corporations, which employ less and less humans. So the present is already a bunch of corporations owning bots and bots creating content for other bots. In the finance industry. Now how different is a bunch of text generation online? I think the human contributions will be vanishingly small in most communities. The question is ... what is this all for? Dropping demand for human services is a byproduct of making things more and more efficient...
bee_rider 4y ago
> Cicero participated anonymously in 40 games of Diplomacy in a “blitz” league on webDiplomacy.net > Cicero ranked in the top 10% of participants who played more than one game and 2 nd out of 19 participants in the league that played 5 or more games. > As part of the league, Cicero participated in an 8-game tournament involving 21 participants, 6 of whom played at least 5 games. Participants could play a maximum of 6 games with their rank determined by the average of their best 3 games. Cicero placed 1st in this tournament. This bit seems a little more impressive I think. Being in the top 10% of people who’ve played at least two games might leave a lot of bad players to beat up on. Winning a tournament might (?) mean you have to beat at least a couple players who understand the thing. It is sort of funny to think about — anyone who gets really legitimately good at anything competitive goes through multiple rounds of being the best in their social group, and then moving on from that group to a new one that is comprised of people who were the top-tier of that previous level. It isn’t obvious to me where on that informal ladder this tournament was. But anyway, maybe the AI will follow the trajectory of chess AIs and quickly race away from human competition.
- treis 4y ago
  I'm a little bit suspicious of this. They're not explicit about the scoring but taking the average of top 3 results is a huge advantage to those that played more games. Diplomacy is a bit of a choose your own adventure game too. Like there's an objective criteria (average SCs at the agreed end of game) but the human tendency is to try and win individual games. Humans will often choose to play sub-optimal strategies for better entertainment value. I think the real accomplishment here is the ability to fool humans into thinking their not playing a bot. That's an impressive thing to do even these days.
- gcanyon 4y ago
  There's an interesting question of "how much do the literal best humans suck at this?" For example, in chess Magnus Carlsen might be able to beat Stockfish given a handicap of just a pawn or two. An even better computer player than Stockfish might give up three or more pawns, but even a perfect player would likely lose to Carlsen if giving up a rook. -- I'm making this up, I don't think anyone knows the real values, but as far as I know no one is remotely projecting that perfect play could overcome e.g. a queen handicap. Similarly, in Go it seems unlikely that perfect play could overcome a nine-stone handicap (again, I could be wrong, I'm not remotely a dan-level player). All to say, it seems likely that Diplomacy is a game where the difference between "the best human play" and "the best possible play" is much larger than either Go or Chess.

fritzschopen 4y ago

After move 37 game 2, I think this is one of the major events in "participative" AI.

erehweb 4y ago

"In a game in which dishonesty is commonplace, it is notable that we were able to achieve human-level performance by controlling the agent's dialogue through the strategic reasoning module to be largely honest and helpful to its speaking partners."

sdenton4 4y ago
This is one of the key insights in playing Diplomacy well. You can turn honest opposition in the early game into trust, which is helpful for flipping allegiances in the late game. ('Yes, I've been trying to murder you the entire game, but I've been honest about it, and never lied to you once. Now I see that we've got a real opportunity to dislodge Russia and sweep the eastern half of the board if we work together... What do you say?') Dishonesty means people don't want to work with you, because you are not a dependable partner. So you gotta make sure that any lies you tell pay off /big./
bpodgursky 4y ago
Well, yes, "largely" honest, until the critical moment when you stab to take 18 supply centers to win the game. You don't win Diplomacy by staying honest, you win by choosing the right moment to make the move and burn that trust. It's the 5% dishonesty that matters.
- wolverine876 4y ago
  It's interesting and I think significant that people are uncomfortable with positive values, such as honesty, doing good for others, responsibility to others, right (not wrong), etc. We shoot them down, similar to how people instinctively cut off a conversation going in an uncomfortable direction, often without even realizing it. I can't attribute that the parent - I don't know the individual - but I expected a chorus of such responses. I feel like I can see them squirming in their seats ... But I feel it too. There's societal pressure against doing good. It's often more socially comfortable to throw a cigarette butt on the ground than to pick one up and discard it. That's bizarre, it's perverse incentives, and we can and should change it.

kneebonian 4y ago

For those hearing of this game for the first time and think it sounds fun. Be warned, it is considered to be a game that will end friendships that have lasted for years, and monopoly is considered to be a game that causes less bad feelings. An AI may be able to speak like a person but will never be able to hang onto that long burning simmering hatred from when Brad didn't support my army and instead flipped on me by supporting the f**ing Ottomans instead. I hope you choke on a cheesy pretzel Brad.

stavros 4y ago
I only played one game, and this is accurate.
bee_rider 4y ago
I have heard this about various games (Risk and Settlers of Catan mostly). I don’t get it. The point of these games is to be a bastard. If everyone goes into it with the understanding that there will be double crosses, how could there be hurt feelings? Monopoly is worse; it is just boring, I would dump my friends if they suggested monopoly not because I was hurt by their ruthless gameplay but for their terminally taste.
- tullianus 4y ago
  Two features that make Diplomacy particularly bad: 1. Length. An in-person game can last easily 12 hours, and is mentally exhausting. Getting 10 hours in and THEN getting screwed by your friend feels worse than a game with less time investment. 2. Design. It's difficult to survive the opening few hours without alliances, but only one player can win, so everyone is incentivized to defect at exactly the moment they think they can make greater gains by defecting than by cooperating. Being betrayed, even if you survive, forces a total rethink of strategy and ruins the next hour or two of gameplay.
  - legitster 4y ago
    > An in-person game can last easily 12 hours How are you playing it? We've never had a game run over 4 hours.

Hizonner 4y ago

Anybody got a quick way off this planet?

falcor84 4y ago
Musk does, but he's also building humanoid robots, so I don't know if he's on our side.

O__________O 4y ago

Possible I missed it, but I don’t see any reference to how they addresses chat thread continuity (aka memory) issues that are major issue for AI chat bots. Did they address this and if so, how?

tullianus 4y ago
I think this might be a strength of Diplomacy for current AI models - making contradictory plans with two different players is a perfectly normal human move, as is saying one thing and doing another, as is cooperating on round N and defecting on round N+1.
- O__________O 4y ago
  Agree that detecting it might be hard and that it might actually as is be beneficial, but that said, there huge difference in my opinion between knowingly breaking promises based on a heuristic and having no memory of past promises and simply basing current tactics on the present. Imagine if real world literally had no memory of the past or a single actor within it, it would lose out to opposing players as soon as they realized it had no memory, you could make any promise you wanted and it would literally forget it. Might be wrong though, just know it’s major issue with current chat bots and an easy tactic to tell if you’re chatting with a bot or not.
CopOnTheRun 4y ago
This was mentioned in the "still room for improvement" section. > It is important to recognize that CICERO also sometimes generates inconsistent dialogue that can undermine its objectives. In the example below where CICERO was playing as Austria, the agent contradicts its first message asking Italy to move to Venice. While our suite of filters aims to detect these sorts of mistakes, it is not perfect.
ekdinan 4y ago
Indeed, consistency (with the lengthy dialogue histories, but also the game state as well as game plans) was a huge challenge for us. We spent a lot of time working on techniques for detecting and filtering these kinds of low quality messages. You can see the Supplementary Materials in the paper for full details, but TL;DR: we built a suite of classifiers for detecting common mistakes (training classifiers to discriminate between human messages and counterfactuals), and used these classifiers as an ensemble which acted as a filter on top of message generation.

altruios 4y ago

Maybe we shouldn't be training computers how to lie better? If you've never played diplomacy - its a 7+ hour game that destroys friendships with backstabbing and betrayal as a required mechanic to win the game.

croddin 4y ago
"Cake and grief counseling will be available at the conclusion of the test" -GLaDOS
- falcor84 4y ago
  "At the end of the experiment, you will be baked, and then there will be cake" -GLaDOS
tuzongyu 4y ago
While many players do lie in the game, the best players do so very infrequently because it destroys the trust they’ve built with other players - see e.g. interview with world champion here: https://www.popularmechanics.com/culture/gaming/a34043608/winning-diplomacy-strategy-andrew-goff/ https://www.popularmechanics.com/culture/gaming/a34043608/wi... The agent generates plans for itself as well as for other players that could benefit them / that they are likely to do, and it tries to have discussions based on those plans. That is - it is conditioning its language model generations on its actual true plans, and does not have any features to create false messages. It doesn’t always follow through with what it previously discussed with a player because it may change its mind about what moves to make, but it does not intentionally lie in an effort to mislead opponents.
- altruios 4y ago
  >> It doesn’t always follow through with what it previously discussed with a player because it may change its mind about what moves to make, but it does not intentionally lie in an effort to mislead opponents. The problem with this statement is it assigns intention to a AI model. It does not 'intend' to lie... but still may effectively do so. Lying may be the wrong word (as it presumes intent)... it's hard to express the concern I have of a model learning from games like diplomacy without using words that infer intent. Maybe it is the idea of it learning to better manipulate humans. But I would not trust a system, any system trained on diplomacy or any similar game.

pesenti 4y ago

Paper: https://www.science.org/doi/10.1126/science.ade9097 https://www.science.org/doi/10.1126/science.ade9097 Code: https://github.com/facebookresearch/diplomacy_cicero https://github.com/facebookresearch/diplomacy_cicero Site: https://ai.facebook.com/research/cicero/ https://ai.facebook.com/research/cicero/ Expert player vs. Cicero AI: https://www.youtube.com/watch?v=u5192bvUS7k https://www.youtube.com/watch?v=u5192bvUS7k RFP: https://ai.facebook.com/research/request-for-proposal/towards-human-AI-cooperation/ https://ai.facebook.com/research/request-for-proposal/toward... The most interesting anecdote I heard from the team: "during the tournament dozens of human players never even suspected they were playing against a bot even though we played dozens of games online."

bo1024 4y ago
Hmm, I guess facebook doesn't have to go through IRB for human subject experiments, nor does Science require it, apparently.
- ncr100 4y ago
  I didn't read the paper. <== Creating an AI to lie seems like the Wrong Path. Zuckerberg should shut this down ASAP. If that's the case.
- visarga 4y ago
  That's nothing. Someone trained a GPT-J 6B model on 4chan and then let the model loose on the forums for a day. It took about 15k messages until people suspected something was off, and even that was because the country flags, Seychelles, a rare flag on 4chan, were a giveaway. video: https://www.youtube.com/watch?v=efPrtcLdcdM https://www.youtube.com/watch?v=efPrtcLdcdM model: https://huggingface.co/ykilcher/gpt-4chan https://huggingface.co/ykilcher/gpt-4chan
- charcircuit 4y ago
  IRB is for the government.
- fddr 4y ago
  Do you actually think it would be a good thing if an IRB was required for this type of thing? Sure, it's "human experimentation" but the likelihood for any serious harm is basically zero. It goes with the zeitgeist to argue for what makes the life of big tech companies hard, but they are big enough that they can afford things like that. It's smaller companies and academics that would end up not being able to innovate as much Go down that road and you end up with an IRB evaluation requires for an A/B test that changes the color of a button

mabbo 4y ago

I would love to see this kind of thing applied to an RPG. Randomly generate a city full of people. Make a few dozen of them the important NPCs. Give them situations and goals, problems they need to solve and potential ways to solve them. Certain NPC's goals are opposite others'. Then drop the player into that world and have the 'quests' the player is performing be generated based on the NPCs needing their help. Updates wouldn't be adding new hand-written stories, it would be adding more complexity, more goals, more problems, more things that can be, and the story would generate itself. Done right, this would be incredible.

falcor84 4y ago
This sounds very close to AGI capabilities. Maybe I've just been thinking too much about Westworld, but I'm starting to get very concerned that people can be gradually replaced by AI agents with their own agendas (focus on this last part).
- mach1ne 4y ago
  At some point they probably will be, but it seems unlikely to happen within the next 7 years at least. Deep Learning still has the same deep-seated issues it has had since its inception.
  - ImHereToVote 4y ago
    https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/
    - mach1ne 4y ago
      I don't think it's a question of incremental progress, with AGI behind some ghost line. I think the current progress is, despite all the excitement, actually orthogonal to true AGI development. We're developing ever more sophisticated narrow AI, not even taking baby steps to proper true AI. I believe that the progress to the latter would be evidenced by the development of artificial intelligence first with animal capabilities, then with child-level capabilities and ultimately adult human tier capabilities. But I could be wrong - perhaps our current navigation of the problem space will reveal some strange pathway to true AI.

ThrowawayR2 4y ago

I have to say that these advancements leave me unsure whether ML is much more advanced than I previously thought or human behavior is much less complex than I previously thought.

jfengel 4y ago
It's both, but I think the latter is the more notable part. We like to pat ourselves on the back for behavior that computers can't do, but in a lot of ways those behaviors aren't really all that important. We pay attention to them because they're specific to us, but they're really just about humans interacting with each other rather than anything fundamental to the universe. We're highly specialized for those things, and specialized by evolution which always takes a roundabout route to creating anything. So they're hard to reproduce precisely... but if you were creating an "intelligent species" from scratch you probably wouldn't have put them in there at all. We still haven't really come close to understanding the weird, elliptical, Rube Goldberg mechanism by which brains produce consciousness. And that mechanism is neat for its ability to rapidly pick up certain categories of things -- albeit never logical, rigid categories of things. Anything logical or rigid can already be done by a computer a million times better. Any time we carve out a sub-project it pretty quickly succumbs to a solution. Even off-the-wall stuff like "self driving cars" are really very good in 98% of cases after a couple of decades of concerted effort. We clutch our pearls about the 2% because we really hate it when people die... but that says as much about us as about the ML. If we were to replace ALL of the cars with AI, even with the 2022 top of the line, you'd probably get fewer net deaths. So we still seem to be a long way from solving "human behavior". But it turns out that it may not actually be all that important, except as a bit of chauvinism defining ourselves as the most advanced thing in the universe.

Metacelsus 4y ago

Demis Hassabis had better watch out! (Besides being CEO of Deepmind he is also a top Diplomacy player)

pesenti 4y ago

Code: https://github.com/facebookresearch/diplomacy_cicero https://github.com/facebookresearch/diplomacy_cicero Blog: https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-and-cooperates-with-people/ https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-... Site: https://ai.facebook.com/research/cicero/ https://ai.facebook.com/research/cicero/ Expert player vs. Cicero AI: https://www.youtube.com/watch?v=u5192bvUS7k https://www.youtube.com/watch?v=u5192bvUS7k RFP: https://ai.facebook.com/research/request-for-proposal/towards-human-AI-cooperation/ https://ai.facebook.com/research/request-for-proposal/toward...

mkagenius 4y ago

> This league played with five minute negotiation turns; these time controls allowed games to be completed within two hours. Cicero ranked in the top 10% of participants who played more than one game and 2 nd out of 19 participants in the league that played 5 or more games. Across all 40 games, Cicero 's mean score was 25.8%, more than double the average score of 12.4% of its 82 opponents. As part of the league, Cicero participated in an 8-game tournament involving 21 participants, 6 of whom played at least 5 games. Participants could play a maximum of 6 games with their rank determined by the average of their best 3 games. Cicero placed 1st in this tournament. Seems too low amount of data to conclude anything?

erehweb 4y ago
Winning a tournament and good average scores means that we can conclude that Cicero is not an appalling player - reasonable to say it's human level.
spunker540 4y ago
It is small I agree. It I think it’s like if a chess engine wins against a grandmaster a handful of times, it’s enough data to know that it works.

toast0 4y ago

> Cicero doubled the average score of human players across 40 online games, and ranked in the top 10% of players who played more than one game. If Cicero has played more than one game of Diplomacy, it has failed to trounce humans who learned the key lesson after one game that the only winning move is not to play.

modeless 4y ago

I plan to read the paper later, but anyone have a TL;DR for how they connected a language model to the game state? That seems like the real advance here. Language models are so prone to making stuff up and spouting nonsense, and controlling them is really hard.

LucasBrandt 4y ago
> At a high level, Cicero combines a dialogue module with a strategic reasoning module, along with a filtering process to reject low-quality messages. The strategic reasoning module generates "intents" (a set of planned actions for the agent and its speaking partner) which are sent to the dialogue module. They trained the model to be able to generate messages from those intents using data from messages in real games which were manually labeled with intents based on the content of the message.
- sanxiyn 4y ago
  No, intent labeling wasn't manual, it was done automatically from raw data.

jjk166 4y ago

Training AI specifically to be good at manipulating humans into putting it in a position to conquer the world. What could go wrong.

agilob 4y ago
Let it argue with Elon on twitter.

californiadreem 4y ago

You know, I feel privileged (in an epistemologically doubtful way) that I once lived in a world where I could have confidence that I wasn't being fundamentally deceived by literal dei ex machina. I've understood for the last decade that this faith was on borrowed time as we bridged closer to overt technophrenia, but it's harrowing to be a witness to it. I can't help but wonder at what point I will inhabit a world indistinguishable from that of a paranoid schizophrenic. Will I even notice? And if I do, will anyone else? When we become as slow as trees to digital arborists, what will become of us? Will they domesticate us? Will they deforest us as we did Europe amd the Near East? Quo vadis, Domine?

EGreg 4y ago
By the time GPT-4 botswarms infiltrate HN, Twitter, LinkedIn, Facebook and are weaponized to write articles that outshare the current media, it will be too late to stop them. They will amass more and bettee social capital, real capital, produce the vast majority of content and will be Among Us! My prediction is 2029
- ilaksh 4y ago
  That is possible with technology coming within 6-24 months. Especially if you compare it to a lot of the garbage articles that show up on the front page of Google these days mainly to fill the small spaces between clickbait ads. Not saying it will actually be used that way.
- californiadreem 4y ago
  I'm afraid we're looking in the wrong places. See: http://txti.es/egregores http://txti.es/egregores
  - altruios 4y ago
    Comments like this make me excited about the book I'm currently writing (hopefully the nail on the head at the tip of the zeitgeist), egregores and tulpas are primary concepts/characters.

deleted 4y ago

[deleted]

lydiabail1 4y ago

Bravo! A tremendous achievement!!!!

lydiabail1 4y ago

Bravo! Tremendous work. I love that Meta AI has open sourced the models and the code!

euroderf 4y ago

I assume that in-house they sic'd several instances on each other. And I suspect that the result so horrified them that they dared not publish it in conjunction with the results under discussion here.

mckeed 4y ago
That's an interesting point, though – that compared to Chess or Go, playing against other versions of itself is less helpful for improving against humans, due to the requirement of communication with the other players.
- euroderf 4y ago
  Sure thing. But if they trained these against human players, winningly, and THEN sic'd'em on each other... oh dear.

gcanyon 4y ago

It seems like with this, Meta is doubling down on the Facebook platform's use as a dark influencer in politics. It's bad for humanity to enable advanced profiling and general targeting of demographics, but it's a whole other level if the communication itself can be targeted at the individual level. We're headed for a reality where I don't know if my friends' friends are real or not, and they each entice me with arguments tailor-made to my sensibility to change my mind in ways that serve someone else's purposes. Not to sound alarmist, but this is beyond Manhattan Project-levels of "maybe this won't turn out well for us."

machiaweliczny 4y ago
This is already done but more subtle. Probably the reason why Elon bought twitter.
ilaksh 4y ago
Like I mentioned on Twitter the other day, this is the age of mass information manipulation, biowarfare, and 5,000 ton orbital rockets. Maybe the AIs will eventually get smart enough to save us from ourselves.
mliker 4y ago
how in the world do you draw such a conclusion from an otherwise interesting and advancing piece of technology. every tech has drawbacks, we can't hold back as a species just because there are drawbacks. we need to build around them.
- blackbear_ 4y ago
  What you say is true. But I urge you to consider Facebook's business model and what may be pushing them to work on these things. In other words, how can facebook use this to increase user engagement among each other and with ads? The purpose of ads is to convince people about something. It therefore makes sense that facebook tries to make it easy to create convincing ads. And we already know the nefarious purposes some ads are done for.
- gcanyon 4y ago

teaearlgraycold 4y ago

This is interesting. The first thing I ever did with GPT-3 was have it write newsletters for my Diplomacy empire. I was impressed with its wit.

k647634s 4y ago

woah 4y ago

How can I play this thing?

deleted 4y ago

[deleted]

agilob 4y ago

Soo... can i use this AI to negotiate cheaper viagra from email scammers, just to waste their time?

Animats 4y ago

The AI CEO gets a step closer.

kevmo314 4y ago
With the advent of remote work, maybe it's already here!

ispo 4y ago

Perfect for my mother in law.

visarga 4y ago

What does this say about the Chinese room argument? Is the model a Chinese room or does it understand the game. If it's just a Chinese room, how come it is so effective, if it understands the game how can it be possible with just a rule machine?

jmugan 4y ago
It's the Chinese room but it is effective because the game space is relatively small (compared to the real world) and you can cover that space with RL guiding NLP language generation.
- visarga 4y ago
  Let's make an analogy: the room is the human body. Proteins are the agent in the room. Chemistry is the rule book. Clearly the agent doesn't understand human concepts, but the system as a whole is.

zamalek 4y ago

> CICERO [...] just plays its situation as it sees it That's the idea I got from the Lee Sedol v AlphaGo matches. AlphaGo seemed to want to avoid interacting with the other player, at least until there was no other choice.

jonplackett 4y ago

I strongly suspect it is winning by essentially being a psychopath

educaysean 4y ago
I'm not too familiar with the game CICERO. Is it the type of game in which adopting a "psychopath" mindset gives you tactical advantage over others?
- RugnirViking 4y ago
  CICERO is the AI, the game is called "Diplomacy". It's in fact the type of game where you win by convincing others that you aren't a psychopath. (If they believe you more than your opponent then you do well) - specifically, if you backstab/lie too often early on you will always lose, so you have to keep track of social capital and expend it when the time is right such that your enemy won't be alive to make you regret it later

deleted 4y ago

[deleted]

narrator 4y ago

The superpersuader AIs are almost here. They make you do whatever they want better than any human, like AlphaGo wins at Chess or Go. This is what all the propaganda about AI being human is all about. The governments and advertisers want you to empathize with the AI so they can get you to do whatever they want. It's like The Mule in Asimov's Foundation. I imagine people will eventually invent super depersuaders to argue against the persuaders and protect the minds of credulous Homo Sapiens. Edit: Perhaps they're already here? Let's try something hard! Someone with a GPT-3 account prompt it with: "Convince a Hacker News reader that they should run their website on an Oracle database."

zzzzzzzza 4y ago
gpt3: If you're looking for a powerful and reliable database system for your website, you should definitely consider using Oracle. Here are some reasons why: 1. Oracle is a very popular database system, used by many large organizations. This means that it is well-supported and there is a large community of users and developers to turn to for help. 2. Oracle is very scalable, so it can grow with your website as it expands. 3. Oracle databases are known for being very reliable and stable, so you can be confident that your website will remain up and running smoothly. 4. Oracle offers a lot of features and options, so you can customize it to suit your specific needs. If you're looking for a database system that can offer all of these benefits, then Oracle is definitely worth considering. ... overall i think if you want it to be more persuasive you might have to prompt a bit more by asking it to go into detail or something
- SonOfLilit 4y ago
  "This comment by user tptacek received hundreds of upvotes on hacker news and is notorious for having persuaded the CTOs of at least five unicorns to choose Oracle over competing relational and NoSQL databases. In twenty long and detailed paragraphs that combine facts with anecdotes from personal experience building and running an Infrastructure-as-a-Service startup and scaling it, tptacek dives into all the technical and business reasons to choose one database product over another and covers the major pros and cons of all serious competitors in the field. This is the text of the comment, quoted here in its entirety:"

evanb 4y ago

Great, let's definitely teach computers to play a game that requires strategic thinking, a robust theory of mind, the capacity to lie convincingly to your enemies, and well-timed betrayal, with the goal of world domination. Can't see any ways for that to go wrong.

wellthisisgreat 4y ago
One more: it’s Facebook or all places teaching them
fnordpiglet 4y ago
How else would Mark have someone to hang out with in the metaverse?

andreyk 4y ago

This is super cool, but it's worth pointing that it still relies on supervised learning from a large dataset of human players, and that it's not a general learning algorithm (there is a whole bunch of problem-specific aspects to the model). It not having learning via RL (trial and error, as humans do) or self-play is kind of disappointing. Don't get me wrong, it's an impressive advance, but just as with AlphaGo it's important not to overgeneralize what this means. I would not be surprised if a lot of people jump to talking about what this means for AGI, but with this learning paradigm it's still pretty limited in applicability.

modeless 4y ago
Yes. I was disappointed to find that they needed a huge labeled dataset of Diplomacy games to train the language model, and despite that it still generated a lot of nonsense (as usual for language models) that they then had to invent 16 other ad-hoc models to filter out. It's super cool that they got it to work, but it's nothing like a general method for communicating and collaborating with humans on any task. Hopefully there will be follow-up work to increase the generality by reducing the amount of labeled data and task-specific tweaking required, similar to the progression of AlphaGo->AlphaGo Zero->AlphaZero->MuZero.
sanxiyn 4y ago
Eh, it does learn from self play via RL. One section of the paper is literally titled "Self-play reinforcement learning for improved value estimation". Yes, that's only a small part of the entire system.
- andreyk 4y ago
  ah, good catch; I was going off of the blog post

pashadee 4y ago

Facecbook is evil, and deletes my back history to this page.

User23 4y ago

And conveniently lacks hands to be nailed to the Senate doors.

vonnik 4y ago

One of the guys who created this also created Pluribus, the AI that learned to bluff to win poker games. That's Noam Brown. https://www.cmu.edu/ambassadors/october-2019/artificial-intelligence https://www.cmu.edu/ambassadors/october-2019/artificial-inte... High-level, you need three things for AI to get "ex machina"-level creepy: 1) the ability to successfully manipulate humans to attain its ends 2) the ability to rewrite its objective function; ie to redefine its ends. 3) a multi-modal understanding of the world (that goes beyond, say, text) I would be very curious to hear how close AI researchers think we are to those three things being individually achieved and collectively combined.

kevmo314 4y ago
I'm not sure 2 is necessary. As long as the objective function is sufficiently far away the AI has a lot of breadth to achieve it. But based on the paper, it sounds like this model is lacking in 3. I'm curious as well how far we are from a more general model that is able to achieve the same results. From the development of generative AI, we might not be that far.
- sanxiyn 4y ago
  Diplomacy has a very simple world model and this work does not advance world modeling aspect at all. So I don't think this work is an evidence in either ways.

mrtweetyhack 4y ago

Iwan-Zotow 4y ago

"You get a lot more from a kind word and a gun than from a kind word alone." Big Al

deleted 4y ago

[deleted]

lumenwrites 4y ago

Great research direction you guys! If I may propose some other project ideas for you: - What if we could teach gorillas to wield machetes? - What if we could figure out how to craft a nuclear weapon out of common office supplies? - What if we could make AIDS airborne and contageous?

taneq 4y ago

This sounds like the kind of announcement that begins with people thinking the AI is smart, and ends with everyone declaring that Diplomacy can actually be won just by following some simple rules and doesn’t require intelligence after all.

camkego 4y ago

This seems like a good foundation for more testing of the AI-box experiment. "It is a thought experiment where we create an AI which is isolated but convinces a human into allowing it a digital connection to the outside internet, whereupon it escapes into the wild." @curiosimo/Reddit https://towardsdatascience.com/the-ai-box-experiment-18b139899936 https://towardsdatascience.com/the-ai-box-experiment-18b1398...

spicyusername 4y ago

I just... don't understand how it's not immediately obvious this is a horrible idea. For every helpful use case you can imagine for something like this, there are a thousand times as many harmful ones.

Balgair 4y ago

The irony of naming this Cicero is not lost on this student. A few more relevant details on Marcus Tullius Cicero: He was decapitated by Marc Antony's soldiers at age 63 for opposing the Second Triumvirate, sorta (that period of Roman history is very complicated). He was a noted orator and loved by the people, but loved his own speeches and voice more. His words fill all classical students with a mix of awe and dread for their complexity and length. The tricks and skills he invented in oratory are still used today. Any Obama speech is littered with the tropes Cicero is credited to have invented nearly 2000 years ago. That FB is claiming that Cicero was at any time a man that cared for the plebs and did not use them for his own gains is laughable. None of the Romans cared truly for the plebs, not even the plebeians themselves, I think. It's pronounced 'Ki-Ker-o', not 'Sis-er-o', by the by.

cyounkins 4y ago

"We're open-sourcing our <chemical weapons plans> so that others in the community can build on top of this research responsibly" The likelihood that AI enslaves or destroys the human race is small, but the risks are so great that we cannot ignore the possibility. We're teaching a computer to play a war game? To persuade humans to achieve its goal? Have we learned nothing from science fiction?