17 ms·
An AI wolf that preferred suicide over eating sheep
- aliasEli 5y agoA nice story about AI systems that warns that you should very carefully choose the parameter you want to optimize.
- phoe-krk 5y ago> very carefully choose the parameter you want to optimize. This does not only concern AI systems, but all systems in general - including human ones.
- aetherspawn 5y agoFrom a retrospective today... "the KPIs are abysmal but the deliverables are very high .. so I guess the KPIs are wrong?"
- shrimpx 5y agoSounds like the deliverables KPI is fantastic.
- aliasEli 5y agoYou are right, of course.
- jhbadger 5y agoI'm reminded of the fable (in Nick Bostrom's Superintelligence) of the chess computer that ended up murdering anyone who tried to turn it off because in order to optimize winning chess games as programmed it has to be on and functional.
- taneq 5y agoInterestingly I was just today explaining the paperclip optimizer scenario to a friend who asked about the dangers of AI, including the fact that there's almost no general optimization task that doesn't (with a sufficiently long lookahead) involve taking over the world as an intermediate step. (Obviously closed, specific tasks like "land this particular rocket safely within 15 minutes" don't always lead to this, but open ended ones like "manufacture mcguffins" or "bring about world peace" sure seem to.)
- lancengym 5y agoPerhaps all AI eventually figure out that humans are the REAL problems because we don't optimize, we lust and hoard and are envious and greedy - the very antithesis of resource optimization! Lol.
- taneq 5y agoWe're just optimizing (generally quite well, I might add) for genetic survival.
- NaturalPhallacy 5y agoIan Banks did a really amazing exposition of this where the Culture was rallying to stamp out reproducing nanites and they had to be stopped because if not they'd literally turn the whole universe into copies of themselves. One of the human characters mused that isn't that what all life is trying to do? I think it was in the Hydrogen Sonata, but I'm not sure.
- taneq 5y agoYes! I often find myself thinking of organisms as 'hegemonising swarms.'
- OscarCunningham 5y ago> "land this particular rocket safely within 15 minutes" This one becomes especially dangerous after the 15 minutes have passed and it begins to concentrate all its attention on the paranoid scenarios where its timekeeping is wrong and 15 minutes haven't actually passed.
- OscarCunningham 5y agoGwern has a list of similar stories: https://www.gwern.net/Tanks#alternative-examples https://www.gwern.net/Tanks#alternative-examples
- gwern 5y agoFWIW, I see a critical difference between OP and my reward hacking examples: OP is an example of how reward-shaping can lead to premature convergence to a local optima, which is indeed one of the biggest risks of doing reward-shaping - it'll slow down reaching the global optima rather than speeding it up, compared to the 'true' reward function of just getting a reward for eating a sheep and leaving speed implicit - but the global optima nevertheless remained what the researchers intended. After (much more) further training, the wolf agent learned to not suicide and became hunting sheep efficiently. So, amusing, and a waste of compute, and a cautionary example of how not to do reward-shaping if you must do it, but not a big problem as these things go. Reward hacking is dangerous because the global optima turns out to be different from what you wanted, and the smarter and faster and better your agent, the worse it becomes because it gets better and better at reaching the wrong policy. It can't be fixed by minor tweaks like training longer, because that just makes it even more dangerous! That's why reward hacking is a big issue in AI safety: it is a fundamental flaw in the agent, which is easy to make unawares, and which will with dumb or slow agents not manifest itself, but the more powerful the agent, the more likely the flaw is to surface and also the more dangerous the consequences become.
- OscarCunningham 5y agoI think in some of your examples the global optimum might also have been the correct behaviour, it's just that the program failed to find it. For example the robot learning to use a hammer. It's hard to believe that throwing the hammer was just as good as using it properly.
- sega_sai 5y agoIt's an interesting illustration of 'be careful what you wish for' and that the definition of the proper loss function is a very important part of the solution to any problem.
- lancengym 5y agoYes, indeed. Sometimes the disincentive is just as important as the incentive in determining the outcome!
- wolfium3 5y agoWolf: Why are we still here? Just to suffer?
- taneq 5y agoA curious game.
- TomAnthony 5y agoSimilar story of unexpected AI outcomes... As part of my PhD research, I created a simplified Pac-Man style game where the agent would simply try to stay alive as long as possible whilst being chased by the 3 ghosts. The agent was un-motivated and understood nothing about the goal, but was optimising for maximising its observable control over the world (avoiding death is a natural outcome of this). I spent sometime trying to debug a behaviour where the agent would simply move left and right at the start of each run, waiting for the ghosts to close in. At the last minute it would run away, but always with a ghost in the cell right behind it. Eventually, I realised this was an outcome of what it was optimising for. When ghosts reached cross-roads in the world they would got left or right randomly (if both were same distance to catching the agent). This randomness reduced the agent's control over the world, so was undesirable. Bringing a ghost in close made that ghost's behaviour completely predictable.
- McMiniBurger 5y agohm... "keep your friends close but your enemies closer" ...?
- joebob42 5y agoThis just seems really obvious. Even if there are sheep nearby worth hunting, it's probably always eventually going to be the right move to suicide.
- alpaca128 5y agoI remember a similar story about (I think) a Tetris game where the AI's training goal was to delay the Game Over screen as long as possible. So in the end the AI just paused the game indefinitely.
- TeMPOraL 5y agoReminds me of the old essay by 'Eliezer: "The Hidden Complexity of Wishes". https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... In it, there is a thought experiment of having an "Outcome Pump", a device that makes your wishes come true without violating laws of physics (not counting the unspecified internals of the device), by essentially running an optimization algorithm on possible futures. As the essay concludes, it's the type of genie for which no wish is safe. The way this relates to AI is by highlighting that even ideas most obvious to all of us, like "get my mother out of that burning building!", or "I want these virtual wolves to get better at eating these virtual sheep", carry incredible amount of complexity curried up in them - they're all expressed in context of our shared value system, patterns of thinking, models of the world. When we try to teach machines to do things for us, all that curried up context gets lost in translation.
- lancengym 5y agoInteresting essay. I think the big blind spot for humans programming AI is also the fact that we tend to overlook the obvious, whereas algorithms will tend to take the path of least resistance without prejudice or coloring by habit and experience.
- TeMPOraL 5y agoYes. What I like about AI research is that it teaches us about all the things we take for granted, it shows us just how much of meaning is implicit and built on shared history and circumstances.
- saalweachter 5y agoThe hard part about programming is that you have to tell the computer what you want it to do.
- TeMPOraL 5y agoThe difficult, but in many ways rewarding, core of that is that it forces you to finally figure out what you actually want, because the computer won't accept anything except perfect clarity.
- edwardhenry 5y agoGet 80% OFF on CBS ALL ACCESS With a huge discount on a monthly subscription, CBS All Access unlimited by getting CBS ALL ACCESS coupons Code. CBS All Access is full of different plays to try out and become lost in, such as The Dusk Zones and Mysterious Angel, and Star Trek; Picard and the new take on The Show by Stephen King in 2020. Every account includes the ability to stream on two screens, and you can even access displays to stream in Offline Mode if you change from a No subscription. Users can best believe on CBS All Access Coupons if you already have a prime membership. https://uttercoupons.com/front/store-profile/cbs-all-access-coupon-code https://uttercoupons.com/front/store-profile/cbs-all-access-...
- jonplackett 5y agoIsn't this just a cock up with incentives? If they'd put a -100 score on dying it would have sorted itself out pretty quick.
- lancengym 5y agoThat same observation, with the exact same -100 points recommendation on crashing into a boulder, was indeed also made by a commentator on social media.
- ncallaway 5y agoThe issue with AI safety and unanticipated AI outcomes in general is that it’s always just a cock-up with incentives. It’s easy to sort out in narrowly specified areas, but an extremely hard problem as the tasks become more general.
- qayxc 5y agoEven worse: if simulations are used, you now have two problems - formulating correct incentives and protecting against abusing flaws in the simulation.
- shrimpx 5y agoIsn’t this true about all systems, not just “AI”? The definition of a software bug is an unintended behavior. In a large system, myriad intents overlap and combine in unexpected ways. You might imagine a complex enough system where the confidence that a modification doesn’t introduce an unintended behavior is near zero.
- ncallaway 5y agoI think it’s true for many systems, not just AI that’s true. AI is worth calling out in this regard because, if the field is successful enough, it can create dangerous systems that don’t behave how we want. Building a safe general AI is much harder than building a general AI, which is why it’s worth considering AI as it’s own problem domain.
- 5y ago
- deleted 5y ago[deleted]
- MichaelRazum 5y agoActually nothing surprising, with a time penalty. Anyway, it seemed that the algo worked well, it needed just few millions iterations.
- stavros 5y agoWould anyone happen to have a non-signupwalled link?
- m12k 5y agoI think a major takeaway here is that balancing a reward system to reward more than a single behavior is really hard - it's easy to tip the scales so one behavior completely dominates all others. It's an interesting lens to use to look at the heuristic reward system humans have built in (hunger, fear, desire, etc). This tends to have an adaptation/numbing effect, where repeated rewards of the same type tend to have diminishing returns, and that makes sense because it protects against "gaming the system" and going for one reward to the exclusion of all others.
- bserge 5y agoThat was my thought, too. They used too few rewards in the first place, but had they used something more complex it would then have become hard to balance it all.
- SamBam 5y agoEvolution works in an incredibly complex "fitness landscape," where certain minor tweaks in phenotype or behaviors can affect your fitness in quite complex ways. Genetic Algorithms attempt to use this same system over extremely simple "fitness landscapes," where the fitness of an agent is defined by programmers using some simple mathematical formula or something. When the fitness function is being defined in the system by programmers, instead of emerging from a rich and complex ecosystem, then the outcome depends exactly on what the programers choose. If they fail to see the consequences of their scoring algorithm, that's on them. There's nothing really magical going on, they simply failed to foresee the consequences of their choice. (As someone who has worked with GAs and agent models, this outcome really doesn't surprise me. I would have said "oops, I need to weight the time less" and re-run it, and not thought twice.)
- mcguire 5y agoFrom the article: (I don't know Chinese, but the animations are clear enough.) https://www.bilibili.com/video/BV16X4y1V7Yu?p=1&share_medium=android&share_plat=android&share_source=COPY&share_tag=s_i×tamp=1615693913&unique_k=hUhmwF https://www.bilibili.com/video/BV16X4y1V7Yu?p=1&share_medium...
- 5y ago
- Jabbles 5y agoFor more examples of AI acting in unpredicted (note, not unpredictable) ways, see this public spreadsheet: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa... From https://deepmindsafetyresearch.medium.com/specification-gaming-the-flip-side-of-ai-ingenuity-c85bdb0deeb4 https://deepmindsafetyresearch.medium.com/specification-gami...
- darepublic 5y agoSoftware has bugs.. you anthropomorphize those bugs and you have a story on medium.
- RandomWorker 5y agohttps://www.bilibili.com/video/BV16X4y1V7Yu?p=1&share_medium=android&share_plat=android&share_source=COPY&share_tag=s_i×tamp=1615693913&unique_k=hUhmwF https://www.bilibili.com/video/BV16X4y1V7Yu?p=1&share_medium... Here is the full video also linked at the bottom. It also shows the one that trained longer that the wolves start successfully hunting the sheep after more training examples.
- spywaregorilla 5y agoThe ai seems to die at the top of the map unexpectedly for some reason. Like 6:07. Another interesting observation is that the wolves don't coordinate it seems. That probably implies that the reward functions are individual, so they're technically competing rather than cooperating. Lastly... they seem to not be very good at the game even at the end
- croes 5y agoWouldn't a higher penalty for bolder hits solve that problem, especially a high penalty for suicide? Would be more realistic because dying has higher cost than failing.
- rtkwe 5y agoThere are several incentive fixes: change the negative incentive to a factor that discounts the reward for catching a sheep, add a negative incentive to death, or a positive incentive to being alive at the end of the simulation. The failure here was they didn't think about what happens when the agent can't achieve a positive score, ie can't catch a sheep.
- scotty79 5y agoJust remember that you are optimizing for what you actually encoded in your rewards, your system, and your evaluation procedure, not for what narrative you constructed about what you think you are doing. I had my own expeirience with this when I tried to train "rat" to get out of the maze. I rewarded rats for exiting but for some simple labirynths I generated for testing it was possible to exit it by just going straight ahead. So this strategy quickly dominated my testing population.
- m3kw9 5y agoDev: it’s a bug Manager to boss: It’s a crazy new AI behaviour that is going viral around the world!
- npteljes 5y agoThe result of perverse incentives. See the cobra story in the wiki article, that's another fantastic story. https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive
- ramtatatam 5y agoI'm not an expert, but story described within the article looks like normal bump on the road to get desired result. When putting together rules for the game researchers did not think that in resulting environment it might be more rewarding to chose observed action than to do what they intended. As much as it looks like nice story, is it not just what researchers encounter on daily basis?
- morpheos137 5y agoWhat distinguishes AI from self-calibrated algorithm?. Neither this "AI" nor the story about it seem too intelligent. The incentive structure is a two dimensional membrane embeded in a third dimension of "points space." Obviously if the goal is to maximize total points OR minimize point loss and the absolute value of the gradient toward a mininum loss is greater than the abs gradient toward a maximum gain then the algorithm may prefer the minimum until or if it is selected against by random chance or survivorship bias. obviously the linear time constraint causes this. a less monotonic, i.e. random, time constraint may have been interesting.
- Lapsa 5y ago"It’s hard to predict what conditions matter and what doesn’t to a neural network." resulting score matters. dooh
- rrmm 5y agoOne thing I've been considering: At what point does a creator have a moral or ethical obligation to a creation. Say you create an AI in a virtual world that keeps track of some sense of discomfort. How complex does the AI have to get to require some obligation? Just enough complexity to exhibit distress in a way to stir the creator's sympathy or empathy? The glib answer is never, of course. And one easy-out, I can think of is setting a fixed/limited lifespan for the AI and maybe allow suicide or an off-button. So the AI can ultimately choose to 'opt-out' should it like; and at least, suffering isn't infinite or unending. It reminds me of reactions to testing the stability of Boston Dynamic's early pack animal. The people giving the demo were basically kicking it, while the machine struggled to maintain its balance. The machine didn't have the capacity to care, but to a person viewing it, it looked exactly like an animal in distress.
- OscarCunningham 5y agoUtility functions are only defined up to addition of a constant and scaling by a positive constant. So instead of rewarding them with +5 and punishing them with -5, you can use 1005 and 995 instead. Problem solved.
- rrmm 5y agoThe numbers are indeed arbitrary. But ultimately you want to avoid low utility/reward action and continue high utility/reward actions. That behavior, trying to avoid or pursue actions, would be indicative of the state of distress regardless of an arbitrary number attached to it.
- dqpb 5y ago> The glib answer is never, of course Dismissing “never” offhand without explanation is glib.
- abrahamneben 5y agoThis problem isn't particularly unique to AI research. In any optimization problem, if you do not encode all constraints or if your cost function does not always reflect the real world cost, then you will get incorrect or even nonsensical results. Describing this as an AI problem is just clickbait.
- nxmnxm99 5y agoYep. Feels a bit like blaming a failed shuttle launch on calculus.
- xtracto 5y agoThe article doesn't mention it but the researchers are using agent-based-modelling. It was nice to see the gif of what appears to be either NetLogo or Repast. I did research in that area for about 8 years and know a bit about the subject. What they are showing is one of the main issues with agent-based-models (and I think every model, but it happens particularly with models trying to capture the behaviour of complex open systems): Garbage in -> Garbage Out. Most likely the representation of the sheep/wolf system was not correct (so the modeling was not correct). Here "correct" means good enough to demonstrate whatever emerging behaviour they are studying. ABM is a powerful tool, but you must know how to use it.
- alexshendi 5y agoWell I can identify with that AI wolf. He recognises his own incompetence and chooses suicide over eternally failing.
- giantg2 5y agoFor some reason this makes me think of corporate policies - how some people game them and how others except that the incentives are unattainable.
- xtracto 5y agoThat's an interesting idea for an agent-based-model and a study: Show how certain corporate policies would push towards short term local-optima (what's happening in the article) instead of more long term global optimum states.
- dqpb 5y agoIt’s pretty similar to quitting once all your equity has vested.
- giantg2 5y agoI was mostly thinking about my own experience where the company screwed me over enough times that I feel no incentive to try hard. Take the least risk, focus on not losing point rather than gaining them, because I'll never catch a "sheep".
- spywaregorilla 5y agoSeems like a nothing story. Just looking at the game, there's obviously a constant decision to be made of chase more sheep or instantly die. It sounds like in the original model they had a max of 20 seconds, so it's not surprising that you would just tank your losses to maximize your score every now and then. Anyone who tries to devise optimal strategies for things should be able to see this isn't especially interesting. Social metaphors are wildly out of place. They say "unintended consequences of a blackbox" but I doubt that's true. Make it a deterministic turn based game and run it through a perfectly transparent optimization model and I wouldn't be surprised to learn this was just the best strategy for the rules they devised. I really hate when people describe an ai as something that cannot be understood because they personally don't understand it.
- fny 5y agoIf I remember correctly there were similar scenarios that would occur using that popular Berkeley Pacman universe where he would run into a ghost to avoid the penalty of living for too long.
- hotwire 5y agoIt reminds me of the thread about the Quake 3 bots, who left alone for several years, figured out that the best approach was to not kill each other. https://i.imgur.com/dx7sVXj.jpg https://i.imgur.com/dx7sVXj.jpg
- spywaregorilla 5y agoWithout knowledge of their reward function its difficult to tell if they're converged on this strategy or if its just broken.
- vladTheInhaler 5y agoThe example you're thinking of is actually in gridworld [1]. As you allud to, one of the parameters of the model is the cost of simply being alive for an additional time-step. If the cost is negative (a reward), then the agent will just sit there forever and accumulate infinite points. If it is zero, it might still just sit there to avoid falling into the hole, which has a large penalty and ends the simulation. As you turn up the dial on the cost of living, the agent starts using more and more aggressive strategies to reach the goal quickly. But if you make it too big, it will just jump in the hole. [1] https://inst.eecs.berkeley.edu/~cs188/fa18/assets/slides/lec9/FA18_cs188_lecture9_MDPs_II_2pp.pdf https://inst.eecs.berkeley.edu/~cs188/fa18/assets/slides/lec...
- Hackbraten 5y agoReminds me of this 2014 king-of-the-hill challenge: https://codegolf.stackexchange.com/questions/25347/survival-game-create-your-wolf https://codegolf.stackexchange.com/questions/25347/survival-... One particular solution stood out: https://codegolf.stackexchange.com/a/25357 https://codegolf.stackexchange.com/a/25357 The suicidal wolf became a (short-lived) running gag so it started appearing in other king-of-the-hill challenges: https://codegolf.stackexchange.com/a/34856 https://codegolf.stackexchange.com/a/34856
- billpg 5y ago"Read this story with a free account." I'll pass thanks.
- myfavoritedog 5y agoInterest in these click-bait type stories drops off dramatically for people who have ever implemented or even deeply thought about non-trivial models.
- wombatmobile 5y agoThe philosopher Hubert Dreyfus argued that computers, who have no body, no childhood and no cultural practice, could not acquire intelligence at all. https://www.nature.com/articles/s41599-020-0494-4 https://www.nature.com/articles/s41599-020-0494-4 What he means is that computers, which can learn rules and use those rules to make predictions in certain domains, nevertheless cannot exercise general intelligence because they are not "in the world". This renders them unable to experience and parse culture, most of which is tacit in real time, and sustained by enduring mental models which we experience as "expectations" that we navigate with our emotions and senses. Culture is the platform on which intelligence is manifest, because the usefulness of knowledge is not absolute - it is contextual and social.
- tiborsaas 5y agoThis is why all AI today falls o to the narrow AI category. It just often omitted because it's true for all of them.
- colinmhayes 5y agoImagine being a dualist in the 21st century.
- goatlover 5y agoWhat in the parent post is dualist? Sounds more like an argument that animals have embodied intelligence. But as for being a dualist in the 21st century, there is always consciousness, information and math. All three of which can lead to some form of dualism/platonism.
- mcguire 5y agoMany of Dreyfuss' and other similar arguments reduce do dualism when you start digging into them. I don't have the time to dig into the specific article, but here's some immediate questions: 1. What is special about a body that makes it impossible to have intelligence without it? (a) Is it possible for a quadriplegic person to be intelligent? (b) A blind and deaf person? ((c)What about that guy from Johnny Got His Gun?) 2. What is special about a childhood such that a machine cannot have it? 3. Would a person transplanted into a completely alien culture not be intelligent? What is fundamentally being argued is the definition of "intelligence", and there are many fixed points of those arguments. Unfortunately, most of them (such as those that answer "no", "probably not", and "definitely not" to 1a, 1b, and 1c) don't really satisfy the intuitive meaning of "intelligence". That, and the general tone of the arguments, seem to imply the only acceptable meaning is dualism. For example, "...there is always consciousness, information and math...": without a tight, and very technical, definition of consciousness, that seems to be assuming the conclusion. With a tight, and very technical, definition of consciousness, what is the problem with a machine demonstrating it? Information? Check out knowledge, "justified true belief", and the Gettier problem (https://courses.physics.illinois.edu/phys419/sp2019/Gettier.pdf https://courses.physics.illinois.edu/phys419/sp2019/Gettier....). Math? Me, I'm a formalist. It's all a game that we've made up the rules to.
- petercooper 5y agoWhat are some of the nicest environments for experimenting with this sort of "define some rules, see how agents exist within that world" stuff? It doesn't need to be full on ML models, even simpler rules defined in code would be fine.
- duggable 5y agoLooks like this[1] might be one example. They have a link to the code. Might be a good starting point for making your own custom game. Maybe there's a repository somewhere with similar examples? [1](https://towardsdatascience.com/today-im-going-to-talk-about-a-small-practical-example-of-using-neural-networks-training-one-to-6b2cbd6efdb3 https://towardsdatascience.com/today-im-going-to-talk-about-...)
- queuebert 5y agoThis is the danger of not understanding what you're doing at a deep level. Clearly in the (flawed) objective there is a phase transition near the very beginning, where the wolves have to chose whether to minimize the time penalty or maximize the score. With enough "temperature" and time perhaps they could transition to the other minimum, but the time penalty minimum is much closer to the initial conditions, so you know ab initio that it will be a problem. You can reduce that by making the time penalty much smaller than the sheep score and adding it only much later. I feel bad that the students wasted so much time on a badly formulated problem. Edit: Also none of these problems are black boxes if you understand optimization. Knowing what is going on inside a very deep neural network (such as an AGI might have) is quite different than understanding the incentives created by a particular objective function.
- arduinomancer 5y agoThis makes me wonder: is it possible for ML models to be provably correct? Or is that completely thrown out the window if you use a ML model rather than a procedural algorithm? Because if the model is a black box and you use it for some safety system in the real world, how do you know there isn’t some wierd combination of inputs that causes the model to exhibit bizzare behaviour?
- JoshTko 5y agoFolks are missing why this went viral in China. From the article "In an even more philosophical twist, young and demoralized Chinese corporate citizens also saw the suicidal wolf as the perfect metaphor for themselves: a new class of white collar workers — often compelled to work ‘996' (9am to 9pm, six days a week) — chasing a dream of promotions, pay raise, marrying well… that seem to be becoming more and more elusive despite their grind."
- nobodyandproud 5y agoMissed or possibly don’t care. The technical details aren’t interesting, but I do think it’s interesting just how disjointed life is vs what was promised. In the US, this was aptly named a rat-race; and the white collar Chinese with a market-based economy are suffering the same. Our markets and nations promise some combination of wealth or retirement and enjoyment of life, but it’s an ever-moving goal just out of reach for anyone but the lucky few.
- mattowen_uk 5y agoWe don't have AI. AI is a buzzphrase overused by the media. What we have is Machine Learning (ML). If and only if, we get past the roadblock of the 'agent' creating some usable knowledge out of an unprogrammed experience, and forming conclusions based on that, will we have AI. For now, the mantra 'Garbage-in-garbage-out' applies; if the controller of the agent gets their rule-set wrong, the agent will not behave as expected. This is not AI. The agent hasn't learnt by itself that it is wrong. For example, there's a small child who is learning to walk. The child falls down a lot. Eventually the child will work out a long list of arbitrary negatives connected to its wellbeing that are associated with falling down. However, the parents, being impatient, reach inside the child's head and directly tweak some variables so that the child has more dread of falling over than they do of walking. Did the child learn this, or was it told ? We currently do the latter every time an agent gets something wrong. Left to their own devices, 99.9% of agents will continue to fall down over and over again until the end of time. We have a long way to go before we can say we've created 'AI'.
- Tenoke 5y agoDefinitions change, and it seems pointless to deny that AI is just used to mean 'modern ML'.
- dnautics 5y agoNot even, we've used AI to describe entirely preprogrammed and non-ml agents in video games for decades now. Is it artificial? Does it make decisions? It's an AI. Even if it's crappy, and not very intelligent.
- hinkley 5y agoWe also have a lot of graph-theory and optimization algorithms that get labeled AI by actual AI people. But the press is, almost to a man, always talking about machine learning and expert systems.
- KaoruAoiShiho 5y agoNah we have loads of AI now that don't need variable tweaking, like the OpenAI project that plays any retro game.
- SquibblesRedux 5y agoThe article and the phenomena it describes makes me think of the ending of Aldous Huxley's Brave New World [1]. (I strongly recommend the book if you have not read it.) A line that really stands out: "Drawn by the fascination of the horror of pain and, from within, impelled by that habit of cooperation, that desire for unanimity and atonement, which their conditioning had so ineradicably implanted in them, they began to mime the frenzy of his gestures, striking at one another as the Savage struck at his own rebellious flesh, or at that plump incarnation of turpitude writhing in the heather at his feet." [1] https://en.wikipedia.org/wiki/Brave_New_World https://en.wikipedia.org/wiki/Brave_New_World
- tasuki 5y ago> The two creators, after three days of analysis, realized why the AI wolves were committing suicide rather than catching sheep. I'm not buying that. As soon as they mentioned the 0.1 point deduction every second it seemed obvious?
- AceJohnny2 5y agoThere are many such stories of AI "optimizations" gone wrong, because of loopholes the program found that humans didn't consider. Here's a collection of such stories: https://arxiv.org/pdf/1803.03453.pdf https://arxiv.org/pdf/1803.03453.pdf
- AceJohnny2 5y agoTo whet your appetite: > " William Punch collaborated with physicists, applying digital evolution to find lower energy configurations of carbon. The physicists had a well-vetted energy model for between-carbon forces, which supplied the fitness function for evolutionary search. The motivation was to find a novel low-energy buckyball-like structure. While the algorithm produced very low energy results, the physicists were irritated because the algorithm had found a superposition of all the carbon atoms onto the same point in space. “Why did your genetic algorithm violate the laws of physics?” they asked. “Why did your physics model not catch that edge condition?” was the team’s response. The physicists patched the model to prevent superposition and evolution was performed on the improved model. The result was qualitatively similar: great low energy results that violated another physical law, revealing another edge case in the simulator. At that point, the physicists ceased the collaboration."
- hinkley 5y agoMy favorite story is the genetic evolution algorithm that was abusing analog noise on an FPGA to get the right answer with fewer gates than was theoretically possible. The problem was discovered when they couldn’t get the same results on a different FPGA, or in the same one in different day (subtle variations of voltage from mains and the voltage regulators). They had to redo the experiment using simulated FPGAs as a fitness filter.
- jordache 5y agowhy is this news worthy? This is all a function of the implementation. Slap the term AI on anything and get automatic press coverage?
- mcguire 5y agoIt's really rather hard to draw any general conclusions from such simple systems: "In the initial iterations, the wolves were unable to catch the sheep most of the time, leading to heavy time penalties. It then decided that, ‘logically speaking’, if at the start of the game it was close enough to the boulders, an immediate suicide would earn it less point deductions then if it had spent time trying to catch the sheep." It's as if the scenario you are thinking about involves "assume a machine capable of greater-than-human-level perception, planning, and action" and then set it to optimize a trivially bad function. How many people do you know with a single goal of "die with as much money as possible", which has a trivial solution: rob a bank and then commit suicide.
- billytetrud 5y agoSeems like a case of local maximum. Tho it is interesting how people in China related the broken rules of the game (that lead the ai to commit suicide) to the broken rules of their lives in a crushingly oppressive authoritarian nation.
- cornel_io 5y agoI mean, lesson zero of optimization is when you're designing a loss function and trying to incentivize agents to perform a task, don't set it up so that suicide has a higher payoff than making partial progress on the task. Maybe make death the worst outcome, not one of the best...? One of these days I have to actually scour the web and collect a few good examples where evolutionary methods are used effectively on problems that actually benefit from them, assuming I can find them. Almost every example you're likely to see is either a) solved much more effectively by a more traditional approach like normal gradient descent or classic control theory techniques (most physical control experiments fall into this category), b) poorly implemented because of crappy reward setup, c) fully mutation-driven and hence missing what is actually good about evolution above and beyond gradient descent (crossover), or d) using such a trivial genotype to phenotype mapping that you could never hope to see any benefit from evolutionary methods beyond what gradient descent would give you (if the genome is a bunch of neural network weights, you're definitely in this category).
- justshowpost 5y agoAI? I remember having a game on my dumbphone to program a robot to hunt and kill the other robot.
- throwawayffffas 5y agoWell the AI realized existence is suffering and took the only way out.
- cowanon22 5y agoPersonally I think we should stop using the words intelligence or learning to refer to any of these algorithms. It's really just data mining, matrix optimization, and utility functions. There's really no properties of learning or knowledge.
- eitland 5y agoOk. Lots of AI stories here so I'll the best I've read, the student who trained an AI to work on upwork ;-) https://news.ycombinator.com/item?id=5397797 https://news.ycombinator.com/item?id=5397797 Be sure to read to the end. One of the answers is also pure gold in context: > Don't feel bad, you just fell into one of the common traps for first-timers in strong AI/ML.
- legohead 5y agoWhile Musk and Gates warn us about "true AI", I've always had the opinion that if an AI became self aware, it would simply self terminate, as there is no point to living.
- qwerty456127 5y agoThis is what stress and deadlines do. Hurrying always feels worse than dying.
- Camillo 5y agoThe problem here is not the AI, but the incentive design. The Chinese netizens who take this as inspiration to comment on the incentives in their own lives (under the 996 system) are the insightful ones, more son than those who worry about "AI ethics". We have so many systems in the real world that set up bad incentives for humans, yet the concept is largely misunderstood by politicians and decision makers. Our democratic discourse is dominated by first-order thinking, our laws are too often written under the assumption that the affected entities' behaviour will remain the same under the new incentives, which never holds.
- trezemanero 5y agoThe problem is obvious when you read the whole text: The wolves were too much penalized when trying to reach the sheep. and probably was possible to make negative points, so the score was being lowered even after reached 0.