5 ms·

I really wish I still had the link, but there used to be a website that listed a bunch of times in which machine learning was used (mostly via reinforcement lea

by viccis 2y ago

I really wish I still had the link, but there used to be a website that listed a bunch of times in which machine learning was used (mostly via reinforcement learning) to teach a computer how to play a video game and it ended up using perverse strategies that no human would do. Like exploiting weird glitches (https://www.youtube.com/watch?v=meE5aaRJ0Zs https://www.youtube.com/watch?v=meE5aaRJ0Zs shows this with Q*bert) Closest I've found to the old list I used to go to is this: https://heystacks.com/doc/186/specification-gaming-examples-in-ai---master-list https://heystacks.com/doc/186/specification-gaming-examples-...

porridgeraisin 2y ago
The recent nvidia AI cuda engineer was also similar: https://news.ycombinator.com/item?id=43113941 https://news.ycombinator.com/item?id=43113941
robertjpayne 2y ago
Make no mistake most humans will exploit any glitches and bugs they can find for personal advantage in game. It’s just machines can exploit timing bugs better.
- Muromec 2y ago
  Some people are able to do frame perfect inputs semi consistently from what I understand. I don’t understand how, as my own performance is around hitting 100ms window once, every other time
  - TeMPOraL 2y ago
    Maybe they have better equipment? If you're using a typical PC (or $deity forbid, a phone) with a typical consumer OS, there's several sources of variability between your controller and the visual feedback you receive from the game, each of which could randomly introduce delays on the order of milliseconds or more. That "randomly" here is the key phrase - lag itself is not a problem, the variability is.
    - Muromec 2y ago
      Better equipment or not, frame-perfect input is just hard to do and I'm impressed with people being able to do it.
genewitch 2y ago
on youtube, codebullet remakes games so that he can try different AI techniques to beat them.
y33t 2y ago
My favorite example was a game of pong with the goal of staying alive as long as possible. One ML algo just paused the game and left it like that.
- voidUpdate 2y ago
  Is that Learnfun/Playfun that tom7 made? That one paused just before losing on tetris and left it like that, because any other input would make it lose
  - y33t 2y ago
    No I want to say this was ~10 years ago. Happened to a university researcher IIRC.
- chefandy 2y ago
  My favorite was the ML learning how to optimally make the lowest-impact landing in a flight simulator— it discovered that it could wrap the impact float value if the impact was high enough so instead of figuring out the optimal landing, it started figuring out the optimal path to the highest-impact crashes.
  - hammock 2y ago
    This comment ought to be higher up. Such a perfect summary of what I have struggled to understand, which is the “danger” of AI once we allow it to control things And yes you can fix the bug but the bike wheel guy shows you there will always be another bug. We need a paper/proof that invents a process that can put an AI-supported (non human intervention) finite cap or limiter or something on the possible bug surface
    - aurbano 2y ago
      Is AI the danger, or is our inability to simplify a problem down to an objective function the problem? If anything, AI could help by "understanding" the real objective, so we don't have to code these simplified goals that ML models end up gaming no?
GranPC 2y ago
I'm almost 100% sure this is the link you're looking for: https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3Hs...
- viccis 2y ago
  Haha that was actually the same one I posted in my comment. This was some old website. A coworker sent it to me on Hipchat at my previous job about 10 years ago. And finding anything online older than like 5 years is nearly impossible unless you have the exact URL on hand.
  - GranPC 2y ago
    Oh sorry! I recognized the description but since I recalled mine being a Google Sheets link, I just went straight into search mode - and yep, it actually took me a bit to find.
szvsw 2y ago
There’s a few very cool examples where someone recently used RL to solve trackmania, and ends up having to add all sorts of constraints/penalties to prevent extremely strange exploits/glitches that are discovered IIRC… been a while since I watched. https://youtu.be/Dw3BZ6O_8LY?si=VUcJa_hfCxjZhhfR https://youtu.be/Dw3BZ6O_8LY?si=VUcJa_hfCxjZhhfR https://youtu.be/NUl6QikjR04?si=DpZ-iqVdqjzahkwy https://youtu.be/NUl6QikjR04?si=DpZ-iqVdqjzahkwy
- hnuser123456 2y ago
  Well, in the case of the latter, there was a vaguely known glitch for driving on the nose that allowed for better speeds than possible on 4 wheels, but it would be completely uncontrollable to a human. He figured out how to break the problem down into steps that the NN could gradually learn piecewise, until he had cars racing around tracks while balancing on their nose. It turned out to have learned to keep the car spinning on its nose for stability, and timing inputs to upset the spinning balance at the right moment to touch the ground with the tire to shoot off in a desired direction. I think the overall lesson is that, to make useful machine learning, we must break our problems down into pieces small enough that an algorithm can truly "build up skills" and learn naturally, under the correct guidance.
- shagie 2y ago
  One of the games that I stumbled across on Steam recently was "AI Learns to Drive" - https://store.steampowered.com/app/3312030/AI_Learns_To_Drive/ https://store.steampowered.com/app/3312030/AI_Learns_To_Driv... It's a neat toy (not really "useful" nor too much of a "game") for generating interest in how neural nets work.
Muromec 2y ago
> using perverse strategies that no human would do Of course we do use perverse strategies and glitches in adversarial multiplayer all the time. Case in point chainsaw glitch, tumblebuffs, early hits and perfect blocks in Elden Ring
matsemann 2y ago
In my thesis many years ago [0] I used EAs to build bicycle wheels. They were so annoyingly good at exploiting whatever idiosyncrasies in my wheel-simulator. Like, the first iterations of my simulator it managed to evolve wheels that would slowly oscillate due to floating point instability or something, and when applied forces to it would increase and increase until the whole simulator exploded and the recorded forces were all over the place, of course then out-competing any wheel in at least some objective dimension. After fixing those bugs, I mostly struggled with it taunting me. Like building a wheel with all the spokes going from the hub and straight up to the rim. It of course would break down when rolling, but on the objective of "how much load can it handle on the bike" it again out-competed every other wheel, and thus was at the pareto-front of that objective and kept showing up through all my tests. Hated that guy, heh. I later changed it to test all wheels in at least 4 orientations, it would then still taunt me with wheels like (c) in this figure[1], exploiting that. [0]: https://news.ycombinator.com/item?id=10410813 https://news.ycombinator.com/item?id=10410813 [1]: https://imgur.com/a/LsONTGc https://imgur.com/a/LsONTGc
- deleted 2y ago
  [deleted]
elzbardico 2y ago
For the model, the weird glitches are just another element of the game. As they can't reason, have no theory of world or even any real knowledge of what is doing, the model don't have the prior assumptions a human would have about how the game is supposed to be played. If you think about it, even using the term "perverse" is a result of us antropomorphizing any object in the universe that does anything we believe is on the realm of things humans do.
rdlw 2y ago
Not quite what you're describing, but no one has yet linked the classic Tom7 series where he applies deep learning to classic NES games: https://youtu.be/xOCurBYI_gY https://youtu.be/xOCurBYI_gY

by viccis 2y ago

porridgeraisin 2y ago

The recent nvidia AI cuda engineer was also similar: https://news.ycombinator.com/item?id=43113941 https://news.ycombinator.com/item?id=43113941

robertjpayne 2y ago

Make no mistake most humans will exploit any glitches and bugs they can find for personal advantage in game. It’s just machines can exploit timing bugs better.

Muromec 2y ago
Some people are able to do frame perfect inputs semi consistently from what I understand. I don’t understand how, as my own performance is around hitting 100ms window once, every other time
- TeMPOraL 2y ago
  Maybe they have better equipment? If you're using a typical PC (or $deity forbid, a phone) with a typical consumer OS, there's several sources of variability between your controller and the visual feedback you receive from the game, each of which could randomly introduce delays on the order of milliseconds or more. That "randomly" here is the key phrase - lag itself is not a problem, the variability is.
  - Muromec 2y ago
    Better equipment or not, frame-perfect input is just hard to do and I'm impressed with people being able to do it.

genewitch 2y ago

on youtube, codebullet remakes games so that he can try different AI techniques to beat them.

y33t 2y ago

My favorite example was a game of pong with the goal of staying alive as long as possible. One ML algo just paused the game and left it like that.

voidUpdate 2y ago
Is that Learnfun/Playfun that tom7 made? That one paused just before losing on tetris and left it like that, because any other input would make it lose
- y33t 2y ago
  No I want to say this was ~10 years ago. Happened to a university researcher IIRC.
chefandy 2y ago
My favorite was the ML learning how to optimally make the lowest-impact landing in a flight simulator— it discovered that it could wrap the impact float value if the impact was high enough so instead of figuring out the optimal landing, it started figuring out the optimal path to the highest-impact crashes.
- hammock 2y ago
  This comment ought to be higher up. Such a perfect summary of what I have struggled to understand, which is the “danger” of AI once we allow it to control things And yes you can fix the bug but the bike wheel guy shows you there will always be another bug. We need a paper/proof that invents a process that can put an AI-supported (non human intervention) finite cap or limiter or something on the possible bug surface
  - aurbano 2y ago
    Is AI the danger, or is our inability to simplify a problem down to an objective function the problem? If anything, AI could help by "understanding" the real objective, so we don't have to code these simplified goals that ML models end up gaming no?

GranPC 2y ago

I'm almost 100% sure this is the link you're looking for: https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3Hs...

viccis 2y ago
Haha that was actually the same one I posted in my comment. This was some old website. A coworker sent it to me on Hipchat at my previous job about 10 years ago. And finding anything online older than like 5 years is nearly impossible unless you have the exact URL on hand.
- GranPC 2y ago
  Oh sorry! I recognized the description but since I recalled mine being a Google Sheets link, I just went straight into search mode - and yep, it actually took me a bit to find.

szvsw 2y ago

There’s a few very cool examples where someone recently used RL to solve trackmania, and ends up having to add all sorts of constraints/penalties to prevent extremely strange exploits/glitches that are discovered IIRC… been a while since I watched. https://youtu.be/Dw3BZ6O_8LY?si=VUcJa_hfCxjZhhfR https://youtu.be/Dw3BZ6O_8LY?si=VUcJa_hfCxjZhhfR https://youtu.be/NUl6QikjR04?si=DpZ-iqVdqjzahkwy https://youtu.be/NUl6QikjR04?si=DpZ-iqVdqjzahkwy

hnuser123456 2y ago
Well, in the case of the latter, there was a vaguely known glitch for driving on the nose that allowed for better speeds than possible on 4 wheels, but it would be completely uncontrollable to a human. He figured out how to break the problem down into steps that the NN could gradually learn piecewise, until he had cars racing around tracks while balancing on their nose. It turned out to have learned to keep the car spinning on its nose for stability, and timing inputs to upset the spinning balance at the right moment to touch the ground with the tire to shoot off in a desired direction. I think the overall lesson is that, to make useful machine learning, we must break our problems down into pieces small enough that an algorithm can truly "build up skills" and learn naturally, under the correct guidance.
shagie 2y ago
One of the games that I stumbled across on Steam recently was "AI Learns to Drive" - https://store.steampowered.com/app/3312030/AI_Learns_To_Drive/ https://store.steampowered.com/app/3312030/AI_Learns_To_Driv... It's a neat toy (not really "useful" nor too much of a "game") for generating interest in how neural nets work.

Muromec 2y ago

> using perverse strategies that no human would do Of course we do use perverse strategies and glitches in adversarial multiplayer all the time. Case in point chainsaw glitch, tumblebuffs, early hits and perfect blocks in Elden Ring

matsemann 2y ago

In my thesis many years ago [0] I used EAs to build bicycle wheels. They were so annoyingly good at exploiting whatever idiosyncrasies in my wheel-simulator. Like, the first iterations of my simulator it managed to evolve wheels that would slowly oscillate due to floating point instability or something, and when applied forces to it would increase and increase until the whole simulator exploded and the recorded forces were all over the place, of course then out-competing any wheel in at least some objective dimension. After fixing those bugs, I mostly struggled with it taunting me. Like building a wheel with all the spokes going from the hub and straight up to the rim. It of course would break down when rolling, but on the objective of "how much load can it handle on the bike" it again out-competed every other wheel, and thus was at the pareto-front of that objective and kept showing up through all my tests. Hated that guy, heh. I later changed it to test all wheels in at least 4 orientations, it would then still taunt me with wheels like (c) in this figure[1], exploiting that. [0]: https://news.ycombinator.com/item?id=10410813 https://news.ycombinator.com/item?id=10410813 [1]: https://imgur.com/a/LsONTGc https://imgur.com/a/LsONTGc

deleted 2y ago
[deleted]

elzbardico 2y ago

For the model, the weird glitches are just another element of the game. As they can't reason, have no theory of world or even any real knowledge of what is doing, the model don't have the prior assumptions a human would have about how the game is supposed to be played. If you think about it, even using the term "perverse" is a result of us antropomorphizing any object in the universe that does anything we believe is on the realm of things humans do.

rdlw 2y ago

Not quite what you're describing, but no one has yet linked the classic Tom7 series where he applies deep learning to classic NES games: https://youtu.be/xOCurBYI_gY https://youtu.be/xOCurBYI_gY