3 ms·

I’m also curious if an AI could process the screen feed quickly enough to compete in first-person shooter games. Seems like it would be difficult without extrem

I’m also curious if an AI could process the screen feed quickly enough to compete in first-person shooter games. Seems like it would be difficult without extremely high end hardware for the foreseeable future?

Thaxll 9mo ago
It already exists.
ModernMech 9mo ago
I had students build this kind of thing in 2020 by screenshotting the game and processing it with a standard OpenCV pipeline. No GenAI needed.
- throwaway894345 9mo ago
  Thank you for educating me. How does OpenCV work from the perspective of recognizing things in an image? Is there some kind of underlying model there that learns what a target looks like or not?
  - ModernMech 9mo ago
    The way they did it, they were writing an aimbot. So the pipeline was: - take a screenshot - run massive skeletal detection on it to get the skeletons of any humanoids - of those skeletons, pick a target closest to the player - for that target, get coordinates of head node - run a PID to control the cursor to where the head node is located - move the camera one frame, repeat the whole process. If you can fit that pipeline to 16ms it can run in real time.
    - throwaway894345 9mo ago
      Wow, that's fascinating. Were they able to fit the whole thing inside the 16ms frame?
      - ModernMech 9mo ago
        oh yeah, with little problem, especially with a GPU it's not hard at all.
Scion9066 9mo ago
There's already models specifically for things like identifying players in Counter-Strike 2, including which team they're on. Someone has even rigged up a system like that to a TENS system to stimulate the nerves in their arm and hand to move the mouse in the correct direction and fire when the crosshair is over the enemy. We are definitely already there.
- fragmede 9mo ago
  They documented it on YouTube for us to see: https://youtu.be/9alJwQG-Wbk https://youtu.be/9alJwQG-Wbk holy shit that's amazing.

Thaxll 9mo ago

It already exists.

ModernMech 9mo ago

I had students build this kind of thing in 2020 by screenshotting the game and processing it with a standard OpenCV pipeline. No GenAI needed.

throwaway894345 9mo ago
Thank you for educating me. How does OpenCV work from the perspective of recognizing things in an image? Is there some kind of underlying model there that learns what a target looks like or not?
- ModernMech 9mo ago
  The way they did it, they were writing an aimbot. So the pipeline was: - take a screenshot - run massive skeletal detection on it to get the skeletons of any humanoids - of those skeletons, pick a target closest to the player - for that target, get coordinates of head node - run a PID to control the cursor to where the head node is located - move the camera one frame, repeat the whole process. If you can fit that pipeline to 16ms it can run in real time.
  - throwaway894345 9mo ago
    Wow, that's fascinating. Were they able to fit the whole thing inside the 16ms frame?
    - ModernMech 9mo ago
      oh yeah, with little problem, especially with a GPU it's not hard at all.

Scion9066 9mo ago

There's already models specifically for things like identifying players in Counter-Strike 2, including which team they're on. Someone has even rigged up a system like that to a TENS system to stimulate the nerves in their arm and hand to move the mouse in the correct direction and fire when the crosshair is over the enemy. We are definitely already there.

fragmede 9mo ago
They documented it on YouTube for us to see: https://youtu.be/9alJwQG-Wbk https://youtu.be/9alJwQG-Wbk holy shit that's amazing.