4 ms·

I built (and tried to get funding for) http://gameref.io/ http://gameref.io/ a few years ago. Many critics argued that a deep learning or AI solution would be b

by dvt 9y ago

I built (and tried to get funding for) http://gameref.io/ http://gameref.io/ a few years ago. Many critics argued that a deep learning or AI solution would be better. For what it's worth, I also played Counter-Strike at a professional level from '04-'07 -- my CS:Source team was ranked #7 in North America in 2005 (according to GotFrag.com). After some research and a few POCs I am 100% certain a ML solution will not work in detecting cheats other than completely trivial "spinbots" or "aim-lock" aimbots. For example, it by definition cannot catch an entire class of cheats (wall hacks), but it even falters when trying to catch aimbots; the presentation here argues for a "heuristic-based" method, but, again, these methods often fail when (if you look at the source), aimbots are often coded to (a) move smoothly/naturally and (b) introduce random noise. In my case, I tried to at the very least combat one class of cheats with close to 100% confidence. These kinds of talks are great at conferences, but, as they say, the proof is in the pudding. Cheaters are very hard to catch, and cheat coders are much smarter than generalized AI methods.

rudedogg 9y ago
gameref looks like a really clever and simple solution as a hardware device. > After some research and a few POCs I am 100% certain a ML solution will not work in detecting cheats other than completely trivial "spinbots" or "aim-lock" aimbots. I'm curious why you say this. I'm a machine learning noob, but it seems pretty straightforward that it would work. Relating to wallhacks, they can use your last 1,000 matches, and if you're never/rarely surprised by a camping enemy, you're probably cheating. Your camera position moments before interaction with an obscured enemy is probably another indicator.
- dvt 9y ago
  > I'm curious why you say this. Well the answer is surprisingly simple. Suppose you only have one person that's wallhacking on your team. He's calling out positions for everyone else. He can play terribly, but since the calls are going out, your team still wins. (This is not a hypothetical and has actually happened in high level matches before.)
  - jzwinck 9y ago
    How high level can a team get in CS (in terms of national or world rankings) without playing in a controlled environment where this sort of cheating would be more easily detected by local human observers? Could my team be ranked in the top ten in the US or Korea or wherever if we only ever play via the internet?
    - dvt 9y ago
      > How high level can a team get in CS (in terms of national or world rankings) without playing in a controlled environment where this sort of cheating would be more easily detected by local human observers? Very high. You can see for yourself here[1]. Anecdotally, I had a player on my team that cheated in an (online) semi-final match. We had no idea, but were moved down a league, were disgraced, and obviously cut him. A year later, he was one of the people that toured the country with the XFX team. I couldn't believe it. Every one knew this guy cheated just a few months prior, now he was offered a contract. I was very active in the Socal LAN scene, and know at least 3-4 players that played professionally and cheated. As far as your second point is concerned, some aimbots are so tricky, even local human observers couldn't catch them. I saw this proof of concept made by a famous cheat maker where the mouse would "tug" at the direcction of the player through a wall. The movie would be almost imperceptible by a spectator, but definitely "felt" by the player. [1] https://www.hltv.org/blog/13538/cheating-on-professional-level-of-csgo https://www.hltv.org/blog/13538/cheating-on-professional-lev...
- metjm 9y ago
  I'm failing to see a use case for gameref. At the big tournaments the computer is provided by the organizer. For online tournaments, wallhacks and maphacks won't be detected by this device. > I'm curious why you say this. I'm a machine learning noob, but it seems pretty straightforward that it would work. Relating to wallhacks, they can use your last say 1,000 matches, and if you're never/rarely surprised by a camping enemy, you're probably cheating. Exactly. And not just that, wouldn't ML in theory also be able to detect subtle reactions to changes that the player shouldn't see? For example if a cheater has a map hack that only shows dots on the minimap, and does some small action like twitch with the mouse when an enemy he's following and can't see or hear ingame, changes direction?
  - dvt 9y ago
    > I'm failing to see a use case for gameref. At the big tournaments the computer is provided by the organizer. For online tournaments, wallhacks and maphacks won't be detected by this device. There have been cases of people smuggling aimbots in onboard mice drivers. There was also a Steam oversight where hacks could be loaded via the workshop feature. A semi-pro German player was caught in 2015 I believe.
    - chi3 9y ago
      There has never been a proven case of either as far as I know, only theories and "insiders" claiming these things. The german guy who was caught was saying a lot of these things which obviously weren't true, like claiming that a lot of other players in top teams were cheating on LAN, but in hindsight that hardly seems believable. A driver cheat which would auto-inject code on insertion would as far as I know require a 0-day exploit on Windows, and I have a hard time believing one of the cheat devs was sitting on one of those.
      - dvt 9y ago
        > The german guy who was caught was saying a lot of these things which obviously weren't true, like claiming that a lot of other players in top teams were cheating on LAN, but in hindsight that hardly seems believable. I mean, there were 3 pro players banned in like a week. I doubt this was coincidence [1]. > A driver cheat which would auto-inject code on insertion would as far as I know require a 0-day exploit on Windows, and I have a hard time believing one of the cheat makers was sitting on one of those It's not completely out of the question. Check out this DEF CON hack which illustrates something similar[2]. [1] https://www.pcgamer.com/csgo-competitive-scene-embroiled-in-hacking-scandal-as-three-players-are-banned/ https://www.pcgamer.com/csgo-competitive-scene-embroiled-in-... [2] https://theoutline.com/post/2032/how-to-hack-a-mouse-to-win-millions-at-esports?zd=1&zi=d6mb2onj https://theoutline.com/post/2032/how-to-hack-a-mouse-to-win-...
- trinitry3 9y ago
  > Relating to wallhacks, they can use your last 1,000 matches, and if you're never/rarely surprised by a camping enemy, you're probably cheating. Your camera position moments before interaction with an obscured enemy is probably another indicator. Do you actually play CS:GO proficiently? Because it doesn't sound like you do.
netheril96 9y ago
> (a) move smoothly/naturally and (b) introduce random noise. Moving naturally is not that easy to code. Otherwise Google would not be able to tell if you are a robot or not simply by analyzing how you click a checkbox.
- LoSboccacc 9y ago
  it's also hard to define "move naturally" - if you look for smoothness you'd get a ton of false positive. check this out https://streamable.com/swz3y https://streamable.com/swz3y - twitchiness is a byproduct of scouting for target in motions.
- mynewtb 9y ago
  Well, Google can't. Or maybe I am a robot.
- makomk 9y ago
  Google can't. They use your history of online activity across their services to decide if you're a robot. If you click the checkbox the exact same way as usual, but do so in an incognito window without any established Google cookies, they require you to go through a captcha.
  - paulie_a 9y ago
    Off topic: but why the fuck does Google operate two different captchas? One that is useable, and for internal services one that is incredibly difficult top get right on the first 9 attemps
ygra 9y ago
It should be noted, though, that this ML model is based on Overwatch (that's where they get their training data), which is only good at detecting blatant cheats as well. So they probably are fairly sure already that this system won't be able to catch 100 % of cheaters. However, as John noted in the beginning, those blatant cheats mostly have one reason to exist: to actively and obviously spoil the fun of others involved, so to improve things to players of that game, it's a very good starting point.
- dvt 9y ago
  > However, as John noted in the beginning, those blatant cheats mostly have one reason to exist: to actively and obviously spoil the fun of others involved, so to improve things to players of that game, it's a very good starting point. This is very true (and very important), but I don't think you need AI to detect blatant cheats. IMO, simple heuristics can do the job better.
  - ygra 9y ago
    Perhaps with a worse conviction rate of the reports or by catching fewer cheaters. One major part of what makes this successful is that the reports by VACnet are high-quality and very often lead to convictions (i.e. they're not false positives). This keeps the humans who're actually judging engaged. From experience I found Overwatch fairly tedious if there's nothing fishy going on at all for ten cases or so, so participation rate has dropped over time.
- switz 9y ago
  To clarify for anyone reading, "Overwatch" is not referring to the Blizzard game, but an in-game Counterstrike tool in which players review recorded gameplay of a suspected cheater. When enough reviewers agree that a player is cheating, that person is banned. Each player has a hidden successful conviction rating and on occasion the system will send you professional players or non-cheating players in an effort to verify your ability. This is only meant to convict the most blatant of cheaters. It's fairly successful for what it's intended to do, but of course, it does not even remotely catch every cheater.
leetcrew 9y ago
what team if you don't mind me asking?

by dvt 9y ago

rudedogg 9y ago

gameref looks like a really clever and simple solution as a hardware device. > After some research and a few POCs I am 100% certain a ML solution will not work in detecting cheats other than completely trivial "spinbots" or "aim-lock" aimbots. I'm curious why you say this. I'm a machine learning noob, but it seems pretty straightforward that it would work. Relating to wallhacks, they can use your last 1,000 matches, and if you're never/rarely surprised by a camping enemy, you're probably cheating. Your camera position moments before interaction with an obscured enemy is probably another indicator.

dvt 9y ago
> I'm curious why you say this. Well the answer is surprisingly simple. Suppose you only have one person that's wallhacking on your team. He's calling out positions for everyone else. He can play terribly, but since the calls are going out, your team still wins. (This is not a hypothetical and has actually happened in high level matches before.)
- jzwinck 9y ago
  How high level can a team get in CS (in terms of national or world rankings) without playing in a controlled environment where this sort of cheating would be more easily detected by local human observers? Could my team be ranked in the top ten in the US or Korea or wherever if we only ever play via the internet?
  - dvt 9y ago
    > How high level can a team get in CS (in terms of national or world rankings) without playing in a controlled environment where this sort of cheating would be more easily detected by local human observers? Very high. You can see for yourself here[1]. Anecdotally, I had a player on my team that cheated in an (online) semi-final match. We had no idea, but were moved down a league, were disgraced, and obviously cut him. A year later, he was one of the people that toured the country with the XFX team. I couldn't believe it. Every one knew this guy cheated just a few months prior, now he was offered a contract. I was very active in the Socal LAN scene, and know at least 3-4 players that played professionally and cheated. As far as your second point is concerned, some aimbots are so tricky, even local human observers couldn't catch them. I saw this proof of concept made by a famous cheat maker where the mouse would "tug" at the direcction of the player through a wall. The movie would be almost imperceptible by a spectator, but definitely "felt" by the player. [1] https://www.hltv.org/blog/13538/cheating-on-professional-level-of-csgo https://www.hltv.org/blog/13538/cheating-on-professional-lev...
metjm 9y ago
I'm failing to see a use case for gameref. At the big tournaments the computer is provided by the organizer. For online tournaments, wallhacks and maphacks won't be detected by this device. > I'm curious why you say this. I'm a machine learning noob, but it seems pretty straightforward that it would work. Relating to wallhacks, they can use your last say 1,000 matches, and if you're never/rarely surprised by a camping enemy, you're probably cheating. Exactly. And not just that, wouldn't ML in theory also be able to detect subtle reactions to changes that the player shouldn't see? For example if a cheater has a map hack that only shows dots on the minimap, and does some small action like twitch with the mouse when an enemy he's following and can't see or hear ingame, changes direction?
- dvt 9y ago
  > I'm failing to see a use case for gameref. At the big tournaments the computer is provided by the organizer. For online tournaments, wallhacks and maphacks won't be detected by this device. There have been cases of people smuggling aimbots in onboard mice drivers. There was also a Steam oversight where hacks could be loaded via the workshop feature. A semi-pro German player was caught in 2015 I believe.
  - chi3 9y ago
    There has never been a proven case of either as far as I know, only theories and "insiders" claiming these things. The german guy who was caught was saying a lot of these things which obviously weren't true, like claiming that a lot of other players in top teams were cheating on LAN, but in hindsight that hardly seems believable. A driver cheat which would auto-inject code on insertion would as far as I know require a 0-day exploit on Windows, and I have a hard time believing one of the cheat devs was sitting on one of those.
    - dvt 9y ago
      > The german guy who was caught was saying a lot of these things which obviously weren't true, like claiming that a lot of other players in top teams were cheating on LAN, but in hindsight that hardly seems believable. I mean, there were 3 pro players banned in like a week. I doubt this was coincidence [1]. > A driver cheat which would auto-inject code on insertion would as far as I know require a 0-day exploit on Windows, and I have a hard time believing one of the cheat makers was sitting on one of those It's not completely out of the question. Check out this DEF CON hack which illustrates something similar[2]. [1] https://www.pcgamer.com/csgo-competitive-scene-embroiled-in-hacking-scandal-as-three-players-are-banned/ https://www.pcgamer.com/csgo-competitive-scene-embroiled-in-... [2] https://theoutline.com/post/2032/how-to-hack-a-mouse-to-win-millions-at-esports?zd=1&zi=d6mb2onj https://theoutline.com/post/2032/how-to-hack-a-mouse-to-win-...
trinitry3 9y ago
> Relating to wallhacks, they can use your last 1,000 matches, and if you're never/rarely surprised by a camping enemy, you're probably cheating. Your camera position moments before interaction with an obscured enemy is probably another indicator. Do you actually play CS:GO proficiently? Because it doesn't sound like you do.

netheril96 9y ago

> (a) move smoothly/naturally and (b) introduce random noise. Moving naturally is not that easy to code. Otherwise Google would not be able to tell if you are a robot or not simply by analyzing how you click a checkbox.

LoSboccacc 9y ago
it's also hard to define "move naturally" - if you look for smoothness you'd get a ton of false positive. check this out https://streamable.com/swz3y https://streamable.com/swz3y - twitchiness is a byproduct of scouting for target in motions.
mynewtb 9y ago
Well, Google can't. Or maybe I am a robot.
makomk 9y ago
Google can't. They use your history of online activity across their services to decide if you're a robot. If you click the checkbox the exact same way as usual, but do so in an incognito window without any established Google cookies, they require you to go through a captcha.
- paulie_a 9y ago
  Off topic: but why the fuck does Google operate two different captchas? One that is useable, and for internal services one that is incredibly difficult top get right on the first 9 attemps

ygra 9y ago

It should be noted, though, that this ML model is based on Overwatch (that's where they get their training data), which is only good at detecting blatant cheats as well. So they probably are fairly sure already that this system won't be able to catch 100 % of cheaters. However, as John noted in the beginning, those blatant cheats mostly have one reason to exist: to actively and obviously spoil the fun of others involved, so to improve things to players of that game, it's a very good starting point.

dvt 9y ago
> However, as John noted in the beginning, those blatant cheats mostly have one reason to exist: to actively and obviously spoil the fun of others involved, so to improve things to players of that game, it's a very good starting point. This is very true (and very important), but I don't think you need AI to detect blatant cheats. IMO, simple heuristics can do the job better.
- ygra 9y ago
  Perhaps with a worse conviction rate of the reports or by catching fewer cheaters. One major part of what makes this successful is that the reports by VACnet are high-quality and very often lead to convictions (i.e. they're not false positives). This keeps the humans who're actually judging engaged. From experience I found Overwatch fairly tedious if there's nothing fishy going on at all for ten cases or so, so participation rate has dropped over time.
switz 9y ago
To clarify for anyone reading, "Overwatch" is not referring to the Blizzard game, but an in-game Counterstrike tool in which players review recorded gameplay of a suspected cheater. When enough reviewers agree that a player is cheating, that person is banned. Each player has a hidden successful conviction rating and on occasion the system will send you professional players or non-cheating players in an effort to verify your ability. This is only meant to convict the most blatant of cheaters. It's fairly successful for what it's intended to do, but of course, it does not even remotely catch every cheater.

leetcrew 9y ago

what team if you don't mind me asking?