6 ms·

Using Computer Vision to Win at Duck Hunt

51 points by jonbaer 5y ago

jmah 5y ago
Images are broken; 403 from googleusercontent.com
- yeldarb 5y ago
  Hey thanks, someone just let me know that this post from a while back ended up on HN; let me look and see if we have those broken images saved anywhere! Edit: fixed!
  - throwroboducks 5y ago
    Appreciate you fixing the links, though is less than a year actually "a while back"? Seems like a relatively short time frame for links to atrophy. </grump>
    - yeldarb 5y ago
      Yeah, not ideal. We typically re-host images we share on our blog but had apparently hyperlinked these from something the user had shared.
HanayamaTriplet 5y ago
I was hoping that this would be a tongue-in-cheek post about how a training a CV model and a robot with enough range of motion would eventually figure out that aiming the light gun off-screen towards a white wall would result in 100% accuracy.
burlesona 5y ago
Tangential, but fascinating to me: I realized a few years ago that you cannot play duck hunt with an original light gun and original NES connected to a flatscreen TV. I’m sure there are people on HN who can explain this more eloquently than I can, but basically the flatscreen TVs have some latency that the original NES hardware cannot handle. You pull the trigger and the gun looks for the white square immediately, but the TV has some digital processing going on that takes some fraction of a second before the image is rendered, and it’s enough that the NES doesn’t think you hit the target. I seem to recall some thing about the CRT refresh rate as well but I don’t remember what that had to do with it. There are apparently after market mods and stuff that people have come up with to work around this, but it takes some effort. As you can imagine I discovered this by trying to play duck hunt and being perplexed when I was literally unable to hit any target, which made a very poor demo for my kids haha.
- cdumler 5y ago
  The fix is actually simple: shoot at a light bulb. It will think you always hit. I figured this out accidentally as a kid one day when I was randomly firing around the room and hit a duck.
  - aynyc 5y ago
    Mind = Blown!
  - deleted 5y ago
    [deleted]
  - NobodyNada 5y ago
    I haven’t tested this with Duck Hunt, but I believe most NES Zapper games actually detect this exploit by blanking the screen for one frame before drawing the target in white, and checking that the gun saw one dark frame followed by one light frame.
  - deleted 5y ago
    [deleted]
- jonbaer 5y ago
  I think what you meant was the CRT refresh rate was used for determining WHICH (multi)target was hit, explained pretty well @ https://www.youtube.com/watch?v=cu83tZIAzlA https://www.youtube.com/watch?v=cu83tZIAzlA
dmurray 5y ago
I'm not sure this solves the hard part of the problem. I was assuming there would be an actual robot arm involved. This is obviously not a difficult CV problem. I don't want to disparage the post too much, though - the Roboflow tools to make this process accessible for a non-expert look very good, and it's a cool demo of those which is the real point of the blog.
d--b 5y ago
> I annotated the images with Roboflow and was able to easily export the data > Roboflow's Dataset Health Check helped me > I found Roboflow’s Model Library to be the best > Roboflow would be my go-to platform > Matt Brems, Growth Manager @ Roboflow Please call this a demo or “how I use Roboflow to win a duck hunt”
- unwind 5y ago
  You missed one: > The user experience for preprocessing is amazing Agreed, this was a bit over the top, but I could have improved my expectations by paying more attention to the domain name. :)
jordigh 5y ago
I don't understand this example. Duck Hunt has only a few sprites, maybe four or six. It's really easy to just scan the whole image and match those sprites, exactly pixel-by-pixel. This doesn't really need anything that could reasonably be called machine learning or computer vision. Synthetic 8-bit images can be handled with 8-bit algorithms. We don't need much art and we certainly don't need its state either. Why pick such an example to advertise roboflow? The article itself also has pretty much no details on how Roboflow works. Not a single code sample! Just links to Roboflow docs. This is blogspam.
- awal2 5y ago
  When all you have is a hammer
  - foobar33333 5y ago
    Don't really agree with this. It like saying "why did you use a phone to do that math when a simple calculator or even pen and paper would work" Sure it would work, but the phone also works and does way more. You might be able to use some trivial processing on duck hunt but it won't work on anything slightly more complex so why would you bother learning a method that only works on the most basic of games when you can develop something that can be applied everywhere.
- jimmies 5y ago
  I agree with you for the most part. This problem definitely doesn't require ML to achieve. However, if doing it with a classical approach (matching sprites) takes a person half a day, and doing it with the new fangled proprietary ml takes a person 20 minutes, I do see that doing it with the new fangled ml approach does have some merits. People want to get stuff done. I'm a barely passable programmer, so not exactly a shining benchmark, so maybe my view is distorted. I don't think I can do duck detection flawlessly with classical programming in half a day and it would be a total pain in the ass to do it that way, to be completely honest with you.
  - awal2 5y ago
29athrowaway 5y ago
That is not the real Duck Hunt, it's a replica. The fonts are different, the UI is different and the UI elements are misaligned, the scenario is larger.
no_time 5y ago
I experimented with using super basic computer vision to cheat at videogames but It never worked quite right. My target was venge.io, pretty bad game but I can tamper with the files freely with some userscripts. I injected bright purple textures for all the characters and used a simple color range matching to find the target. My choice of language was python with pywin32,numba and a screen duplication library which name I forgot. The biggest problem was that by the time I finished scanning the screenbuffer contents the enemies on screen have already moved. If I had paid attention in university math classes I could've come up with a direction tracking and prediction system. :/ But then I threw the whole thing out the window and rewrote everything with good old fashioned readmemory()/writememory() :^)

51 points by jonbaer 5y ago

jmah 5y ago

Images are broken; 403 from googleusercontent.com

yeldarb 5y ago
Hey thanks, someone just let me know that this post from a while back ended up on HN; let me look and see if we have those broken images saved anywhere! Edit: fixed!
- throwroboducks 5y ago
  Appreciate you fixing the links, though is less than a year actually "a while back"? Seems like a relatively short time frame for links to atrophy. </grump>
  - yeldarb 5y ago
    Yeah, not ideal. We typically re-host images we share on our blog but had apparently hyperlinked these from something the user had shared.

HanayamaTriplet 5y ago

I was hoping that this would be a tongue-in-cheek post about how a training a CV model and a robot with enough range of motion would eventually figure out that aiming the light gun off-screen towards a white wall would result in 100% accuracy.

burlesona 5y ago

Tangential, but fascinating to me: I realized a few years ago that you cannot play duck hunt with an original light gun and original NES connected to a flatscreen TV. I’m sure there are people on HN who can explain this more eloquently than I can, but basically the flatscreen TVs have some latency that the original NES hardware cannot handle. You pull the trigger and the gun looks for the white square immediately, but the TV has some digital processing going on that takes some fraction of a second before the image is rendered, and it’s enough that the NES doesn’t think you hit the target. I seem to recall some thing about the CRT refresh rate as well but I don’t remember what that had to do with it. There are apparently after market mods and stuff that people have come up with to work around this, but it takes some effort. As you can imagine I discovered this by trying to play duck hunt and being perplexed when I was literally unable to hit any target, which made a very poor demo for my kids haha.

cdumler 5y ago
The fix is actually simple: shoot at a light bulb. It will think you always hit. I figured this out accidentally as a kid one day when I was randomly firing around the room and hit a duck.
- aynyc 5y ago
  Mind = Blown!
- deleted 5y ago
  [deleted]
- NobodyNada 5y ago
  I haven’t tested this with Duck Hunt, but I believe most NES Zapper games actually detect this exploit by blanking the screen for one frame before drawing the target in white, and checking that the gun saw one dark frame followed by one light frame.
- deleted 5y ago
  [deleted]
jonbaer 5y ago
I think what you meant was the CRT refresh rate was used for determining WHICH (multi)target was hit, explained pretty well @ https://www.youtube.com/watch?v=cu83tZIAzlA https://www.youtube.com/watch?v=cu83tZIAzlA

dmurray 5y ago

I'm not sure this solves the hard part of the problem. I was assuming there would be an actual robot arm involved. This is obviously not a difficult CV problem. I don't want to disparage the post too much, though - the Roboflow tools to make this process accessible for a non-expert look very good, and it's a cool demo of those which is the real point of the blog.

d--b 5y ago

> I annotated the images with Roboflow and was able to easily export the data > Roboflow's Dataset Health Check helped me > I found Roboflow’s Model Library to be the best > Roboflow would be my go-to platform > Matt Brems, Growth Manager @ Roboflow Please call this a demo or “how I use Roboflow to win a duck hunt”

unwind 5y ago
You missed one: > The user experience for preprocessing is amazing Agreed, this was a bit over the top, but I could have improved my expectations by paying more attention to the domain name. :)

jordigh 5y ago

I don't understand this example. Duck Hunt has only a few sprites, maybe four or six. It's really easy to just scan the whole image and match those sprites, exactly pixel-by-pixel. This doesn't really need anything that could reasonably be called machine learning or computer vision. Synthetic 8-bit images can be handled with 8-bit algorithms. We don't need much art and we certainly don't need its state either. Why pick such an example to advertise roboflow? The article itself also has pretty much no details on how Roboflow works. Not a single code sample! Just links to Roboflow docs. This is blogspam.

awal2 5y ago
When all you have is a hammer
- foobar33333 5y ago
  Don't really agree with this. It like saying "why did you use a phone to do that math when a simple calculator or even pen and paper would work" Sure it would work, but the phone also works and does way more. You might be able to use some trivial processing on duck hunt but it won't work on anything slightly more complex so why would you bother learning a method that only works on the most basic of games when you can develop something that can be applied everywhere.
jimmies 5y ago
I agree with you for the most part. This problem definitely doesn't require ML to achieve. However, if doing it with a classical approach (matching sprites) takes a person half a day, and doing it with the new fangled proprietary ml takes a person 20 minutes, I do see that doing it with the new fangled ml approach does have some merits. People want to get stuff done. I'm a barely passable programmer, so not exactly a shining benchmark, so maybe my view is distorted. I don't think I can do duck detection flawlessly with classical programming in half a day and it would be a total pain in the ass to do it that way, to be completely honest with you.
- awal2 5y ago

29athrowaway 5y ago

That is not the real Duck Hunt, it's a replica. The fonts are different, the UI is different and the UI elements are misaligned, the scenario is larger.

no_time 5y ago

I experimented with using super basic computer vision to cheat at videogames but It never worked quite right. My target was venge.io, pretty bad game but I can tamper with the files freely with some userscripts. I injected bright purple textures for all the characters and used a simple color range matching to find the target. My choice of language was python with pywin32,numba and a screen duplication library which name I forgot. The biggest problem was that by the time I finished scanning the screenbuffer contents the enemies on screen have already moved. If I had paid attention in university math classes I could've come up with a direction tracking and prediction system. :/ But then I threw the whole thing out the window and rewrote everything with good old fashioned readmemory()/writememory() :^)