4 ms·

Shepherd's Dog is a game I've wanted to create for a long time, but I never got the sheep flocking behaviour just right. The goal of the game is to herd all the

by vnglst 2y ago

Shepherd's Dog is a game I've wanted to create for a long time, but I never got the sheep flocking behaviour just right. The goal of the game is to herd all the sheep into the pen before nightfall. I've asked several models to create this game and I'm particularly impressed with what Claude 3.7 could do with a one-shot prompt. - You can play the Claude game here (note: doesn't work on Safari for some reason): https://html-preview.github.io/?url=https://raw.githubusercontent.com/vnglst/when-ai-fails/refs/heads/main/shepards-dog/claude-3.7/index.html https://html-preview.github.io/?url=https://raw.githubuserco... - o3-mini's version is here: https://html-preview.github.io/?url=https://raw.githubusercontent.com/vnglst/when-ai-fails/refs/heads/main/shepards-dog/o3-mini/index.html https://html-preview.github.io/?url=https://raw.githubuserco... Results of other models and a leaderboard is here: https://github.com/vnglst/when-ai-fails/blob/main/shepards-dog/README.md#leaderboard https://github.com/vnglst/when-ai-fails/blob/main/shepards-d... Some videos: https://hachyderm.io/@vnglst/114125938185826311 https://hachyderm.io/@vnglst/114125938185826311

breckenedge 2y ago
Since you’re releasing the code to GitHub, do you think you’ll eventually run into issues with the training data including prior versions of the game?
- tdy_err 2y ago
  The implied scenario being that the memory of its own output would result in the model producing degraded future output? Why is that a given?
  - Chaosvex 2y ago
    Read about model collapse. The TL;DR is garbage in, garbage out. https://en.wikipedia.org/wiki/Model_collapse https://en.wikipedia.org/wiki/Model_collapse
  - mythrwy 2y ago
    Probably the same reason that close relatives marrying each other for generations produces genetic problems.
    - Etherlord87 2y ago
      Not the same reason at all. In genetics the reason is that you're losing gene variety and eventually recessive genes aren't suppressed anymore. In case of LLM it's just error accumulation.
      - mythrwy 2y ago
        It's a few days late but "losing gene variety" isn't the cause. What happens is genetic errors compound and are more likely to be expressed. I.E. "error accumulation".
        Etherlord87 2y ago
        You're wrong. You clearly have the Internet, I don't understand why won't you just google it and learn about it instead of claiming stuff that is bs.
        mythrwy 2y ago
        How about a number of grad level genetics courses? Does that beat your google search? Because that is what I have. And what I am telling you is what happens. This is really easily searched (as you said). You might read up on it if interested. Check out why inbreeding can lead to expression of genetic defects. What is the mechanism? (hint: it's not "losing gene diversity" or "suppression").
        Etherlord87 2y ago
        Very bad courses then. https://biology.stackexchange.com/questions/58769/what-are-the-consequences-of-inbreeding https://biology.stackexchange.com/questions/58769/what-are-t...
jchw 2y ago
Here is an attempt using Google Gemini 2.0 Pro Experimental. https://gist.github.com/jchv/e8869a7cbe2d854a0ec93e946030d90a https://gist.github.com/jchv/e8869a7cbe2d854a0ec93e946030d90... It seems like it has some issues, but the result is interesting nonetheless. Just a one-shot like the others, needed a single "Keep going" but otherwise this is the vanilla output from the prompt. Edit: Looks like you can share an HTML preview of a gist using html-preview.github.io, so here's that. https://html-preview.github.io/?url=https://gist.githubusercontent.com/jchv/e8869a7cbe2d854a0ec93e946030d90a/raw/06d9366e67f3619df86090a31fcbd7c81052d0d7/index.html https://html-preview.github.io/?url=https://gist.githubuserc... - It'll go to level 2 if you refresh the page and hit Restart, but I don't think it's possible to clear Level 2. The flock stays too far apart to fit enough sheep in the pen.
AustinDev 2y ago
Just tried a 1-shot on Grok3 - Thinking and it couldn't get past the start button. Throws an error: | "<a class='gotoLine' href='#67:39'>67:39</a> Uncaught ReferenceError: startGame is not defined" Scope issue. No barking or dog player model but pretty similar in style to Claude's output. What's interesting to me about playing with AI Codegen is each model has specific and sometimes overlapping output errors. Claude 3.7 really like to solve errors by returning dummy data as a 'fallback' when doing client or server calls. A little prompting can reduce this but not eliminate it. 'The tests always pass if you return dummy data' https://jsfiddle.net/aL3ugtj1/ https://jsfiddle.net/aL3ugtj1/
n4r9 2y ago
I just played the Claude attempt and found that the "fence" in level 3 doesn't actually obstruct either the dog or the sheep. Otherwise pretty fun.
swyx 2y ago
great demos. one shotting isnt really fair imo, i feel like that might be hard even for a human to do (working without feedback). i'd be curious what deepseek would do with a bit more feedback.
Keyframe 2y ago
Shepherd's Dog is a game I've wanted to create for a long time Not sure if you're aware, but there was a game like that for playstation and GBA, called Sheep! https://en.wikipedia.org/wiki/Sheep_(video_game) https://en.wikipedia.org/wiki/Sheep_(video_game) Here's some gameplay footage (player here didn't chose a dog to play with for some reason): https://www.youtube.com/watch?v=SP058CHQj20 https://www.youtube.com/watch?v=SP058CHQj20 Premise of the game is the same, you run the sheep to the designated area over obstacles.
- vnglst 2y ago
  Ah thanks for this. The game above is lovely and it’s really similar to what I had in mind (I was also thinking of lemmings!). I see in the other comments below that this idea of mine has been created as a game a lot of times already. Seems like I’m not as original as I thought haha
  - Keyframe 2y ago
    Seems like I’m not as original as I thought haha in creative work that's absolutely irrelevant. Don't even think about that. Everything has been done before; It's your take that counts, your vision!

by vnglst 2y ago

breckenedge 2y ago

Since you’re releasing the code to GitHub, do you think you’ll eventually run into issues with the training data including prior versions of the game?

tdy_err 2y ago
The implied scenario being that the memory of its own output would result in the model producing degraded future output? Why is that a given?
- Chaosvex 2y ago
  Read about model collapse. The TL;DR is garbage in, garbage out. https://en.wikipedia.org/wiki/Model_collapse https://en.wikipedia.org/wiki/Model_collapse
- mythrwy 2y ago
  Probably the same reason that close relatives marrying each other for generations produces genetic problems.
  - Etherlord87 2y ago
    Not the same reason at all. In genetics the reason is that you're losing gene variety and eventually recessive genes aren't suppressed anymore. In case of LLM it's just error accumulation.
    - mythrwy 2y ago
      It's a few days late but "losing gene variety" isn't the cause. What happens is genetic errors compound and are more likely to be expressed. I.E. "error accumulation".
      - Etherlord87 2y ago
        You're wrong. You clearly have the Internet, I don't understand why won't you just google it and learn about it instead of claiming stuff that is bs.
        mythrwy 2y ago
        How about a number of grad level genetics courses? Does that beat your google search? Because that is what I have. And what I am telling you is what happens. This is really easily searched (as you said). You might read up on it if interested. Check out why inbreeding can lead to expression of genetic defects. What is the mechanism? (hint: it's not "losing gene diversity" or "suppression").
        Etherlord87 2y ago
        Very bad courses then. https://biology.stackexchange.com/questions/58769/what-are-the-consequences-of-inbreeding https://biology.stackexchange.com/questions/58769/what-are-t...

jchw 2y ago

Here is an attempt using Google Gemini 2.0 Pro Experimental. https://gist.github.com/jchv/e8869a7cbe2d854a0ec93e946030d90a https://gist.github.com/jchv/e8869a7cbe2d854a0ec93e946030d90... It seems like it has some issues, but the result is interesting nonetheless. Just a one-shot like the others, needed a single "Keep going" but otherwise this is the vanilla output from the prompt. Edit: Looks like you can share an HTML preview of a gist using html-preview.github.io, so here's that. https://html-preview.github.io/?url=https://gist.githubusercontent.com/jchv/e8869a7cbe2d854a0ec93e946030d90a/raw/06d9366e67f3619df86090a31fcbd7c81052d0d7/index.html https://html-preview.github.io/?url=https://gist.githubuserc... - It'll go to level 2 if you refresh the page and hit Restart, but I don't think it's possible to clear Level 2. The flock stays too far apart to fit enough sheep in the pen.

AustinDev 2y ago

Just tried a 1-shot on Grok3 - Thinking and it couldn't get past the start button. Throws an error: | "<a class='gotoLine' href='#67:39'>67:39</a> Uncaught ReferenceError: startGame is not defined" Scope issue. No barking or dog player model but pretty similar in style to Claude's output. What's interesting to me about playing with AI Codegen is each model has specific and sometimes overlapping output errors. Claude 3.7 really like to solve errors by returning dummy data as a 'fallback' when doing client or server calls. A little prompting can reduce this but not eliminate it. 'The tests always pass if you return dummy data' https://jsfiddle.net/aL3ugtj1/ https://jsfiddle.net/aL3ugtj1/

n4r9 2y ago

I just played the Claude attempt and found that the "fence" in level 3 doesn't actually obstruct either the dog or the sheep. Otherwise pretty fun.

swyx 2y ago

great demos. one shotting isnt really fair imo, i feel like that might be hard even for a human to do (working without feedback). i'd be curious what deepseek would do with a bit more feedback.

Keyframe 2y ago

Shepherd's Dog is a game I've wanted to create for a long time Not sure if you're aware, but there was a game like that for playstation and GBA, called Sheep! https://en.wikipedia.org/wiki/Sheep_(video_game) https://en.wikipedia.org/wiki/Sheep_(video_game) Here's some gameplay footage (player here didn't chose a dog to play with for some reason): https://www.youtube.com/watch?v=SP058CHQj20 https://www.youtube.com/watch?v=SP058CHQj20 Premise of the game is the same, you run the sheep to the designated area over obstacles.

vnglst 2y ago
Ah thanks for this. The game above is lovely and it’s really similar to what I had in mind (I was also thinking of lemmings!). I see in the other comments below that this idea of mine has been created as a game a lot of times already. Seems like I’m not as original as I thought haha
- Keyframe 2y ago
  Seems like I’m not as original as I thought haha in creative work that's absolutely irrelevant. Don't even think about that. Everything has been done before; It's your take that counts, your vision!