5 ms·

Very cool! As a further variation on this idea, I'm imagining training a reinforcement learning agent on atari games / super mario, but with an additional music

Very cool! As a further variation on this idea, I'm imagining training a reinforcement learning agent on atari games / super mario, but with an additional music-based reward/input, to try to get a "musical" looking playthrough... (Not sure how good it would look / whether it would be worth it though...)

dberst 2y ago
I'm a novice at machine learning, but Open AI made a python library for reinforcement learning in video games, and a fork of it is still actively maintained [1]. It's been a few years, but I remember being able to get it up and running in a day or two (maybe a weekend). It used the Retroarch emulator, which is compatible with a huge number of emulators and consoles. https://github.com/Farama-Foundation/Gymnasium https://github.com/Farama-Foundation/Gymnasium There also SethBling's excellent video on YouTube about machine learning specifically with Super Mario World: https://www.youtube.com/watch?v=qv6UVOQ0F44 https://www.youtube.com/watch?v=qv6UVOQ0F44 I encourage you to give it a try! I feel that video games are a bit underrated by current AI buzz, I think there's lots of potential for a machine to learn a skill through playing a game. And lots of potential for a game being selected or even created with the goal of teaching a specific skill. However at that point maybe it's better to forego the audio and visuals and speak to the machine in text or with pure data. On the other hand, I have seen a video about convolutional neural networks that feed on each pixel of an image. So perhaps training with sound data, or with the pixels of a spectrogram, could have some positive results. And certainly I would be amused to see a game played in time with music, or possibly even dancing with a songs melody, harmony, and story as well. Anything that's ever been created by humans, existed first in the imagination of a human brain. And you've got one of those. A mental vision pursued and brought from the mind into physical reality is a beautiful thing, a gift for all of humanity in my eyes. I think it's quite worthwhile. But that's just my perspective. Thank you for sharing your imagination. Have a nice day
deleted 2y ago
[deleted]
jncfhnb 2y ago
Not very good. Mario doesn’t have enough rhythm to evoke a lot of musicality. The original game’s sound was tied to the frame rate so this vaguely happened by default. Later ports to PAL broke this because it ran at a slower frame rate.
vunderba 2y ago
Crypt of the NecroDancer explores this idea of rhythmically timing your character's movement to get bonuses in game.
- Sirizarry 2y ago
  Don’t forget the legend of Zelda spinoff “cadence of hyrule” which I’m pretty sure was made by the same guys as crypt of the necrodancer.
- jerf 2y ago
  Bit.Trip Beat is a game based on almost exactly the concept in the linked page: https://youtu.be/LHbg-sNqe4w?t=47 https://youtu.be/LHbg-sNqe4w?t=47 , as well as its later sequel Flux.

dberst 2y ago

I'm a novice at machine learning, but Open AI made a python library for reinforcement learning in video games, and a fork of it is still actively maintained [1]. It's been a few years, but I remember being able to get it up and running in a day or two (maybe a weekend). It used the Retroarch emulator, which is compatible with a huge number of emulators and consoles. https://github.com/Farama-Foundation/Gymnasium https://github.com/Farama-Foundation/Gymnasium There also SethBling's excellent video on YouTube about machine learning specifically with Super Mario World: https://www.youtube.com/watch?v=qv6UVOQ0F44 https://www.youtube.com/watch?v=qv6UVOQ0F44 I encourage you to give it a try! I feel that video games are a bit underrated by current AI buzz, I think there's lots of potential for a machine to learn a skill through playing a game. And lots of potential for a game being selected or even created with the goal of teaching a specific skill. However at that point maybe it's better to forego the audio and visuals and speak to the machine in text or with pure data. On the other hand, I have seen a video about convolutional neural networks that feed on each pixel of an image. So perhaps training with sound data, or with the pixels of a spectrogram, could have some positive results. And certainly I would be amused to see a game played in time with music, or possibly even dancing with a songs melody, harmony, and story as well. Anything that's ever been created by humans, existed first in the imagination of a human brain. And you've got one of those. A mental vision pursued and brought from the mind into physical reality is a beautiful thing, a gift for all of humanity in my eyes. I think it's quite worthwhile. But that's just my perspective. Thank you for sharing your imagination. Have a nice day

deleted 2y ago

[deleted]

jncfhnb 2y ago

Not very good. Mario doesn’t have enough rhythm to evoke a lot of musicality. The original game’s sound was tied to the frame rate so this vaguely happened by default. Later ports to PAL broke this because it ran at a slower frame rate.

vunderba 2y ago

Crypt of the NecroDancer explores this idea of rhythmically timing your character's movement to get bonuses in game.

Sirizarry 2y ago
Don’t forget the legend of Zelda spinoff “cadence of hyrule” which I’m pretty sure was made by the same guys as crypt of the necrodancer.
jerf 2y ago
Bit.Trip Beat is a game based on almost exactly the concept in the linked page: https://youtu.be/LHbg-sNqe4w?t=47 https://youtu.be/LHbg-sNqe4w?t=47 , as well as its later sequel Flux.