6 ms·
I was never great at Guitar Hero and couldn't fathom how people would play Through The Fire And The Flames, so I made a robot that could do it for me. The vide
by nik12795 5y ago
I was never great at Guitar Hero and couldn't fathom how people would play Through The Fire And The Flames, so I made a robot that could do it for me.
The video explains some of the technical details but I'll highlight a few other key points.
- My first attempt at note recognition was looking for specific pixels on the screen. The thought was that I would be able to detect if a pixel changed color and, depending on the color, it would know which note it was. The game outputs at 30 fps and on faster songs, it would miss nearly 30% of notes.
- The note recognition was eventually built using computer vision from OpenCV and specifically blob detection. It took a fair amount of effort to hone in the perfect filtering to make note recognition was highly accurate.
- Implementing the solenoids threw me for a loop. I ended up burning through a lot of them since I'm new to robotics and didn't understand the relationship between power, volts, and current. At one point I was giving these solenoids 4x the power it was expecting. Needless to say, they burnt out quickly. I eventually talked with a robotics guy and he gave me a 30 min crash course which straightened me out.
- The whole project, from idea to the YouTube video I linked in the post, took 2 months. It probably cost nearly $1,000 for everything (raspberry pi, pvc, solenoids, AC/DC power converter, relay modules, paint, stickers, etc). That might seem high, but like I said, I was learning about volts/current and part of that learning process was destroying many solenoids (oops).
- StavrosK 5y agoGreat work! Though, wouldn't it be easier to just open the controller up and use transistors to close the contacts? Transistors are cheaper!
- freedomben 5y agoNice work man, and posting it to HN was a great idea to start gaining some subscribers ;-) Giving away my PS2 and Guitar Hero is one of the only things I regret by the way. If someone created a Guitar Hero that ran on Linux I would gladly pay.
- LocalH 5y agohttps://clonehero.net/ https://clonehero.net/
- geococcyxc 5y agoThere's Frets on Fire: http://fretsonfire.sourceforge.net/ http://fretsonfire.sourceforge.net/
- Gamemaster1379 5y agoPhase Shift is by far the most comprehensive "knock off" Guitar Hero. Blows Frets on Fire and Clone Hero out of the water. Netplay, multi format support. Etc. All previous game charts are available in formats compatible with Phase Shift.
- ghusbands 5y agoIt seems to be abandoned and unavailable. They switched over to Early Access on Steam in 2019 or so, but it's now not possible to buy it there, and there's been no sign of the devs for a long time, now.
- LocalH 5y agoClone Hero has netplay in the public test builds and wider format support than FoF (CH supports .chart format and afaik is the only significant 5-lane guitar/band game to do so). Phase Shift also unfortunately plays like complete ass, especially on guitar. The entire Guitar Hero community has pretty much shifted to Clone Hero, with a few diehard fans who still mod GH2/RB
- grecy 5y agoI second this comment, and if the OP is looking to grow his channel, I suggest watching a few from "Stuff Made Here". His channel has skyrockted from a few thousand subs less than a year ago to a few million now, and growing VERY fast.
- LocalH 5y ago>The game outputs at 30 fps and on faster songs, it would miss nearly 30% of notes. That 30fps is interlaced though, so there is 60Hz motion. All of the PS2 GH/RB games I know run at 60fps internally.
- nik12795 5y agoI don't understand exactly what you mean. There were only 30 frames coming in per second that I could parse. Do you think there might be a way to get 60fps from the PS2 and parse that in the OpenCV code?
- toast0 5y agoWhat's your video setup? A look around says Guitar Hero 2 will do 480p with component cables. I'd assume it puts out a new frame each time, but it could be 480p with doubled frames.
- LocalH 5y agoYeah that's probably the easiest and most latency-free way. It's absolutely 60fps of unique frames, especially with GH2. GH1 can also be hacked for 480p, even though the option isn't present in the normal game.
- utopcell 5y agoyou bring shame to the world of robots! there are humans that can achieve 100% notes. mere, humans!
- nik12795 5y agoHahaha
- Robotbeat 5y agoHonestly, I think people underestimate how good humans are mechanically. Like, a human arm is a 7 degree of freedom arm capable of incredibly fast motion. Some of the songs on Beat Saber, on Expert Plus, are effectively impossible for any existing 7 axis serial robot arms to actuate fast enough to equal the best human players if you strapped VR controllers to where the end effectors are. Maybe some sort of parallel robot (like a SCARA) robot could do it. Two robot arms with 7 degrees of freedom (the extra degree to allow them to avoid colliding with each other), plus the camera has to also move to avoid obstacles. It'd be really tough to beat the best human players, from a mechanical point of view. You'd need custom robots. https://www.youtube.com/watch?v=LLb7cLTrSCE https://www.youtube.com/watch?v=LLb7cLTrSCE
- kortex 5y agoNot only that, but each arm can produce dozens of newton-meters of torque, with possibly sub-millimeter precision, and like you said, fast. Just building an arm with comparable specs is quite the challenge. You end up needing hydraulics because even the finest, most expensive transmissions with belts to move the motors closer to the body, still end up being too heavy to achieve human agility+strength. I looked into building a robot arm which could cook breakfast. It's almost ludicrous just speccing out the torque requirements to pick up a skillet. You end up having to pick harmonic drives and less speed but more stiffness, or hydraulics, which are lighter, but messy, less stiff (can be both good and bad) and less reproducible.
- Robotbeat 5y agoYup. I think people also underestimate how much Boston Dynamics benefitted from using high performance hydraulics for some of their more impressive robot demos (vs others that relied on more conventional electric robotic actuators).
- layer8 5y agoNice! If you're looking for a follow-up project, you could pick up on this guy's work and build a Dance Dance Revolution bot: https://www.youtube.com/watch?v=M4DvlAb4eA4 https://www.youtube.com/watch?v=M4DvlAb4eA4 (...and then challenge Boston Dynamics to a DDR duel ;))
- nik12795 5y agoLol challenging Boston Dynamics would be awesome
- fbelzile 5y agoVery nice work! I noticed that if there were notes missed, they all seemed like the long hold ones. Was that a limitation of the AI software or was that by design to keep the strumming part easier to make?
- nik12795 5y agoThere wasn't any limitation from the mechanical perspective. The AI did a nearly perfect job of recognizing notes. I don't know exactly why it missed some notes, but my hypothesis is that it was the speed of notes in quick succession. If I do a follow up video with Jon Bot Jovi, I'll dig into it more!
- fbelzile 5y agoNice, thanks for the response! The video was well produced too, congrats! :)
- nik12795 5y agoThanks! Appreciate you saying that :)
- bredren 5y agoYa I was glad you cut it to specific pointed explanations. Maybe I missed it, but it looks like there are 3d printed parts. They aren’t what makes jbj’s brain work, but they might be worth going over the models (even if crude) as part of a follow up video. If you posted that to /r/3dprinting it would be another audience to pull in.
- bshep 5y agoit might have been a mechanical issue on the guitar itself ( i remember a similar issue when i played the game, it would sometimes detect a double strum or miss it ) Anyway it looks awesome!
- hinoki 5y agoVery nice! For debugging missed notes, do you replay the video to the bot? You could capture clips by detecting when the score multiplier drops.
- jsjohnst 5y agoDid you consider potentially not using solenoids at all? Underneath each of the GH buttons is a momentary switch, so theoretically you can remove all mechanical forms of potential failure by eliminating the switches and solenoids.
- qyi 5y ago> couldn't fathom how people would play Through The Fire And The Flames Either the game is slow enough to reason about in real time (certainly, a sweep like 4-3-2-1-2-3-4 as seen in the video wouldn't trip up any seasoned player) or the player resorts to memorizing finger positions (2+3 [2nd button held down at the same time as 3rd], 3+4, 1+4, etc) as well as sequences (1,1,1, 2+3, 1,1,1, 2+3). Hopefully one can pause at a certain segment and redo it as much as needed (like with real instruments), instead of starting from the beginning of the song each time.