12 ms·
Show HN: I built a Guitar Hero robot [video]
- nik12795 5y agoI was never great at Guitar Hero and couldn't fathom how people would play Through The Fire And The Flames, so I made a robot that could do it for me. The video explains some of the technical details but I'll highlight a few other key points. - My first attempt at note recognition was looking for specific pixels on the screen. The thought was that I would be able to detect if a pixel changed color and, depending on the color, it would know which note it was. The game outputs at 30 fps and on faster songs, it would miss nearly 30% of notes. - The note recognition was eventually built using computer vision from OpenCV and specifically blob detection. It took a fair amount of effort to hone in the perfect filtering to make note recognition was highly accurate. - Implementing the solenoids threw me for a loop. I ended up burning through a lot of them since I'm new to robotics and didn't understand the relationship between power, volts, and current. At one point I was giving these solenoids 4x the power it was expecting. Needless to say, they burnt out quickly. I eventually talked with a robotics guy and he gave me a 30 min crash course which straightened me out. - The whole project, from idea to the YouTube video I linked in the post, took 2 months. It probably cost nearly $1,000 for everything (raspberry pi, pvc, solenoids, AC/DC power converter, relay modules, paint, stickers, etc). That might seem high, but like I said, I was learning about volts/current and part of that learning process was destroying many solenoids (oops).
- StavrosK 5y agoGreat work! Though, wouldn't it be easier to just open the controller up and use transistors to close the contacts? Transistors are cheaper!
- freedomben 5y agoNice work man, and posting it to HN was a great idea to start gaining some subscribers ;-) Giving away my PS2 and Guitar Hero is one of the only things I regret by the way. If someone created a Guitar Hero that ran on Linux I would gladly pay.
- LocalH 5y agohttps://clonehero.net/ https://clonehero.net/
- geococcyxc 5y agoThere's Frets on Fire: http://fretsonfire.sourceforge.net/ http://fretsonfire.sourceforge.net/
- Gamemaster1379 5y agoPhase Shift is by far the most comprehensive "knock off" Guitar Hero. Blows Frets on Fire and Clone Hero out of the water. Netplay, multi format support. Etc. All previous game charts are available in formats compatible with Phase Shift.
- ghusbands 5y agoIt seems to be abandoned and unavailable. They switched over to Early Access on Steam in 2019 or so, but it's now not possible to buy it there, and there's been no sign of the devs for a long time, now.
- LocalH 5y agoClone Hero has netplay in the public test builds and wider format support than FoF (CH supports .chart format and afaik is the only significant 5-lane guitar/band game to do so). Phase Shift also unfortunately plays like complete ass, especially on guitar. The entire Guitar Hero community has pretty much shifted to Clone Hero, with a few diehard fans who still mod GH2/RB
- grecy 5y agoI second this comment, and if the OP is looking to grow his channel, I suggest watching a few from "Stuff Made Here". His channel has skyrockted from a few thousand subs less than a year ago to a few million now, and growing VERY fast.
- LocalH 5y ago>The game outputs at 30 fps and on faster songs, it would miss nearly 30% of notes. That 30fps is interlaced though, so there is 60Hz motion. All of the PS2 GH/RB games I know run at 60fps internally.
- nik12795 5y agoI don't understand exactly what you mean. There were only 30 frames coming in per second that I could parse. Do you think there might be a way to get 60fps from the PS2 and parse that in the OpenCV code?
- toast0 5y agoWhat's your video setup? A look around says Guitar Hero 2 will do 480p with component cables. I'd assume it puts out a new frame each time, but it could be 480p with doubled frames.
- LocalH 5y agoYeah that's probably the easiest and most latency-free way. It's absolutely 60fps of unique frames, especially with GH2. GH1 can also be hacked for 480p, even though the option isn't present in the normal game.
- utopcell 5y agoyou bring shame to the world of robots! there are humans that can achieve 100% notes. mere, humans!
- nik12795 5y agoHahaha
- Robotbeat 5y agoHonestly, I think people underestimate how good humans are mechanically. Like, a human arm is a 7 degree of freedom arm capable of incredibly fast motion. Some of the songs on Beat Saber, on Expert Plus, are effectively impossible for any existing 7 axis serial robot arms to actuate fast enough to equal the best human players if you strapped VR controllers to where the end effectors are. Maybe some sort of parallel robot (like a SCARA) robot could do it. Two robot arms with 7 degrees of freedom (the extra degree to allow them to avoid colliding with each other), plus the camera has to also move to avoid obstacles. It'd be really tough to beat the best human players, from a mechanical point of view. You'd need custom robots. https://www.youtube.com/watch?v=LLb7cLTrSCE https://www.youtube.com/watch?v=LLb7cLTrSCE
- kortex 5y agoNot only that, but each arm can produce dozens of newton-meters of torque, with possibly sub-millimeter precision, and like you said, fast. Just building an arm with comparable specs is quite the challenge. You end up needing hydraulics because even the finest, most expensive transmissions with belts to move the motors closer to the body, still end up being too heavy to achieve human agility+strength. I looked into building a robot arm which could cook breakfast. It's almost ludicrous just speccing out the torque requirements to pick up a skillet. You end up having to pick harmonic drives and less speed but more stiffness, or hydraulics, which are lighter, but messy, less stiff (can be both good and bad) and less reproducible.
- Robotbeat 5y agoYup. I think people also underestimate how much Boston Dynamics benefitted from using high performance hydraulics for some of their more impressive robot demos (vs others that relied on more conventional electric robotic actuators).
- layer8 5y agoNice! If you're looking for a follow-up project, you could pick up on this guy's work and build a Dance Dance Revolution bot: https://www.youtube.com/watch?v=M4DvlAb4eA4 https://www.youtube.com/watch?v=M4DvlAb4eA4 (...and then challenge Boston Dynamics to a DDR duel ;))
- nik12795 5y agoLol challenging Boston Dynamics would be awesome
- fbelzile 5y agoVery nice work! I noticed that if there were notes missed, they all seemed like the long hold ones. Was that a limitation of the AI software or was that by design to keep the strumming part easier to make?
- nik12795 5y agoThere wasn't any limitation from the mechanical perspective. The AI did a nearly perfect job of recognizing notes. I don't know exactly why it missed some notes, but my hypothesis is that it was the speed of notes in quick succession. If I do a follow up video with Jon Bot Jovi, I'll dig into it more!
- fbelzile 5y agoNice, thanks for the response! The video was well produced too, congrats! :)
- nik12795 5y agoThanks! Appreciate you saying that :)
- bredren 5y agoYa I was glad you cut it to specific pointed explanations. Maybe I missed it, but it looks like there are 3d printed parts. They aren’t what makes jbj’s brain work, but they might be worth going over the models (even if crude) as part of a follow up video. If you posted that to /r/3dprinting it would be another audience to pull in.
- bshep 5y agoit might have been a mechanical issue on the guitar itself ( i remember a similar issue when i played the game, it would sometimes detect a double strum or miss it ) Anyway it looks awesome!
- hinoki 5y agoVery nice! For debugging missed notes, do you replay the video to the bot? You could capture clips by detecting when the score multiplier drops.
- jsjohnst 5y agoDid you consider potentially not using solenoids at all? Underneath each of the GH buttons is a momentary switch, so theoretically you can remove all mechanical forms of potential failure by eliminating the switches and solenoids.
- qyi 5y ago> couldn't fathom how people would play Through The Fire And The Flames Either the game is slow enough to reason about in real time (certainly, a sweep like 4-3-2-1-2-3-4 as seen in the video wouldn't trip up any seasoned player) or the player resorts to memorizing finger positions (2+3 [2nd button held down at the same time as 3rd], 3+4, 1+4, etc) as well as sequences (1,1,1, 2+3, 1,1,1, 2+3). Hopefully one can pause at a certain segment and redo it as much as needed (like with real instruments), instead of starting from the beginning of the song each time.
- StavrosK 5y agoI love this, especially the sound the solenoids make when hitting notes. Well done!
- exdeve 5y agoThis is so funny! Jon-bot-jovi rocks!
- nik12795 5y agoHaha thanks! Yeah it’s pretty hilarious
- jmfldn 5y agoThis is awesome. I was heavily involved with designing the audio and gameplay for DJ Hero, the sequel to Guitar Hero. I'd love to see you do this for DJ Hero next!
- nik12795 5y agoWow that’s sweet! Glad you liked it. What was the process like to design the audio/gameplay? Sounds like a great gig
- jmfldn 5y agoHey there, nice to meet some fans! Yeah it did really well critically and we shifted a lot of units but the marketing investment killed the profit as we spent on it like it was going to be the next Guitar Hero. It was always going to be more niche than that so this was its achilles heel. Activision loved what we did almost too much and killed the profitability. When I started to see Eminem and Jay Z turning up to promo events I thought "wow" / "oh crap how much did this cost us?!" Working on it was amazing. My job was remixing music for a video game and turning it into gameplay so what was not to love! I prototyped some of the earliest mixes in the game so helped to get the project through it's earliest milestones. I met a lot of famous people and got to remix Daft Punk, DJ Shadow and a few other heroes of mine. My favorite moment was opening the Ableton project sent to us by Daft Punk containing their stems. The music nerd / fanboy in me couldn't believe I got to play around with these stems that very few people ever had access to. They basically don't work with anyone as a rule so the fact that they collaborated on our project was insane. We had their likenesses in the game and the big pyramid stage. Was surreal! It was a great gig and I feel we produced a cult classic game that stood the test of time. The financials just didn't stack up.
- adrianh 5y agoThanks for sharing this — that is really cool! What are you up to now? Still in the music/tech space?
- 5y ago
- Baeocystin 5y agoI love this kind of silly/fun stuff. Thanks for sharing your project with us, it's great.
- nik12795 5y agoYou’re welcome! Thanks for checking it out
- kevinherron 5y agoWhat a trip... I also built a Guitar Hero "bot" some 13 years ago. Mine was not mechanical like this. Instead I spliced into the guitar with a microcontroller that I controlled via serial port from my laptop. I wrote software that analyzed recordings of the note chart, saved them to a file, and could play them back over the serial port by telling my microcontroller program what buttons to press. I don't have any pictures of the thing any more, but some of the YouTube videos live on: https://www.youtube.com/watch?v=n1YE8Wwo2rs https://www.youtube.com/watch?v=n1YE8Wwo2rs
- nik12795 5y agoWow how crazy is that! That’s really great. Obviously I had the privilege to leverage modern tech with CV, but amazing that you were able to accomplish it 13 years ago!
- kevinherron 5y agoYour version is so much cooler. It’s more like what I had in my head but wasn’t able to accomplish :)
- nik12795 5y agoThanks!! It was super challenging but it was a fun time and the payoff of it shredding was well worth it
- LocalH 5y agoI remember TomHannu
- vuciv1 5y agoGreat job. Very well presented. Keep it ul!
- nik12795 5y agoThank you!!
- coverband 5y agoThis was great, thanks for the clear explanation and no-fluff presentation. On a different topic, based on your other video —- what was the result from your Shark Tank audition?
- nik12795 5y agoHaha thanks for checking out the videos! That company ended up failing. I was almost on Shark Tank again for my company DirtByMail.com (we were talking with the producers of ABC), but didn’t make it on.
- njacobs5074 5y agoHave so many great memories of playing this game with my family. What an absolutely fantastic project you made for yourself!
- nik12795 5y agoThanks a lot!! Really glad you enjoyed
- dom96 5y agoNice! Big Mark Rober vibes in the video, was it a major inspiration for how you put this together? :)
- nik12795 5y agoThanks! Yes I love Mark and his videos. Huge inspiration for sure :)
- jbverschoor 5y agoOne minor nit pick: It only strums one direction
- ziml77 5y agoThat's how I used to play. For some reason strumming in both directions really threw my timing off. Damn did that hurt my arm for fast sections though (if it got too painful I would change to strumming both ways simply because the alternative of slowing down would end up with me missing more notes).
- jugg1es 5y agoI'm not sure it is physically possible for a human to strum only one direction on some of those songs on 'expert'. Our muscles just don't twitch that fast more than a few times in a row.
- kodt 5y agoAlso it isn’t necessary to strum for every note in Guitar Hero.
- foobar33333 5y agoYes but if you miss a single note you end up with a long chain of missed notes. The hammer ons are only really useful for the really fast sections where you can't physically strum fast enough for that section.
- nik12795 5y agoI did think of ways to have alternative strumming, but there's something hilarious about it just mashing the strum button as fast as it can. And since the solenoid was able to handle it, I kept it as is :)
- aledalgrande 5y agoInteresting, was expecting RL, but computer vision works too!
- nik12795 5y agoAh yes. And it worked beautifully!
- maratc 5y agoAbsolutely amazing. I probably spent more hours playing Rock Band than all other games combined. One thing was a bit unclear: is Raspberry Pi doing the blob detection, or is it running on the Macbook? Also, why only 97%?
- stefanmichael 5y agoa couple of nuances make the remaining 3% a bit more difficult to get, two that I noticed is that the bot is bad at holding sustain notes and also tends to make mistakes during long hammer-on sections because it hits two notes at the same time instead of realizing that they are individual notes. the second one is probably easy to fix by spreading the notes out a bit (hyperspeed mode) but the first one is a bit more annoying to program, probably
- nik12795 5y agoAgreed the sustained notes would be a bit of a programming annoyance, but if I had spent more time, I think I could have cracked it. I did get some MVP working for it but it was throwing some other things off, so I abandoned it in an effort to maximize accuracy.
- maratc 5y agoI think it might be easy implementing sustain on any and all notes until the next note needs to be hit. So instead of press-release-wait you do press-wait-release.
- maratc 5y agoNot holding sustain is not a failure per se in the game. If you've hit the note but did not sustain it, you just get less points.
- nik12795 5y agoThe Raspberry Pi is doing the blob detection! In terms of 97% and why not 100%: > There wasn't any limitation from the mechanical perspective. The AI did a nearly perfect job of recognizing notes. I don't know exactly why it missed some notes, but my hypothesis is that it was the speed of notes in quick succession. If I do a follow up video with Jon Bot Jovi, I'll dig into it more!
- brunojppb 5y agoThis is amazing work man! Creative and fun work while learning cool tech. Through The Fire And The Flames Is still in my playlist to this day and reminds me about how cool was to play my PS2 back then.
- nik12795 5y agoThanks a lot!! Yeah it's an epic song and is so nostalgic.
- ziml77 5y agoI love that you actually made it press the buttons instead of wiring directly into the guitar. Get all that authentic clacking :) Looks really cool!
- nik12795 5y agoThanks!! Appreciate it
- sodality2 5y agoThis is incredible, hope this video gets more traction! Subscribed.
- nik12795 5y agoThank you and I appreciate your subscription :) Hope to continue to entertain you in future videos!
- kregasaurusrex 5y agoVery cool mixture of software and hardware for your project! I've been wanting to build a Go-playing robot for the longest time, hoping the cost of 6DoF robotic arms comes down in the near future. Do you have future tweaks in mind for improving your score, like increasing the signal of the notes when they're first starting to appear on the screen?
- nik12795 5y agoA Go-playing robot would be sweet! I think the biggest improvement for score would be implementing logic to hold down long notes and then implement star power at the ideal time. That would make a dramatic difference!
- throw7 5y ago"The thing that makes this all work is artificial intelligence". How so? That was an interesting statement that I was keen on understanding.
- frankbreetz 5y agoHe use computer vision to get information. Computer vision is a subcategory of ai
- throw7 5y agoInteresting. I never realized computer scientists categorize "computer vision" as "artificial intelligence". Seems to me OP could have gotten 100% correctness from the "vision" input without going the "computer vision" route then.
- clircle 5y agoWhat is your suggested alternative solution?
- lars_thomas 5y agoAbsolutely is not. There was nothing AI in this, simple classical image processing tools only.
- frankbreetz 5y agoThe project is obviously super awesome. I am incapable of such a feat. What worries me is that if the AI CV algorithioms can't catch all the guitar chords, how are they ever going to see all the people crossing the street? 97% is pretty bad when it comes to identifying people in the crosswalk
- phalangion 5y agoThat is a fair concern, but also keep in mind that this is one guy’s project. And with even more effort and debugging, I’m sure he could get it to 100%.
- Robotbeat 5y agoI really think sensor fusion is how we get there, like LIDAR or imaging radar on top of a really, REALLY good multispectral machine vision system. It’s similarly disconcerting how bad I am (and other humans) at spotting people crossing the street at night if they’re wearing dark clothes. I gotta imagine there are analogous corner cases for machine vision, and a sensor fusion approach should help solve that.
- foobar33333 5y agoAnd now you see why self driving cars aren't single developer hobby projects released in to the wild. Also you may be overestimating human vision. I doubt normal drivers have 97% accuracy spotting all hazards on the road. We get by mostly on the fact that we keep hazards off the road whenever possible.
- fny 5y agoNeat project, but where is the AI exactly? I doesn't look liked you used anything "AI" related from OpenCV.
- eliasbagley 5y agoOnce upon a time blob detection was considered "AI"
- jugg1es 5y agoWhat a crazy project. I love that you went all the way with it using the actual guitar. I would think you could have just reverse engineered the guitar signals but then it wouldn't be a robot.
- nik12795 5y agoYes exactly! Thank you!!
- blamazon 5y agoGreat job on this video. You took an interesting concept, implemented a solution, and conveyed it to the audience in a concise way that was respectful of our time. I love to see folks flexing their creative brains in this way across software and hardware. Video editing and production too. Subscribed and rang the bell. One nitpick on presentation--I am a youtube addict and the first thing I notice on any vlog-style video is mic quality. I don't have words to describe the audiology of it all but I instantly know when someone sounds 'pro' vs. 'amateur'. This video and the others you've posted recently, have narration that sounds 'amateur' to my youtube addled mind. I don't know anything about microphones and stuff, but if you can tackle that I think it will noticeably elevate your videos! Cheers.
- nik12795 5y agoThanks for taking the time to write that. It means a lot to me and I'm really happy you're enjoying the videos. Good call on the microphone. I'm still testing some things out so will keep that in mind as I produce the next few videos. Thanks for subscribing and looking forward to hearing what you think of the next video!
- hncommenter13 5y agoThis is obviously very impressive, and far better than I could ever do myself. But I must say it bears more than a passing resemblence to the version below, first uploaded to YouTube in 2009. Including the specific appearance of the robot and the graphic design of its eyes: https://www.youtube.com/watch?v=0mWHVvKb1hM https://www.youtube.com/watch?v=0mWHVvKb1hM Perhaps worth acknowledging that prior effort? (I have no relationship, just feels like that effort--with 2009 technology--deserves some credit.)
- nik12795 5y agoThanks! Actually had a similar comment on my video: > Yes! I didn't realize this other robot existed until I was partially through the project and was doing more research. I loved their form factor so I used it for Jon Bot Jovi. IIRC, they struggled to beat TTFATF (they got 60-70%?) because their note detection software struggled to decipher long blocks of notes tightly together
- qwertox 5y agoCode on screen at https://youtu.be/htk6eXxpSNA?t=23 https://youtu.be/htk6eXxpSNA?t=23 `if y < prevValue or prevValue is None:` should be `if prevValue is None or y < prevValue:` because the first one will make a comparison for `y < None`, while the second one won't. `y < None` will always return `False` while `prevValue is None` will then always return `True`, making the entire statement `True` if `prevValue` is `None`. It is not a bug but it's something which I would correct. In the second one `y < prevValue` would not get evaluated if `prevValue` is `None` and it's not really clear what `y < None` means. -- Great work which reminded me of the Ace of Spades-Robots. It also shows how insanely good humans can be at playing guitars.
- nik12795 5y agoAppreciate you looking through the code and offering that suggestion. I'll take a closer look ;) Also was not aware of the Ace of Spades robots. Just watched the video and it's epic!
- Yajirobe 5y agothat's a condescending comment
- qwertox 5y agoMy apologies to nik12795 if he felt that way, it certainly wasn't meant to be condescending at all. It's really impressive what he archived there, not only the image recognition, considering the high speed of certain parts of the video, but also the hardware implementation. It was my mistake to only focus so much on the code-thing. Thanks for pointing it out.
- quickthrower2 5y agoHi there I have a question - why relays and not transistors? I would have though relays would add to the latency, and might be why some notes were missed.
- nik12795 5y agoHonestly, I'm pretty new to robotics and haven't had a chance to mess around with transistors. I was able to find some videos on solenoids and relays and that made it easy for me to get something up and running quickly. The latency wasn't an issue though. Even on expert mode on TTFATF, I still had to program a delay to time everything up. But that's great to know for future projects if I need a super low-latency solution.
- quickthrower2 5y agoI don’t have massive experience but about 25 years ago I purchased a unit using transistors to power large stepper motors (probably a similar style of load to a solenoid - inductive with quick on/off) and it worked well, and didn’t get too hot. But don’t ask me what transistor tech it used, or the name of the company selling it! But if you don’t need the response time or longivity then relays are probably much simpler to set up so a good choice.
- anonu 5y agovery cool stuff. feels Mark Rober-y too which is great. Whats the tech behind the image recognition?
- nik12795 5y agoThanks! The image recognition is done by OpenCV which I highly recommend checking out.
- drivers99 5y agoFor some reason I emotionally want it to play perfectly. I would probably take the note data from multiple runs and use it as a basis to learn/memorize what the notes should be (like a human). I guess I’m looking at it like a TAS (tool assisted speed run) and try to perfect the inputs, even manually if necessary.
- njkumarr 5y agothis is so dope
- Zedronar 5y agoNobody talking about Rocksmith here!? Really!?