6 ms·
Piano on Paper
- elwell 3y agoObviously a bit rough, but a start. This is the kind of content I like on HN.
- mayureshsatam 3y agoyeah! it was a hella challenging to detect touch.
- tmountain 3y agoI had an idea for a similar project. Basically, a tabletop drum kit. In this case, you’d tap on a few objects on the table that make distinct sounds and then map that back to a virtual drum kit in your DAW using midi. I never got around to building it, but I am still keen on the idea.
- timlod 3y agoThat already exists in some forms, most notably perhaps in Dubler 2. I’m also working on an open source version of something similar, albeit quite involved and aiming at professional real-time performance and drumkit augmentation (if all goes well to share on HN this month for the first time :))
- mayureshsatam 3y agoHey that is cool! lemme know if you need any help, send mail if required!
- iefbr14 3y agoReminds me of virtual laser keyboards, eg: https://nerdknowbetter.com/best-virtual-laser-keyboards https://nerdknowbetter.com/best-virtual-laser-keyboards
- userbinator 3y agoI'm surprised they actually exist instead of being a rendered design concept that I saw many years ago, and are apparently common and numerous enough that someone would write a listicle about them. They don't seem to cost more than a good regular keyboard either, but I've found reviews saying that the typing experience is about as uncomfortable as you'd expect from tapping on a hard surface.
- robinsonb5 3y agoThe typing experience is <chyrosran22>Hhhhhhhideous</chyrosran22>[1] [1] https://www.youtube.com/watch?v=1Cy1dPlo8No https://www.youtube.com/watch?v=1Cy1dPlo8No (NSFW)
- napier 3y agoYou don't need to type on a hard surface (but anyway, most people tap away on their phone screens for hours in a day). You can type on an octopus if you want. Anything can be a projection surface keyboard with enough inspiration.
- layer8 3y ago[flagged]
- deleted 3y ago[deleted]
- a1o 3y agoThis is pretty cool work, love the concept. I am curious if there are fast, easy to use and efficient ways to also detect the full hand. Maybe by placing the camera from above would make it easier to process but at the cost of complicating the setup.
- latchkey 3y agoPreviously: The pianist who learnt to play on a paper piano https://news.ycombinator.com/item?id=21823936 https://news.ycombinator.com/item?id=21823936 (December 18, 2019 — 1 points, 0 comments) It is a video of a guy in 2019 who couldn't afford a piano, so he drew it out on paper and learned that way and became an award winning musician.
- macintux 3y agoObviously in no way comparable, but you reminded me that many, many years ago, when I was taking typing in high school, I'd sit in class "typing" in mid-air (hands at my side) transcribing what the teacher was saying. It was a little annoying to people around me, but ended up being very effective.
- deleted 3y ago[deleted]
- dhosek 3y agoThere was a scene like this in the Talented Mr Ripley (1999).
- proee 3y agoWhen I was in 3rd grade we learned to type on a “keyboard” that was printed on a sheet of paper. I suppose it’s a novel way to teach kids to type with a budget of zero.
- dhosek 3y agoWhen I was in high school in the 80s, there was a piano class at the high school and the students had non-sounding, non-moving plastic piano keyboards to practice on at their desks. Nowadays, they get 60-key keyboards with headphones.
- devsda 3y agoI know that this is better than nothing, but as I understand, precise fingering is an important part of learning piano. Is there a risk of kids picking up wrong habits on a paper compared to a (cheap) real one ?
- xandrius 3y agoI guess it's always better bad habits than no habits at all. The context in which this is done is rather important. Also, it might prevent having a ton of lost and worthless keyboards in everyone's attic :D
- persnickety 3y agoI have a long password which I don't remember. I can type it on my laptop, but not recall it. Last holiday season I unexpectedly needed to access my master keyring, but all I had with me was my phone. Which doesn't have a laptop-sized or -shaped keyboard. No amount of conscious typing helped me remember. As a last-ditch effort, I found squared paper and a pen, drew a full-sized keyboard, and "typed" my password in on that. It took a couple of attempts, but I did eventually reverse-engineer my own password from my own muscle memory by carefully observing what my hands are doing.
- zarzavat 3y agoYou should write it down. I had such a password that only my fingers knew, not my head. Then I went a couple of months without using it. When I wanted to type the password again, I had completely forgotten it. Without even knowing the first letter, I had no hope of guessing it.
- p1mrx 3y agoI think this idea would work better using 3D printed keys. It's possible to print flexible springs, and filament can be used as a hinge pin. The computer just needs to recognize which keys have moved.
- mayureshsatam 3y agoyeah! but main motive was to make it as free as possible. Someone like me who does not have much of money to buy a piano but to play one see this as absolute win with trade offs ofc!
- p1mrx 3y agoI just made this piano key; the cost of filament is around $0.15: https://files.catbox.moe/kvop4m.webm https://files.catbox.moe/kvop4m.webm So it's in the ballpark of a couple dollars per octave. I agree that's not very competitive with paper.
- mayureshsatam 3y agoThat's cool, Sounds good. I bet you can sell keyboards with that sound.
- pama 3y agoI’ve often wondered about the reverse. Get the finest grand piano action, like the ones used in the Fazioli, and create a minimal mechanical unit that has just that piece and the pedals (with the action mechanics that connect the left/right pedals to the keys) and then nothing else but a couple of sensors and a couple fast USB-C. The Kawai Novus series did an OK action and left pedal, but then they connect it to a whole furniture and their sound synthesis and vibrations are imperfect. With modern powerful GPU the sound should be possible to become near perfect and easy to tune to one’s preference.
- SamBam 3y agoI am no master pianist, but I imagine much of the feel of such an instrument comes from the physical weight of the keys, and the feel as they strike the strings. Although good quality keyboards have "weighted" keys, they don't feel exactly the same. I think it would be hard to reproduce this exactly.
- pama 3y agoI agree. Perhaps I was unclear. I was hoping that someone can use the exact action of the grand piano, but skip the strings, soundboard and so on.
- ksherlock 3y agocraigslist is full of free upright pianos and occasionally free baby grands. Of course if you show up with a chainsaw instead of 5 men and a dolly...
- jasomill 3y agoSilent grand pianos exist, though the ones I'm aware of also include the acoustic bits as an option, e.g., https://usa.yamaha.com/products/musical_instruments/pianos/silent_piano/sh/index.html https://usa.yamaha.com/products/musical_instruments/pianos/s... https://www.boesendorfer.com/en/pianos/silent-edition https://www.boesendorfer.com/en/pianos/silent-edition
- 3y ago
- yakito 3y agoI've previously used PianoVision, and it worked quite well. The concept seems similar, now augmented with VR. I anticipate that VR pianos will evolve to such an extent that they'll replace traditional pianos in many scenarios (like teaching/learning) https://www.meta.com/experiences/5271074762922599/ https://www.meta.com/experiences/5271074762922599/
- josephg 3y ago> they'll replace traditional pianos in many scenarios I'm not convinced. Totally VR pianos are obviously a bad idea just like totally VR keyboards for typing are rubbish for anything more complex than a wifi password. And as for learning using VR - well, I'm not convinced by things like pianovision. How do you learn a piece with that? With sheet music I can play the music at my own speed, when and how I want. I can go back and play a measure a few times to experiment with timing. Speed up and slow down, and develop my own ear for how I want it to sound. If I'm feeling cheeky, I'll improvise extra notes in the song, or take some notes out if I'm not feeling it. At an instrument, sheet music is a polite suggestion. I can imagine playing something like pianovision and getting good at pushing the buttons at the right time like guitar hero. But being a human shaped CD player feels like a very small part of what makes a musician.
- superb_dev 3y agoGetting good at pushing the buttons is exactly it. Just being able to hit the note that you want to hit on an instrument is a skill
- m3at 3y agoI agree for pianos, because the mechanical feedback is part of what makes playing enjoyable. I'm not as sure for keyboards in VR! There has been a lot of research on non-invasive brain computer interface (BCI), including predictive systems that guess what you want to type, instead of where you're actually typing (example from 8y ago [1]). Simpler gesture recognition is already on the market (like this one, from 2020, for the apple watch [2]). And now bigger VR players are investing in the tech [3]. I expect useful brain interface to be integrated in common VR devices in a couple of generations. [1] Air Keyboard: Mid-Air Text Input Using Wearable EMG Sensors and a Predictive Text Modeland a Predictive Text Model; https://digitalcommons.dartmouth.edu/cgi/viewcontent.cgi?article=1113&context=senior_theses https://digitalcommons.dartmouth.edu/cgi/viewcontent.cgi?art... [2] https://mudra-band.com/ https://mudra-band.com/ [3] https://www.androidcentral.com/gaming/virtual-reality/zuckerberg-says-neural-interface-controls-are-close-to-ready https://www.androidcentral.com/gaming/virtual-reality/zucker...
- two_handfuls 3y agoThis requirement is logical but also.. ouch > Hand with all fingers.
- abrichr 3y agoCongratulations on shipping! You might find this interesting, a comment on https://news.ycombinator.com/item?id=35376138 https://news.ycombinator.com/item?id=35376138 (https://news.ycombinator.com/item?id=35379983 https://news.ycombinator.com/item?id=35379983): > I had been playing with the idea of creating a browser-based virtual piano for when I'm travelling and don't have access to a real piano but have my laptop with me. The idea would be to point the webcam down at the table between me and the laptop, and play on the table as if a piano were there. Then use the mediapipe framework [1] to capture finger positions, and use those to update a virtual environment like the one you have here. > I put it on hold due to the significant engineering required, but it seems you have already implemented (and open sourced!) the browser-based piano simulation component. > A quick scan through your repo indicates that this is all implemented in Python. I see that you are using mujoco_wasm [2]. [1] https://google.github.io/mediapipe/ https://google.github.io/mediapipe/ [2] https://github.com/zalo/mujoco_wasm https://github.com/zalo/mujoco_wasm
- deleted 3y ago[deleted]
- ipeev 3y agoWhere is the file touch_detection_model.h5?
- mayureshsatam 3y agowell actually you need to train the model then the file will be generated
- phlakaton 3y agoI'm glad you're finding ways of getting into music inexpensively. But my advice is: haptics are super important in music interfaces! Spend the extra effort to make something that you can satisfyingly press and which can give your fingers some mechanical feedback. Your playing will be sooooo much more rewarding.
- deleted 3y ago[deleted]
- mayureshsatam 3y agoyeaa you are right, but those haptics cost money :(
- phlakaton 3y agoSee if you can find – or make! – a cheap one.
- zerojames 3y agoThis is a delightful project. Congratulations on shipping it! I would be curious to see ways to do this where the piece of paper had lines so you could visually see on the paper which key was which. Doing this would be tricky, though. On first thought, an object detection model that detects each key from a perspective, then saves the coordinates, may work. Then you would calculate the Intersection over Union between the finger annd each region. This may only work with a top-down perspective, which might be less practical in real-world use. In any case, I always miss my keyboard when I am traveling. A project like this feels a lot more practical than the “portable” travel keyboards.
- mayureshsatam 3y agoyess I am currently working on that too!
- alwinaugustin 3y agoAppreciate your work. Happy coding !!
- SonOfLilit 3y agoMayuresh, I'd like to donate a keyboard instrument to you. Please choose between an AKAI MPK Mini and a Hohner Melodica and email me at my nick on gmail with a shipping address. Also, if you consider this effort much cheaper than a $100 instrument, maybe I could hook you up with a programming job?
- deleted 3y ago[deleted]
- napier 3y agoI suspect it's less about the music and more about the finger tracking, but to the extent it's about the music - billions of people have smartphones with cameras that would work with this code, but would unlikely be able to afford piano lessons or convince / inspire anyone to offer to purchase them an instrument without first demonstrating aptitude in some area. Definitely not intended at a dig at you and hope you manage to hook Mayuresh up with a keyboard instrument and a job!
- SonOfLilit 3y agoPhones have touch screens that are better touch-based MIDI controllers than paper+camera :) I've successfully played songs on phone pianos. However, neither solution is even close to a tactile instrument.
- mayureshsatam 3y agoYou are absolutely right, my intention behind it was that you could really play the piano not on a small screen (like a phone) but like an actual sized piano. I didn't want to buy one because they were expensive and I am more like "free is the better" guy. So I decided to make one on paper. This being rather a prototype, I am currently working on designing and mapping keys, improving upon touch recognition. ( also I have mailed you )
- SonOfLilit 3y ago
- lostinwonder 3y agoI'm a pianist and have actually resorted to drawing a keyboard on paper in the past when I've been without a piano for extended periods (yes I'm addicted lol). Although VR keyboards absolutely won't replace real pianos for obvious reasons they actually do have some use. They are certainly better than nothing and better than drawing a piano on physical paper. The accuracy of finger tracking still needs improvement but that will come and as haptics improves I'm sure a virtual piano one day might start to feel more realistic. Technology needs to catch up. I've been able to memorise simple pieces with PianoVisions air piano during free play. While playing with songs within the app I've been able to remember some harder works. Still connecting a midi piano is 100 times better no argument about that.but the virtual piano certainly has its place at the moment and will become stronger as tech improves.