6 ms·

I get that this leaves non-midi keyboards behind though, as well as all other instruments. If anyone has ideas on that topic I am all ears! Initially I do plan

I get that this leaves non-midi keyboards behind though, as well as all other instruments. If anyone has ideas on that topic I am all ears! Initially I do plan to keep the scope small to see how well it performs, identify the problem areas, etc.

jrumbut 6y ago
I encountered a keyboard with a very primitive, single player version of that feature and I was playing interesting tunes in a weekend having repeatedly made zero progress in traditional lessons. It's so helpful to just get into the flow of playing, then the things that teachers tell you make sense.
- corytheboyd 6y ago
  Yeah I can relate so much to that. Most lessons we cover concepts that don’t immediately click until I next sit down on my own and just focus on playing. After getting past the familiarity building, and into the flow state, all the things my teacher said start to make more sense haha. Some of the time at least, but that’s what weekly checkins are for :) It’s a slow process learning to read music in my thirties, but it’s just as possible as learning anything else, don’t listen to people who say you can’t learn music as an adult :)
m3at 6y ago
For non midi keyboards complemented with a webcam, hand tracking could do the trick, maybe paired with basic audio analysis to get the timing right. Some off the shelf tools are available [1] and pretty decent. It does go against your plan to start small though, but if you're interested feel free to shoot me an email (in profile) [1] https://google.github.io/mediapipe/solutions/hands.html https://google.github.io/mediapipe/solutions/hands.html
- corytheboyd 6y ago
  Hey! That looks interesting I’ll check it out and add it to the list of topics to revisit. Summarized in another comment but I am a bit worried about hand tracking for a couple reasons. Also mentioned in reply to another comment, but will summarize here for convenience as well! 1. While I know there is promising tech in this area (and I am very interested in it!) the tolerance for error while playing/performing music is slim to none. Will a solution end up being 75% accurate? 90%? Both are two low IMO. Pure MIDI keeps things in a relatively more controllable realm 2. Camera rigging. There are just so many variables here for things to go wrong/right. Quality of the camera, position, lighting. It’s challenging just finding a good method of positioning a camera even if you have the right equipment to locate it. All that said, I’ll still reinvestigate this zone of ideas in the future and your insight is much appreciated! First, need to prove that I can make the simple case work as well as I think I can :)
  - m3at 6y ago
    You're right to be wary, I re-read your use case and hand tracking alone won't cut it. However I still think that hand tracking _combined_ with audio input can be precise enough (above 99%). Here is an example of raw audio to midi: https://magenta.tensorflow.org/onsets-frames https://magenta.tensorflow.org/onsets-frames Regardless of the feasibility, it won't be as reliable as direct midi input, that's for sure! I think your goal of getting things working for a simpler case first is right, and wish you best luck :)

jrumbut 6y ago

I encountered a keyboard with a very primitive, single player version of that feature and I was playing interesting tunes in a weekend having repeatedly made zero progress in traditional lessons. It's so helpful to just get into the flow of playing, then the things that teachers tell you make sense.

corytheboyd 6y ago
Yeah I can relate so much to that. Most lessons we cover concepts that don’t immediately click until I next sit down on my own and just focus on playing. After getting past the familiarity building, and into the flow state, all the things my teacher said start to make more sense haha. Some of the time at least, but that’s what weekly checkins are for :) It’s a slow process learning to read music in my thirties, but it’s just as possible as learning anything else, don’t listen to people who say you can’t learn music as an adult :)

m3at 6y ago

For non midi keyboards complemented with a webcam, hand tracking could do the trick, maybe paired with basic audio analysis to get the timing right. Some off the shelf tools are available [1] and pretty decent. It does go against your plan to start small though, but if you're interested feel free to shoot me an email (in profile) [1] https://google.github.io/mediapipe/solutions/hands.html https://google.github.io/mediapipe/solutions/hands.html

corytheboyd 6y ago
Hey! That looks interesting I’ll check it out and add it to the list of topics to revisit. Summarized in another comment but I am a bit worried about hand tracking for a couple reasons. Also mentioned in reply to another comment, but will summarize here for convenience as well! 1. While I know there is promising tech in this area (and I am very interested in it!) the tolerance for error while playing/performing music is slim to none. Will a solution end up being 75% accurate? 90%? Both are two low IMO. Pure MIDI keeps things in a relatively more controllable realm 2. Camera rigging. There are just so many variables here for things to go wrong/right. Quality of the camera, position, lighting. It’s challenging just finding a good method of positioning a camera even if you have the right equipment to locate it. All that said, I’ll still reinvestigate this zone of ideas in the future and your insight is much appreciated! First, need to prove that I can make the simple case work as well as I think I can :)
- m3at 6y ago
  You're right to be wary, I re-read your use case and hand tracking alone won't cut it. However I still think that hand tracking _combined_ with audio input can be precise enough (above 99%). Here is an example of raw audio to midi: https://magenta.tensorflow.org/onsets-frames https://magenta.tensorflow.org/onsets-frames Regardless of the feasibility, it won't be as reliable as direct midi input, that's for sure! I think your goal of getting things working for a simpler case first is right, and wish you best luck :)