5 ms·

NeuralPi: Raspberry Pi guitar pedal using neural networks

97 points by rcarmo 4y ago

nickkao 4y ago
Silly question but why is the neural network necessary?
- ruined 4y ago
  tone generated by analog circuits is notoriously difficult to reproduce digitally. the conceptual behavior is often easy to model, but hardware deviates from theory in ways that are technically subtle but audibly apparent. it's deterministic, but the parameters may be unknown and approximate values must often be discovered by iterative guess-and-check. researching and manually modeling an approximation can be incredibly tedious and still fall short. this is exactly the kind of application that machine learning excels at.
  - claxo 4y ago
    Ohhh... people over chiptunes complains that the Commodore SID is hard to emulate because the analog parts...
    - eurasiantiger 4y ago
      The analog parts provide some interesting undocumented features. It’s possible to play 4 channels of 8-bit samples, filter them and still have 2 of the 3 SID channels free for other audio. https://www.youtube.com/watch?v=Y6mXLxUZvzg https://www.youtube.com/watch?v=Y6mXLxUZvzg
      - eurasiantiger 4y ago
        Hey downvoters, I never wrote playing 4 channels was a feature—it’s still possible, though.
- pjbk 4y ago
  Not a silly question. One of the things I always miss from these studies is comparison with more classical inference models. In particular, back in the days we had to train against outputs for PCA models (linear and non-linear) in order to get a feasible production version with the best reduced parameters and justify equivalent performance. Today we have so much computing power that nobody cares. Audio is a particular good application of this. For example, the old ADPCM algorithms have evolved naturally into their ML counterparts. Some have even less parameters and thus are more computationally efficient because of the advantages of flexible feedback of the training or production models (e.g. RNNs).
JKCalhoun 4y ago
Game changer. Pedal/amp vendors will likely scramble to figure out if they can sue an implementation of a neural network trained with their hardware. I know there have been amp simulation for over a decade now but this feels different.
- lkschubert8 4y ago
  I believe this already existed in a proprietary implementation with https://neuraldsp.com/quad-cortex https://neuraldsp.com/quad-cortex.
  - bjelkeman-again 4y ago
    I have a Quad Cortex. It is very good, but the Pi based solution beats it handily on price. :)
- aconsult1 4y ago
  There's a similar interesting solution here: https://forum.moddevices.com/t/ai-research-at-mod/8368 https://forum.moddevices.com/t/ai-research-at-mod/8368
armchairhacker 4y ago
IMO audio is ripe for neural network advancements like pictures with DALL-E and Stable Diffusion. Even if it's not the same, maybe it's just neural networks to emulate expensive instruments and improve quality of digital filters and effects
- niccl 4y ago
  I understand that the Peter Jackson Beatles documentary did some very clever things with ML to separate the voices and instruments to make a decent soundtrack from a fairly mediocre input stream.
SMAAART 4y ago
Disruption ahead.
cognivorous 4y ago
Cool. There’s a startup working on something very similar, and they also have a fast audio inference engine and raspberry pi implementation: https://waveshaper.ai https://waveshaper.ai
pmontra 4y ago
Wonderful project, if you already have a stock of Raspberry Pis at home. I wonder if we'll ever be able to buy a Pi again. They've been out of stock everywhere since forever. I'm starting to consider it a dead platform and looking at different hardware (example Odroid.)
- mindcrime 4y ago
  I wonder if we'll ever be able to buy a Pi again. They've been out of stock everywhere since forever. You can get them now, if one or more of a few things is true: 1. You're willing to pay exorbitant scalper prices. In that case, just head over to Ali-Express and you can pick up a Pi with no problem. 2. You're very patient and willing to watch rpilocator[1] a lot in order to score a Pi. I do that and managed to score one more 8GB model during a 24 hour (or so) long window when Elektor had some in stock last week. Just keep checking, they pop up here and there. 3. You live near a Microcenter store. Apparently they get stock in every now and then as well. Sadly that only benefits the people who live near by. A group which I can't count myself among. sigh I'm starting to consider it a dead platform and looking at different hardware (example Odroid.) Things are going to get better eventually. That's very close to 100% certain. Pi's aren't in short supply because of anything the RPi folks are doing wrong, it's down to the global chip shortage that's affecting just about everything.[2][3] But the bullwhip effect will almost certainly play out, like it always does and at some point there will be glut of all the chips that are in short supply today and probably a glut of Raspberry Pi's as well. The only issue is predicting when that will happen. [1]: https://rpilocator.com/ https://rpilocator.com/ [2]: https://www.yahoo.com/now/chipageddon-could-last-until-2023-132341226.html https://www.yahoo.com/now/chipageddon-could-last-until-2023-... [3]: https://en.wikipedia.org/wiki/2020%E2%80%93present_global_chip_shortage https://en.wikipedia.org/wiki/2020%E2%80%93present_global_ch...
  - ninjagoo 4y ago
    > Pi's aren't in short supply because of anything the RPi folks are doing wrong, it's down to the global chip shortage that's affecting just about everything Aren't the RPI leaders on record saying that the vast bulk of RPI production is being allocated to commercial entities instead of retail? It would be interesting to see data on the # of RPIs being manufactured (over time - to see impact and recovery from Covid), and contrast it with retail availability.
0x20cowboy 4y ago
Very cool, but can’t we already do this with impulse responses? This seems kind of overly complicated to emulate something that already exists. However, I can imagine this becoming much more interesting - describing some tone and effects and having it make a custom cocktail of sounds would be nuts.
- dharma1 4y ago
  Impulse responses don’t capture distortion or compression, just frequency response
  - jcelerier 4y ago
    Basic impulses don't capture non-linearity but Volterra series do. See e.g. https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=Practical+modelling+of+nonlinear+audio+systems+using+the+Volterra+series&btnG=#d=gs_qabs&t=1662889262708&u=%23p%3DOB_Iw0BJ-rwJ https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=Prac...
ComplexSystems 4y ago
Some have asked how this is different from what an impulse response convolution plugin can do. The simple answer is that IR convolution is a linear, time-invariant system. It can adjust the amplitudes and phases of different frequencies, but it can't create new ones. What this is doing is basically a nonlinear generalization of that, which makes it better able to model overdrive, amp saturation, and so on. One thing I am curious about is exactly how this works. How do neural networks make black box nonlinear system modeling possible, and how does this relate to something like a Volterra series? Would love an explanation and/or some sources if anyone has any.
manmal 4y ago
Guitarists who wanted to use realistic amp emulations have been able to use profiler amps from Kemper for years: https://www.kemper-amps.com/products/profiler/line-up https://www.kemper-amps.com/products/profiler/line-up With those you can either load profiles (models) made by other people, or „profile“ your own amps or pedals by playing back and at the same time recording a 2-3 minute track (sounds like various kinds of white & pink noise) through them. I have one at home, and it sounds incredibly good. Once you‘ve loaded a bunch of amps into a van after an exhausting gig you come to appreciate having a lightweight option instead.
- williamcotton 4y ago
  Although I don’t use an amp modeler I ripped the guts out of my Twin Reverb and replaced the heavy ceramic speakers with neodymium speakers and an external solid-state amp head from Quilter. No more moving cart for gigging and I still have a pleasure of standing next to an open-backed 2x12! I’m too much of a minimalist for a modeling amp. I want physical dials and switches that are easily tweaked during performance and I want to be familiar with the entire range of settings along the audio path!
dharma1 4y ago
This is based on the work from Aalto University / NeuralDSP (which the latter has later commercialised neuraldsp.com) - https://www.mdpi.com/2076-3417/10/3/766/htm https://www.mdpi.com/2076-3417/10/3/766/htm https://github.com/damskaggep/WaveNetVA https://github.com/damskaggep/WaveNetVA IK multimedia has a similar app coming - https://www.ikmultimedia.com/products/aimachinemodeling/ https://www.ikmultimedia.com/products/aimachinemodeling/ Love to see this as an open source project though
davidy123 4y ago
Silly question but why is the Pi necessary? Isn't this a Linux project, or actually a C++ project? My guess is it's because device names are a -mess- between Linux (etc) devices (starting with, the same device has different names ex /dev, lsusb, alsa|pulse|pipewire list devices; multiple layers of indirection does not make for a great casual experience), and this simplifies the mess to one well-known device to make it less of a hair-pulling exercise. Which is telling, because Pi is sold as a learning kit, and it takes away this most unhelpful level of knowledge. I guess looking for a meta project where the driver was a specification, 'stereo-96+khz-active' (that survived reboots/hot plugging) would just create one more meta project? Like https://editorconfig.org/ https://editorconfig.org/. But maybe its about time, cause "Linux" and even "open source" loses a lot of value when getting something running is 90% arbitrary yak-shaving, where people need deep magic to get audio working the way they like. I mean, you have to be a 90s-2020s cruft (no other word for it) expert to play with supposedly user friendly software.
- i_am_proteus 4y ago
  Pi+Hifiberry remains the most accessible small computer the public can use for such an audio project. An alternative might be https://www.electro-smith.com/daisy https://www.electro-smith.com/daisy
- lloydatkinson 4y ago
  For the SEO
- squarefoot 4y ago
  > Silly question but why is the Pi necessary? Not sure about this project, but generally it is not. All it needs is a small board capable of running Linux and the necessary drivers for external ADCs/DACs where necessary, plus the digital fx software. As an example, Guitarix runs also on ARM and can work on cheaper boards such as the Orange PI, Nano PI and many others cheaper and more obtanium than the Raspberry PI. https://guitarix.org/ https://guitarix.org/ In some cases you don't even need to run Linux. There are many effects projects using a cheap Teensy board plus its piggybacked audio card; it features a really powerful audio library and is compatible with the Arduino IDE. https://www.pjrc.com/teensy/ https://www.pjrc.com/teensy/ https://www.pjrc.com/teensy/td_libs_Audio.html https://www.pjrc.com/teensy/td_libs_Audio.html The Teensy is truly amazing, to the point one can build synthesizers that just a few years ago would cost hundreds of bucks. Take a look for example at the TSynth, 100% Open Hardware & Open Source, also available in kit. https://electrotechnique.cc/ https://electrotechnique.cc/ Demo here: https://www.youtube.com/watch?v=uCA2L7CeWSE https://www.youtube.com/watch?v=uCA2L7CeWSE
wdfx 4y ago
Nice project - I guess I'll be spending today getting it running on my Pi pedal https://doug.lon.dev/software/hardware/2020/07/26/guitarix-pedal.html https://doug.lon.dev/software/hardware/2020/07/26/guitarix-p... https://doug.lon.dev/software/hardware/2022/03/01/louper.html https://doug.lon.dev/software/hardware/2022/03/01/louper.htm...
ewuhic 4y ago
Is there such a nicely packaged neural network, but for voice as well?
djmips 4y ago
Paper on Efficient neural networks for real-time modeling of analog dynamic range compression. https://arxiv.org/pdf/2102.06200.pdf https://arxiv.org/pdf/2102.06200.pdf Might be interesting to those reading in this post.

97 points by rcarmo 4y ago

nickkao 4y ago

Silly question but why is the neural network necessary?

ruined 4y ago
tone generated by analog circuits is notoriously difficult to reproduce digitally. the conceptual behavior is often easy to model, but hardware deviates from theory in ways that are technically subtle but audibly apparent. it's deterministic, but the parameters may be unknown and approximate values must often be discovered by iterative guess-and-check. researching and manually modeling an approximation can be incredibly tedious and still fall short. this is exactly the kind of application that machine learning excels at.
- claxo 4y ago
  Ohhh... people over chiptunes complains that the Commodore SID is hard to emulate because the analog parts...
  - eurasiantiger 4y ago
    The analog parts provide some interesting undocumented features. It’s possible to play 4 channels of 8-bit samples, filter them and still have 2 of the 3 SID channels free for other audio. https://www.youtube.com/watch?v=Y6mXLxUZvzg https://www.youtube.com/watch?v=Y6mXLxUZvzg
    - eurasiantiger 4y ago
      Hey downvoters, I never wrote playing 4 channels was a feature—it’s still possible, though.
pjbk 4y ago
Not a silly question. One of the things I always miss from these studies is comparison with more classical inference models. In particular, back in the days we had to train against outputs for PCA models (linear and non-linear) in order to get a feasible production version with the best reduced parameters and justify equivalent performance. Today we have so much computing power that nobody cares. Audio is a particular good application of this. For example, the old ADPCM algorithms have evolved naturally into their ML counterparts. Some have even less parameters and thus are more computationally efficient because of the advantages of flexible feedback of the training or production models (e.g. RNNs).

JKCalhoun 4y ago

Game changer. Pedal/amp vendors will likely scramble to figure out if they can sue an implementation of a neural network trained with their hardware. I know there have been amp simulation for over a decade now but this feels different.

lkschubert8 4y ago
I believe this already existed in a proprietary implementation with https://neuraldsp.com/quad-cortex https://neuraldsp.com/quad-cortex.
- bjelkeman-again 4y ago
  I have a Quad Cortex. It is very good, but the Pi based solution beats it handily on price. :)
aconsult1 4y ago
There's a similar interesting solution here: https://forum.moddevices.com/t/ai-research-at-mod/8368 https://forum.moddevices.com/t/ai-research-at-mod/8368

armchairhacker 4y ago

IMO audio is ripe for neural network advancements like pictures with DALL-E and Stable Diffusion. Even if it's not the same, maybe it's just neural networks to emulate expensive instruments and improve quality of digital filters and effects

niccl 4y ago
I understand that the Peter Jackson Beatles documentary did some very clever things with ML to separate the voices and instruments to make a decent soundtrack from a fairly mediocre input stream.

SMAAART 4y ago

Disruption ahead.

cognivorous 4y ago

Cool. There’s a startup working on something very similar, and they also have a fast audio inference engine and raspberry pi implementation: https://waveshaper.ai https://waveshaper.ai

pmontra 4y ago

Wonderful project, if you already have a stock of Raspberry Pis at home. I wonder if we'll ever be able to buy a Pi again. They've been out of stock everywhere since forever. I'm starting to consider it a dead platform and looking at different hardware (example Odroid.)

mindcrime 4y ago
I wonder if we'll ever be able to buy a Pi again. They've been out of stock everywhere since forever. You can get them now, if one or more of a few things is true: 1. You're willing to pay exorbitant scalper prices. In that case, just head over to Ali-Express and you can pick up a Pi with no problem. 2. You're very patient and willing to watch rpilocator[1] a lot in order to score a Pi. I do that and managed to score one more 8GB model during a 24 hour (or so) long window when Elektor had some in stock last week. Just keep checking, they pop up here and there. 3. You live near a Microcenter store. Apparently they get stock in every now and then as well. Sadly that only benefits the people who live near by. A group which I can't count myself among. sigh I'm starting to consider it a dead platform and looking at different hardware (example Odroid.) Things are going to get better eventually. That's very close to 100% certain. Pi's aren't in short supply because of anything the RPi folks are doing wrong, it's down to the global chip shortage that's affecting just about everything.[2][3] But the bullwhip effect will almost certainly play out, like it always does and at some point there will be glut of all the chips that are in short supply today and probably a glut of Raspberry Pi's as well. The only issue is predicting when that will happen. [1]: https://rpilocator.com/ https://rpilocator.com/ [2]: https://www.yahoo.com/now/chipageddon-could-last-until-2023-132341226.html https://www.yahoo.com/now/chipageddon-could-last-until-2023-... [3]: https://en.wikipedia.org/wiki/2020%E2%80%93present_global_chip_shortage https://en.wikipedia.org/wiki/2020%E2%80%93present_global_ch...
- ninjagoo 4y ago
  > Pi's aren't in short supply because of anything the RPi folks are doing wrong, it's down to the global chip shortage that's affecting just about everything Aren't the RPI leaders on record saying that the vast bulk of RPI production is being allocated to commercial entities instead of retail? It would be interesting to see data on the # of RPIs being manufactured (over time - to see impact and recovery from Covid), and contrast it with retail availability.

0x20cowboy 4y ago

Very cool, but can’t we already do this with impulse responses? This seems kind of overly complicated to emulate something that already exists. However, I can imagine this becoming much more interesting - describing some tone and effects and having it make a custom cocktail of sounds would be nuts.

dharma1 4y ago
Impulse responses don’t capture distortion or compression, just frequency response
- jcelerier 4y ago
  Basic impulses don't capture non-linearity but Volterra series do. See e.g. https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=Practical+modelling+of+nonlinear+audio+systems+using+the+Volterra+series&btnG=#d=gs_qabs&t=1662889262708&u=%23p%3DOB_Iw0BJ-rwJ https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=Prac...

ComplexSystems 4y ago

Some have asked how this is different from what an impulse response convolution plugin can do. The simple answer is that IR convolution is a linear, time-invariant system. It can adjust the amplitudes and phases of different frequencies, but it can't create new ones. What this is doing is basically a nonlinear generalization of that, which makes it better able to model overdrive, amp saturation, and so on. One thing I am curious about is exactly how this works. How do neural networks make black box nonlinear system modeling possible, and how does this relate to something like a Volterra series? Would love an explanation and/or some sources if anyone has any.

manmal 4y ago

Guitarists who wanted to use realistic amp emulations have been able to use profiler amps from Kemper for years: https://www.kemper-amps.com/products/profiler/line-up https://www.kemper-amps.com/products/profiler/line-up With those you can either load profiles (models) made by other people, or „profile“ your own amps or pedals by playing back and at the same time recording a 2-3 minute track (sounds like various kinds of white & pink noise) through them. I have one at home, and it sounds incredibly good. Once you‘ve loaded a bunch of amps into a van after an exhausting gig you come to appreciate having a lightweight option instead.

williamcotton 4y ago
Although I don’t use an amp modeler I ripped the guts out of my Twin Reverb and replaced the heavy ceramic speakers with neodymium speakers and an external solid-state amp head from Quilter. No more moving cart for gigging and I still have a pleasure of standing next to an open-backed 2x12! I’m too much of a minimalist for a modeling amp. I want physical dials and switches that are easily tweaked during performance and I want to be familiar with the entire range of settings along the audio path!

dharma1 4y ago

This is based on the work from Aalto University / NeuralDSP (which the latter has later commercialised neuraldsp.com) - https://www.mdpi.com/2076-3417/10/3/766/htm https://www.mdpi.com/2076-3417/10/3/766/htm https://github.com/damskaggep/WaveNetVA https://github.com/damskaggep/WaveNetVA IK multimedia has a similar app coming - https://www.ikmultimedia.com/products/aimachinemodeling/ https://www.ikmultimedia.com/products/aimachinemodeling/ Love to see this as an open source project though

davidy123 4y ago

Silly question but why is the Pi necessary? Isn't this a Linux project, or actually a C++ project? My guess is it's because device names are a -mess- between Linux (etc) devices (starting with, the same device has different names ex /dev, lsusb, alsa|pulse|pipewire list devices; multiple layers of indirection does not make for a great casual experience), and this simplifies the mess to one well-known device to make it less of a hair-pulling exercise. Which is telling, because Pi is sold as a learning kit, and it takes away this most unhelpful level of knowledge. I guess looking for a meta project where the driver was a specification, 'stereo-96+khz-active' (that survived reboots/hot plugging) would just create one more meta project? Like https://editorconfig.org/ https://editorconfig.org/. But maybe its about time, cause "Linux" and even "open source" loses a lot of value when getting something running is 90% arbitrary yak-shaving, where people need deep magic to get audio working the way they like. I mean, you have to be a 90s-2020s cruft (no other word for it) expert to play with supposedly user friendly software.

i_am_proteus 4y ago
Pi+Hifiberry remains the most accessible small computer the public can use for such an audio project. An alternative might be https://www.electro-smith.com/daisy https://www.electro-smith.com/daisy
lloydatkinson 4y ago
For the SEO
squarefoot 4y ago
> Silly question but why is the Pi necessary? Not sure about this project, but generally it is not. All it needs is a small board capable of running Linux and the necessary drivers for external ADCs/DACs where necessary, plus the digital fx software. As an example, Guitarix runs also on ARM and can work on cheaper boards such as the Orange PI, Nano PI and many others cheaper and more obtanium than the Raspberry PI. https://guitarix.org/ https://guitarix.org/ In some cases you don't even need to run Linux. There are many effects projects using a cheap Teensy board plus its piggybacked audio card; it features a really powerful audio library and is compatible with the Arduino IDE. https://www.pjrc.com/teensy/ https://www.pjrc.com/teensy/ https://www.pjrc.com/teensy/td_libs_Audio.html https://www.pjrc.com/teensy/td_libs_Audio.html The Teensy is truly amazing, to the point one can build synthesizers that just a few years ago would cost hundreds of bucks. Take a look for example at the TSynth, 100% Open Hardware & Open Source, also available in kit. https://electrotechnique.cc/ https://electrotechnique.cc/ Demo here: https://www.youtube.com/watch?v=uCA2L7CeWSE https://www.youtube.com/watch?v=uCA2L7CeWSE

wdfx 4y ago

Nice project - I guess I'll be spending today getting it running on my Pi pedal https://doug.lon.dev/software/hardware/2020/07/26/guitarix-pedal.html https://doug.lon.dev/software/hardware/2020/07/26/guitarix-p... https://doug.lon.dev/software/hardware/2022/03/01/louper.html https://doug.lon.dev/software/hardware/2022/03/01/louper.htm...

ewuhic 4y ago

Is there such a nicely packaged neural network, but for voice as well?

djmips 4y ago

Paper on Efficient neural networks for real-time modeling of analog dynamic range compression. https://arxiv.org/pdf/2102.06200.pdf https://arxiv.org/pdf/2102.06200.pdf Might be interesting to those reading in this post.