8 ms·

JackTrip WebRTC: high quality, uncompressed, low-delay audio streaming

156 points by jarmitage 6y ago

jarmitage 6y ago
For anyone who is fatigued by web and telecomms audio quality, I highly recommend trying out JackTrip: https://ccrma.stanford.edu/groups/soundwire/software/jacktrip/ https://ccrma.stanford.edu/groups/soundwire/software/jacktri... If you've ever been in a recording studio with headphone monitoring / studio talkback, it's like that. Really intimate. This project uses RTCDataChannel for audio, which is very neat, but it's a shame that once again audio on the web has to be hacked to perform well.
- rezonant 6y ago
  Agreed, data channels for uncompressed audio is certainly the most interesting idea of this project for me, but it appears they are having trouble with the audio worklet processing time, so it's not an immediate slam dunk :-/
TomMasz 6y ago
As a musician, I am thrilled that JackTrip and Jamulus are available to make real-time collaboration possible but they're not simple to install and configure for everyone. I spent numerous sessions trying to get a non-techie friend's Ubuntu laptop setup with Jamulus but we both got so frustrated we gave up. I hope the next generation brings with it easier setup, there's just so much potential.
- declnz 6y ago
  +1 I get the problem with JackTrip (though JamTrip[1] looks a good start). Out of interest, what were the problems with Jamulus? I've found it easier than expected (once I'd got Jack set up and happy / understood). On Windows / MacOS it's actually easier as there are are clear, well-trodden install guides (and no Jack / low-latency kernels to explain). [1] https://github.com/vicwomg/jamtrip https://github.com/vicwomg/jamtrip
  - spacechild1 6y ago
    Last time I've checked, Jamulus is not peer-to-peer but rather server-client and therefore has more latency.
  - TomMasz 6y ago
    The problem was he's not a techie and only has a Linux laptop because of cost. Trying to get him to understand how to work the shell cost me a year of life.
- wmleler 6y ago
  I've just published an article that goes into depth about how to get a JackTrip Virtual Studio installed. This uses an inexpensive Raspberry Pi instead of requiring installing finicky software on a general purpose laptop or other computer. I wrote this while helping a choir get online, and it worked very well. I hope this helps! https://wmleler1.medium.com/setting-up-your-virtual-studio-6a9fbea56f08 https://wmleler1.medium.com/setting-up-your-virtual-studio-6...
zaroth 6y ago
Truly is it uncompressed PCM WAV audio? In that case 44kHz 16-bit 1-channel is 705 kbps. Are there not any good nonproprietary lossless low latency audio codecs? A brief look at Wikipedia shows a couple that are 5ms or lower, but either they are proprietary or they are low bitrate.
- tleb_ 6y ago
  Opus has frame sizes down to 2.5ms. See https://en.m.wikipedia.org/wiki/Opus_(audio_format)#cite_note-ValinAES135-8 https://en.m.wikipedia.org/wiki/Opus_(audio_format)#cite_not...
  - staticfloat 6y ago
    You pay a price for the super-small frame sizes; much worse compression ratios. Frame sizes smaller than 10ms disable the LPC and hybrid coding modes, which are quite advantageous. [0] In my experiments with transmitting realtime audio signals using Opus, 10ms frame sizes are acceptable for 1-way synchronicity (e.g. if you want the user to perform an action and hear the result as simultaneous with the action) but it's definitely the upper bound. From what I remember my signal processing professor say, the threshold to try and hit is 7ms total latency for truly undetectable processing, but I can't find a reference for that, so take it with a grain of salt. [0] https://www.opus-codec.org/docs/html_api/group__opusencoder.html https://www.opus-codec.org/docs/html_api/group__opusencoder....
    - jarmitage 6y ago
      See for example: Robert H. Jack, Tony Stockman, and Andrew McPherson. 2016. Effect of latency on performer interaction and subjective quality assessment of a digital musical instrument. In Proceedings of the Audio Mostly 2016 (AM '16). Association for Computing Machinery, New York, NY, USA, 116–123. DOI: https://doi.org/10.1145/2986416.2986428 https://doi.org/10.1145/2986416.2986428 https://scholar.google.com/scholar?cluster=13472622195552899495 https://scholar.google.com/scholar?cluster=13472622195552899...
  - declnz 6y ago
jefftk 6y ago
This says ultra low delay, but how low? It seems to me like it's still going to be limited by the browser, and then you're going to have network latency on top of that? Testing in Chrome on Mac OS 11.1, the lowest latency I can get is 23ms while in Firefox it's 14ms: https://www.jefftk.com/test/latency-demo-5 https://www.jefftk.com/test/latency-demo-5
- gregsadetsky 6y ago
  30ms here in Chrome and 23ms in Firefox here. But yeah, same questions -- is this over the network (or no, that's the point of a p2p connection)? What is this measuring..?
  - jefftk 6y ago
    My tool is local: it measures the time from when JS says "send this to the speaker" until JS receives that audio back from the mic. It's the client-side round-trip latency. (As far as I know, there is no way to get the browser to tell you this latency, and you have to measure it.)
rexreed 6y ago
Is it possible to connect this to an RTMPS stream as well or does that defeat the purpose? Can only people Joined in the room get the stream?
PaulDavisThe1st 6y ago
if you're in this space, consider trying out sonobus also. not the author (though he is a friend of mine), haven't tried it yet myself, but seems to have all the right stuff.
- spacechild1 6y ago
  Yes, SonoBus looks really promising! I was surprised and also a bit proud to learn that it uses a fork of my AOO library (https://git.iem.at/cm/aoo/ https://git.iem.at/cm/aoo/). I have been in touch with Jesse for the last couple of days and I'm (passively) following the development of SonoBus. AOO is a C/C++ library for flexible on-the-fly multi-channel audio streaming and OSC messaging over local networks and the internet. The repo also includes Pure Data externals and Supercollider extensions. WARNING: AOO is still alpha and there are breaking changes between the pre-releases, but I hope to publish a final release soon!
- sonosaurus 6y ago
  Thanks Paul.... happy to try it with you sometime :)
morsch 6y ago
I'd love for this to make a difference for simple video calls with relatives -- in my imagination, I set it up, and we find out we've been in the audio equivalent of the uncanny valley this whole time. But I'm afraid in reality, the default audio codecs used in WebRTC are probably good enough to make poor microphones and pure connection latency the bigger issue. Am I wrong? Obviously, even if I'm right, this has a target audience -- musicians, interviewers and other people who talk for a living -- for which it might be fantastic.
ximm 6y ago
I am wondering whether using AudioWorklets actually has any benefits in this case. AFAIU the benefit in general is that they run in the audio thread so the data does not need to be handed over to a different thread for processing. In this case however, the data is handed over to the main thread anyway to be send over the data channel. Has anyone measured the impact of using AudioWorklets this way?
lxe 6y ago
What's the bitrate for these streams? If you're on mobile, or corporate net, you might not be able to get "punched through" by the other party for actual p2p connection and will need a TURN server. I'm guessing that an uncompressed data channel with audio in it might send a lot of traffic through it.
WalterGR 6y ago
Er, what is this project exactly? The README says “Multi-machine network music performance over the Internet is achieved” but then later describes how to create multi-person video chat sessions. Comments mention music. What does this have to do with music? Is it just that remote live music collaboration and multi-machine audio playback are use cases for low-latency codec implementations? Because it seems like this is an audio codec implemented in the browser that comes with a video chat demo. Am I missing something?
- musicale 6y ago
  It's a browser implementation of JackTrip, which is low-latency audio conferencing software for musicians, because video conferencing (Zoom etc.) has terrible latency. The way to have rehearsals over the internet is usually to use JackTrip for the audio and (Zoom etc.) for the video (which can be out of sync; obviously conductors have to tap in an audio metronome channel since video won't be in sync.)
- bschwindHN 6y ago
  This video might have some more context in how it relates to musicians: https://www.youtube.com/watch?v=Sj6Ij1Oxe1s https://www.youtube.com/watch?v=Sj6Ij1Oxe1s
fxtentacle 6y ago
"low-delay", "WebRTC" ... choose one. I really don't get why everything needs to be in a web browser nowadays. WebRTC is simply the wrong choice for low-delay audio streaming. Audinate's DANTE protocol has been around for 14 years, is affordable, hardware-accelerated, and has practically no avoidable latency when sending audio over the network. Why add 20+ms of latency just to use WebRTC?
- musicale 6y ago
  Good question; JackTrip normally uses UDP and I think most browsers do not support sending UDP packets directly without an extension. If you could send UDP directly then WebRTC could presumably be avoided.
  - mdonahoe 6y ago
    Doesn’t webrtc use UDP?
    - fxtentacle 6y ago
      In some cases on some web browsers WebRTC can be configured to use UDP. But that's exactly the problem. Usually, nobody knows what's actually happening in that 20000+ lines of code WebRTC stack.
- deleted 6y ago
  [deleted]
- jdc 6y ago
  Dante is proprietary for one thing.
  - fxtentacle 6y ago
    Here's a GPL implementation https://github.com/jsharkey/wycliffe https://github.com/jsharkey/wycliffe
    - 6y ago
musicale 6y ago
This is great - should make it a lot easier to use JackTrip.

156 points by jarmitage 6y ago

For anyone who is fatigued by web and telecomms audio quality, I highly recommend trying out JackTrip: https://ccrma.stanford.edu/groups/soundwire/software/jacktrip/ https://ccrma.stanford.edu/groups/soundwire/software/jacktri... If you've ever been in a recording studio with headphone monitoring / studio talkback, it's like that. Really intimate. This project uses RTCDataChannel for audio, which is very neat, but it's a shame that once again audio on the web has to be hacked to perform well.

rezonant 6y ago
Agreed, data channels for uncompressed audio is certainly the most interesting idea of this project for me, but it appears they are having trouble with the audio worklet processing time, so it's not an immediate slam dunk :-/

TomMasz 6y ago

As a musician, I am thrilled that JackTrip and Jamulus are available to make real-time collaboration possible but they're not simple to install and configure for everyone. I spent numerous sessions trying to get a non-techie friend's Ubuntu laptop setup with Jamulus but we both got so frustrated we gave up. I hope the next generation brings with it easier setup, there's just so much potential.

declnz 6y ago
+1 I get the problem with JackTrip (though JamTrip[1] looks a good start). Out of interest, what were the problems with Jamulus? I've found it easier than expected (once I'd got Jack set up and happy / understood). On Windows / MacOS it's actually easier as there are are clear, well-trodden install guides (and no Jack / low-latency kernels to explain). [1] https://github.com/vicwomg/jamtrip https://github.com/vicwomg/jamtrip
- spacechild1 6y ago
  Last time I've checked, Jamulus is not peer-to-peer but rather server-client and therefore has more latency.
- TomMasz 6y ago
  The problem was he's not a techie and only has a Linux laptop because of cost. Trying to get him to understand how to work the shell cost me a year of life.
wmleler 6y ago
I've just published an article that goes into depth about how to get a JackTrip Virtual Studio installed. This uses an inexpensive Raspberry Pi instead of requiring installing finicky software on a general purpose laptop or other computer. I wrote this while helping a choir get online, and it worked very well. I hope this helps! https://wmleler1.medium.com/setting-up-your-virtual-studio-6a9fbea56f08 https://wmleler1.medium.com/setting-up-your-virtual-studio-6...

zaroth 6y ago

Truly is it uncompressed PCM WAV audio? In that case 44kHz 16-bit 1-channel is 705 kbps. Are there not any good nonproprietary lossless low latency audio codecs? A brief look at Wikipedia shows a couple that are 5ms or lower, but either they are proprietary or they are low bitrate.

tleb_ 6y ago
Opus has frame sizes down to 2.5ms. See https://en.m.wikipedia.org/wiki/Opus_(audio_format)#cite_note-ValinAES135-8 https://en.m.wikipedia.org/wiki/Opus_(audio_format)#cite_not...
- staticfloat 6y ago
  You pay a price for the super-small frame sizes; much worse compression ratios. Frame sizes smaller than 10ms disable the LPC and hybrid coding modes, which are quite advantageous. [0] In my experiments with transmitting realtime audio signals using Opus, 10ms frame sizes are acceptable for 1-way synchronicity (e.g. if you want the user to perform an action and hear the result as simultaneous with the action) but it's definitely the upper bound. From what I remember my signal processing professor say, the threshold to try and hit is 7ms total latency for truly undetectable processing, but I can't find a reference for that, so take it with a grain of salt. [0] https://www.opus-codec.org/docs/html_api/group__opusencoder.html https://www.opus-codec.org/docs/html_api/group__opusencoder....
  - jarmitage 6y ago
    See for example: Robert H. Jack, Tony Stockman, and Andrew McPherson. 2016. Effect of latency on performer interaction and subjective quality assessment of a digital musical instrument. In Proceedings of the Audio Mostly 2016 (AM '16). Association for Computing Machinery, New York, NY, USA, 116–123. DOI: https://doi.org/10.1145/2986416.2986428 https://doi.org/10.1145/2986416.2986428 https://scholar.google.com/scholar?cluster=13472622195552899495 https://scholar.google.com/scholar?cluster=13472622195552899...
- declnz 6y ago

jefftk 6y ago

This says ultra low delay, but how low? It seems to me like it's still going to be limited by the browser, and then you're going to have network latency on top of that? Testing in Chrome on Mac OS 11.1, the lowest latency I can get is 23ms while in Firefox it's 14ms: https://www.jefftk.com/test/latency-demo-5 https://www.jefftk.com/test/latency-demo-5

gregsadetsky 6y ago
30ms here in Chrome and 23ms in Firefox here. But yeah, same questions -- is this over the network (or no, that's the point of a p2p connection)? What is this measuring..?
- jefftk 6y ago
  My tool is local: it measures the time from when JS says "send this to the speaker" until JS receives that audio back from the mic. It's the client-side round-trip latency. (As far as I know, there is no way to get the browser to tell you this latency, and you have to measure it.)

rexreed 6y ago

Is it possible to connect this to an RTMPS stream as well or does that defeat the purpose? Can only people Joined in the room get the stream?

PaulDavisThe1st 6y ago

if you're in this space, consider trying out sonobus also. not the author (though he is a friend of mine), haven't tried it yet myself, but seems to have all the right stuff.

spacechild1 6y ago
Yes, SonoBus looks really promising! I was surprised and also a bit proud to learn that it uses a fork of my AOO library (https://git.iem.at/cm/aoo/ https://git.iem.at/cm/aoo/). I have been in touch with Jesse for the last couple of days and I'm (passively) following the development of SonoBus. AOO is a C/C++ library for flexible on-the-fly multi-channel audio streaming and OSC messaging over local networks and the internet. The repo also includes Pure Data externals and Supercollider extensions. WARNING: AOO is still alpha and there are breaking changes between the pre-releases, but I hope to publish a final release soon!
sonosaurus 6y ago
Thanks Paul.... happy to try it with you sometime :)

morsch 6y ago

I'd love for this to make a difference for simple video calls with relatives -- in my imagination, I set it up, and we find out we've been in the audio equivalent of the uncanny valley this whole time. But I'm afraid in reality, the default audio codecs used in WebRTC are probably good enough to make poor microphones and pure connection latency the bigger issue. Am I wrong? Obviously, even if I'm right, this has a target audience -- musicians, interviewers and other people who talk for a living -- for which it might be fantastic.

ximm 6y ago

I am wondering whether using AudioWorklets actually has any benefits in this case. AFAIU the benefit in general is that they run in the audio thread so the data does not need to be handed over to a different thread for processing. In this case however, the data is handed over to the main thread anyway to be send over the data channel. Has anyone measured the impact of using AudioWorklets this way?

lxe 6y ago

What's the bitrate for these streams? If you're on mobile, or corporate net, you might not be able to get "punched through" by the other party for actual p2p connection and will need a TURN server. I'm guessing that an uncompressed data channel with audio in it might send a lot of traffic through it.

WalterGR 6y ago

Er, what is this project exactly? The README says “Multi-machine network music performance over the Internet is achieved” but then later describes how to create multi-person video chat sessions. Comments mention music. What does this have to do with music? Is it just that remote live music collaboration and multi-machine audio playback are use cases for low-latency codec implementations? Because it seems like this is an audio codec implemented in the browser that comes with a video chat demo. Am I missing something?

musicale 6y ago
It's a browser implementation of JackTrip, which is low-latency audio conferencing software for musicians, because video conferencing (Zoom etc.) has terrible latency. The way to have rehearsals over the internet is usually to use JackTrip for the audio and (Zoom etc.) for the video (which can be out of sync; obviously conductors have to tap in an audio metronome channel since video won't be in sync.)
bschwindHN 6y ago
This video might have some more context in how it relates to musicians: https://www.youtube.com/watch?v=Sj6Ij1Oxe1s https://www.youtube.com/watch?v=Sj6Ij1Oxe1s

fxtentacle 6y ago

"low-delay", "WebRTC" ... choose one. I really don't get why everything needs to be in a web browser nowadays. WebRTC is simply the wrong choice for low-delay audio streaming. Audinate's DANTE protocol has been around for 14 years, is affordable, hardware-accelerated, and has practically no avoidable latency when sending audio over the network. Why add 20+ms of latency just to use WebRTC?

musicale 6y ago
Good question; JackTrip normally uses UDP and I think most browsers do not support sending UDP packets directly without an extension. If you could send UDP directly then WebRTC could presumably be avoided.
- mdonahoe 6y ago
  Doesn’t webrtc use UDP?
  - fxtentacle 6y ago
    In some cases on some web browsers WebRTC can be configured to use UDP. But that's exactly the problem. Usually, nobody knows what's actually happening in that 20000+ lines of code WebRTC stack.
deleted 6y ago
[deleted]
jdc 6y ago
Dante is proprietary for one thing.
- fxtentacle 6y ago
  Here's a GPL implementation https://github.com/jsharkey/wycliffe https://github.com/jsharkey/wycliffe
  - 6y ago

musicale 6y ago

This is great - should make it a lot easier to use JackTrip.