13 ms·

Show HN: Low-latency jamming over the internet

236 points by weepy 5y ago

tytrdev 5y ago
This is super cool. I’ve been formulating a similar idea for a while now. I’m building a desktop utility for guitarists using clojurescript and Tauri. I also wonder if this can also capture ASIO audio streams in a useful way? My goal was to do something like that to allow streaming of processed audio. Some differences in what I was planning and accompanying thoughts: Clojurescript. I do like that it’s using Svelte, I just wanted more idiomatic support for datalog stuff for the purpose of building metadata-driven music theory tools. Svelte is super cool though, and is my go to JS tool right now. There’s always Datalevin, a portable datalog implementation that I found recently. Currently I’m using a locally running XTDB instance for development, but for the final shippable I may switch to Datalevin. If anyone is interested in doing some similar you could try XTDB over http or figure out a nice way to interface with Datalevin from other languages. Electron -> Tauri. Better native feel and the ability to hit Rust code directly. May not be worth it for this project since it seems like C++ is being used for some stuff. But for me Rust is a better fit. As a side note I think the Tauri team is working on support for interchangeable back ends, so soon you could replace Rust with Go or whatever. Tauri also makes including accompanying binaries easy. Not that I’m saying electron doesn’t, I have no idea. Capturing ASIO streams. Super important for getting good sound for most people, allowing people to play audio through interfaces and mixers while still capturing it. I’m not expert in ASIO or audio streaming, but from my understanding capturing ASIO streams directly is tricky. Reastream (a reaper plugin) is the only thing I’ve found that lets this happen, and sadly it doesn’t work well with other DAWs. Why this would be useful IMO: people can stream audio while still listening to the processed output through whatever means they already do. Guitarists could process audio in a DAW or plugin and both listen to and stream that audio. People using DAWs can stream the output of the DAWs master channel without compromising how they listen to it. I’m not saying sub.live doesn’t accomplish that, I just think it’s important either way. Typically this is the missing link that makes other methods of audio streaming difficult. Open source. Makes me sad that it isn’t. Could have been a good building block and I definitely would have tried to be involved right away. Feel free to correct me if this is actually open source and I just misunderstood.
- weepy 5y ago
  Thanks! I haven't ruled out opensourcing, but honestly I already have limited time and in my experience open source takes _more_ time commitment (I get that you will get free help eventually). I'm making a VST plugin to stream output from a DAW. Problem with Tauri is that you have to support the native browser, rather than just chrome, so it's more work to build and maintain. Good luck with your project! J
weepy 5y ago
Heya - I recently made a desktop app that lets you jam live with your friends over the internet. You can read more about how it works here => https://sub.live https://sub.live - but essentially the latency we achieve means I can jam between UK and Denmark with <20ms ! I also added video because it really helps to see the other person. If you're interested - the tech stack is : * Electron embedding a Svelte app * Websockets to a node server to handle the users/room * C++ sub-process that handles the UDP networking and audio BTW It's also free to use for both Windows and Mac.
analog31 5y ago
Thanks for making this. It looks worth giving a try. Some friends and I get together roughly weekly using Jamulus. We're playing mostly jazz standards. We have gotten latency down into the 20 ms range, when we are all in the same town. That's the ping time. Add another ~ 40 ms that seems to be eaten up by my local computer and wired home network. With that said, online jamming is hard work. I'm the bassist. Fortunately I can set my double bass aside and use my electric bass, so I'm not hearing my acoustic and delayed sound at once. The wind instruments and singer, not so lucky. For the bassist it's a constant chore to hold the tempo, leaving little room for anything expressive. I'm glad to be doing this, beats not playing at all, but yet there is still no substitute for playing together in person.
- capableweb 5y ago
  > Add another ~ 40 ms that seems to be eaten up by my local computer and wired home network. Maybe you have a weak computer? Having 40ms being eaten up by your local network would be absolutely bananas, the latency should be closer to 1-3ms if not lower. If I ping 8.8.4.4 it's not until the switch outside the city I live in that the latency comes up to above 10ms, so something sounds wrong/broken in your setup if it's really your local LAN having 40ms latency.
  - weepy 5y ago
    If you are on Windows you need to use an ASIO soundcard ?
    - capableweb 5y ago
      Ah, it's possible analog31 isn't using ASIO card, I assumed they are as they seem to have tried to jam before so also assumed they were a musician and I haven't met many who doesn't use ASIO in the first place.
      - analog31 5y ago
        Ah, that's interesting. I'm using a USB audio interface. It claims to be ASIO compatible. Oddly enough I tried the same interface in different computers, and have tried other computers, but haven't dug much further than that. I've also tried all of this on a decent Ubuntu box, and I've now got it running on a Raspberry Pi 3 since the overall latency has been more less the same in all cases. We're all analog musicians playing alcohol powered instruments. ;-) Some of us are techies, others not. While I'm a techie, that side of my life has always been somewhat separate from the musical side, so I haven't paid that much attention to the digital technology until the pandemic came along. But I'm happy to learn, especially if something simple can make it work better.
koopuluri 5y ago
Wow. I really like the positioning of your product: the audience and value proposition are crystal clear. I'm going to share this with my musician friends and see what they have to say. I know they had challenges jamming at the start of the pandemic lockdown that stemmed from a) variance in internet connection strengths and b) innate latency in Zoom. All the best & kudos for launching!
- weepy 5y ago
  Thanks !
capableweb 5y ago
Would be amazing if this could be released as a Linux binary as well. Me and many of my friends are on a P2P mesh network, I'm sure we can push the latency down to below 20ms which would be amazing to see/hear!
- weepy 5y ago
  Within the same city you can get <5ms
NikolaNovak 5y ago
One of the concerns I always have about things I love is that they'll work but eventually disappear. Is there a way to donate/pay/tier for this? I assume it comes at non-zero cost to you in terms of time bandwidth hosting etc.
- capableweb 5y ago
  Development is never free, but this seems to be driven by a P2P protocol so hosting/bandwidth costs should be close to zero (maybe signalling server is hosted by creator but should be relatively cheap to host). I agree that donation would be nice, but won't prevent it from disappearing. Only solution I can think of to completely make it unable to disappear would be to open source the code base, either now or having a guarantee that the code base would be open sourced if the author looses interest.
  - weepy 5y ago
    There's a "buy me a coffee" on the page if you're feeling generous! It's true the cost is certainly the dev time not so much the servers. Though open sourcing doesn't protect things from dying - it's people not using them!
    - capableweb 5y ago
      Awesome! If Linux binaries become available I'll for sure "buy you a coffee" :) Open sourcing is more to enable people to be able to build the thing in the future with new libraries. OSes move forward every day and something that runs/builds today might not be able to run/build tomorrow, so open sourcing makes sure it's possible to change libraries/API usage if needed.
k__ 5y ago
I don't get these realtime collaborative streaming services. Streaming at home via wifi is already crappy, how should this ever work with more distance?
- capableweb 5y ago
  Don't use WiFi? Ethernet has way less latency. Combine with everyone being in the same city/region and using fiber network and latency should be really good. Otherwise, find friends that you have visibility with and setup a P2P WiFi via antennas/radios, you'll get insanely good latency.
  - k__ 5y ago
    Yeah, I shouldn't use tech that is basically used all over the world for convenient internet connections to access a new tech that is basically built on top of that. Sounds reasonable...
    - capableweb 5y ago
      The suggestion of a P2P mesh is only if you physically could set that up. Otherwise, using Ethernet + Fiber connection should be good enough for most. Only if you really wanna lower the latency should you invest in additional hardware.
      - weepy 5y ago
        You can play with Wifi - it's OK - but you get probably 5-10ms better latency with Wired LAN. Also I think you have a lower tolerance for latency of your own actions vs the latency of another person's.
        wmwmwm 5y ago
        Wifi might not be too bad, though it can spontaneously degrade badly if you get interference. Not great in the middle of a jam!
emerged 5y ago
I had assumed this would be a VST plug-in so you could integrate everyone into (for example) an Ableton Live set with whatever other audio/midi manipulation you want.
- capableweb 5y ago
  Using audio devices directly makes it possible for any audio application to use it, not just applications supporting VSTs. Makes it more flexible, not less. A VST could be built on top of the current solution, but if it was done vice-versa, it wouldn't.
  - weepy 5y ago
    Exactly. I have a prototype VST plugin which pipes the audio from the DAW into the application.
  - emerged 5y ago
    VST frameworks like JUCE export to VST or stand-alone. So you get both for free. Much easier and more flexible. Also DAWs don’t typically allow you to interface with multiple audio devices. There are ways of doing it but they have major downsides.
    - weepy 5y ago
      Yes JUCE is da bomb
andai 5y ago
Hey, are you using Opus for audio? I remember the whole reason Opus was so cool, is that it was the first codec that combined low latency with high quality. Most voice chat software uses Opus nowadays so I'm surprised to hear existing solutions are so bad.
- weepy 5y ago
  Yes Opus - the other benefit is that it can handle a degree of missing packets which is very handy.
  - fenesiistvan 5y ago
    And that feature adds ~20 msec more latency
    - weepy 5y ago
      You must have set it up wrong. It doesn't add any latency, but you do need a higher bitrate to get similar quality.
      - ad8e 5y ago
        I googled "opus packet loss" and found https://www.asterisk.org/asterisk-opus-packet-loss-fec/ https://www.asterisk.org/asterisk-opus-packet-loss-fec/, which says "an Opus encoder can embed redundant data about the preceding packet in-band in the current packet". That method would cause 1 packet of delay. Are you using something different, or avoiding the delay through another method like tiny packets?
        ad8e 5y ago
        Here's probably the answer: https://opus-codec.org/docs/opus_api-1.2/group__opus__decoder.html https://opus-codec.org/docs/opus_api-1.2/group__opus__decode... "Lost packets can be replaced with loss concealment by calling the decoder with a null pointer and zero length for the missing packet."
deeblering4 5y ago
I think https://endlesss.fm https://endlesss.fm got the approach to jamming over the internet right. Instead of trying to reduce latency (which you simply cannot do beyond speed of light) endlesss implements a shared “multiplayer” 8 track looper. Jammers add layers to the loop, which has a clock and supports sync via ableton live, external audio input, etc. Plus, every addition to the looper is saved/versioned. You can move backwards through loops and export the audio stems. Its a great creative tool, and quick way to build on ideas with friends. It’s way better than any of the live internet jamming software I’ve used, by far.
- weepy 5y ago
  Endlesss is cool - in fact I know the founder. It's a different experience though - when it's live you feel someone else's presence compared with offline collab. Up to you what you prefer.
- rzzzt 5y ago
  NINJAM also works on the same principle: https://cockos.com/ninjam/ https://cockos.com/ninjam/ > The NINJAM client records and streams synchronized > intervals of music between participants. Just as the > interval finishes recording, it begins playing on > everyone else's client. So when you play through an > interval, you're playing along with the previous > interval of everybody else, and they're playing along > with your previous interval.
  - deeblering4 5y ago
    Ninjam is a bit different, in that its not a looper. The timing is offset to keep music sounding in sync to all clients, but what you play plays once and does not loop. In endlesss phrases will loop until they are changed or removed.
deleted 5y ago
[deleted]
tomxor 5y ago
It's good that latency is considered to be so critical, but for the same reason I'm sceptical that this would work well for the majority beyond quite local ranges with very good internet connections i.e some kind of fiber... which most people don't have, although I realise a lot of the tech crowd is unaware of this (most of the worlds user end points are some kind of DSL or cell network with a 20-40ms minimum). If anyone has tried playing a processing heavy software synth on a pc in real-time you will have experienced how unplayable it is as soon as latency goes beyond 10-20ms - you can't play music if there is noticeable delay between your fingers and ears, and we are much more sensitive to sound latency, it would be the same problem trying to play in time with each other. I like the idea, but feel like the internet isn't there yet for the majority of users, and latency hasn't exactly been improving at a great pace. I expect it will still be useful in internet rich areas like city to city.
- weepy 5y ago
  Actually I can jam with ease with my friends in London from Denmark. That's over 1000km. And if you don't believe me - check the testimonials on https://sub.live https://sub.live ^_^ Also you have a much lower tolerance of latency for your own actions that other people's. I'm hopefully going to record a video this weekend to show everyone!
  - weepy 5y ago
    Also I live in a village in Denmark near Aarhus. I believe my connection goes via Copenhagen which is actually the wrong direction! Though it is true that Denmark is very well connected.
- digitallyfree 5y ago
  Yeah, the hop to the internet exchange point is the main issue with DSL or cable. I live in a major city and it takes around 10ms to hit the internet exchange point (using Ethernet the LAN latency is neglegible) and thus I get a minimum of 20ms or so just connecting to a server hosted by someone in the same neighborhood. I've actually tried this experiment with some of my nearby friends - pinging each other's public IPs on DSL or cable takes around 20ms. With fiber, you can hit the exchange point in 1-2ms. So I suppose this tool would work well fine over fiber assuming everyone is in the same city. I've even heard of people successfully using protocols like Dante (professional audio over IP intended for LAN) over gig fiber lines as well. From my experience, a couple milliseconds of latency is fine for keyboard or guitar processing, but anything more than that starts to mess me up. There are artists who insist on using a full analog chain for monitoring for that reason, and refuse to use digital mixers or digital wireless systems that can add several ms of latency.
spmurrayzzz 5y ago
This is very cool (former touring musician turned software engineer here!) I'm curious about the networking/UDP side of this. How are you handling retransmits? Do you treat this like a video game would and just keep sending the latest data? Or doing something more advanced like forward error correction?
- weepy 5y ago
  no retransmits - Opus handles a degree of packet loss pretty well. Not using FEC as it adds extra latency.
jp57 5y ago
I'm very interested in this! Can you say how this compares to something like https://jamkazam.com https://jamkazam.com? One of my issues with JK is that it seems to require all participants to be members, and to pay for relatively pricy memberships to partcipate. I'm sympathetic that nothing is free and I'm willing to pay for a good service, but I'd love to have a model where one person could pay for a session that others could join (with video) for free. That said, I'm not sure if you plan to start charging.
- weepy 5y ago
  I would like to make some money with it to continue the support, but it's not clear yet the best way forward. Currently it's donationware, but other models could be a subscription, or some sort of premium/freemium model. Really I just want to see how people use it and figure it out from there.
PaulDavisThe1st 5y ago
I'm guessing that most people reading this are not familiar with some of the other options available for this kind of thing. In particular, they likely don't know about Sonobus: https://sonobus.net/ https://sonobus.net/ (and that's likely because it's libre software and the developer doesn't really do marketing)
- danrochman 5y ago
  Wow!! I'm one of those people - TIL about Sonobus. Thank you for that, because it's amazing, how-did-I-not-know software. With iOS and Android builds, too!
  - spacechild1 5y ago
    And it's also available as a VST plugin!
- spacechild1 5y ago
  I was about to mention Sonobus. Fun fact: under the hood it uses a fork of my AOO library (https://git.iem.at/cm/aoo/-/tree/develop https://git.iem.at/cm/aoo/-/tree/develop). AOO is still alpha but I'm hoping to release a stable version till end of the year.
darepublic 5y ago
Curious about using this to stream a jukebox playing analog records to myself
- capableweb 5y ago
  You can achieve that with a much simpler solution, look into RTMP.
- weepy 5y ago
  you could but you don't need higher latency.
rdtennent 5y ago
My experience with Jamulus might be instructive. Members of our group live in the same city (Kingston, Ontario) but use two ISPs. Packets between users on the same ISP were fine but packets from one ISP to the other were being routed via Toronto and then Chicago and ultimately back to Kingston. It's not the distance travelled that's the problem (speed of light), it's the latency introduced by each intermediate node. Solved the problem by setting up a Jamulus server on AWS in Montreal. Both ISPs provide low-latency connections to Montreal, much better than one mile across town! Of course each participant has to use ethernet rather than WiFi and has to use a low-latency audio device, not a laptop sound card.
- nemetroid 5y ago
  Had the same experience in Sweden. Between certain pairs of ISPs, all traffic was routed via one of the capital cities (Stockholm, Copenhagen, Oslo), adding 10-20 ms and jitter. Solved it by setting up a VPS at a provider with good connections to all ISPs involved.
- porcc 5y ago
  How can you detect these node transfers? I run into similar problems often with WebRTC and I have found it troublesome to diagnose
  - rdtennent 5y ago
    traceroute
- __turbobrew__ 5y ago
  Something similar was happening in Calgary where multiple ISPs in Calgary peered through Seattle and Toronto. Luckily YYCIX was formed and it appears that things are much better now: https://yycix.ca/talks/cuug-2013-06-18/mgp00001.html https://yycix.ca/talks/cuug-2013-06-18/mgp00001.html
- weepy 5y ago
  The issue with Jamulus is that it requires a central server - which means it needs to be close to everywhere . It also needs double buffering and double compression. P2P is the way forward here IMHO.
arctan5x 5y ago
Checkout https://jacktrip.org/studio.html https://jacktrip.org/studio.html. We enable low latency audio connection through open source technology developed at Stanford. We can scale to hundreds of users signing at the same time. Check out hundreds of choir members signing together https://www.youtube.com/watch?v=SJgB5QmyDfU https://www.youtube.com/watch?v=SJgB5QmyDfU.
- deleted 5y ago
  [deleted]
lwansbrough 5y ago
One thing you might be interested in is a company called NetworkNext. (No affiliation, I just think it’s a great idea.) They provide a private network marketplace that will give you pretty much optimized latency anywhere in the world.
iandanforth 5y ago
It would be fun to try to add a predictive layer to this: - Given the score and what each person has just played predict what the next few sounds are going to be - Given a high frame rate video stream of a person predict what the next note to be played In the same way that Nvidia has extremely low bandwidth but high resolution video enabled by face keypoint tracking and facial reconstruction / puppeteering maybe there's a place for prediction and/or sound reconstruction from extremely low bitrate streams.* * Obviously not the exact usecase here since a premium is being placed on not processing, but still fun to think about.
- mandor 5y ago
  We're doing exactly this to teleoperate humanoid robots on high-latency networks! Paper: https://arxiv.org/abs/2107.01281 https://arxiv.org/abs/2107.01281 Video: https://www.youtube.com/watch?v=N3u4ot3aIyQ https://www.youtube.com/watch?v=N3u4ot3aIyQ "We introduce a system in which a humanoid robot executes commands before it actually receives them, so that the visual feedback appears to be synchronized to the operator, whereas the robot executed the commands in the past. To do so, the robot continuously predicts future commands by querying a machine learning model that is trained on past trajectories and conditioned on the last received commands. In our experiments, an operator was able to successfully control a humanoid robot (32 degrees of freedom) with stochastic delays up to 2 seconds in several whole-body manipulation tasks, including reaching different targets, picking up, and placing a box at distinct locations."
debuggerpk 5y ago
This is revolutionary ...
fouc 5y ago
Does anyone know of an alternative that keeps both audio & video in complete sync, but possibly pauses/buffers at times in order to maximize the segments where everything is kept in sync?
- mgamache 5y ago
  Not possible to be in complete sync and have ultra low latency. You would have to have a jitter buffer (at least).
prendo1234 5y ago
Hey @weepy. Awesome project! As a webrtc dev - interested to hear about your choices around opus packet durations, FEC percentages, ARQ strategies, jitterbuffer length and packet redundancy.
- weepy 5y ago
  I used the standard 2.5ms. I know you can go lower if you want, but then you need a higher bitrate as it's "custom". I turned off FEC as it adds latency. Jitter buffer right now is just user controlled. A bit lame, - I should make it automatic, but need to get the right heuristic. With a LAN connection, the buffer can be as low as one or two packets. I don't use any packet redundancy either and there's no ARQ as if you have to ask for a retransmit you've already lost the war! How about you ?
  - prendo1234 5y ago
    Yeah, awesome. Have to say this is the best version of something like this that I have tried. The packet capture didn't seem to be RTP, are you using libwebrtc under the hood? The system seems to have the settings right for realtime collab. In fact I found it much better for chatting too if you can guarantee they have a headset and good mic - I hate current-gen AEC algorithms. Interested to see how it fares in a 3+ participant setting. I had a go at jamming with my brother yesterday. We had to adjust down the delay to 2ms (I think default was 5ms) in order to counteract the "lagging" effect you get when you lock into the remote beat. Once we had that tuned it worked really well. We would occasionally suffer burst losses, but you can play through it and it's still synced afterward which is the best you can do with an unreliable network. Some feature requests: - Join room by name (we were confused that there was a different code than the name we chose) - Auto register new audio devices when you plug them in/edit in settings. - An easier way to join with multiple audio inputs. We had trouble setting up our instrument, but also adding a mic for chatting. - Local recording of your end with some shared sync markers, that you could manually or automatically sync up post-hoc The use case I use WebRTC for is slightly different, where quality >> latency , so we have it tuned to the other end and run the full gambit of FEC, ARQ, packet redundancy etc. Interesting to hear about other's solutions though!
audray 5y ago
I may be missing something but this appears to me to be a WebRTC application. It is very easy to build something like this using any modern browser's WebRTC APIs, just disable audio processing in the media constraints and munge the opus SDP to stereo, maybe play with the network buffer setting if you want to lower latency but the audio hardware and physical distance is going matter more there. The developer states that no other software was suitable, and also that it's the first of its kind. Both of those statements are not accurate: there is nothing innovative in sublive that isn't in any of the apps listed below. Props to the developer though for scratching an itch. https://en.wikipedia.org/wiki/Comparison_of_Remote_Music_Performance_Software https://en.wikipedia.org/wiki/Comparison_of_Remote_Music_Per...
- deleted 5y ago
  [deleted]
- weepy 5y ago
  It uses WebRTC for the video, but the audio latency of WebRTC is too large and uncontrollable. As stated in the post, the audio uses a custom C++ UDP solution. As far as I know it's the first video calling app with very low latency audio.
  - telesilla 5y ago
    How do you deal with firewalls in that case? Sonobus has a similar problem that if you don't have accessible NAT you can't connect. You need a relay or central server which can get really expensive!
    - weepy 5y ago
      Basically similar issue to Sonobus. Relay could work buy it probably adds latency and it certainly adds complexity. I may know a way to improve some NAT configurations but need to do more research.
    - audray 5y ago
      Yeah one reason I like Jacktrip, you can optionally use servers to connect. Low latency is nice but it doesn't help in a performance when you can't connect to your group of musicians
alberth 5y ago
>”We've been able to achieve sub 20ms latency from UK to Denmark - more than 1000km” It’s pretty solid given that the speed-of-light would like 3.3ms to travel that same distance.
- weepy 5y ago
  Honestly I was a bit blown away when I got it working the first time ! It kind of mental that I can capture audio and compress it - send it through all these layers and machines - and receive it 1000km away 20ms later. All on consumer grade internet.

236 points by weepy 5y ago

This is super cool. I’ve been formulating a similar idea for a while now. I’m building a desktop utility for guitarists using clojurescript and Tauri. I also wonder if this can also capture ASIO audio streams in a useful way? My goal was to do something like that to allow streaming of processed audio. Some differences in what I was planning and accompanying thoughts: Clojurescript. I do like that it’s using Svelte, I just wanted more idiomatic support for datalog stuff for the purpose of building metadata-driven music theory tools. Svelte is super cool though, and is my go to JS tool right now. There’s always Datalevin, a portable datalog implementation that I found recently. Currently I’m using a locally running XTDB instance for development, but for the final shippable I may switch to Datalevin. If anyone is interested in doing some similar you could try XTDB over http or figure out a nice way to interface with Datalevin from other languages. Electron -> Tauri. Better native feel and the ability to hit Rust code directly. May not be worth it for this project since it seems like C++ is being used for some stuff. But for me Rust is a better fit. As a side note I think the Tauri team is working on support for interchangeable back ends, so soon you could replace Rust with Go or whatever. Tauri also makes including accompanying binaries easy. Not that I’m saying electron doesn’t, I have no idea. Capturing ASIO streams. Super important for getting good sound for most people, allowing people to play audio through interfaces and mixers while still capturing it. I’m not expert in ASIO or audio streaming, but from my understanding capturing ASIO streams directly is tricky. Reastream (a reaper plugin) is the only thing I’ve found that lets this happen, and sadly it doesn’t work well with other DAWs. Why this would be useful IMO: people can stream audio while still listening to the processed output through whatever means they already do. Guitarists could process audio in a DAW or plugin and both listen to and stream that audio. People using DAWs can stream the output of the DAWs master channel without compromising how they listen to it. I’m not saying sub.live doesn’t accomplish that, I just think it’s important either way. Typically this is the missing link that makes other methods of audio streaming difficult. Open source. Makes me sad that it isn’t. Could have been a good building block and I definitely would have tried to be involved right away. Feel free to correct me if this is actually open source and I just misunderstood.

weepy 5y ago
Thanks! I haven't ruled out opensourcing, but honestly I already have limited time and in my experience open source takes _more_ time commitment (I get that you will get free help eventually). I'm making a VST plugin to stream output from a DAW. Problem with Tauri is that you have to support the native browser, rather than just chrome, so it's more work to build and maintain. Good luck with your project! J

weepy 5y ago

Heya - I recently made a desktop app that lets you jam live with your friends over the internet. You can read more about how it works here => https://sub.live https://sub.live - but essentially the latency we achieve means I can jam between UK and Denmark with <20ms ! I also added video because it really helps to see the other person. If you're interested - the tech stack is : * Electron embedding a Svelte app * Websockets to a node server to handle the users/room * C++ sub-process that handles the UDP networking and audio BTW It's also free to use for both Windows and Mac.

analog31 5y ago

Thanks for making this. It looks worth giving a try. Some friends and I get together roughly weekly using Jamulus. We're playing mostly jazz standards. We have gotten latency down into the 20 ms range, when we are all in the same town. That's the ping time. Add another ~ 40 ms that seems to be eaten up by my local computer and wired home network. With that said, online jamming is hard work. I'm the bassist. Fortunately I can set my double bass aside and use my electric bass, so I'm not hearing my acoustic and delayed sound at once. The wind instruments and singer, not so lucky. For the bassist it's a constant chore to hold the tempo, leaving little room for anything expressive. I'm glad to be doing this, beats not playing at all, but yet there is still no substitute for playing together in person.

capableweb 5y ago
> Add another ~ 40 ms that seems to be eaten up by my local computer and wired home network. Maybe you have a weak computer? Having 40ms being eaten up by your local network would be absolutely bananas, the latency should be closer to 1-3ms if not lower. If I ping 8.8.4.4 it's not until the switch outside the city I live in that the latency comes up to above 10ms, so something sounds wrong/broken in your setup if it's really your local LAN having 40ms latency.
- weepy 5y ago
  If you are on Windows you need to use an ASIO soundcard ?
  - capableweb 5y ago
    Ah, it's possible analog31 isn't using ASIO card, I assumed they are as they seem to have tried to jam before so also assumed they were a musician and I haven't met many who doesn't use ASIO in the first place.
    - analog31 5y ago
      Ah, that's interesting. I'm using a USB audio interface. It claims to be ASIO compatible. Oddly enough I tried the same interface in different computers, and have tried other computers, but haven't dug much further than that. I've also tried all of this on a decent Ubuntu box, and I've now got it running on a Raspberry Pi 3 since the overall latency has been more less the same in all cases. We're all analog musicians playing alcohol powered instruments. ;-) Some of us are techies, others not. While I'm a techie, that side of my life has always been somewhat separate from the musical side, so I haven't paid that much attention to the digital technology until the pandemic came along. But I'm happy to learn, especially if something simple can make it work better.

koopuluri 5y ago

Wow. I really like the positioning of your product: the audience and value proposition are crystal clear. I'm going to share this with my musician friends and see what they have to say. I know they had challenges jamming at the start of the pandemic lockdown that stemmed from a) variance in internet connection strengths and b) innate latency in Zoom. All the best & kudos for launching!

weepy 5y ago
Thanks !

capableweb 5y ago

Would be amazing if this could be released as a Linux binary as well. Me and many of my friends are on a P2P mesh network, I'm sure we can push the latency down to below 20ms which would be amazing to see/hear!

weepy 5y ago
Within the same city you can get <5ms

NikolaNovak 5y ago

One of the concerns I always have about things I love is that they'll work but eventually disappear. Is there a way to donate/pay/tier for this? I assume it comes at non-zero cost to you in terms of time bandwidth hosting etc.

capableweb 5y ago
Development is never free, but this seems to be driven by a P2P protocol so hosting/bandwidth costs should be close to zero (maybe signalling server is hosted by creator but should be relatively cheap to host). I agree that donation would be nice, but won't prevent it from disappearing. Only solution I can think of to completely make it unable to disappear would be to open source the code base, either now or having a guarantee that the code base would be open sourced if the author looses interest.
- weepy 5y ago
  There's a "buy me a coffee" on the page if you're feeling generous! It's true the cost is certainly the dev time not so much the servers. Though open sourcing doesn't protect things from dying - it's people not using them!
  - capableweb 5y ago
    Awesome! If Linux binaries become available I'll for sure "buy you a coffee" :) Open sourcing is more to enable people to be able to build the thing in the future with new libraries. OSes move forward every day and something that runs/builds today might not be able to run/build tomorrow, so open sourcing makes sure it's possible to change libraries/API usage if needed.

k__ 5y ago

I don't get these realtime collaborative streaming services. Streaming at home via wifi is already crappy, how should this ever work with more distance?

capableweb 5y ago
Don't use WiFi? Ethernet has way less latency. Combine with everyone being in the same city/region and using fiber network and latency should be really good. Otherwise, find friends that you have visibility with and setup a P2P WiFi via antennas/radios, you'll get insanely good latency.
- k__ 5y ago
  Yeah, I shouldn't use tech that is basically used all over the world for convenient internet connections to access a new tech that is basically built on top of that. Sounds reasonable...
  - capableweb 5y ago
    The suggestion of a P2P mesh is only if you physically could set that up. Otherwise, using Ethernet + Fiber connection should be good enough for most. Only if you really wanna lower the latency should you invest in additional hardware.
    - weepy 5y ago
      You can play with Wifi - it's OK - but you get probably 5-10ms better latency with Wired LAN. Also I think you have a lower tolerance for latency of your own actions vs the latency of another person's.
      - wmwmwm 5y ago
        Wifi might not be too bad, though it can spontaneously degrade badly if you get interference. Not great in the middle of a jam!

emerged 5y ago

I had assumed this would be a VST plug-in so you could integrate everyone into (for example) an Ableton Live set with whatever other audio/midi manipulation you want.

capableweb 5y ago
Using audio devices directly makes it possible for any audio application to use it, not just applications supporting VSTs. Makes it more flexible, not less. A VST could be built on top of the current solution, but if it was done vice-versa, it wouldn't.
- weepy 5y ago
  Exactly. I have a prototype VST plugin which pipes the audio from the DAW into the application.
- emerged 5y ago
  VST frameworks like JUCE export to VST or stand-alone. So you get both for free. Much easier and more flexible. Also DAWs don’t typically allow you to interface with multiple audio devices. There are ways of doing it but they have major downsides.
  - weepy 5y ago
    Yes JUCE is da bomb

andai 5y ago

Hey, are you using Opus for audio? I remember the whole reason Opus was so cool, is that it was the first codec that combined low latency with high quality. Most voice chat software uses Opus nowadays so I'm surprised to hear existing solutions are so bad.

weepy 5y ago
Yes Opus - the other benefit is that it can handle a degree of missing packets which is very handy.
- fenesiistvan 5y ago
  And that feature adds ~20 msec more latency
  - weepy 5y ago
    You must have set it up wrong. It doesn't add any latency, but you do need a higher bitrate to get similar quality.
    - ad8e 5y ago
      I googled "opus packet loss" and found https://www.asterisk.org/asterisk-opus-packet-loss-fec/ https://www.asterisk.org/asterisk-opus-packet-loss-fec/, which says "an Opus encoder can embed redundant data about the preceding packet in-band in the current packet". That method would cause 1 packet of delay. Are you using something different, or avoiding the delay through another method like tiny packets?
      - ad8e 5y ago
        Here's probably the answer: https://opus-codec.org/docs/opus_api-1.2/group__opus__decoder.html https://opus-codec.org/docs/opus_api-1.2/group__opus__decode... "Lost packets can be replaced with loss concealment by calling the decoder with a null pointer and zero length for the missing packet."

deeblering4 5y ago

I think https://endlesss.fm https://endlesss.fm got the approach to jamming over the internet right. Instead of trying to reduce latency (which you simply cannot do beyond speed of light) endlesss implements a shared “multiplayer” 8 track looper. Jammers add layers to the loop, which has a clock and supports sync via ableton live, external audio input, etc. Plus, every addition to the looper is saved/versioned. You can move backwards through loops and export the audio stems. Its a great creative tool, and quick way to build on ideas with friends. It’s way better than any of the live internet jamming software I’ve used, by far.

weepy 5y ago
Endlesss is cool - in fact I know the founder. It's a different experience though - when it's live you feel someone else's presence compared with offline collab. Up to you what you prefer.
rzzzt 5y ago
NINJAM also works on the same principle: https://cockos.com/ninjam/ https://cockos.com/ninjam/ > The NINJAM client records and streams synchronized > intervals of music between participants. Just as the > interval finishes recording, it begins playing on > everyone else's client. So when you play through an > interval, you're playing along with the previous > interval of everybody else, and they're playing along > with your previous interval.
- deeblering4 5y ago
  Ninjam is a bit different, in that its not a looper. The timing is offset to keep music sounding in sync to all clients, but what you play plays once and does not loop. In endlesss phrases will loop until they are changed or removed.

deleted 5y ago

[deleted]

tomxor 5y ago

It's good that latency is considered to be so critical, but for the same reason I'm sceptical that this would work well for the majority beyond quite local ranges with very good internet connections i.e some kind of fiber... which most people don't have, although I realise a lot of the tech crowd is unaware of this (most of the worlds user end points are some kind of DSL or cell network with a 20-40ms minimum). If anyone has tried playing a processing heavy software synth on a pc in real-time you will have experienced how unplayable it is as soon as latency goes beyond 10-20ms - you can't play music if there is noticeable delay between your fingers and ears, and we are much more sensitive to sound latency, it would be the same problem trying to play in time with each other. I like the idea, but feel like the internet isn't there yet for the majority of users, and latency hasn't exactly been improving at a great pace. I expect it will still be useful in internet rich areas like city to city.

weepy 5y ago
Actually I can jam with ease with my friends in London from Denmark. That's over 1000km. And if you don't believe me - check the testimonials on https://sub.live https://sub.live ^_^ Also you have a much lower tolerance of latency for your own actions that other people's. I'm hopefully going to record a video this weekend to show everyone!
- weepy 5y ago
  Also I live in a village in Denmark near Aarhus. I believe my connection goes via Copenhagen which is actually the wrong direction! Though it is true that Denmark is very well connected.
digitallyfree 5y ago
Yeah, the hop to the internet exchange point is the main issue with DSL or cable. I live in a major city and it takes around 10ms to hit the internet exchange point (using Ethernet the LAN latency is neglegible) and thus I get a minimum of 20ms or so just connecting to a server hosted by someone in the same neighborhood. I've actually tried this experiment with some of my nearby friends - pinging each other's public IPs on DSL or cable takes around 20ms. With fiber, you can hit the exchange point in 1-2ms. So I suppose this tool would work well fine over fiber assuming everyone is in the same city. I've even heard of people successfully using protocols like Dante (professional audio over IP intended for LAN) over gig fiber lines as well. From my experience, a couple milliseconds of latency is fine for keyboard or guitar processing, but anything more than that starts to mess me up. There are artists who insist on using a full analog chain for monitoring for that reason, and refuse to use digital mixers or digital wireless systems that can add several ms of latency.

spmurrayzzz 5y ago

This is very cool (former touring musician turned software engineer here!) I'm curious about the networking/UDP side of this. How are you handling retransmits? Do you treat this like a video game would and just keep sending the latest data? Or doing something more advanced like forward error correction?

weepy 5y ago
no retransmits - Opus handles a degree of packet loss pretty well. Not using FEC as it adds extra latency.

jp57 5y ago

I'm very interested in this! Can you say how this compares to something like https://jamkazam.com https://jamkazam.com? One of my issues with JK is that it seems to require all participants to be members, and to pay for relatively pricy memberships to partcipate. I'm sympathetic that nothing is free and I'm willing to pay for a good service, but I'd love to have a model where one person could pay for a session that others could join (with video) for free. That said, I'm not sure if you plan to start charging.

weepy 5y ago
I would like to make some money with it to continue the support, but it's not clear yet the best way forward. Currently it's donationware, but other models could be a subscription, or some sort of premium/freemium model. Really I just want to see how people use it and figure it out from there.

PaulDavisThe1st 5y ago

I'm guessing that most people reading this are not familiar with some of the other options available for this kind of thing. In particular, they likely don't know about Sonobus: https://sonobus.net/ https://sonobus.net/ (and that's likely because it's libre software and the developer doesn't really do marketing)

danrochman 5y ago
Wow!! I'm one of those people - TIL about Sonobus. Thank you for that, because it's amazing, how-did-I-not-know software. With iOS and Android builds, too!
- spacechild1 5y ago
  And it's also available as a VST plugin!
spacechild1 5y ago
I was about to mention Sonobus. Fun fact: under the hood it uses a fork of my AOO library (https://git.iem.at/cm/aoo/-/tree/develop https://git.iem.at/cm/aoo/-/tree/develop). AOO is still alpha but I'm hoping to release a stable version till end of the year.

darepublic 5y ago

Curious about using this to stream a jukebox playing analog records to myself

capableweb 5y ago
You can achieve that with a much simpler solution, look into RTMP.
weepy 5y ago
you could but you don't need higher latency.

rdtennent 5y ago

My experience with Jamulus might be instructive. Members of our group live in the same city (Kingston, Ontario) but use two ISPs. Packets between users on the same ISP were fine but packets from one ISP to the other were being routed via Toronto and then Chicago and ultimately back to Kingston. It's not the distance travelled that's the problem (speed of light), it's the latency introduced by each intermediate node. Solved the problem by setting up a Jamulus server on AWS in Montreal. Both ISPs provide low-latency connections to Montreal, much better than one mile across town! Of course each participant has to use ethernet rather than WiFi and has to use a low-latency audio device, not a laptop sound card.

nemetroid 5y ago
Had the same experience in Sweden. Between certain pairs of ISPs, all traffic was routed via one of the capital cities (Stockholm, Copenhagen, Oslo), adding 10-20 ms and jitter. Solved it by setting up a VPS at a provider with good connections to all ISPs involved.
porcc 5y ago
How can you detect these node transfers? I run into similar problems often with WebRTC and I have found it troublesome to diagnose
- rdtennent 5y ago
  traceroute
__turbobrew__ 5y ago
Something similar was happening in Calgary where multiple ISPs in Calgary peered through Seattle and Toronto. Luckily YYCIX was formed and it appears that things are much better now: https://yycix.ca/talks/cuug-2013-06-18/mgp00001.html https://yycix.ca/talks/cuug-2013-06-18/mgp00001.html
weepy 5y ago
The issue with Jamulus is that it requires a central server - which means it needs to be close to everywhere . It also needs double buffering and double compression. P2P is the way forward here IMHO.

arctan5x 5y ago

Checkout https://jacktrip.org/studio.html https://jacktrip.org/studio.html. We enable low latency audio connection through open source technology developed at Stanford. We can scale to hundreds of users signing at the same time. Check out hundreds of choir members signing together https://www.youtube.com/watch?v=SJgB5QmyDfU https://www.youtube.com/watch?v=SJgB5QmyDfU.

deleted 5y ago
[deleted]

lwansbrough 5y ago

One thing you might be interested in is a company called NetworkNext. (No affiliation, I just think it’s a great idea.) They provide a private network marketplace that will give you pretty much optimized latency anywhere in the world.

iandanforth 5y ago

It would be fun to try to add a predictive layer to this: - Given the score and what each person has just played predict what the next few sounds are going to be - Given a high frame rate video stream of a person predict what the next note to be played In the same way that Nvidia has extremely low bandwidth but high resolution video enabled by face keypoint tracking and facial reconstruction / puppeteering maybe there's a place for prediction and/or sound reconstruction from extremely low bitrate streams.* * Obviously not the exact usecase here since a premium is being placed on not processing, but still fun to think about.

mandor 5y ago
We're doing exactly this to teleoperate humanoid robots on high-latency networks! Paper: https://arxiv.org/abs/2107.01281 https://arxiv.org/abs/2107.01281 Video: https://www.youtube.com/watch?v=N3u4ot3aIyQ https://www.youtube.com/watch?v=N3u4ot3aIyQ "We introduce a system in which a humanoid robot executes commands before it actually receives them, so that the visual feedback appears to be synchronized to the operator, whereas the robot executed the commands in the past. To do so, the robot continuously predicts future commands by querying a machine learning model that is trained on past trajectories and conditioned on the last received commands. In our experiments, an operator was able to successfully control a humanoid robot (32 degrees of freedom) with stochastic delays up to 2 seconds in several whole-body manipulation tasks, including reaching different targets, picking up, and placing a box at distinct locations."

debuggerpk 5y ago

This is revolutionary ...

fouc 5y ago

Does anyone know of an alternative that keeps both audio & video in complete sync, but possibly pauses/buffers at times in order to maximize the segments where everything is kept in sync?

mgamache 5y ago
Not possible to be in complete sync and have ultra low latency. You would have to have a jitter buffer (at least).

prendo1234 5y ago

Hey @weepy. Awesome project! As a webrtc dev - interested to hear about your choices around opus packet durations, FEC percentages, ARQ strategies, jitterbuffer length and packet redundancy.

weepy 5y ago
I used the standard 2.5ms. I know you can go lower if you want, but then you need a higher bitrate as it's "custom". I turned off FEC as it adds latency. Jitter buffer right now is just user controlled. A bit lame, - I should make it automatic, but need to get the right heuristic. With a LAN connection, the buffer can be as low as one or two packets. I don't use any packet redundancy either and there's no ARQ as if you have to ask for a retransmit you've already lost the war! How about you ?
- prendo1234 5y ago
  Yeah, awesome. Have to say this is the best version of something like this that I have tried. The packet capture didn't seem to be RTP, are you using libwebrtc under the hood? The system seems to have the settings right for realtime collab. In fact I found it much better for chatting too if you can guarantee they have a headset and good mic - I hate current-gen AEC algorithms. Interested to see how it fares in a 3+ participant setting. I had a go at jamming with my brother yesterday. We had to adjust down the delay to 2ms (I think default was 5ms) in order to counteract the "lagging" effect you get when you lock into the remote beat. Once we had that tuned it worked really well. We would occasionally suffer burst losses, but you can play through it and it's still synced afterward which is the best you can do with an unreliable network. Some feature requests: - Join room by name (we were confused that there was a different code than the name we chose) - Auto register new audio devices when you plug them in/edit in settings. - An easier way to join with multiple audio inputs. We had trouble setting up our instrument, but also adding a mic for chatting. - Local recording of your end with some shared sync markers, that you could manually or automatically sync up post-hoc The use case I use WebRTC for is slightly different, where quality >> latency , so we have it tuned to the other end and run the full gambit of FEC, ARQ, packet redundancy etc. Interesting to hear about other's solutions though!

audray 5y ago

I may be missing something but this appears to me to be a WebRTC application. It is very easy to build something like this using any modern browser's WebRTC APIs, just disable audio processing in the media constraints and munge the opus SDP to stereo, maybe play with the network buffer setting if you want to lower latency but the audio hardware and physical distance is going matter more there. The developer states that no other software was suitable, and also that it's the first of its kind. Both of those statements are not accurate: there is nothing innovative in sublive that isn't in any of the apps listed below. Props to the developer though for scratching an itch. https://en.wikipedia.org/wiki/Comparison_of_Remote_Music_Performance_Software https://en.wikipedia.org/wiki/Comparison_of_Remote_Music_Per...

deleted 5y ago
[deleted]
weepy 5y ago
It uses WebRTC for the video, but the audio latency of WebRTC is too large and uncontrollable. As stated in the post, the audio uses a custom C++ UDP solution. As far as I know it's the first video calling app with very low latency audio.
- telesilla 5y ago
  How do you deal with firewalls in that case? Sonobus has a similar problem that if you don't have accessible NAT you can't connect. You need a relay or central server which can get really expensive!
  - weepy 5y ago
    Basically similar issue to Sonobus. Relay could work buy it probably adds latency and it certainly adds complexity. I may know a way to improve some NAT configurations but need to do more research.
  - audray 5y ago
    Yeah one reason I like Jacktrip, you can optionally use servers to connect. Low latency is nice but it doesn't help in a performance when you can't connect to your group of musicians

alberth 5y ago

>”We've been able to achieve sub 20ms latency from UK to Denmark - more than 1000km” It’s pretty solid given that the speed-of-light would like 3.3ms to travel that same distance.

weepy 5y ago
Honestly I was a bit blown away when I got it working the first time ! It kind of mental that I can capture audio and compress it - send it through all these layers and machines - and receive it 1000km away 20ms later. All on consumer grade internet.