3 ms·

To learn. Mike Ash is a well known developer with a lot of great things to say about Apple technologies (I believe his day job is with Rogue Amoeba). He regula

by ary 11y ago

To learn. Mike Ash is a well known developer with a lot of great things to say about Apple technologies (I believe his day job is with Rogue Amoeba). He regularly breaks down notable parts of the Apple software stack and reimplements them as a learning exercise. His entire blog is worth a read.

DiabloD3 11y ago
Rogue Amoeba does a lot of neat things that, honestly, should have been built into OSX and other Apple products from day one, such as SoundFlower (an audio router using fake devices, inherited the stewardship from Cycling 74) and LineIn (just plays an input to an output with no magic). AirFoil is also really neat, but I wish there was an AirFoil to AirFoil zero latency PCM mode that can plug into SoundFlower and just send my audio to my Windows desktop with no otherwise magic... AirFoil via the AirPlay protocol (or anything via the AirPlay protocol, even an Apple product to any other Apple product) has at least two seconds of lag, and also wastes time by encoding it (and is probably the source of the latency). Literally, the latency is the only thing preventing me from adopting the product.
- mikeash 11y ago
  As noted in a sibling comment I'm not with Rogue Amoeba anymore, but I'll still weigh in. Airfoil is limited by the decision to stick to Apple's protocol. Obviously, this is necessary to talk to Apple's devices. It's not strictly necessary when going from Airfoil to Airfoil Speakers, but it makes life much simpler to just use the same protocol for everything. AirTunes (what Apple used to call the audio version of AirPlay, which is totally unrelated to the video version of AirPlay... confused yet?) isn't really suited to low latencies. The transmitter basically sends audio data, and then separately sends commands every so often saying, "At time T you should be playing sample number X in the audio stream." This allows sending to multiple outputs and having them all be synchronized. But for low latency you really just want "play this instantly," and to make sure you get audio data to it in a timely fashion. If it's fast enough, then you don't need to synchronize multiple outputs, since they'll be synchronized with "real time." The major challenge is dealing with packet loss. A typical WiFi network might see 1% packet loss, which you need to recover from somehow. For non-realtime operations, it's easy. You detect the loss, retransmit the data, and carry on. For realtime stuff it's harder. With video you can drop frames. It's not ideal, but it's not too bad. Dropping audio sounds way worse than dropping video looks, though, so that's not really an option for something like Airfoil. One option I've always wanted to explore was to use forward error correction on the transmitted data. Send out enough redundancy that the receiver can put the original data together even with some loss in the middle. I never have tried it out, but I think it could really cut down on latency. Without that, you're left with detecting loss and retransmitting, which eats up valuable time. Combine it with code written to do synchronized output with a substantial buffer, and you just can't do realtime transmission. I don't think the encoding/decoding step (AirTunes uses Apple Lossless encoding for the audio, for those unaware) adds all that much latency to the process, but I will admit that I never went in and measured the bits on the millisecond level, since we were looking at 2000ms end-to-end latency. If you want to experiment, there is a hidden setting in there somewhere that lets you adjust the latency Airfoil tells the receiver to use. It might be in the window that pops up when you hold option while launching the apps, or you might need to track down a hidden defaults key, I don't quite recall anymore. In my experience you probably won't be able to get it below about 200ms without lots of audible problems, and that's still plenty for a highly perceptible delay. OK, I'd better stop this before it turns into something long enough to be a blog post on its own!
  - grandinj 11y ago
    Rather that FEC, send a lower quality and bitrate version alongside that can be used to fill in the gaps.
  - thewarrior 11y ago
    Mike could you do a breakdown of how to write a simple version of soundflower on OS X , say if I wanted to apply a global system wide audio compression to my audio ? Apple used to have some sample code but I can no longer find it.
    - mikeash 11y ago
      I'm afraid I've never touched the Soundflower code or done any related development with it. It is open source though, so you can download the code and do what you will with it: https://github.com/RogueAmoeba/Soundflower https://github.com/RogueAmoeba/Soundflower
  - nitrogen 11y ago
    Forward error correction implies extra latency, so of course it has to make up for enough packet loss to provide a net reduction in latency to be useful. Is there a generic UDP-based FEC transport library that one could use to wrap a simple Opus or FLAC stream and test this out? Edit: I found http://openfec.org/ http://openfec.org/ but it seems to implement something other than packet loss recovery
    - mikeash 11y ago
      Yep. Fortunately audio is relatively low-bandwidth compared to a typical LAN so you can pile on a lot of redundancy. If you want to go crazy you could transmit everything twice from the start. AirTunes uses 352-frame packets. At 44100Hz, that's about 8ms per packet. So you definitely do not want to do FEC over more than a fairly small number of packets, or else you'll hit audible latency. (In my testing, if two sets of speakers are playing identical audio but with different delays, latency differences can be heard down to about 10ms with the right audio. Under 50ms is usually OK.) Of course, if you're doing your own protocol you could use whatever size you want for the packets, keeping in mind that overhead will start to dominate if you make them really tiny. Another problematic question is, what causes the packet loss? If it's just random, that's one thing, but what if it's outside interference? If some outside transmitter blasts your network for, say, 200ms and causes all data to be lost in that time, then a 2s buffer can recover, but a low-latency scheme is screwed no matter what. I don't know what the answer is here.
aaronbrethorst 11y ago
Mike apparently joined Plausible Labs in 2011. https://www.linkedin.com/in/mikeash https://www.linkedin.com/in/mikeash

by ary 11y ago

DiabloD3 11y ago

Rogue Amoeba does a lot of neat things that, honestly, should have been built into OSX and other Apple products from day one, such as SoundFlower (an audio router using fake devices, inherited the stewardship from Cycling 74) and LineIn (just plays an input to an output with no magic). AirFoil is also really neat, but I wish there was an AirFoil to AirFoil zero latency PCM mode that can plug into SoundFlower and just send my audio to my Windows desktop with no otherwise magic... AirFoil via the AirPlay protocol (or anything via the AirPlay protocol, even an Apple product to any other Apple product) has at least two seconds of lag, and also wastes time by encoding it (and is probably the source of the latency). Literally, the latency is the only thing preventing me from adopting the product.

mikeash 11y ago
As noted in a sibling comment I'm not with Rogue Amoeba anymore, but I'll still weigh in. Airfoil is limited by the decision to stick to Apple's protocol. Obviously, this is necessary to talk to Apple's devices. It's not strictly necessary when going from Airfoil to Airfoil Speakers, but it makes life much simpler to just use the same protocol for everything. AirTunes (what Apple used to call the audio version of AirPlay, which is totally unrelated to the video version of AirPlay... confused yet?) isn't really suited to low latencies. The transmitter basically sends audio data, and then separately sends commands every so often saying, "At time T you should be playing sample number X in the audio stream." This allows sending to multiple outputs and having them all be synchronized. But for low latency you really just want "play this instantly," and to make sure you get audio data to it in a timely fashion. If it's fast enough, then you don't need to synchronize multiple outputs, since they'll be synchronized with "real time." The major challenge is dealing with packet loss. A typical WiFi network might see 1% packet loss, which you need to recover from somehow. For non-realtime operations, it's easy. You detect the loss, retransmit the data, and carry on. For realtime stuff it's harder. With video you can drop frames. It's not ideal, but it's not too bad. Dropping audio sounds way worse than dropping video looks, though, so that's not really an option for something like Airfoil. One option I've always wanted to explore was to use forward error correction on the transmitted data. Send out enough redundancy that the receiver can put the original data together even with some loss in the middle. I never have tried it out, but I think it could really cut down on latency. Without that, you're left with detecting loss and retransmitting, which eats up valuable time. Combine it with code written to do synchronized output with a substantial buffer, and you just can't do realtime transmission. I don't think the encoding/decoding step (AirTunes uses Apple Lossless encoding for the audio, for those unaware) adds all that much latency to the process, but I will admit that I never went in and measured the bits on the millisecond level, since we were looking at 2000ms end-to-end latency. If you want to experiment, there is a hidden setting in there somewhere that lets you adjust the latency Airfoil tells the receiver to use. It might be in the window that pops up when you hold option while launching the apps, or you might need to track down a hidden defaults key, I don't quite recall anymore. In my experience you probably won't be able to get it below about 200ms without lots of audible problems, and that's still plenty for a highly perceptible delay. OK, I'd better stop this before it turns into something long enough to be a blog post on its own!
- grandinj 11y ago
  Rather that FEC, send a lower quality and bitrate version alongside that can be used to fill in the gaps.
- thewarrior 11y ago
  Mike could you do a breakdown of how to write a simple version of soundflower on OS X , say if I wanted to apply a global system wide audio compression to my audio ? Apple used to have some sample code but I can no longer find it.
  - mikeash 11y ago
    I'm afraid I've never touched the Soundflower code or done any related development with it. It is open source though, so you can download the code and do what you will with it: https://github.com/RogueAmoeba/Soundflower https://github.com/RogueAmoeba/Soundflower
- nitrogen 11y ago
  Forward error correction implies extra latency, so of course it has to make up for enough packet loss to provide a net reduction in latency to be useful. Is there a generic UDP-based FEC transport library that one could use to wrap a simple Opus or FLAC stream and test this out? Edit: I found http://openfec.org/ http://openfec.org/ but it seems to implement something other than packet loss recovery
  - mikeash 11y ago
    Yep. Fortunately audio is relatively low-bandwidth compared to a typical LAN so you can pile on a lot of redundancy. If you want to go crazy you could transmit everything twice from the start. AirTunes uses 352-frame packets. At 44100Hz, that's about 8ms per packet. So you definitely do not want to do FEC over more than a fairly small number of packets, or else you'll hit audible latency. (In my testing, if two sets of speakers are playing identical audio but with different delays, latency differences can be heard down to about 10ms with the right audio. Under 50ms is usually OK.) Of course, if you're doing your own protocol you could use whatever size you want for the packets, keeping in mind that overhead will start to dominate if you make them really tiny. Another problematic question is, what causes the packet loss? If it's just random, that's one thing, but what if it's outside interference? If some outside transmitter blasts your network for, say, 200ms and causes all data to be lost in that time, then a 2s buffer can recover, but a low-latency scheme is screwed no matter what. I don't know what the answer is here.

aaronbrethorst 11y ago

Mike apparently joined Plausible Labs in 2011. https://www.linkedin.com/in/mikeash https://www.linkedin.com/in/mikeash