20 ms·

Android’s 10 Millisecond Problem explained

431 points by lmedinas 12y ago

afandian 12y ago
Anyone with any insight about how Apple does this?
- pja 12y ago
  Same way they do with anything else they care about: by controlling everything in the stack from the hardware up.
  - raverbashing 12y ago
    I don't see how this has to do with hardware rather than architectural decisions AudioFlinger + Alsa take a lot of time, as seen per the graph But Android seems to take the option of least effort and "works most of the time" (which they have their reasons to)
    - epochwolf 12y ago
      By controlling the hardware, they need fewer drivers and they can skip one of the abstraction layers.
      - madez 12y ago
        > they need fewer drivers This suggests that you can work focused on one driver and therefore save developer-time. However, the drivers on Android are made by a bigger workforce, which must be taken into account. > they can skip one of the abstraction layers The post explicitly states that the HAL ought to add no latency at all.
        vidarh 12y ago
        Besides the point. When you know the hardware you can shave away the upper layers. Consider e.g. AmigaOS. AmigaOS let you obtain a pointer directly to the screen bitmap to update your window contents with no buffering or clipping. It could do that because originally all the hardware was the same, or close enough. Then graphics cards came along, and you didn't necessarily have a way of writing directly to the bitmap. Suddenly you had to use WritePixel() and ReadPixel() and similar, which would obtain the screen pointer for the window, and obtain the display the screen is on, and find the driver corresponding to the screen, and call the appropriate driver function via a jump table. Similarly, the AmigaOS had functions to e.g. install copper lists (the copper was a very primitive co-processor that could be used to do things like change the palette at specific scan lines), which also wouldn't work at all on graphics cards. This is why knowing the hardware is part of a limited set matters: You can define your API to match the hardware very precisely, or even expose hardware features directly.
      - __david__ 12y ago
        That whole theory is blown by the fact that CoreAudio supports 3rd party audio hardware. There is a thriving market of soundcards and audio peripherals for the professional Mac OS X market. Those all run CoreAudio, which is exactly what iOS runs.
    - mhax 12y ago
      The hardware you intend run on will impact your architectural decisions. Given the wide scope of hardware targeted by android, it's not that surprising that it performs less well than a system targeting a very limited set of devices. Having said that, it performs 'well enough' for the vast majority of use cases.
- tim333 12y ago
  Some answers on quora. TLDR Apple have been working on audio a long time. http://www.quora.com/Why-do-iPhones-have-better-professional-audio-apps-than-Android-phones http://www.quora.com/Why-do-iPhones-have-better-professional...
  - creshal 12y ago
    Or, rather, actually care about audio. Google (/Motorola Mobility, when it was owned by it) doesn't even care if a phone ships with a broken headphone port and refuses to fix it.
    - TillE 12y ago
      Seriously, the absolute garbage headphone jacks on the two flagship Android phones I owned (Motorola Milestone and Galaxy Nexus) are the reason I now own an iPhone 6. I still mostly prefer Android as an OS, but at least Apple is willing to spend the extra 50 cents on a headphone jack that lasts for more than a couple weeks.
- easytiger 12y ago
  By having their devices all roughly the same, it is easier to mitigate in app development than the variance in latency on android devices
- miahi 12y ago
  Probably by custom and known hardware. Let's say it takes 1000 working hours to produce a low latency driver/firmware; Apple only needs to do that for a few devices, but every Android manufacturer has different audio chips and most of them only care that audio works, not that it works well - so they do not invest that time for each device type. Having known hardware also works for the application developers, because they can optimize their code for the specific device (knowing that you always have a 40ms latency is way better than knowing that the latency is between 35 and 300ms). Also, Apple is much more motivated to get the audio path right, having started with the iPods and selling a lot of music.
  - madez 12y ago
    The latency problems arise with ALSA (mostly hardware-independent) and AudioFlinger (totally hardware-independent).
    - szantog 12y ago
      ALSA is super. The HAL connecting ALSA to AudioFlinger is the major problem, plus the audio stack's "push" philosophy.
    - kllrnohj 12y ago
      That's technically correct but an extremely inaccurate summary. The reason ALSA and AudioFlinger add latency is to hide hardware-dependent differences as well as kernel-caused scheduling issues & policy decisions. To achieve low-latency you need real-time scheduling, something Linux has with SCHED_FIFO but it's a bit kludgy, and getting the policy right on that is tricky (obviously you don't want a random app to be able to set a thread to SCHED_FIFO and preempt the entire system). So you have to restrict the CPU budge of a SCHED_FIFO thread, and you have to only allow apps to have a single SCHED_FIFO thread. But how much CPU time you give it needs to depend on the CPU's performance in combination with the audio buffer size that the underlying audio chip needs (and those chips also have different sample rates, is it 44.1khz or 48khz or etc...). tl;dr: this is insanely hardware-dependent.
- po 12y ago
  Look at the architecture of the Core Audio API and compare it to the parent article: https://developer.apple.com/library/mac/documentation/MusicAudio/Conceptual/CoreAudioOverview/CoreAudioEssentials/CoreAudioEssentials.html https://developer.apple.com/library/mac/documentation/MusicA... Besides the fact that you are writing in C/Objective C, most heavy lifting is being provided by Audio Units which are specifically designed with common datatypes to be chained, composed and executed in low-latency situations. Furthermore, most common things you would need in your app like mixing, conversion, timing, etc are provided as highly optimized services by the system.
  - JoeAltmaier 12y ago
    Also CoreAudio drivers (HAL) calls back every 11ms to empty/fill a buffer. So it starts with lower latency?
- coreyoconnor 12y ago
  One aspect of CoreAudio that, IMO, contributes to low latency: CoreAudio provides the buffers to process to the AudioUnit. Not the other way around. This permits CoreAudio to chain AudioUnits while minimizing buffer use. In the ideal case where all AudioUnits support the same audio stream format they are all provided the same buffers to process. Which can get close to zero-copy, zero-allocation audio rendering. I don't know enough about android to say that this is the key aspect tho. Perhaps the low level API of android is the same? I didn't find an easy reference in a quick search.
mianos 12y ago
Does anyone actually believe the reason " the Google Play store, the Music category is not even a top five revenue producing app category." is due to audio latency? I am not disagreeing with the fact that there is a lot of audio latency (no worse than Windows 8) but maybe it's time to explain cause vs correlation? Interesting article all the same.
- mcroft 12y ago
  I think the argument comes down to the fact that audio latency is preventing the makers of the most popular/lucrative apps from developing for Android, the implication being that they would develop for Android if the latency wasn't there and the music category would therefore increase in revenue numbers.
- serve_yay 12y ago
  And so your preferred hypothesis is, what, everyone just forgot to write nice professional audio and music tools for Android?
  - szantog 12y ago
    They did not forgot to write nice professional audio and music tools for Android. They can't, because of audio latency and dropouts.
- likeclockwork 12y ago
  It wouldn't be satisfying. I'm an Android user and have given thought to developing on Android but the only things I'm interesting in doing on a mobile platform are synthesis and sequencing. Based on everything I've read.. it seems like it would be a waste of time. Forget commercial viability, it doesn't even sound like it would be worth it to make something for personal use. I get 7ms roundtrip on my Linux music workstation, with my audio interface, JACK, ALSA, Bitwig.. total cost maybe $1600(when buying the machine I was trying to figure out how much I could leave out/how much of a cheapskate I could be).. not a huge sum but it shows how important low latency is. A lot of the people buying music apps(synthesizers, toys) own hardware synthesizers, have computer recording/sequencing setups, or play acoustic instruments.. some of them spend multiples of 10K on their setups over time.. when they pick up a phone or a tablet, they're not comparing it to a flash site or the performance from an integrated soundcard on a $300 laptop.. they're comparing it directly to the immediate physical response of their instruments or the low-latency response of the computer audio setup they've invested in. High latency feels incredibly sluggish and harms your sense of rhythm; unpredictable latency is just murder. Given that most makers of music software are themselves music makers.. when they pick up a device and see its audio performance is so poor the thought process is something like "If I can't even make something I would use myself on this, what's the point in trying to release something to the public?" There are some good music apps for Android, some that I would even say are very good efforts, but they're very few, and still suffer from latency pretty badly.
LoneWolf 12y ago
Developing with the NDK will not solve this problem? Does it still have to go through alsa and audioflinger?
- madez 12y ago
  Even if you write an application in C for a GNU/Linux distribution you have to use whatever sound Linux-subsystem is deployed, unless you have root permissions and want to mess concurrently with the audio stack (you don't). The problem is not the fact that they use ALSA and AudioFlinger. ALSA and AudioFlinger just use at this moment too much time. This could be improved by decreasing the period size.
- szantog 12y ago
  All of the measurements and the latency graph shows you a pure NDK implementation.
Terretta 12y ago
> “Consumers ... have a strong desire to buy such apps on Android, as shown by revenue data on iOS...” That's like saying, “Consumers have a strong desire to buy gourmet steaks from McDonald's, as shown by revenue data from Ruth's Chris.” No, McDonald's serves billions of meals by understanding its own market, not by catering to diners at Ruth's Chris. And naturally, comparing top sellers at each will give very different lists.
- DanielStraight 12y ago
  This is not a good analogy. McDonald's and Ruth's Chris are not direct competitors, they are targeting completely different markets. Android and iOS are in direct competition. If we have to use a food analogy, I would propose Starbucks and a competing coffee shop. Almost all the major players in app development make corresponding Android and iOS versions of their apps. Imagine if Twitter or Snapchat only had an iOS or only had an Android app. Similarly, customers expect to have certain things available at all coffee shops, regardless the brand behind them: cappuccinos, flavored syrups, alternative milk choices, etc. So a better analogy would be one coffee shop chain not supplying their stores with espresso machines, or flavored syrups, or alternate milk choices... or maybe just not letting them have refrigerators at all to keep milk cold. The prevalence of orders for highly-sweetened, cold, milk-based drinks at Starbucks almost certainly means that customers would order the same thing at similarly positioned coffee shops if it was available, and indeed most coffee shops offer such drinks now. There is no reason to think that somehow Android users are in such a different market that they would have no interest in these apps on Android even though in almost every other case platform parity is expected from large players in the app market.
  - patrickaljord 12y ago
    > This is not a good analogy. I'd say it's a bad analogy because most people who don't live in the US have no idea what Ruth's Chris is.
    - megaman22 12y ago
      Nor do most people who do live in the US, I would imagine... I infer that it's a high-end steak joint, but I've never heard of it.
Lewton 12y ago
This squares pretty well with my personal experience. Pretty much every music geek I know chose an iPhone over an Android because of the music apps
- buro9 12y ago
  I'm a music geek, I just chose to carry a dedicated music player.
  - Someone 12y ago
    The issue isn't about having your phone play music, it's about you playing music on your phone, in some cases with friends who play on their own phones.
    - buro9 12y ago
      Ah, for that I have a workstation. But as my guitar playing is very poor it doesn't help a great deal.
      - dvfjsdhgfv 12y ago
        It seems you're talking about two different things still. If you have an iOS device, you can connect an electric guitar to it via a dedicated (but relatively inexpensive) interface and have it sound like a reasonably amp plus some effect processors. It's really great for mobile musicians. I don't even mention Garageband and other apps which basically allow you to compose quite decent music on the go.
  - Lewton 12y ago
    Yeah I wasn't talking about that kind of music geek :) There are some pretty great instrument apps available for the iphone, like the ikaossilator by korg, a tonne of great drum machines, loop apps, synths etc. And mixed with the Audiobus app that lets you route the sound of one app into the input of another (as long as both apps support Audiobus which most seem to do), the possibilities for creating music entirely on your phone are limitless
lallysingh 12y ago
This is a pet peeve of mine. Hackers, when measuring time for software performance, please use something smaller than milliseconds. 0ms is a Dirty Lie!
- mrob 12y ago
  In this case the performance measure in question is whether the latency is perceptible to humans or not. Milliseconds is the appropriate unit. Nobody is going to notice differences of less than 1ms.
  - madez 12y ago
    Accumulation. If x is the unit of measure for the end-result and we have n components that add together, then we need at least x/n as unit of measure for the performance of each component. x/(2*n) is more reasonable to not deviate from the target performance more than one x after rounding in the worst-case.
    - AndrewDucker 12y ago
      Yes. But if some of your stages have delays of tens of milliseconds then there's no point knowing how big the delays are in the stages that are lower than 1 millisecond. Once they're all below 1ms it's worth increasing the resolution.
  - chinathrow 12y ago
    Well, e.g. 20x 0.4ms (rounded to 0ms) is going to be noticed quite soon.
- wpietri 12y ago
  Their goal is to produce and "easily digestible overview", which necessarily means removing a lot of the detail that a hardcore performance measurement person would find interesting. They were right to stick with milliseconds throughout.
nness 12y ago
Can someone explain in layman's what the issue actually is, because the article, neither its comments here or on Reddit, seem to dumb it down for non audiophiles?
- Joona 12y ago
  Imagine if you were playing an instrument, and your hand moved a little (36ms) before any sound came out. Or imagine trying to listen to your own voice, but having the sound delayed by that 35ms, so it feels like someone is talking over you every time you open your mouth.
- fredley 12y ago
  Say you want to create an app that adds a sound-effect to your voice, live. On Android, the time it takes for the audio from the microphone to get into the app, be processed, and sent back out to the speakers is long enough that it'll be jarring to use. For musical instrument apps, the latency between touching the screen and the speakers responding is too long to be useful.
- analog31 12y ago
  Audio processing on Android devices is Slow. Even seemingly small delays (10 milliseconds) is enough to make apps seem clunky and erratic. This puts the Android platform at a disadvantage for developers whose product ideas involve delivering high quality audio performance. The embedded graph with the U shaped figure pretty much sums up the author's search for the components in the software that are the culprits.
  - valdiorn 12y ago
    > Even seemingly small delays (10 milliseconds) is enough to make apps seem clunky and erratic Not just that, if you try to play guitar with a 10ms latency from input to output, you'll find that it's impossible to keep rhythm at all. It's the same effect as a speech jammer: http://www.stutterbox.co.uk/ http://www.stutterbox.co.uk/ And I don't think many people outside the the tech audio community realize how many people have ditched their guitar amplifiers and are playing guitar purely through the iOS devices. It's a great use case for tablets, but is completely infeasible to do on Android at the moment.
stinos 12y ago
So if I get it correctly the problem is twofold: there is some extra intermediate processing and the buffer size and sampling rate are fixed to 256 samples and 48kHz respectively? And which of these two does Superpowered fix? Or both? And what would be the lowest possible latency on for example the Nexus 9?
- cnvogel 12y ago
  The lowest possible input-to-output latency of an audio workstation is always two times the period size (samples/buffer) (plus a few microseconds of internal delays in the ADCs, controllers, ...) The sound chips operate on integral periods or a fixed number of samples, and when you get a period worth of audio data from your ADC, the DAC will already have started putting out the first samples of the next period. Hence, you prepare your audio samples for the second-to-next period. Assume you have audio processing code roughly looking like this: while (1) { poll(); /* some API function waiting for the "next period" */ read(soundcard, block_of_samples); /* or let DMA do it */ process_samples(block_of_samples); write(soundcard, block_of_samples); /* or let DMA do it */ } Let's try some ASCII art: v- audio samples going into your soundcard _ _ _ _ _ _ _ _ _ _ _ _ /1\ /2\ /3\ /4\ /5\ /6\ /7\ /8\ /9\ /0\ /1\ /2\ ... \_/ \_/ \_/v \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \ | \________________/\________________/\________________/\________________/ Period 1 Period 2 Period 3 Period 4 | | [*] here, Period 1 has been DMA'ed from the soundcard to an mmapped buffer of your audio application <~~~~~~~~~~~> here processing of your audio takes place [*] this is the latest point at which processing | must complete so that there will be data for | the soundcard to output. DMA will start | from the buffer to the soundcard DAC. | _ _ _ _ _ _ v _ _ _ _ _ _ / \ / \ / \ / \ / \ / \ /1\ /2\ /3\ /4\ /5\ /6\ ... \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \ \________________/\________________/\________________/\________________/ ^- processed audio samples will come out of your laptop's speakers... Obviously, your machine has to be fast enough to do the whole real-time audio computation in a little less than the time between two interrupts, that's the period size. And it must reliably be able to do this, because if it misses the time to have a block of samples ready for the DAC. If it misses that goal, a "underrun" will take place, and the audio application will have to resynchronize, possibly causing some clicking, intermittent audio, ...
supertruth 12y ago
I feel like I learned something about the problem. I was expecting to also learn about the solution but instead got hit with sales pitch at the end there. How does Superpowered get around ALSA and the Audio Flinger?
sandGorgon 12y ago
interestingly, a 2012 article[1] by Arun Raghavan mentions: On the Galaxy Nexus, for example, the best latency I can get appears to be 176 ms. This is pretty high for certain types of applications, particularly ones that generate tones based on user input. With PulseAudio, where we dynamically adjust buffering based on what clients request, I was able to drive down the total buffering to approximately 20 ms (too much lower, and we started getting dropouts). There is likely room for improvement here, and it is something on my todo list, but even out-of-the-box, we’re doing quite well. Let's get systemd on android and then see ! [1] http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-fight/ http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-f...
rglullis 12y ago
Isn't this kind of a known issue in Linux land? As much as it has improved, latency has always been the bane of audio applications in the Linux kernel. I remember in the days of kernel 2.2 that even XMMS would stop playing any music if I started using more than one or two applications. Recently I got one of those cheap USB interfaces to connect my guitar. I spent some good 4 hours changing the kernel to "low latency" one provided by ubuntu, then trying to setup aRTs to run, then trying to make pulseaudio work with it, then figuring out how to keep both aRTs and pulseaudio dependent applications happy. In the end I got most of it working and I could run some kind of guitar effects application, but the next minute I realized that the volume control media keys stopped working. It was enough for me to throw the usb interface in the drawer and give up on linux for audio applications on workstations: buying a 50€ effect pedal was cheaper than all the time devoted to it.
- jebblue 12y ago
  It was and I'm guessing still is a problem on Windows too: http://en.wikipedia.org/wiki/Latency_(audio) http://en.wikipedia.org/wiki/Latency_(audio) Linux has a Real Timer kernel you can run, I used to run it when doing audio stuff, it works great.
  - copx 12y ago
    Windows overcame its audio latency issues in the Vista days by introducing WASAPI (Windows Audio Session API), an API especially designed for professional audio processing. It grands an application exclusive, direct control of the audio hardware drivers, bypassing the higher-level audio layer of the OS. That + real-time threads (I do not know how old that feature is) allows for pro-grade latencies.
  - stinos 12y ago
    I had the exact opposite experience (about 10 years ago, didn't try any audio since on linux): even after all possible RT patches and whatnot we were happy if we would get skip-free roundtrip latencies of 50mSec on Debian with pretty much top of the line pro soundcards. Might have been bad drivers, or maybe we were just screwing up, but on Windows (with ASIO)/OsX we would just install driver, open control panel, turn down buffer size to minimum and sampling rate to maximum and be looking at about 2mSec of latency with a simple loop I->O test app (IIRC but I'm pretty sure it was 64 samples @ 96kHz)
72deluxe 12y ago
One piece of software that does do very well on Android is Caustic3. Admittedly, I am likely creating loops instead of live playing but the experience is a good one. Even better is that you can run the OSX/Windows version for £0.00 and copy your projects over to work on a "real" machine after tinkering on your phone throughout the day.
guelo 12y ago
Up to the ALSA driver step these delays would be the same on any Linux system. Do Linux desktop systems experience these types of delays? I have experienced these types of delays trying to setup a Windows box as karaoke machine. In fact, I've never seen a DJ use anything but a Mac. That leaves the question, how does Apple do it?
- stinos 12y ago
  I would guess the driver model works closely together with the hardware which in practice roughly means you get samples from the ADC and put them into a buffer which is then directly accessible by the toplevel userland software. Which in turn puts output samples into a buffer which is then directly fed to the DAC. At least that is also how it works for ASIO or for example for data acquisition cards from major players like National Instruments.
- krakensden 12y ago
  ALSA isn't the problem- JACK runs on top of ALSA, and people comfortably run well under 10ms. The problem is higher up in the stack. Even switching to the much-maligned pulseaudio could be a huge improvement: http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-fight/ http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-f...
lnikkila 12y ago
If you’re interested in this topic, you should listen to episode 20 of the Android Developers Backstage podcast. [1] Raph Levien talks about audio latency and how they started working on minimising it in Lollipop. He mentions that it’s still an ongoing process. The relevant part starts at 35 minutes into the episode. [1]: http://androidbackstage.blogspot.com/2015/01/episode-20-fonts-and-audio.html http://androidbackstage.blogspot.com/2015/01/episode-20-font...
- mackey 12y ago
  Also this: Related presentation at Google I/O 2013. https://developers.google.com/events/io/2013/sessions/325993827 https://developers.google.com/events/io/2013/sessions/325993... This was probably the best presentation I saw during Google I/O.
S_A_P 12y ago
Reading about this takes me back to the late 90s. I had just gotten serious about recording music on my computer and purchased an 866mhz dell pc. I was a using windows 98 and ASIO was still mostly a Steinberg only development and I couldn't afford cubase. I got a hand me down version of cakewalk pro audio 8. I remember the latency of that set up was about 35ms and thinking this is pretty good. There was some degree of having to start playing just slightly early so you could stay on time. With guitar and keys this was doable but I'm sure it would have been maddening with drums. I think it was when fl studio 3 came out that I had the first "daw" that supported ASIO. I remember being blown away at how much tighter everything sounded from that point on. For those interested in why iOS is better than android, a good summary would just be to say Core Audio was designed from the ground up to be a low level low latency audio API. There are fewer layers in it vs android. When designing OS X Apple knew they had a large professional media market so that got priority. I am interested to see the results of these guys efforts and have wondered when an ASIO equivalent would pop up for android.
- nathanb 12y ago
  Thank you, that's exactly what I was wondering. I didn't see anything in the article that gave good reasons for why iOS would be a significant improvement.
  - mSparks 12y ago
    Hmm, or it could just be there are a ton of very good free audio apps for android so no interest in paying for them?
- amelius 12y ago
  > There are fewer layers in it vs android. Fewer layers usually also means less flexibility.
  - SEJeff 12y ago
    Actually you have it inverted. Fewer layers means fewer abstractions. Less abstraction means you're lower level, thereby having increased flexibility. Edit: Anyone care to elaborate why the downvotes? From a technical standpoint, I'm absolutely right.
ynniv 12y ago
TL;DR the Linux layer (ALSA) and the Java layer (Audio Flinger) use widely compatible but high latency techniques, whereas Apple designed their API's and hardware such that these layers can be optimized to almost nothing. (From the article: http://superpowered.com/wp-content/uploads/2015/04/Android-Audio-Path-Latency-Superpowered-Audio700px.gif http://superpowered.com/wp-content/uploads/2015/04/Android-A...)
- szantog 12y ago
  Audio Flinger is not Java. On the graph you see absolutely no part which runs on Java. Android's media architecture with it's "push" design is not more compatible than anything else with a "pull". The many layers doesn't add any flexibility, they are "only" the result of many years hacking.
- likeclockwork 12y ago
  I don't know about Android but you can get quite good audio latency with a full Linux install, using ALSA(via JACK), provided your audio interface is up to the task.
- nileshtrivedi 12y ago
  ALSA with JACK has been shown to give latencies less than 2ms.
- cnvogel 12y ago
  ALSA is not the problem, it works very fine for the low-latency case, I can reliably run soundcards with light processing load at 96 kHz @ 64 frames/period (20.7ms --> 1.4 ms latency) on a quad core i5, e.g. for running a reverb effect, but most of the time I only record and will settle for 1024 frames/period or so. (210ms --> 20ms). The period size, just for completeness, is the number of samples recorded on each block that is forwarded to the audio processing application. If whatever audio framework you use doesn't allow to run processing with a input-to-output delay (latency) of two times the period size, it's broken (probably the case for Audio Flinger at Android, don't know much about it). ➜ ~ jackd -d alsa -p 64 -r 96000 jackdmp 1.9.10 Copyright 2001-2005 Paul Davis and others. Copyright 2004-2014 Grame. (...) creating alsa driver hw:0|hw:0|64|2|96000|0|0|nomon|swmeter|-|32bit configuring for 96000Hz, period = 64 frames (0.7 ms), buffer = 2 periods ALSA: final selected sample format for capture: 32bit integer little-endian ALSA: use 2 periods for capture ALSA: final selected sample format for playback: 32bit integer little-endian ALSA: use 2 periods for playback (this is on my laptop, just for illustration purposes)
amelius 12y ago
I hope they can fix this for Linux in general, and not just for Android.
- tveita 12y ago
  Properly configured JACK should already achieve single-digit millisecond latency on Linux.
vlaskovits 12y ago
Hi everyone -- Gabor and I wrote the piece -- let us know if you have any questions we can help answer for you.
- CountSessine 12y ago
  Excellent research - thank you for doing it.
  - vlaskovits 12y ago
    Glad you enjoyed it!
- madez 12y ago
  What are the concrete steps you are taking to tackle the issue? Idea: A latency ranking for devices would put pressure on manufactures. Will you work together with the Linux community so we all benefit from it?
  - vlaskovits 12y ago
    Hi there -- latency ranking you ask? As Westley from The Princess Bride might reply, "As you wish" http://superpowered.com/latency http://superpowered.com/latency :)
    - madez 12y ago
      Awesome, thank you! It's remarkable how apple cares about certain quality aspects of their devices. They would be interesting options if they wouldn't jail and lock you down :/ However, I wonder why newer and supposedly faster devices like the iPhone 6 have higher latency. I'd think to surpass the latter generation would be the goal for each successor. Also, it seems Samsung is taking the challenge seriously.
MrBra 12y ago
it looks it's currently down?
ohitsdom 12y ago
Is 10ms really that big of a deal? I'm an amateur musician so have some experience playing in bands, but I have a hard time believing 10ms would feel off when playing with others.
- marquis 12y ago
  9ms is the industry-recognised time that it takes before the brain notices that sync is out. If you have 2 woodblocks clapping and they are more than 9ms out of sync, you will hear one as an echo instead of 2 blocks at the same time. So if you are a band wanting to play remotely and you have 100ms network latency plus all the other latencies of the software layer, it's not going to be possible. Similarly if you are trying to drum in time with a track: more than 10ms latency from when you tap your finger to when you hear the audio, you'll have a hard time staying in sync.
  - ohitsdom 12y ago
    Thanks for the 9ms reference, that's helpful. I'm honestly surprised it's that low.
  - hyperpallium 12y ago
    IIRC it's a similar latency figure for VR headtracking (15ms), and also for touch screens (10ms).
- abruzzi 12y ago
  I think the biggest problem is that 10ms is more than enough to throw off a performer because it feels like a lag between when you hit the key and the note sounds. This makes it hard to play proficiently for anything with a fast tempo because it feels like the sound engine isn't keeping up with you. For an audience, 10ms is going to feel like sloppy timing. That will be more of an issue with tightly timed music (techno/EDM) and less with a slow jazz ballad.
  - Synaesthesia 12y ago
    Really 10ms is ok, and you can cmompensate for it mentally. As they note, it's because Android music apps actually have a latency of 100ms or worse - that it's a problem.
nsajko 12y ago
What downsides would there be to just halving the period size?
- CountSessine 12y ago
  Given that each sample means a kernel mode-user mode trip and various context switches, I'm going to guess that halving the sample size globally would make your phone's audio handling consume more CPU.
- wpietri 12y ago
  My guess is that the large buffers are covering up for scheduling issues. On my devices (Moto X, Nexus 7) I can regularly get Android audio to stutter by connecting the power or switching apps. (And it'll happen on its own sometimes when it picks, as it so often does, a terrible time to update apps.) The easy fix to that is to make the buffers bigger. Doing that, though, increases latency. The hard fix is to do what Apple has done and build for audio from the ground up.
- CamperBob2 12y ago
  The heart of the problem is simply the limited timing and computational resources in a multitasking non-realtime OS. The smaller the audio buffers are, the more prone they'll be to starvation. Only if the application can be guaranteed to receive interrupt service and/or thread timeslices at a 100 Hz rate or better is it possible to achieve audio latency of 10 milliseconds. That's a difficult thing to guarantee in a modern consumer OS. It can be done, but it won't happen by accident, only by design. The penalty for failing to service your 10-ms buffer is a dropout that sounds much worse than slightly higher latency, so there's an incentive to use larger buffers than necessary at every link in the signal chain. From the point of view of the OS vendor, musicians might complain about latency, but everyone will complain about dropouts.
- szantog 12y ago
  Android's audio stack is "pushing" audio down to the audio driver, with the audio threads sleeping between every push. In other words, the audio driver's interrupts are not scheduling the actions. Having smaller period sizes increases the risk of not "guessing" the sleep scheduling right, resulting in many audio dropouts. Using a "pull" method is required for low latency audio, where the audio driver's interrupts are scheduling when audio is passed to and pulled from applications. Another downside of halving the period size is CPU load/battery drain. There are simply too many layers in Android's audio stack, and it has quite a few unoptimised code as well (such as converting between audio sample formats with plain C code).
deleted 12y ago
[deleted]
kchoudhu 12y ago
Any word on how Windows Phone does on the latency metric?
- ygra 12y ago
  There is this thread: https://social.msdn.microsoft.com/Forums/en-US/14f96b68-60e3-47a8-8d96-9cf6fe2c347a/lowlatency-audio-on-windows-phone-8-hint-forget-it-for-now?forum=wpdevelop https://social.msdn.microsoft.com/Forums/en-US/14f96b68-60e3... and it seems to be that anything below 140 ms latency seems impossible on Windows Phone.
troymc 12y ago
There is something I don't understand; maybe someone here can explain: Sound travels at about 340 m/s (in a typical room). That means it travels about 3.4 metres in 10 milliseconds. Therefore another way to get a 10 millisecond problem is to stand 3.4 metres from the orchestra. Most people sit farther than 3.4 metres from the orchestra, yet they don't complain about a lag between when the violin bow moves and the sound is heard. Why not? (The speed of light is so fast that we can assume it's effectively infinite for the purposes of this argument.)
- s73v3r 12y ago
  Because they're sitting back and listening to the music. They're not pressing buttons and expecting feedback within a specific timeframe.
- dandelany 12y ago
  Probably because they're not directly interacting with it, they're just watching it, which makes the delay a lot less noticeable. The 10ms delay is more of a problem for interactive apps where you eg. touch the screen and expect it to instantly make a sound. In this kind of feedback loop, even small delays are distracting and it becomes difficult to keep a beat because the lag is perceptible. If you tried to remotely play an instrument from 3.4 meters away, you would probably notice it too.
  - bentcorner 12y ago
    Sort of like why audience clapping tends to get really messy until the performer does the "hands above the head" thing to get them back in sync.
    - falcor84 12y ago
      Clapping synchronisation is actually a well studied spontaneous occurrence, that just takes O(1) periods, see this article for some analysis: http://arxiv.org/pdf/cond-mat/0006423v1.pdf http://arxiv.org/pdf/cond-mat/0006423v1.pdf
kabdib 12y ago
I'd heard over the years that working with isochronous systems was difficult. I'd done a number of real-time systems before, and written OS schedulers and NTP-like systems and so forth. A little audio work should be a walk in the park, right? A little manly-man programming from the wrist and we move on to real problems. So I walked into an audio project thinking that "Oh, this latency and synchronization stuff, how bad could it really be?" Bad. Nailing down buffer-bloat and sources of latency and jitter in an isochronous system took several months, the last few weeks of which were 80-100 hour weeks just before ship. Several times we thought we'd fixed the issues, only to find that our tests had been inadequate, or that some new component broke what we had built. I remember nearly being in tears when I finally realized what the clock root of the audio system actually was, and that it wasn't what people had been using. From there everything fell into place. Don't pull the number of buffers you have out of thin air ("Oh, six is enough, maybe twelve, don't want to run out in my layer of the system, after all...") Don't assume your local underflow or overflow recovery strategy actually works in the whole system. Don't assume your underlying "real time" hypervisor won't screw you by halting the whole god damned system for 20ms while it mucks around with TLBs and physical pages and then resumes you saying, "Have fun putting all the pieces of your pipeline back together, toodle-oo!" Put debug taps and light-weight logging everywhere and attach them to tests. And know what your clock root is, or you will be sunk without a trace. Isoch is hard.
- weaksauce 12y ago
  A real postmortem of this would be a fascinating read.
  - kabdib 12y ago
    Well, if something bad happens I might have to work there again, so . . . not for a few years, probably :-)
    - walterbell 12y ago
      >when I finally realized what the clock root of the audio system actually was How did you finally figure this out -- any recommended references?
vlaskovits 12y ago
If our site goes down (again), the webcache can be found here: http://webcache.googleusercontent.com/search?q=cache:zvFOaWHye-IJ:superpowered.com/androidaudiopathlatency/+&cd=1&hl=en&ct=clnk&gl=us&client=safari http://webcache.googleusercontent.com/search?q=cache:zvFOaWH...
mox1 12y ago
I spent a lot of hours fighting with the low level Android libraries while writing a Spotify App for Android back in late 2013. There is very little documentation on the web (especially for streaming Audio). Getting this working was a serious headache. If anyone is interested, I pulled the OpenSL ES parts out and posted them to github. https://gist.github.com/mox1/3516112bc52c54aabb98 https://gist.github.com/mox1/3516112bc52c54aabb98
Pxtl 12y ago
I think the article may be ascribing too much technical reasoning to why iOS has a better community of music apps - remember that Apple is also has Garage Band and dominates the online music sales business. It's fair to say that music is a core part of Apple's brand, and so the platform has a much greater draw for people who prioritize music software.
- s73v3r 12y ago
  It's possible for Apple to have stuff like Garage Band because they have the ability to do low latency audio.
  - Pxtl 12y ago
    Garage Band (and Apple's position as the kings of music) predates iOS and Android. That's my point. Even if Android had good low-latency audio, I doubt that Android would still see the kind of audio-community we see with iOS because Apple is the brand for that sort of thing and has been that since the fruity-iMac-days.
code_duck 12y ago
Interesting to read of the details behind this issue. This has been a serious issue for me - it's actually why I own an iPhone. My first modern touch device was an iPod touch 4. I downloaded Garage Band and, as a long time milt instrumentalist and composer, loved it. I was amazed by how well the touch instruments worked and how easily I could record riffs and flesh out small snippets of songs. It ran almost flawlessly on the 4th gen Touch. Next, I decided to buy an Android phone - a Motorola Droid 2. I was surprised to find that despite the power advantage over the iPod, none of he music apps I tried were usable. The drum programs, for instance, we're so lacy and unpredictable as to be worse than useless. Hit a drum, and you may hear it seemingly instantly, maybe 1/8 a second later, maybe half a second, possibly never. Meanwhile the tiny iPod could play Garage Band instruments so well one could use it for live performance. I upgraded my phone twice, first to a Droid X, then a Galaxy S3... Each time was disappointed that the improved specs gave no improvement in the terrible audio performance. Currently I have an iPhone 6 so I can use Garage Band. Kudos to apple for doing this right - it's the best app I've ever used.
- astral303 12y ago
  Thanks for a musician's perspective!
kllrnohj 12y ago
Google has a whole section on this here (if you want more technicals and less sales-pitch): http://source.android.com/devices/audio/latency.html http://source.android.com/devices/audio/latency.html including a bunch of measurements of Nexus devices here (even going as far back as the Nexus One on Gingerbread): http://source.android.com/devices/audio/latency_measurements.html http://source.android.com/devices/audio/latency_measurements...
- tim333 12y ago
  And here are some measurements for iOS as well (Superpowered Mobile Audio Latency Test App for Android and iOS.) The iOS devices come out at 6-18ms, Androids at 17-860ms. The faster Androids have Samsung's Professional Audio SDK. http://superpowered.com/latency/ http://superpowered.com/latency/
acgourley 12y ago
While the specific problems differ, in general iOS is far easier to work with when it comes to video as well as audio. Android video is a never ending headache for us.
towelguy 12y ago
It's interesting that their business bases on Google not fixing this problem. What happens when they fix it?
- macintux 12y ago
  People have been assuming Apple's success depends on one or two minor features that someone else will copy and destroy their business for, well, ever.
guscost 12y ago
Great explanation, thanks for taking the time to write this up. A while back I figured that there had to be complicated structural reasons for the lack of progress on the notorious "issue 3434", and I decided to go with native iOS for mobile audio projects rather than wait. Seems it will be quite a difficult problem to solve (although perhaps a library that is not totally backwards-compatible would be easier to optimize). But kudos for taking a crack at it, and good luck. I'm interested to hear how it goes.
jongraehl 12y ago
Compare to a nice grand piano: 1 meter of air ~3ms. (hammer travel after you bottom out the key - another 10ms? A MIDI keyboard doesn't know the velocity of the note (or send the note at all) until key travel is mostly done too.) I have only 5 ms of ASIO buffering using my PC but I don't actually know how much actual key->sound latency; I do know that using headphones it's only slightly less immediate than a nice grand. I think the low MIDI bit rate (31kbaud) also adds a little latency on chords. It would be nice if keyboards in the future immediately sent a lower latency+precision keypress-initiated notice (before full key travel) so disk-based samplers can make ready for that note (and make any initial attack sound that's appropriate).
mrwilliamchang 12y ago
Audio lag is definitely not the worst of Android's lagginess problems.
lostgame 12y ago
The audio thing is the number one reason I will never use the Android platform. At this point, iOS devs have so many numbers of years ahead of Android devs in the music department, and the simplicity of porting existing Mac OS compatible audio stuff makes it incomparable. Metaphor: Two people are going for a race, over the same distance. One starts 5 years ahead of the other. At that point, is anyone even watching the race anymore? Sorry, despite it's absolutely God-awful flaws such as no user-accessible filesystem, no ability to do a basic task such as download an MP3, without GarageBand (and it's awesome ability to open files I sketch out on the go right in Logic on my Mac...), without iElectribe, and the list goes on, I absolutely just can't even use the platform, and I have no reason to go back 5 years technologically. Sorry, Android, you already lost, here is a user you can never have.
- JoeAltmaier 12y ago
  Not really good metaphors. Its not necessary to cover the intervening ground, to catch up technologically. Fix latency, app developers port and voila - Android looks pretty good again.
  - lostgame 12y ago
    Did you not get the '5 year head start' bit? Doesn't matter how fast developers create applications that are amazing, iOS has already had all these years to create an already fantastic subset of these applications. Doesn't matter how 'good' it looks - for the use case of music, android has simply lost.
    - JoeAltmaier 12y ago
      That was actually my point. There's no need for Android development to cover that ground - it can just start where we are now, with cool apps that can just be ported over. Once the Android audio latency issue is addressed. App developers LOVE to port, its another sale with small effort.
Finster 12y ago
Does this cause audio de-sync issues on Android?
dottrap 12y ago
This is a great description of the lowest level problem. But for those wondering why iOS doesn't have the problem or why only 10ms is such a difference, the answer is that this is only the beginning of the problem of Android audio latency. There are still additional problems that add latency that span the entire Android stack from the actual hardware and drivers, to the kernel and scheduler, to the Android implementation of their audio APIs. Anybody who has tried to do any serious audio on Android knows the infamous Bug 3434: https://code.google.com/p/android/issues/detail?id=3434 https://code.google.com/p/android/issues/detail?id=3434 Google I/O 2013 did a pretty good talk on the problem and shows how there are problems across the entire stack. Glenn Kasten pretty much carries the brunt of all the audio problems with Android. I find it telling that he had to handcraft his own latency measurement device using an oscilloscope and LED because there were no actual tools built into the OS to help them analyze performance. https://www.youtube.com/watch?v=d3kfEeMZ65c https://www.youtube.com/watch?v=d3kfEeMZ65c Audio has been terrible since Android's inception. It has improved a little over time, but unfortunately, 7 years later is is still pretty much unacceptable for any serious work.
- rasz_pl 12y ago
  Whole Androids latency has been terrible since its inception, screen/input/audio. Android is famous for $600 2GHz phones that drop frames when doing simple animations :(.
cfstras 12y ago
Going completely off-topic: Why has everyone recently decided to use ultra-thin fonts everywhere? On my system (24" FullHD, Win8.1, Chrome 42 and Ubuntu 14.04, Chrome 43) the text is thinner than one pixel and thus unreadable below 130% zoom. Sure, I can open up DevTools and fix the font-weight, but seriously?
- emn13 12y ago
  My theory on this is that Windows 8 hyped this. It's particularly problematic when it's being copied in mac circles because OSX has a biased font-smoothing algorithm that adds font weight (it's a design flaw). In other words, to get something to look thin on a mac, especially a non-retina mac, it needs to be very thin, sometimes less than a pixel thin. How other systems display borderline visible strokes varies depending on the system and the details of the font.
  - cfstras 12y ago
    Oh, so I can tell whether the designer uses OSX by the fact that I can't read the fonts. That's nice.
- UberMouse 12y ago
  The website font? I'm viewing it on a 27" 2560x1440 screen, Chrome 42 and OS X 10.10 and the fonts are normal sized for me. In fact larger than the HN font.
deleted 12y ago
[deleted]
Polyphonie 12y ago
One of the reason for the audio latency issue on Android is the complex power management circuitry (PMIC) implemented on most of not all Qualcomm SoCs. That's why to anyone with experience in regards to using Android device in audios production, most would agree that the older 2011 Galaxy Nexus is still the Android device to use. Unlike the the recent Snapdragon based Nexus device, the GNex is based on TI OMAP4460 (dual-core ARM Cortex A9). It might not be a coincidence the Apple kept the number of cores on its A series SoCs to a minimum (only moving to 3 with the A8X). http://createdigitalmusic.com/2013/05/why-mobile-low-latency-is-hard-explained-by-google-galaxy-nexus-still-android-of-choice/ http://createdigitalmusic.com/2013/05/why-mobile-low-latency...
yeureka 12y ago
I am writing a game that implements an audio sequencer and it runs great on iOS and OSX. But I had to write a karaoke app for Android recently and found an amazing 100ms latency there which makes me think I won't be able to port my game to Android... Wish Google fixed this.
faragon 12y ago
Android has many problems: - Poor event handling. - Braindead UI. - Braindead VM approach: WTF JIT on mobile devices??!!! Google: fuck fix the UI, fuck fix the VM bullshit, fuck fix bloating everywhere. NDK is not the solution: the problem is in the architecture.
DigitalSea 12y ago
This is the one downside with using Android I have experienced. I came from an iPhone 4 and as an amateur musician I used it to produce music, record ideas on the go and whatnot. When I moved to a Samsung Galaxy S4 (my first Android phone) I was mortified. I grown to love a couple of drumming applications on iOS, when I moved to Android there were a few sub-par applications and they all had ridiculous latency, to the point they were not usable and I had to bust out my keyboard at home to record drums (no drum ideas on the go anymore). You hit a snare in a drumming app and it felt like hundreds of milliseconds of lag, horrible. It was so bad to the point where I went out and purchased an iPad just so I could have that freedom of recording ideas when I am not at home. The iPad and iPhone offered what sounded like basically no latency at all, I never understood why Android devices struggled (but I speculated and assumed it was how the audio was being processed). I considered moving back to an iPhone, but I love the freedom that Android affords me and the competition, so I stuck it out and kept using the iPad. I recently purchased a Samsung Galaxy S6 Edge and while I still notice some slight latency, it is usable again. I can finally jot down ideas when I am away from home using my phone again. Research seems to yield some improvements that Samsung themselves have made to their hardware and software, not to mention the professional audio driver that allows the use of a third party audio interface (great feature by the way). Google needs to make this a priority, because believe it or not a lot of people use their tablets and phones to produce music. We need to fix this. Music production might not seem like a big deal to Google, but Apple definitely gets it and have from the beginning. The one aspect I miss about owning an Apple device, not enough to make me switch back but definitely a good feature of iOS devices.
elcct 12y ago
This is actually a known bug hanging there since 2009: https://code.google.com/p/android/issues/detail?id=3434 https://code.google.com/p/android/issues/detail?id=3434 And very good video from Google about it: http://www.youtube.com/watch?v=d3kfEeMZ65c http://www.youtube.com/watch?v=d3kfEeMZ65c
bobby5467 12y ago
titties
deleted 12y ago
[deleted]
MrBra 12y ago
Happy to see this side is getting attention!

431 points by lmedinas 12y ago

afandian 12y ago

Anyone with any insight about how Apple does this?

pja 12y ago
Same way they do with anything else they care about: by controlling everything in the stack from the hardware up.
- raverbashing 12y ago
  I don't see how this has to do with hardware rather than architectural decisions AudioFlinger + Alsa take a lot of time, as seen per the graph But Android seems to take the option of least effort and "works most of the time" (which they have their reasons to)
  - epochwolf 12y ago
    By controlling the hardware, they need fewer drivers and they can skip one of the abstraction layers.
    - madez 12y ago
      > they need fewer drivers This suggests that you can work focused on one driver and therefore save developer-time. However, the drivers on Android are made by a bigger workforce, which must be taken into account. > they can skip one of the abstraction layers The post explicitly states that the HAL ought to add no latency at all.
      - vidarh 12y ago
        Besides the point. When you know the hardware you can shave away the upper layers. Consider e.g. AmigaOS. AmigaOS let you obtain a pointer directly to the screen bitmap to update your window contents with no buffering or clipping. It could do that because originally all the hardware was the same, or close enough. Then graphics cards came along, and you didn't necessarily have a way of writing directly to the bitmap. Suddenly you had to use WritePixel() and ReadPixel() and similar, which would obtain the screen pointer for the window, and obtain the display the screen is on, and find the driver corresponding to the screen, and call the appropriate driver function via a jump table. Similarly, the AmigaOS had functions to e.g. install copper lists (the copper was a very primitive co-processor that could be used to do things like change the palette at specific scan lines), which also wouldn't work at all on graphics cards. This is why knowing the hardware is part of a limited set matters: You can define your API to match the hardware very precisely, or even expose hardware features directly.
    - __david__ 12y ago
      That whole theory is blown by the fact that CoreAudio supports 3rd party audio hardware. There is a thriving market of soundcards and audio peripherals for the professional Mac OS X market. Those all run CoreAudio, which is exactly what iOS runs.
  - mhax 12y ago
    The hardware you intend run on will impact your architectural decisions. Given the wide scope of hardware targeted by android, it's not that surprising that it performs less well than a system targeting a very limited set of devices. Having said that, it performs 'well enough' for the vast majority of use cases.
tim333 12y ago
Some answers on quora. TLDR Apple have been working on audio a long time. http://www.quora.com/Why-do-iPhones-have-better-professional-audio-apps-than-Android-phones http://www.quora.com/Why-do-iPhones-have-better-professional...
- creshal 12y ago
  Or, rather, actually care about audio. Google (/Motorola Mobility, when it was owned by it) doesn't even care if a phone ships with a broken headphone port and refuses to fix it.
  - TillE 12y ago
    Seriously, the absolute garbage headphone jacks on the two flagship Android phones I owned (Motorola Milestone and Galaxy Nexus) are the reason I now own an iPhone 6. I still mostly prefer Android as an OS, but at least Apple is willing to spend the extra 50 cents on a headphone jack that lasts for more than a couple weeks.
easytiger 12y ago
By having their devices all roughly the same, it is easier to mitigate in app development than the variance in latency on android devices
miahi 12y ago
Probably by custom and known hardware. Let's say it takes 1000 working hours to produce a low latency driver/firmware; Apple only needs to do that for a few devices, but every Android manufacturer has different audio chips and most of them only care that audio works, not that it works well - so they do not invest that time for each device type. Having known hardware also works for the application developers, because they can optimize their code for the specific device (knowing that you always have a 40ms latency is way better than knowing that the latency is between 35 and 300ms). Also, Apple is much more motivated to get the audio path right, having started with the iPods and selling a lot of music.
- madez 12y ago
  The latency problems arise with ALSA (mostly hardware-independent) and AudioFlinger (totally hardware-independent).
  - szantog 12y ago
    ALSA is super. The HAL connecting ALSA to AudioFlinger is the major problem, plus the audio stack's "push" philosophy.
  - kllrnohj 12y ago
    That's technically correct but an extremely inaccurate summary. The reason ALSA and AudioFlinger add latency is to hide hardware-dependent differences as well as kernel-caused scheduling issues & policy decisions. To achieve low-latency you need real-time scheduling, something Linux has with SCHED_FIFO but it's a bit kludgy, and getting the policy right on that is tricky (obviously you don't want a random app to be able to set a thread to SCHED_FIFO and preempt the entire system). So you have to restrict the CPU budge of a SCHED_FIFO thread, and you have to only allow apps to have a single SCHED_FIFO thread. But how much CPU time you give it needs to depend on the CPU's performance in combination with the audio buffer size that the underlying audio chip needs (and those chips also have different sample rates, is it 44.1khz or 48khz or etc...). tl;dr: this is insanely hardware-dependent.
po 12y ago
Look at the architecture of the Core Audio API and compare it to the parent article: https://developer.apple.com/library/mac/documentation/MusicAudio/Conceptual/CoreAudioOverview/CoreAudioEssentials/CoreAudioEssentials.html https://developer.apple.com/library/mac/documentation/MusicA... Besides the fact that you are writing in C/Objective C, most heavy lifting is being provided by Audio Units which are specifically designed with common datatypes to be chained, composed and executed in low-latency situations. Furthermore, most common things you would need in your app like mixing, conversion, timing, etc are provided as highly optimized services by the system.
- JoeAltmaier 12y ago
  Also CoreAudio drivers (HAL) calls back every 11ms to empty/fill a buffer. So it starts with lower latency?
coreyoconnor 12y ago
One aspect of CoreAudio that, IMO, contributes to low latency: CoreAudio provides the buffers to process to the AudioUnit. Not the other way around. This permits CoreAudio to chain AudioUnits while minimizing buffer use. In the ideal case where all AudioUnits support the same audio stream format they are all provided the same buffers to process. Which can get close to zero-copy, zero-allocation audio rendering. I don't know enough about android to say that this is the key aspect tho. Perhaps the low level API of android is the same? I didn't find an easy reference in a quick search.

mianos 12y ago

Does anyone actually believe the reason " the Google Play store, the Music category is not even a top five revenue producing app category." is due to audio latency? I am not disagreeing with the fact that there is a lot of audio latency (no worse than Windows 8) but maybe it's time to explain cause vs correlation? Interesting article all the same.

mcroft 12y ago
I think the argument comes down to the fact that audio latency is preventing the makers of the most popular/lucrative apps from developing for Android, the implication being that they would develop for Android if the latency wasn't there and the music category would therefore increase in revenue numbers.
serve_yay 12y ago
And so your preferred hypothesis is, what, everyone just forgot to write nice professional audio and music tools for Android?
- szantog 12y ago
  They did not forgot to write nice professional audio and music tools for Android. They can't, because of audio latency and dropouts.
likeclockwork 12y ago
It wouldn't be satisfying. I'm an Android user and have given thought to developing on Android but the only things I'm interesting in doing on a mobile platform are synthesis and sequencing. Based on everything I've read.. it seems like it would be a waste of time. Forget commercial viability, it doesn't even sound like it would be worth it to make something for personal use. I get 7ms roundtrip on my Linux music workstation, with my audio interface, JACK, ALSA, Bitwig.. total cost maybe $1600(when buying the machine I was trying to figure out how much I could leave out/how much of a cheapskate I could be).. not a huge sum but it shows how important low latency is. A lot of the people buying music apps(synthesizers, toys) own hardware synthesizers, have computer recording/sequencing setups, or play acoustic instruments.. some of them spend multiples of 10K on their setups over time.. when they pick up a phone or a tablet, they're not comparing it to a flash site or the performance from an integrated soundcard on a $300 laptop.. they're comparing it directly to the immediate physical response of their instruments or the low-latency response of the computer audio setup they've invested in. High latency feels incredibly sluggish and harms your sense of rhythm; unpredictable latency is just murder. Given that most makers of music software are themselves music makers.. when they pick up a device and see its audio performance is so poor the thought process is something like "If I can't even make something I would use myself on this, what's the point in trying to release something to the public?" There are some good music apps for Android, some that I would even say are very good efforts, but they're very few, and still suffer from latency pretty badly.

LoneWolf 12y ago

Developing with the NDK will not solve this problem? Does it still have to go through alsa and audioflinger?

madez 12y ago
Even if you write an application in C for a GNU/Linux distribution you have to use whatever sound Linux-subsystem is deployed, unless you have root permissions and want to mess concurrently with the audio stack (you don't). The problem is not the fact that they use ALSA and AudioFlinger. ALSA and AudioFlinger just use at this moment too much time. This could be improved by decreasing the period size.
szantog 12y ago
All of the measurements and the latency graph shows you a pure NDK implementation.

Terretta 12y ago

> “Consumers ... have a strong desire to buy such apps on Android, as shown by revenue data on iOS...” That's like saying, “Consumers have a strong desire to buy gourmet steaks from McDonald's, as shown by revenue data from Ruth's Chris.” No, McDonald's serves billions of meals by understanding its own market, not by catering to diners at Ruth's Chris. And naturally, comparing top sellers at each will give very different lists.

DanielStraight 12y ago
This is not a good analogy. McDonald's and Ruth's Chris are not direct competitors, they are targeting completely different markets. Android and iOS are in direct competition. If we have to use a food analogy, I would propose Starbucks and a competing coffee shop. Almost all the major players in app development make corresponding Android and iOS versions of their apps. Imagine if Twitter or Snapchat only had an iOS or only had an Android app. Similarly, customers expect to have certain things available at all coffee shops, regardless the brand behind them: cappuccinos, flavored syrups, alternative milk choices, etc. So a better analogy would be one coffee shop chain not supplying their stores with espresso machines, or flavored syrups, or alternate milk choices... or maybe just not letting them have refrigerators at all to keep milk cold. The prevalence of orders for highly-sweetened, cold, milk-based drinks at Starbucks almost certainly means that customers would order the same thing at similarly positioned coffee shops if it was available, and indeed most coffee shops offer such drinks now. There is no reason to think that somehow Android users are in such a different market that they would have no interest in these apps on Android even though in almost every other case platform parity is expected from large players in the app market.
- patrickaljord 12y ago
  > This is not a good analogy. I'd say it's a bad analogy because most people who don't live in the US have no idea what Ruth's Chris is.
  - megaman22 12y ago
    Nor do most people who do live in the US, I would imagine... I infer that it's a high-end steak joint, but I've never heard of it.

Lewton 12y ago

This squares pretty well with my personal experience. Pretty much every music geek I know chose an iPhone over an Android because of the music apps

buro9 12y ago
I'm a music geek, I just chose to carry a dedicated music player.
- Someone 12y ago
  The issue isn't about having your phone play music, it's about you playing music on your phone, in some cases with friends who play on their own phones.
  - buro9 12y ago
    Ah, for that I have a workstation. But as my guitar playing is very poor it doesn't help a great deal.
    - dvfjsdhgfv 12y ago
      It seems you're talking about two different things still. If you have an iOS device, you can connect an electric guitar to it via a dedicated (but relatively inexpensive) interface and have it sound like a reasonably amp plus some effect processors. It's really great for mobile musicians. I don't even mention Garageband and other apps which basically allow you to compose quite decent music on the go.
- Lewton 12y ago
  Yeah I wasn't talking about that kind of music geek :) There are some pretty great instrument apps available for the iphone, like the ikaossilator by korg, a tonne of great drum machines, loop apps, synths etc. And mixed with the Audiobus app that lets you route the sound of one app into the input of another (as long as both apps support Audiobus which most seem to do), the possibilities for creating music entirely on your phone are limitless

lallysingh 12y ago

This is a pet peeve of mine. Hackers, when measuring time for software performance, please use something smaller than milliseconds. 0ms is a Dirty Lie!

mrob 12y ago
In this case the performance measure in question is whether the latency is perceptible to humans or not. Milliseconds is the appropriate unit. Nobody is going to notice differences of less than 1ms.
- madez 12y ago
  Accumulation. If x is the unit of measure for the end-result and we have n components that add together, then we need at least x/n as unit of measure for the performance of each component. x/(2*n) is more reasonable to not deviate from the target performance more than one x after rounding in the worst-case.
  - AndrewDucker 12y ago
    Yes. But if some of your stages have delays of tens of milliseconds then there's no point knowing how big the delays are in the stages that are lower than 1 millisecond. Once they're all below 1ms it's worth increasing the resolution.
- chinathrow 12y ago
  Well, e.g. 20x 0.4ms (rounded to 0ms) is going to be noticed quite soon.
wpietri 12y ago
Their goal is to produce and "easily digestible overview", which necessarily means removing a lot of the detail that a hardcore performance measurement person would find interesting. They were right to stick with milliseconds throughout.

nness 12y ago

Can someone explain in layman's what the issue actually is, because the article, neither its comments here or on Reddit, seem to dumb it down for non audiophiles?

Joona 12y ago
Imagine if you were playing an instrument, and your hand moved a little (36ms) before any sound came out. Or imagine trying to listen to your own voice, but having the sound delayed by that 35ms, so it feels like someone is talking over you every time you open your mouth.
fredley 12y ago
Say you want to create an app that adds a sound-effect to your voice, live. On Android, the time it takes for the audio from the microphone to get into the app, be processed, and sent back out to the speakers is long enough that it'll be jarring to use. For musical instrument apps, the latency between touching the screen and the speakers responding is too long to be useful.
analog31 12y ago
Audio processing on Android devices is Slow. Even seemingly small delays (10 milliseconds) is enough to make apps seem clunky and erratic. This puts the Android platform at a disadvantage for developers whose product ideas involve delivering high quality audio performance. The embedded graph with the U shaped figure pretty much sums up the author's search for the components in the software that are the culprits.
- valdiorn 12y ago
  > Even seemingly small delays (10 milliseconds) is enough to make apps seem clunky and erratic Not just that, if you try to play guitar with a 10ms latency from input to output, you'll find that it's impossible to keep rhythm at all. It's the same effect as a speech jammer: http://www.stutterbox.co.uk/ http://www.stutterbox.co.uk/ And I don't think many people outside the the tech audio community realize how many people have ditched their guitar amplifiers and are playing guitar purely through the iOS devices. It's a great use case for tablets, but is completely infeasible to do on Android at the moment.

stinos 12y ago

So if I get it correctly the problem is twofold: there is some extra intermediate processing and the buffer size and sampling rate are fixed to 256 samples and 48kHz respectively? And which of these two does Superpowered fix? Or both? And what would be the lowest possible latency on for example the Nexus 9?

cnvogel 12y ago
The lowest possible input-to-output latency of an audio workstation is always two times the period size (samples/buffer) (plus a few microseconds of internal delays in the ADCs, controllers, ...) The sound chips operate on integral periods or a fixed number of samples, and when you get a period worth of audio data from your ADC, the DAC will already have started putting out the first samples of the next period. Hence, you prepare your audio samples for the second-to-next period. Assume you have audio processing code roughly looking like this: while (1) { poll(); /* some API function waiting for the "next period" */ read(soundcard, block_of_samples); /* or let DMA do it */ process_samples(block_of_samples); write(soundcard, block_of_samples); /* or let DMA do it */ } Let's try some ASCII art: v- audio samples going into your soundcard _ _ _ _ _ _ _ _ _ _ _ _ /1\ /2\ /3\ /4\ /5\ /6\ /7\ /8\ /9\ /0\ /1\ /2\ ... \_/ \_/ \_/v \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \ | \________________/\________________/\________________/\________________/ Period 1 Period 2 Period 3 Period 4 | | [*] here, Period 1 has been DMA'ed from the soundcard to an mmapped buffer of your audio application <~~~~~~~~~~~> here processing of your audio takes place [*] this is the latest point at which processing | must complete so that there will be data for | the soundcard to output. DMA will start | from the buffer to the soundcard DAC. | _ _ _ _ _ _ v _ _ _ _ _ _ / \ / \ / \ / \ / \ / \ /1\ /2\ /3\ /4\ /5\ /6\ ... \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \_/ \ \________________/\________________/\________________/\________________/ ^- processed audio samples will come out of your laptop's speakers... Obviously, your machine has to be fast enough to do the whole real-time audio computation in a little less than the time between two interrupts, that's the period size. And it must reliably be able to do this, because if it misses the time to have a block of samples ready for the DAC. If it misses that goal, a "underrun" will take place, and the audio application will have to resynchronize, possibly causing some clicking, intermittent audio, ...

supertruth 12y ago

I feel like I learned something about the problem. I was expecting to also learn about the solution but instead got hit with sales pitch at the end there. How does Superpowered get around ALSA and the Audio Flinger?

sandGorgon 12y ago

interestingly, a 2012 article[1] by Arun Raghavan mentions: On the Galaxy Nexus, for example, the best latency I can get appears to be 176 ms. This is pretty high for certain types of applications, particularly ones that generate tones based on user input. With PulseAudio, where we dynamically adjust buffering based on what clients request, I was able to drive down the total buffering to approximately 20 ms (too much lower, and we started getting dropouts). There is likely room for improvement here, and it is something on my todo list, but even out-of-the-box, we’re doing quite well. Let's get systemd on android and then see ! [1] http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-fight/ http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-f...

rglullis 12y ago

Isn't this kind of a known issue in Linux land? As much as it has improved, latency has always been the bane of audio applications in the Linux kernel. I remember in the days of kernel 2.2 that even XMMS would stop playing any music if I started using more than one or two applications. Recently I got one of those cheap USB interfaces to connect my guitar. I spent some good 4 hours changing the kernel to "low latency" one provided by ubuntu, then trying to setup aRTs to run, then trying to make pulseaudio work with it, then figuring out how to keep both aRTs and pulseaudio dependent applications happy. In the end I got most of it working and I could run some kind of guitar effects application, but the next minute I realized that the volume control media keys stopped working. It was enough for me to throw the usb interface in the drawer and give up on linux for audio applications on workstations: buying a 50€ effect pedal was cheaper than all the time devoted to it.

jebblue 12y ago
It was and I'm guessing still is a problem on Windows too: http://en.wikipedia.org/wiki/Latency_(audio) http://en.wikipedia.org/wiki/Latency_(audio) Linux has a Real Timer kernel you can run, I used to run it when doing audio stuff, it works great.
- copx 12y ago
  Windows overcame its audio latency issues in the Vista days by introducing WASAPI (Windows Audio Session API), an API especially designed for professional audio processing. It grands an application exclusive, direct control of the audio hardware drivers, bypassing the higher-level audio layer of the OS. That + real-time threads (I do not know how old that feature is) allows for pro-grade latencies.
- stinos 12y ago
  I had the exact opposite experience (about 10 years ago, didn't try any audio since on linux): even after all possible RT patches and whatnot we were happy if we would get skip-free roundtrip latencies of 50mSec on Debian with pretty much top of the line pro soundcards. Might have been bad drivers, or maybe we were just screwing up, but on Windows (with ASIO)/OsX we would just install driver, open control panel, turn down buffer size to minimum and sampling rate to maximum and be looking at about 2mSec of latency with a simple loop I->O test app (IIRC but I'm pretty sure it was 64 samples @ 96kHz)

72deluxe 12y ago

One piece of software that does do very well on Android is Caustic3. Admittedly, I am likely creating loops instead of live playing but the experience is a good one. Even better is that you can run the OSX/Windows version for £0.00 and copy your projects over to work on a "real" machine after tinkering on your phone throughout the day.

guelo 12y ago

Up to the ALSA driver step these delays would be the same on any Linux system. Do Linux desktop systems experience these types of delays? I have experienced these types of delays trying to setup a Windows box as karaoke machine. In fact, I've never seen a DJ use anything but a Mac. That leaves the question, how does Apple do it?

stinos 12y ago
I would guess the driver model works closely together with the hardware which in practice roughly means you get samples from the ADC and put them into a buffer which is then directly accessible by the toplevel userland software. Which in turn puts output samples into a buffer which is then directly fed to the DAC. At least that is also how it works for ASIO or for example for data acquisition cards from major players like National Instruments.
krakensden 12y ago
ALSA isn't the problem- JACK runs on top of ALSA, and people comfortably run well under 10ms. The problem is higher up in the stack. Even switching to the much-maligned pulseaudio could be a huge improvement: http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-fight/ http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-f...

lnikkila 12y ago

If you’re interested in this topic, you should listen to episode 20 of the Android Developers Backstage podcast. [1] Raph Levien talks about audio latency and how they started working on minimising it in Lollipop. He mentions that it’s still an ongoing process. The relevant part starts at 35 minutes into the episode. [1]: http://androidbackstage.blogspot.com/2015/01/episode-20-fonts-and-audio.html http://androidbackstage.blogspot.com/2015/01/episode-20-font...

mackey 12y ago
Also this: Related presentation at Google I/O 2013. https://developers.google.com/events/io/2013/sessions/325993827 https://developers.google.com/events/io/2013/sessions/325993... This was probably the best presentation I saw during Google I/O.

S_A_P 12y ago

Reading about this takes me back to the late 90s. I had just gotten serious about recording music on my computer and purchased an 866mhz dell pc. I was a using windows 98 and ASIO was still mostly a Steinberg only development and I couldn't afford cubase. I got a hand me down version of cakewalk pro audio 8. I remember the latency of that set up was about 35ms and thinking this is pretty good. There was some degree of having to start playing just slightly early so you could stay on time. With guitar and keys this was doable but I'm sure it would have been maddening with drums. I think it was when fl studio 3 came out that I had the first "daw" that supported ASIO. I remember being blown away at how much tighter everything sounded from that point on. For those interested in why iOS is better than android, a good summary would just be to say Core Audio was designed from the ground up to be a low level low latency audio API. There are fewer layers in it vs android. When designing OS X Apple knew they had a large professional media market so that got priority. I am interested to see the results of these guys efforts and have wondered when an ASIO equivalent would pop up for android.

nathanb 12y ago
Thank you, that's exactly what I was wondering. I didn't see anything in the article that gave good reasons for why iOS would be a significant improvement.
- mSparks 12y ago
  Hmm, or it could just be there are a ton of very good free audio apps for android so no interest in paying for them?
amelius 12y ago
> There are fewer layers in it vs android. Fewer layers usually also means less flexibility.
- SEJeff 12y ago
  Actually you have it inverted. Fewer layers means fewer abstractions. Less abstraction means you're lower level, thereby having increased flexibility. Edit: Anyone care to elaborate why the downvotes? From a technical standpoint, I'm absolutely right.

ynniv 12y ago

TL;DR the Linux layer (ALSA) and the Java layer (Audio Flinger) use widely compatible but high latency techniques, whereas Apple designed their API's and hardware such that these layers can be optimized to almost nothing. (From the article: http://superpowered.com/wp-content/uploads/2015/04/Android-Audio-Path-Latency-Superpowered-Audio700px.gif http://superpowered.com/wp-content/uploads/2015/04/Android-A...)

szantog 12y ago
Audio Flinger is not Java. On the graph you see absolutely no part which runs on Java. Android's media architecture with it's "push" design is not more compatible than anything else with a "pull". The many layers doesn't add any flexibility, they are "only" the result of many years hacking.
likeclockwork 12y ago
I don't know about Android but you can get quite good audio latency with a full Linux install, using ALSA(via JACK), provided your audio interface is up to the task.
nileshtrivedi 12y ago
ALSA with JACK has been shown to give latencies less than 2ms.
cnvogel 12y ago
ALSA is not the problem, it works very fine for the low-latency case, I can reliably run soundcards with light processing load at 96 kHz @ 64 frames/period (20.7ms --> 1.4 ms latency) on a quad core i5, e.g. for running a reverb effect, but most of the time I only record and will settle for 1024 frames/period or so. (210ms --> 20ms). The period size, just for completeness, is the number of samples recorded on each block that is forwarded to the audio processing application. If whatever audio framework you use doesn't allow to run processing with a input-to-output delay (latency) of two times the period size, it's broken (probably the case for Audio Flinger at Android, don't know much about it). ➜ ~ jackd -d alsa -p 64 -r 96000 jackdmp 1.9.10 Copyright 2001-2005 Paul Davis and others. Copyright 2004-2014 Grame. (...) creating alsa driver hw:0|hw:0|64|2|96000|0|0|nomon|swmeter|-|32bit configuring for 96000Hz, period = 64 frames (0.7 ms), buffer = 2 periods ALSA: final selected sample format for capture: 32bit integer little-endian ALSA: use 2 periods for capture ALSA: final selected sample format for playback: 32bit integer little-endian ALSA: use 2 periods for playback (this is on my laptop, just for illustration purposes)

amelius 12y ago

I hope they can fix this for Linux in general, and not just for Android.

tveita 12y ago
Properly configured JACK should already achieve single-digit millisecond latency on Linux.

vlaskovits 12y ago

Hi everyone -- Gabor and I wrote the piece -- let us know if you have any questions we can help answer for you.

CountSessine 12y ago
Excellent research - thank you for doing it.
- vlaskovits 12y ago
  Glad you enjoyed it!
madez 12y ago
What are the concrete steps you are taking to tackle the issue? Idea: A latency ranking for devices would put pressure on manufactures. Will you work together with the Linux community so we all benefit from it?
- vlaskovits 12y ago
  Hi there -- latency ranking you ask? As Westley from The Princess Bride might reply, "As you wish" http://superpowered.com/latency http://superpowered.com/latency :)
  - madez 12y ago
    Awesome, thank you! It's remarkable how apple cares about certain quality aspects of their devices. They would be interesting options if they wouldn't jail and lock you down :/ However, I wonder why newer and supposedly faster devices like the iPhone 6 have higher latency. I'd think to surpass the latter generation would be the goal for each successor. Also, it seems Samsung is taking the challenge seriously.

MrBra 12y ago

it looks it's currently down?

ohitsdom 12y ago

Is 10ms really that big of a deal? I'm an amateur musician so have some experience playing in bands, but I have a hard time believing 10ms would feel off when playing with others.

marquis 12y ago
9ms is the industry-recognised time that it takes before the brain notices that sync is out. If you have 2 woodblocks clapping and they are more than 9ms out of sync, you will hear one as an echo instead of 2 blocks at the same time. So if you are a band wanting to play remotely and you have 100ms network latency plus all the other latencies of the software layer, it's not going to be possible. Similarly if you are trying to drum in time with a track: more than 10ms latency from when you tap your finger to when you hear the audio, you'll have a hard time staying in sync.
- ohitsdom 12y ago
  Thanks for the 9ms reference, that's helpful. I'm honestly surprised it's that low.
- hyperpallium 12y ago
  IIRC it's a similar latency figure for VR headtracking (15ms), and also for touch screens (10ms).
abruzzi 12y ago
I think the biggest problem is that 10ms is more than enough to throw off a performer because it feels like a lag between when you hit the key and the note sounds. This makes it hard to play proficiently for anything with a fast tempo because it feels like the sound engine isn't keeping up with you. For an audience, 10ms is going to feel like sloppy timing. That will be more of an issue with tightly timed music (techno/EDM) and less with a slow jazz ballad.
- Synaesthesia 12y ago
  Really 10ms is ok, and you can cmompensate for it mentally. As they note, it's because Android music apps actually have a latency of 100ms or worse - that it's a problem.

nsajko 12y ago

What downsides would there be to just halving the period size?

CountSessine 12y ago
Given that each sample means a kernel mode-user mode trip and various context switches, I'm going to guess that halving the sample size globally would make your phone's audio handling consume more CPU.
wpietri 12y ago
My guess is that the large buffers are covering up for scheduling issues. On my devices (Moto X, Nexus 7) I can regularly get Android audio to stutter by connecting the power or switching apps. (And it'll happen on its own sometimes when it picks, as it so often does, a terrible time to update apps.) The easy fix to that is to make the buffers bigger. Doing that, though, increases latency. The hard fix is to do what Apple has done and build for audio from the ground up.
CamperBob2 12y ago
The heart of the problem is simply the limited timing and computational resources in a multitasking non-realtime OS. The smaller the audio buffers are, the more prone they'll be to starvation. Only if the application can be guaranteed to receive interrupt service and/or thread timeslices at a 100 Hz rate or better is it possible to achieve audio latency of 10 milliseconds. That's a difficult thing to guarantee in a modern consumer OS. It can be done, but it won't happen by accident, only by design. The penalty for failing to service your 10-ms buffer is a dropout that sounds much worse than slightly higher latency, so there's an incentive to use larger buffers than necessary at every link in the signal chain. From the point of view of the OS vendor, musicians might complain about latency, but everyone will complain about dropouts.
szantog 12y ago
Android's audio stack is "pushing" audio down to the audio driver, with the audio threads sleeping between every push. In other words, the audio driver's interrupts are not scheduling the actions. Having smaller period sizes increases the risk of not "guessing" the sleep scheduling right, resulting in many audio dropouts. Using a "pull" method is required for low latency audio, where the audio driver's interrupts are scheduling when audio is passed to and pulled from applications. Another downside of halving the period size is CPU load/battery drain. There are simply too many layers in Android's audio stack, and it has quite a few unoptimised code as well (such as converting between audio sample formats with plain C code).

deleted 12y ago

[deleted]

kchoudhu 12y ago

Any word on how Windows Phone does on the latency metric?

ygra 12y ago
There is this thread: https://social.msdn.microsoft.com/Forums/en-US/14f96b68-60e3-47a8-8d96-9cf6fe2c347a/lowlatency-audio-on-windows-phone-8-hint-forget-it-for-now?forum=wpdevelop https://social.msdn.microsoft.com/Forums/en-US/14f96b68-60e3... and it seems to be that anything below 140 ms latency seems impossible on Windows Phone.

troymc 12y ago

There is something I don't understand; maybe someone here can explain: Sound travels at about 340 m/s (in a typical room). That means it travels about 3.4 metres in 10 milliseconds. Therefore another way to get a 10 millisecond problem is to stand 3.4 metres from the orchestra. Most people sit farther than 3.4 metres from the orchestra, yet they don't complain about a lag between when the violin bow moves and the sound is heard. Why not? (The speed of light is so fast that we can assume it's effectively infinite for the purposes of this argument.)

s73v3r 12y ago
Because they're sitting back and listening to the music. They're not pressing buttons and expecting feedback within a specific timeframe.
dandelany 12y ago
Probably because they're not directly interacting with it, they're just watching it, which makes the delay a lot less noticeable. The 10ms delay is more of a problem for interactive apps where you eg. touch the screen and expect it to instantly make a sound. In this kind of feedback loop, even small delays are distracting and it becomes difficult to keep a beat because the lag is perceptible. If you tried to remotely play an instrument from 3.4 meters away, you would probably notice it too.
- bentcorner 12y ago
  Sort of like why audience clapping tends to get really messy until the performer does the "hands above the head" thing to get them back in sync.
  - falcor84 12y ago
    Clapping synchronisation is actually a well studied spontaneous occurrence, that just takes O(1) periods, see this article for some analysis: http://arxiv.org/pdf/cond-mat/0006423v1.pdf http://arxiv.org/pdf/cond-mat/0006423v1.pdf

kabdib 12y ago

I'd heard over the years that working with isochronous systems was difficult. I'd done a number of real-time systems before, and written OS schedulers and NTP-like systems and so forth. A little audio work should be a walk in the park, right? A little manly-man programming from the wrist and we move on to real problems. So I walked into an audio project thinking that "Oh, this latency and synchronization stuff, how bad could it really be?" Bad. Nailing down buffer-bloat and sources of latency and jitter in an isochronous system took several months, the last few weeks of which were 80-100 hour weeks just before ship. Several times we thought we'd fixed the issues, only to find that our tests had been inadequate, or that some new component broke what we had built. I remember nearly being in tears when I finally realized what the clock root of the audio system actually was, and that it wasn't what people had been using. From there everything fell into place. Don't pull the number of buffers you have out of thin air ("Oh, six is enough, maybe twelve, don't want to run out in my layer of the system, after all...") Don't assume your local underflow or overflow recovery strategy actually works in the whole system. Don't assume your underlying "real time" hypervisor won't screw you by halting the whole god damned system for 20ms while it mucks around with TLBs and physical pages and then resumes you saying, "Have fun putting all the pieces of your pipeline back together, toodle-oo!" Put debug taps and light-weight logging everywhere and attach them to tests. And know what your clock root is, or you will be sunk without a trace. Isoch is hard.

weaksauce 12y ago
A real postmortem of this would be a fascinating read.
- kabdib 12y ago
  Well, if something bad happens I might have to work there again, so . . . not for a few years, probably :-)
  - walterbell 12y ago
    >when I finally realized what the clock root of the audio system actually was How did you finally figure this out -- any recommended references?

vlaskovits 12y ago

If our site goes down (again), the webcache can be found here: http://webcache.googleusercontent.com/search?q=cache:zvFOaWHye-IJ:superpowered.com/androidaudiopathlatency/+&cd=1&hl=en&ct=clnk&gl=us&client=safari http://webcache.googleusercontent.com/search?q=cache:zvFOaWH...

mox1 12y ago

I spent a lot of hours fighting with the low level Android libraries while writing a Spotify App for Android back in late 2013. There is very little documentation on the web (especially for streaming Audio). Getting this working was a serious headache. If anyone is interested, I pulled the OpenSL ES parts out and posted them to github. https://gist.github.com/mox1/3516112bc52c54aabb98 https://gist.github.com/mox1/3516112bc52c54aabb98

Pxtl 12y ago

I think the article may be ascribing too much technical reasoning to why iOS has a better community of music apps - remember that Apple is also has Garage Band and dominates the online music sales business. It's fair to say that music is a core part of Apple's brand, and so the platform has a much greater draw for people who prioritize music software.

s73v3r 12y ago
It's possible for Apple to have stuff like Garage Band because they have the ability to do low latency audio.
- Pxtl 12y ago
  Garage Band (and Apple's position as the kings of music) predates iOS and Android. That's my point. Even if Android had good low-latency audio, I doubt that Android would still see the kind of audio-community we see with iOS because Apple is the brand for that sort of thing and has been that since the fruity-iMac-days.

code_duck 12y ago

Interesting to read of the details behind this issue. This has been a serious issue for me - it's actually why I own an iPhone. My first modern touch device was an iPod touch 4. I downloaded Garage Band and, as a long time milt instrumentalist and composer, loved it. I was amazed by how well the touch instruments worked and how easily I could record riffs and flesh out small snippets of songs. It ran almost flawlessly on the 4th gen Touch. Next, I decided to buy an Android phone - a Motorola Droid 2. I was surprised to find that despite the power advantage over the iPod, none of he music apps I tried were usable. The drum programs, for instance, we're so lacy and unpredictable as to be worse than useless. Hit a drum, and you may hear it seemingly instantly, maybe 1/8 a second later, maybe half a second, possibly never. Meanwhile the tiny iPod could play Garage Band instruments so well one could use it for live performance. I upgraded my phone twice, first to a Droid X, then a Galaxy S3... Each time was disappointed that the improved specs gave no improvement in the terrible audio performance. Currently I have an iPhone 6 so I can use Garage Band. Kudos to apple for doing this right - it's the best app I've ever used.

astral303 12y ago
Thanks for a musician's perspective!

kllrnohj 12y ago

Google has a whole section on this here (if you want more technicals and less sales-pitch): http://source.android.com/devices/audio/latency.html http://source.android.com/devices/audio/latency.html including a bunch of measurements of Nexus devices here (even going as far back as the Nexus One on Gingerbread): http://source.android.com/devices/audio/latency_measurements.html http://source.android.com/devices/audio/latency_measurements...

tim333 12y ago
And here are some measurements for iOS as well (Superpowered Mobile Audio Latency Test App for Android and iOS.) The iOS devices come out at 6-18ms, Androids at 17-860ms. The faster Androids have Samsung's Professional Audio SDK. http://superpowered.com/latency/ http://superpowered.com/latency/

acgourley 12y ago

While the specific problems differ, in general iOS is far easier to work with when it comes to video as well as audio. Android video is a never ending headache for us.

towelguy 12y ago

It's interesting that their business bases on Google not fixing this problem. What happens when they fix it?

macintux 12y ago
People have been assuming Apple's success depends on one or two minor features that someone else will copy and destroy their business for, well, ever.

guscost 12y ago

Great explanation, thanks for taking the time to write this up. A while back I figured that there had to be complicated structural reasons for the lack of progress on the notorious "issue 3434", and I decided to go with native iOS for mobile audio projects rather than wait. Seems it will be quite a difficult problem to solve (although perhaps a library that is not totally backwards-compatible would be easier to optimize). But kudos for taking a crack at it, and good luck. I'm interested to hear how it goes.

jongraehl 12y ago

Compare to a nice grand piano: 1 meter of air ~3ms. (hammer travel after you bottom out the key - another 10ms? A MIDI keyboard doesn't know the velocity of the note (or send the note at all) until key travel is mostly done too.) I have only 5 ms of ASIO buffering using my PC but I don't actually know how much actual key->sound latency; I do know that using headphones it's only slightly less immediate than a nice grand. I think the low MIDI bit rate (31kbaud) also adds a little latency on chords. It would be nice if keyboards in the future immediately sent a lower latency+precision keypress-initiated notice (before full key travel) so disk-based samplers can make ready for that note (and make any initial attack sound that's appropriate).

mrwilliamchang 12y ago

Audio lag is definitely not the worst of Android's lagginess problems.

lostgame 12y ago

The audio thing is the number one reason I will never use the Android platform. At this point, iOS devs have so many numbers of years ahead of Android devs in the music department, and the simplicity of porting existing Mac OS compatible audio stuff makes it incomparable. Metaphor: Two people are going for a race, over the same distance. One starts 5 years ahead of the other. At that point, is anyone even watching the race anymore? Sorry, despite it's absolutely God-awful flaws such as no user-accessible filesystem, no ability to do a basic task such as download an MP3, without GarageBand (and it's awesome ability to open files I sketch out on the go right in Logic on my Mac...), without iElectribe, and the list goes on, I absolutely just can't even use the platform, and I have no reason to go back 5 years technologically. Sorry, Android, you already lost, here is a user you can never have.

JoeAltmaier 12y ago
Not really good metaphors. Its not necessary to cover the intervening ground, to catch up technologically. Fix latency, app developers port and voila - Android looks pretty good again.
- lostgame 12y ago
  Did you not get the '5 year head start' bit? Doesn't matter how fast developers create applications that are amazing, iOS has already had all these years to create an already fantastic subset of these applications. Doesn't matter how 'good' it looks - for the use case of music, android has simply lost.
  - JoeAltmaier 12y ago
    That was actually my point. There's no need for Android development to cover that ground - it can just start where we are now, with cool apps that can just be ported over. Once the Android audio latency issue is addressed. App developers LOVE to port, its another sale with small effort.

Finster 12y ago

Does this cause audio de-sync issues on Android?

dottrap 12y ago

This is a great description of the lowest level problem. But for those wondering why iOS doesn't have the problem or why only 10ms is such a difference, the answer is that this is only the beginning of the problem of Android audio latency. There are still additional problems that add latency that span the entire Android stack from the actual hardware and drivers, to the kernel and scheduler, to the Android implementation of their audio APIs. Anybody who has tried to do any serious audio on Android knows the infamous Bug 3434: https://code.google.com/p/android/issues/detail?id=3434 https://code.google.com/p/android/issues/detail?id=3434 Google I/O 2013 did a pretty good talk on the problem and shows how there are problems across the entire stack. Glenn Kasten pretty much carries the brunt of all the audio problems with Android. I find it telling that he had to handcraft his own latency measurement device using an oscilloscope and LED because there were no actual tools built into the OS to help them analyze performance. https://www.youtube.com/watch?v=d3kfEeMZ65c https://www.youtube.com/watch?v=d3kfEeMZ65c Audio has been terrible since Android's inception. It has improved a little over time, but unfortunately, 7 years later is is still pretty much unacceptable for any serious work.

rasz_pl 12y ago
Whole Androids latency has been terrible since its inception, screen/input/audio. Android is famous for $600 2GHz phones that drop frames when doing simple animations :(.

cfstras 12y ago

Going completely off-topic: Why has everyone recently decided to use ultra-thin fonts everywhere? On my system (24" FullHD, Win8.1, Chrome 42 and Ubuntu 14.04, Chrome 43) the text is thinner than one pixel and thus unreadable below 130% zoom. Sure, I can open up DevTools and fix the font-weight, but seriously?

emn13 12y ago
My theory on this is that Windows 8 hyped this. It's particularly problematic when it's being copied in mac circles because OSX has a biased font-smoothing algorithm that adds font weight (it's a design flaw). In other words, to get something to look thin on a mac, especially a non-retina mac, it needs to be very thin, sometimes less than a pixel thin. How other systems display borderline visible strokes varies depending on the system and the details of the font.
- cfstras 12y ago
  Oh, so I can tell whether the designer uses OSX by the fact that I can't read the fonts. That's nice.
UberMouse 12y ago
The website font? I'm viewing it on a 27" 2560x1440 screen, Chrome 42 and OS X 10.10 and the fonts are normal sized for me. In fact larger than the HN font.

deleted 12y ago

[deleted]

Polyphonie 12y ago

One of the reason for the audio latency issue on Android is the complex power management circuitry (PMIC) implemented on most of not all Qualcomm SoCs. That's why to anyone with experience in regards to using Android device in audios production, most would agree that the older 2011 Galaxy Nexus is still the Android device to use. Unlike the the recent Snapdragon based Nexus device, the GNex is based on TI OMAP4460 (dual-core ARM Cortex A9). It might not be a coincidence the Apple kept the number of cores on its A series SoCs to a minimum (only moving to 3 with the A8X). http://createdigitalmusic.com/2013/05/why-mobile-low-latency-is-hard-explained-by-google-galaxy-nexus-still-android-of-choice/ http://createdigitalmusic.com/2013/05/why-mobile-low-latency...

yeureka 12y ago

I am writing a game that implements an audio sequencer and it runs great on iOS and OSX. But I had to write a karaoke app for Android recently and found an amazing 100ms latency there which makes me think I won't be able to port my game to Android... Wish Google fixed this.

faragon 12y ago

Android has many problems: - Poor event handling. - Braindead UI. - Braindead VM approach: WTF JIT on mobile devices??!!! Google: fuck fix the UI, fuck fix the VM bullshit, fuck fix bloating everywhere. NDK is not the solution: the problem is in the architecture.

DigitalSea 12y ago

This is the one downside with using Android I have experienced. I came from an iPhone 4 and as an amateur musician I used it to produce music, record ideas on the go and whatnot. When I moved to a Samsung Galaxy S4 (my first Android phone) I was mortified. I grown to love a couple of drumming applications on iOS, when I moved to Android there were a few sub-par applications and they all had ridiculous latency, to the point they were not usable and I had to bust out my keyboard at home to record drums (no drum ideas on the go anymore). You hit a snare in a drumming app and it felt like hundreds of milliseconds of lag, horrible. It was so bad to the point where I went out and purchased an iPad just so I could have that freedom of recording ideas when I am not at home. The iPad and iPhone offered what sounded like basically no latency at all, I never understood why Android devices struggled (but I speculated and assumed it was how the audio was being processed). I considered moving back to an iPhone, but I love the freedom that Android affords me and the competition, so I stuck it out and kept using the iPad. I recently purchased a Samsung Galaxy S6 Edge and while I still notice some slight latency, it is usable again. I can finally jot down ideas when I am away from home using my phone again. Research seems to yield some improvements that Samsung themselves have made to their hardware and software, not to mention the professional audio driver that allows the use of a third party audio interface (great feature by the way). Google needs to make this a priority, because believe it or not a lot of people use their tablets and phones to produce music. We need to fix this. Music production might not seem like a big deal to Google, but Apple definitely gets it and have from the beginning. The one aspect I miss about owning an Apple device, not enough to make me switch back but definitely a good feature of iOS devices.

elcct 12y ago

This is actually a known bug hanging there since 2009: https://code.google.com/p/android/issues/detail?id=3434 https://code.google.com/p/android/issues/detail?id=3434 And very good video from Google about it: http://www.youtube.com/watch?v=d3kfEeMZ65c http://www.youtube.com/watch?v=d3kfEeMZ65c

bobby5467 12y ago

titties

deleted 12y ago

[deleted]

MrBra 12y ago

Happy to see this side is getting attention!