6 ms·

Anyone with any insight about how Apple does this?

by afandian 12y ago

pja 12y ago
Same way they do with anything else they care about: by controlling everything in the stack from the hardware up.
- raverbashing 12y ago
  I don't see how this has to do with hardware rather than architectural decisions AudioFlinger + Alsa take a lot of time, as seen per the graph But Android seems to take the option of least effort and "works most of the time" (which they have their reasons to)
  - epochwolf 12y ago
    By controlling the hardware, they need fewer drivers and they can skip one of the abstraction layers.
    - madez 12y ago
      > they need fewer drivers This suggests that you can work focused on one driver and therefore save developer-time. However, the drivers on Android are made by a bigger workforce, which must be taken into account. > they can skip one of the abstraction layers The post explicitly states that the HAL ought to add no latency at all.
      - vidarh 12y ago
        Besides the point. When you know the hardware you can shave away the upper layers. Consider e.g. AmigaOS. AmigaOS let you obtain a pointer directly to the screen bitmap to update your window contents with no buffering or clipping. It could do that because originally all the hardware was the same, or close enough. Then graphics cards came along, and you didn't necessarily have a way of writing directly to the bitmap. Suddenly you had to use WritePixel() and ReadPixel() and similar, which would obtain the screen pointer for the window, and obtain the display the screen is on, and find the driver corresponding to the screen, and call the appropriate driver function via a jump table. Similarly, the AmigaOS had functions to e.g. install copper lists (the copper was a very primitive co-processor that could be used to do things like change the palette at specific scan lines), which also wouldn't work at all on graphics cards. This is why knowing the hardware is part of a limited set matters: You can define your API to match the hardware very precisely, or even expose hardware features directly.
    - __david__ 12y ago
      That whole theory is blown by the fact that CoreAudio supports 3rd party audio hardware. There is a thriving market of soundcards and audio peripherals for the professional Mac OS X market. Those all run CoreAudio, which is exactly what iOS runs.
  - mhax 12y ago
    The hardware you intend run on will impact your architectural decisions. Given the wide scope of hardware targeted by android, it's not that surprising that it performs less well than a system targeting a very limited set of devices. Having said that, it performs 'well enough' for the vast majority of use cases.
tim333 12y ago
Some answers on quora. TLDR Apple have been working on audio a long time. http://www.quora.com/Why-do-iPhones-have-better-professional-audio-apps-than-Android-phones http://www.quora.com/Why-do-iPhones-have-better-professional...
- creshal 12y ago
  Or, rather, actually care about audio. Google (/Motorola Mobility, when it was owned by it) doesn't even care if a phone ships with a broken headphone port and refuses to fix it.
  - TillE 12y ago
    Seriously, the absolute garbage headphone jacks on the two flagship Android phones I owned (Motorola Milestone and Galaxy Nexus) are the reason I now own an iPhone 6. I still mostly prefer Android as an OS, but at least Apple is willing to spend the extra 50 cents on a headphone jack that lasts for more than a couple weeks.
easytiger 12y ago
By having their devices all roughly the same, it is easier to mitigate in app development than the variance in latency on android devices
miahi 12y ago
Probably by custom and known hardware. Let's say it takes 1000 working hours to produce a low latency driver/firmware; Apple only needs to do that for a few devices, but every Android manufacturer has different audio chips and most of them only care that audio works, not that it works well - so they do not invest that time for each device type. Having known hardware also works for the application developers, because they can optimize their code for the specific device (knowing that you always have a 40ms latency is way better than knowing that the latency is between 35 and 300ms). Also, Apple is much more motivated to get the audio path right, having started with the iPods and selling a lot of music.
- madez 12y ago
  The latency problems arise with ALSA (mostly hardware-independent) and AudioFlinger (totally hardware-independent).
  - szantog 12y ago
    ALSA is super. The HAL connecting ALSA to AudioFlinger is the major problem, plus the audio stack's "push" philosophy.
  - kllrnohj 12y ago
    That's technically correct but an extremely inaccurate summary. The reason ALSA and AudioFlinger add latency is to hide hardware-dependent differences as well as kernel-caused scheduling issues & policy decisions. To achieve low-latency you need real-time scheduling, something Linux has with SCHED_FIFO but it's a bit kludgy, and getting the policy right on that is tricky (obviously you don't want a random app to be able to set a thread to SCHED_FIFO and preempt the entire system). So you have to restrict the CPU budge of a SCHED_FIFO thread, and you have to only allow apps to have a single SCHED_FIFO thread. But how much CPU time you give it needs to depend on the CPU's performance in combination with the audio buffer size that the underlying audio chip needs (and those chips also have different sample rates, is it 44.1khz or 48khz or etc...). tl;dr: this is insanely hardware-dependent.
po 12y ago
Look at the architecture of the Core Audio API and compare it to the parent article: https://developer.apple.com/library/mac/documentation/MusicAudio/Conceptual/CoreAudioOverview/CoreAudioEssentials/CoreAudioEssentials.html https://developer.apple.com/library/mac/documentation/MusicA... Besides the fact that you are writing in C/Objective C, most heavy lifting is being provided by Audio Units which are specifically designed with common datatypes to be chained, composed and executed in low-latency situations. Furthermore, most common things you would need in your app like mixing, conversion, timing, etc are provided as highly optimized services by the system.
- JoeAltmaier 12y ago
  Also CoreAudio drivers (HAL) calls back every 11ms to empty/fill a buffer. So it starts with lower latency?
coreyoconnor 12y ago
One aspect of CoreAudio that, IMO, contributes to low latency: CoreAudio provides the buffers to process to the AudioUnit. Not the other way around. This permits CoreAudio to chain AudioUnits while minimizing buffer use. In the ideal case where all AudioUnits support the same audio stream format they are all provided the same buffers to process. Which can get close to zero-copy, zero-allocation audio rendering. I don't know enough about android to say that this is the key aspect tho. Perhaps the low level API of android is the same? I didn't find an easy reference in a quick search.

Anyone with any insight about how Apple does this?

pja 12y ago

Same way they do with anything else they care about: by controlling everything in the stack from the hardware up.

raverbashing 12y ago
I don't see how this has to do with hardware rather than architectural decisions AudioFlinger + Alsa take a lot of time, as seen per the graph But Android seems to take the option of least effort and "works most of the time" (which they have their reasons to)
- epochwolf 12y ago
  By controlling the hardware, they need fewer drivers and they can skip one of the abstraction layers.
  - madez 12y ago
    > they need fewer drivers This suggests that you can work focused on one driver and therefore save developer-time. However, the drivers on Android are made by a bigger workforce, which must be taken into account. > they can skip one of the abstraction layers The post explicitly states that the HAL ought to add no latency at all.
    - vidarh 12y ago
      Besides the point. When you know the hardware you can shave away the upper layers. Consider e.g. AmigaOS. AmigaOS let you obtain a pointer directly to the screen bitmap to update your window contents with no buffering or clipping. It could do that because originally all the hardware was the same, or close enough. Then graphics cards came along, and you didn't necessarily have a way of writing directly to the bitmap. Suddenly you had to use WritePixel() and ReadPixel() and similar, which would obtain the screen pointer for the window, and obtain the display the screen is on, and find the driver corresponding to the screen, and call the appropriate driver function via a jump table. Similarly, the AmigaOS had functions to e.g. install copper lists (the copper was a very primitive co-processor that could be used to do things like change the palette at specific scan lines), which also wouldn't work at all on graphics cards. This is why knowing the hardware is part of a limited set matters: You can define your API to match the hardware very precisely, or even expose hardware features directly.
  - __david__ 12y ago
    That whole theory is blown by the fact that CoreAudio supports 3rd party audio hardware. There is a thriving market of soundcards and audio peripherals for the professional Mac OS X market. Those all run CoreAudio, which is exactly what iOS runs.
- mhax 12y ago
  The hardware you intend run on will impact your architectural decisions. Given the wide scope of hardware targeted by android, it's not that surprising that it performs less well than a system targeting a very limited set of devices. Having said that, it performs 'well enough' for the vast majority of use cases.

tim333 12y ago

Some answers on quora. TLDR Apple have been working on audio a long time. http://www.quora.com/Why-do-iPhones-have-better-professional-audio-apps-than-Android-phones http://www.quora.com/Why-do-iPhones-have-better-professional...

creshal 12y ago
Or, rather, actually care about audio. Google (/Motorola Mobility, when it was owned by it) doesn't even care if a phone ships with a broken headphone port and refuses to fix it.
- TillE 12y ago
  Seriously, the absolute garbage headphone jacks on the two flagship Android phones I owned (Motorola Milestone and Galaxy Nexus) are the reason I now own an iPhone 6. I still mostly prefer Android as an OS, but at least Apple is willing to spend the extra 50 cents on a headphone jack that lasts for more than a couple weeks.

easytiger 12y ago

By having their devices all roughly the same, it is easier to mitigate in app development than the variance in latency on android devices

miahi 12y ago

Probably by custom and known hardware. Let's say it takes 1000 working hours to produce a low latency driver/firmware; Apple only needs to do that for a few devices, but every Android manufacturer has different audio chips and most of them only care that audio works, not that it works well - so they do not invest that time for each device type. Having known hardware also works for the application developers, because they can optimize their code for the specific device (knowing that you always have a 40ms latency is way better than knowing that the latency is between 35 and 300ms). Also, Apple is much more motivated to get the audio path right, having started with the iPods and selling a lot of music.

madez 12y ago
The latency problems arise with ALSA (mostly hardware-independent) and AudioFlinger (totally hardware-independent).
- szantog 12y ago
  ALSA is super. The HAL connecting ALSA to AudioFlinger is the major problem, plus the audio stack's "push" philosophy.
- kllrnohj 12y ago
  That's technically correct but an extremely inaccurate summary. The reason ALSA and AudioFlinger add latency is to hide hardware-dependent differences as well as kernel-caused scheduling issues & policy decisions. To achieve low-latency you need real-time scheduling, something Linux has with SCHED_FIFO but it's a bit kludgy, and getting the policy right on that is tricky (obviously you don't want a random app to be able to set a thread to SCHED_FIFO and preempt the entire system). So you have to restrict the CPU budge of a SCHED_FIFO thread, and you have to only allow apps to have a single SCHED_FIFO thread. But how much CPU time you give it needs to depend on the CPU's performance in combination with the audio buffer size that the underlying audio chip needs (and those chips also have different sample rates, is it 44.1khz or 48khz or etc...). tl;dr: this is insanely hardware-dependent.

po 12y ago

Look at the architecture of the Core Audio API and compare it to the parent article: https://developer.apple.com/library/mac/documentation/MusicAudio/Conceptual/CoreAudioOverview/CoreAudioEssentials/CoreAudioEssentials.html https://developer.apple.com/library/mac/documentation/MusicA... Besides the fact that you are writing in C/Objective C, most heavy lifting is being provided by Audio Units which are specifically designed with common datatypes to be chained, composed and executed in low-latency situations. Furthermore, most common things you would need in your app like mixing, conversion, timing, etc are provided as highly optimized services by the system.

JoeAltmaier 12y ago
Also CoreAudio drivers (HAL) calls back every 11ms to empty/fill a buffer. So it starts with lower latency?

coreyoconnor 12y ago

One aspect of CoreAudio that, IMO, contributes to low latency: CoreAudio provides the buffers to process to the AudioUnit. Not the other way around. This permits CoreAudio to chain AudioUnits while minimizing buffer use. In the ideal case where all AudioUnits support the same audio stream format they are all provided the same buffers to process. Which can get close to zero-copy, zero-allocation audio rendering. I don't know enough about android to say that this is the key aspect tho. Perhaps the low level API of android is the same? I didn't find an easy reference in a quick search.