7 ms·
Oboe: A C++ library for low latency audio in Android
- deleted 8y ago[deleted]
- gugagore 8y agoThe proliferation of apps for musicians on iOS (drums, guitar amps) is explained by the iOS's longstanding API for low-latency audio. I hope we can expect the same kinds of apps, with usable latency, for Android now too.
- veli_joza 8y agoI contacted one of Oboe authors and I reached conclusion that this doesn't achieve anything you couldn't already get with openal-soft lib. You can expect some 20ms of audio delay, plus 16-33ms of touchscreen latency . If you are curious about latency, check out my android app "Hexpress" (on play store) that gives you drums and other instruments to play with. I measured roughly 50 ms of tap-to-sound delay, not consistent across devices. That lands it within 'cute audio toy' territory, which is why I stopped investing time.
- f_m 8y agoThat's too bad: on my OnePlus 6, the latencies for your app are great. It's the first time I'm drumming in an Android app, and am not bothered by latency. Thank you for having the source on Github, I'll amuse myself this weekend checking the code.
- romwell 8y agoDoes this mean we will finally get decent musicmaking apps on Android? A decade after iOS, I guess it comes in from the "better late than never department", but I am happy. Apple seems to have lost its focus on the creative usage of their devices, and it has somewhat stagnated (compared to the initial explosion), plus iOS apps tend to be pricey. Apple's decision to not include the headphone jack effectively puts the kibosh on their realtime audio / musicmaking apps. The latency on AirPods is atrocious. You press a key on a virtual keyboard, and count to three before you hear a sound. Yes, this is probably solved by a dongle. Eh, I'd rather carry my Yamaha Reface DX around than think about that -- and the developers are going to have a FunTime(tm) explaining this to the users. (The sole reason musicians love iDevices so much is the JustWorks(tm) aspect - well, that just went out the window). This Android API just might start a new wave. The video appeals to the synth nerd in me. The gear, the T-Shirt, the code samples doing the right thing. I am looking forward to what this will bring us.
- telesilla 8y agoHave you tried Ableton's Link technology on Apple? I know a few DJs who find this solution works well as it will sync with their desktop DAW.
- romwell 8y agoI haven't - it seems to be an orthogonal technology to what is being discussed here.
- puzzle 8y agoRaph Levien worked for years on improving Android audio latency. I had a great chat with him a long ago about emulating the DX7, APIs and so on. I thought they already managed to get numbers to reasonable levels (after adding tighter requirements to the Android certification suite, among others) and even gave a talk at I/O. I'm surprised he's not involved in this, but then this is C++; he's been really into Rust for a while and perhaps back to working on typography alone — assuming he's still at Google, on the Android team.
- raphlinus 8y agoWrong on both counts. However, I'm very much back in the audio synthesis game, and will give a talk at next month's SF Rust Meetup on the topic.
- 90minuteAPI 8y agoLooking forward to it! It has always seemed to me like such a great application of the language, having done a fair amount of low-latency A/V programming in the C[++] realm.
- wwweston 8y agoWould love to hear you're planning to put up video and/or slides -- that way I don't have to drive up from Los Angeles. Or get around the waitlist of 30 people for a full event. :)
- adamnemecek 8y agoYou guys should also check out AudioKit https://github.com/AudioKit/AudioKit https://github.com/AudioKit/AudioKit
- alexforster 8y agoThe title should probably mention "for Android". It's misleading without that context.
- sctb 8y agoThanks, updated!
- telesilla 8y agoWonderful! Latency has been a massive blocker for audio on Android - this makes me want to break open the SDKs again and tinker.
- saagarjha 8y agoUnrelated, but there's a code example on that website: AudioStreamBuilder builder; AudioStream *stream = nullptr; Result result = builder.openStream(&stream); Why even bother initializing stream to nullptr if it's going to be overwritten anyways by AudioStreamBuilder::openStream?
- conradev 8y agoI think it's good practice to avoid having uninitialized memory. Just in case, for example, the initialization gets moved inside of an if statement and no one thinks about the else case.
- FigBug 8y agoYou'll probably get a warning about using an uninitialized variable if you don't. Compiler probably can't tell the variable will only be written and not read. It's also just good practice to initialize your variables.
- MauranKilom 8y agoThought the same recently and omitted the initialization. Then Lint yelled at me. I decided it's not a fight worth fighting since it basically costs nothing and is more robust in the face of change.
- mohitmun 8y agoWould this make it possible to build active noise cancelling using software?
- taneq 8y agoI wouldn't think so. If your cancelling signal is half a period out, you'll double the volume instead of cancelling it. That suggests the required latency is on the order of microseconds, not milliseconds.
- iandcoleman 8y agoAndroid already has an aec module that you can use. It's timestamp based using the timestamp calls to ALSA to perform the cancellation
- nshgraph 8y agoUnfortunately this won’t help with the terrible hardware level latency on many Android devices.
- iandcoleman 8y agoIt's not the hardware per see, ALSA could certainly be improved by vendors, it's the intermediate layers that cause the problem. Android needs to enforce low latency upon vendors, and not continue to give them the easy option of avoiding it.
- hyperpallium 8y ago> Oboe provides the lowest possible audio latency across the widest range of Android devices The "lowest possible" on android might not be low enough for musical instuments (<10-15ms). It's revealing that they don't reveal latency for any devices at all. (If I sound cynical, this isn't the first low-latency offering for android.) It's just "low" for android. funfact: consoles also have terrible audio latency. Rhythm games like rockband/guitar hero don't actually need this, they need only relate it to the audio playing (and also config compensation for input latency). The latency is shocking if you use the mode where you free-"play" the instrument, and is why actual musicians often a tough time with these games. The surprising thing is that delayed sound effects from games in general aren't so jarring. FWIW iphone audio latency also isn't good enough, despite being far better than android. That's how bad android is. EDIT 7ms claimed in screenshot at https://github.com/google/oboe/blob/master/samples/hello-oboe/README.md https://github.com/google/oboe/blob/master/samples/hello-obo... - doesn't include input latency: average latency of the audio stream between data entering the stream and it being presented to the audio device EDIT they know they need latency figures "Publish a list of Android device latency" https://github.com/google/oboe/issues/235 https://github.com/google/oboe/issues/235
- sirwitti 8y agoAudio latency might not event be the main issue here: Touchscreen latency is far worse which makes any input almost useless for someone trying to actually play notes. Note that changing parameters of filters, synths,... is fine because changing these does not require the same precision as playing. Btw if you're looking for a subjective number for tolerable latency I'd say ~5ms. This could be output latency if you're playing notes via midi or round-trip latency if you're processing incoming sound and output it again. In my experience you don't get that acceptable latency on notebooks with the onboard sound cards either :)
- veli_joza 8y ago5ms is bit on the low side. If you're playing from guitar amplifier that's 4 meters from you, you get 12 ms latency just from sound propagation (and most amps introduce additional latency). As extreme example of what the brain can compensate, church organ players have to deal with 100ms+ latency and they still manage to play Bach.
- fulafel 8y agoSo what are the latency numbers for some common devices? (apart from the 7 ms visible in the hello oboe screenshot)
- iandcoleman 8y ago'low_latency' used to be sub 20ms, however 'professional' was introduced a while ago to replace this, with 'low_latency' becoming 40ms
- iandcoleman 8y agoSorry, Actually, it's 45ms. From the docs: android.hardware.audio.low_latency indicates a continuous output latency of 45 ms or less. android.hardware.audio.pro indicates a continuous round-trip latency of 20 ms or less
- iandcoleman 8y agoUnfortunately, this is unlikely to improve latency on devices which already don't have 'good latency'. Reviewing the linked doc, this is a 'high level' API whos goal is possibly more about streamlining the API across multiple recent releases, while I'm sure, addressing latency as best it can. The problem is that this API still sits on top of other lower (and already existing api's) aaudio for current releases and opensles for older. A large part of Android audio latency is down to the HAL layer that each vendor provides. Andoid as a framework offers support for varying degrees of latency, but it's up to the vendor supplying the audio HAL to support this. This is exposed through the low_latency prop check (or 'professional' in recent releases). If the vendor HAL doesn't support 'low latency' through reduced buffer sizes and efficient (or no) locking etc, there's nothing a high level API like this can do, other than bypass the HAL and talk to ALSA directly, but this breaks the whole architecture.
- xtrapolate 8y agoAudio latency issues on PCs are largely alleviated thanks to better softwate (device drivers that shorten the path to/from the hardware - such as ASIO4ALL) combined with better hardware (soundcards interfacing over PCI-E/Firewire/Thunderbolt/USB) at affordable prices. Plenty of audio cards today will offload computation off the CPU via discrete DSPs. On Android devices (and smartphones in general) - hardware would be the limiting factor.
- AstralStorm 8y agoNot even close. You can have 1 millisecond latencies with a well configured kernel (PREEMPT), directly talking to ALSA. I'm talking both Qualcomm and Exynos CPUs. It just takes a SCHED_FIFO task with forced CPU affinity. Android does not make it easy to get one. There are some other hardware issues, like audio input and output using separate clocks on Qualcomms, but that's at most one extra buffer. Speaking of this, you can have sub-millisecond latencies on Beagleboard. These devices in phones are vastly more powerful.
- i88y6 8y agoWhat about all the delays from the use of java? Even if your audio app is in c++ for performance, it's still competing for resources with many other java apps easily using gigabytes of ram just idling. I don't see any way to mitigate that.
- pjmlp 8y agoNot a problem at all, that is just the typical Java FUD and bad coded apps. The real time audio APIs are in native code and they can request for priority use when on foreground. Samsung used to support real time audio on their S models since many years. https://developer.samsung.com/galaxy/professional-audio https://developer.samsung.com/galaxy/professional-audio Which they are now deprecating as they also contributed to the design of AAudio.
- i88y6 8y ago
- pjmlp 8y agoTypical Android NDK library from Google, instead of being added to the NDK, Android Studio and respective documentation, we get a github repository and "good luck" pat in the back. And maybe if they have enough buffer on their 20% we get some of the reported issues fixed. Like cdep, Play Services for C++, FPL libraries...
- resters 8y agoCan anyone recommend the best way to do low latency audio on windows and osx?
- adzm 8y agoFor Windows, ASIO! http://www.asio4all.org/ http://www.asio4all.org/
- raphlinus 8y agoI think wasapi has more or less superseded asio and has more support from Microsoft. If you're programming in Rust, you can use the cpal crate, which wraps all these. I'm not sure cpal has astonishingly good performance in all cases, but I plan to work with the authors. It doesn't have an Android back-end yet, but such a thing should be possible.
- resters 8y agoExcellent. Trying to do sine wave generation, on and off at low latency.
- FigBug 8y agomacOS has very low latency with the standard APIs. If you use JUCE (https://juce.com/ https://juce.com/) you'll get a single API for all the different audio APIs on macOS, Windows, iOS, Android, Linux, etc.
- Wehrdo 8y agoI used Oboe (and its predecessor, Howie) to make a self-balancing robot on Android. We were able to use the headphone jack (RIP) as a low-latency output to send serial data to an Arduino[0]. Android Studio has also finally improved NDK support enough that writing large chunks of your app in C++ isn't painful anymore. [0]: https://davidawehr.com/blog/audioserial/ https://davidawehr.com/blog/audioserial/
- evozer 8y agoI looked at the implementation of the openStream() function: AudioStream *AudioStreamBuilder::build() { AudioStream *stream = nullptr; if (mAudioApi == AudioApi::AAudio && isAAudioSupported()) { stream = new AudioStreamAAudio(*this); // If unspecified, only use AAudio if recommended. } else if (mAudioApi == AudioApi::Unspecified && isAAudioRecommended()) { stream = new AudioStreamAAudio(*this); } else { if (getDirection() == oboe::Direction::Output) { stream = new AudioOutputStreamOpenSLES(*this); } else if (getDirection() == oboe::Direction::Input) { stream = new AudioInputStreamOpenSLES(*this); } } return stream; } Result AudioStreamBuilder::openStream(AudioStream **streamPP) { if (streamPP == nullptr) { return Result::ErrorNull; } *streamPP = nullptr; AudioStream *streamP = build(); if (streamP == nullptr) { return Result::ErrorNull; } Result result = streamP->open(); // TODO review API if (result == Result::OK) { *streamPP = streamP; } return result; } Doesn't this leak memory if streamP->open() fails? I'm also surprised they are using new instead of unique_ptr, especially since one of their listed benefits is "Convenient C++ API (uses the C++11 standard) ".
- libso 8y agoOne the face of it, it does look like the memory pointed by streamP leaks. Unless build () kept another pointer to be used in the destructor.
- lossolo 8y agoYeah, seems like memory leak to me too. streamP is leaked if open fails. And this in modern C++ in year 2018: if (result != Result::OK) { goto error2; }
- The_rationalist 8y agoIt may be interesting to recall that collabora did port the linux professional JACK audio server to Android and it has far less latency than audioflinger (btw in french audioflinger phonetically mean to destroy the audio (quality), which is kinda appropriate). Sadly no OEM ever cared to my knowledge to integrate this work in their ROMs. I also openened an issue on the android bugtracker to feature request replacing audioflinger with alsa/pulseaudio. They never answered nor closed the issue but I haven't checked since 2 years. Edit: here is the android feature request https://issuetracker.google.com/issues/37073168 https://issuetracker.google.com/issues/37073168 Maybe if you comment/upvote it it may gain traction. Digression: If you are interested in audio latency, I recall that Firefox has better webaudio performance than Chrome.