7 ms·
Using AI to compress audio files for quick and easy sharing
- rmnclmnt 4y agoPied Piper’s middle-out!
- telotortium 4y agoImpressive results - on CD-quality audio too, not just speech as with many recent codecs. One thing I was not able to find, either in the blog post or the paper (https://arxiv.org/pdf/2210.13438.pdf https://arxiv.org/pdf/2210.13438.pdf), was the size of the model. EDIT: I downloaded the model from the code (https://github.com/facebookresearch/encodec/blob/3837f0db2a3b0fa00084cdd145cc6e8553048431/encodec/model.py#L259 https://github.com/facebookresearch/encodec/blob/3837f0db2a3...). It looks to have 15M parameters in total (this is a CPU-only model, by the way). For inference, I assume we can get away with 16-bit floats, so this clocks in at about 30 MiB. Not particularly favorable compared with LAME on my system (164 KiB), but not at all unreasonable to use - these days apps take 30 MiB dependencies all the time.
- augustuspolius 4y agoThe phrasing they use is a bit confusing. "Speech" is not a clear quality description, so it's odd to compare speech to a CD quality recording (presumably 44.1 kHz). Speech can also be sampled at CD quality. If they meant to emphasize the difference in frequency range or dynamic range - there were, perhaps, better ways to do that (e.g. "telephone conversation" vs "symphonic orchestra recording").
- deleted 4y ago[deleted]
- bscphil 4y agoMy best guess is that all they meant by "CD quality" was that it was targeting full band stereo replication, rather than narrow band as is typically used for speech. Not that it could achieve transparent compression at any particular bitrate. The tiny music clip in the sample, encoded at 6 kbps, is obviously not any kind of evidence for "CD quality" one way or the other. (The clip itself, if you download it from the page, is re-encoded with 64 kbps AAC.) No way to know how it would stack up against 96 kbps Opus on a stack of CDs with blind testing, I don't think.
- password4321 4y agoYeah, I didn't quite get the "easy sharing" part... need the model for playback!
- guggle 4y ago> we are the first to make it work for 48 kHz sampled stereo audio (i.e., CD quality) "CD quality" is 44.1khz over 16bit. > We achieve an approximate 10x compression rate compared with MP3 at 64 kbps, without a loss of quality. but then later: > The key to lossy compression is to identify changes that will not be perceivable by humans
- augustuspolius 4y agoIt's quite amazing how they can build something so ingenious and yet make such a glaring mistake in the announcement post (48 kHz CD quality). On your second point: they probably meant that there is no loss in perceived quality, of course it's a lossy algorithm. 6kbps sound demo is really impressive! Initially I was turned off by aliasing artifacts (hiss) but to be fair 6kbps is a really really low bitrate.
- guggle 4y agoIt is impressive, but it's also not something I would want to use for music listening. I guess the "10x" goal makes for an impressive chart, but I'd like to know what are the more realistic goals and gains.
- augustuspolius 4y agoThe interesting part to me was that they used specifically a 64kbps example, not a higher bitrate that would be more appropriate for music listening. Just speculating, but if they managed to get 10x higher compression rate compared to 64kbps MP3, could they achieve an even higher compression rate when compared to 320kbps MP3? If the algorithm is so good that it can compress audio down to 6kbps with just a few artifacts, would it sound almost good at 12kbps? 24kbps? Between this effort and the recently announced Google-led multi-channel/immersive audio codec initiative, I am pretty excited about the future of audio streaming and distribution.
- brokenmachine 4y ago>recently announced Google-led multi-channel/immersive audio codec initiative Link?
- vaidhy 4y agoWill the next step be piracy detection? If two audio files compress into the same output, can they be considered the same? Similar to soundex or double metaphone?
- notacanofsoda 4y agoAudio hashing (acoustic fingerprinting) already exist for detection of copyrighted music and are more efficient. See Shazam
- augustuspolius 4y agoShazam produces an embarrassing number of false matches, which I assume are hash collisions or something similar in the world of their fingerprinting system?
- bongobingo1 4y ago> False match? Sorry! Please lodge a complaint for manual review. > Nb: complaints require an account in good standing at as this is your third strike your account is now suspended. Near enough is good enough for most people paying-for/selling the tech.
- IndySun 4y agoShazams correct hit rate has certainly lowered in the many years I've been using it. Presumably because the pool of music it now is being asked to decipher is becoming unwieldy. I am permanently pleased to learn one size does not fit all even in gargantuan saas companies.
- nsgi 4y agoIs there a need for that? Content id seems to work well enough
- tgv 4y agoI doubt it will be that simple. Audio fingerprinting a la Shazam works by identifying key elements in some transformation of the sound, and setting some threshold for the number of matches. I suppose the output of this compression method offers a good way for identifying a song by a bit of sound, but the identification algorithm will be much more complex. I'd like to see what happens when you modify the stream. If the representation is really that compact, making small changes should change the output considerably. Treat it like a synthesizer, really. That should tell us something about its usefulness in identifying audio.
- blaaaaa99a 4y agoAI seems to have incredible potential in compression of audio, images and video. None of which really seems to make any difference in a world of near-limitless bandwith. It just isn't the constraint it once was.
- orlp 4y agoOur world is near-limitless in bandwidth, but highly restrained in cache size and latency. 10x compression means you can keep 10x more stuff in cache. And it doesn't matter what level you operate on, cache and latency is always relevant. Whether it's registers, L1, L2, L3, same-core NUMA RAM, cross-core, SSD, disk controller cache, disk, same-location distribution server, cross-location distribution server, tape archive backup, etc, going up a level of cache is always a lot slower regardless of bandwidth if you're doing a small read.
- sdenton4 4y agoKeep in mind that you also need to put the model weights somewhere...
- zimpenfish 4y ago> None of which really seems to make any difference in a world of near-limitless bandwith. Does to me - I have more than a terabyte of field-recorded FLACs that I'd like to put up on the net for people to listen to or download. Compressing them to q1 OGG (and I'm not sure that's good enough quality for some of them) only gets that down to about 20-25% of the size (eg 200408_0403.flac at 540M goes down to 128M) - even if I host them on S3 or B2, it's still going to be costing me a not inconsiderable sum if people actively listen or download. If this can get them down to 5-10% with usable quality, that makes life a lot easier (but obviously would depend on browser support, etc.)
- deleted 4y ago[deleted]
- Gigachad 4y agoWould be easier to host them as a torrent.
- manholio 4y agoTypically, applications where bandwidth is sufficiently limited to warrant AI heavy lossy compression are also application with tight latency and computational constraints. Not to disparage these results, but the area of very low bitrate codecs is more or less over, voice and audio streaming is a solved problem. In terms of relative bitrate you can do much better with AI, but it will have diminishing returns in the real world. On the other hand, I think AI-informed video compression will be absolutely transformational; it could enable practical transmission and reproduction of high definition holography (reproduction of an entire wavefront, allowing real 3D scenes viewable from multiple angles) that currently requires a small data-center to process.
- sgt101 4y agoThe other thing is price - running this on lots of files could get expensive...
- langsoul-com 4y agoI wonder how useful it is compared to H. 265, AVI? Also what would be the comparable hardware? A laptop can easily compress h. 265 or avi format without requiring intensive hard wire like AI.
- pjc50 4y agoThose are video codec/container standards - the audio in such video files is usually MP3 or AAC.
- kernelsanderz 4y agoI'm excited to see this get implementations for mobile platforms. a 30MiB model sounds like a good tradeoff to stream audio over low bandwidth, or even less reliable mobile connections.
- fsiefken 4y agoO nice, I wonder how 3 kbps speech compares to Google's Lyra. https://github.com/google/lyra/releases https://github.com/google/lyra/releases 1.5 kbps speech is impressive, I wonder how it compares with Codec2 https://github.com/drowe67/codec2 https://github.com/drowe67/codec2 or how all three compare to each other, useful for speech recordings, the only problem is that there is no widespread support yet.
- _zfxr 4y agoThe Audacity.
- mckirk 4y agoI'm curious what the 'artefacts' are going to sound like with schemes like this. Using a generative-adversarial network as a compressor (with a 'perceptual' discriminator, how ever that works) probably means that all introduced artefacts will sound entirely plausible. I.e., it's possible that you didn't mishear the word, but that it was mis-compressed to a perceptually very similar word.
- avian 4y agoUsing compression like this for storage is kind of going back to ancient times when songs and stories were passed down through people listening, learning and re-telling them. Each reproduction was a bit different because the person telling it sometimes couldn't remember the details and filled in the blanks with plausible replacements they came up with on the spot.
- pessimizer 4y agohttps://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... Xerox copier flaw changes numbers in scanned docs [2013]
- genewitch 4y agoMy only experience with Facebook AI has been "meh". I'm using FB research transformers and it subtly changes what is supposed to be deterministic output. I have more fun trying to break stable diffusion than getting interesting pictures anyhow.
- devcat 4y agoMy favorite song for testing these audio compression algorithms is Adele’s hello because any changes to her voice instantly pop out. It did a great job of reducing FLAC size from 100mb to less than 1mb using stereo 24kbps preset but audio quality suffers a lot in some places. Maybe training it with 48kbps or 64kbps would make it a feasible alternative for storing music without much quality loss. In comparison, lame insane preset (320kbps) produces 12mb mp3 with almost indistinguishable quality from flac. For those who want to listen to sample: https://a.pomf.cat/pcjynr.wav https://a.pomf.cat/pcjynr.wav (first flac then encodec)
- MontyCarloHall 4y agoI wonder how well this would work for the basis of a FLAC-like lossless encoder. FLAC works by approximating the audio stream with a lossy linear predictive code, and then storing the LPC encoding and its residuals (i.e. the delta between the original signal and its lossy approximation). It turns out that LPC+residuals are a lot more amenable to lossless compression (via Huffman coding) than the raw audio signal itself. If the LPC were replaced with this neural network based encoding, would the resulting encoding+residuals also be amenable to lossless compression?
- sdenton4 4y agoI think the main difficulty is that a neural decoder is allowed to make up lots of plausible phase information, which likely leads to pretty large L2 errors while retaining perceptual quality. So then you'll end up with large residuals even though you might only barely discern the difference perceptually.
- smoldesu 4y agoReally interesting results, thanks for sharing! The sound of the AI encoder is distinct and probably not suited for music at that bitrate, but would probably serve fine for Facebook videos and podcasts. I'd be really interested in seeing a model optimized to compress human speech...
- kkielhofner 4y agoInteresting that “Hello” could be this generations Tom’s Diner: https://www.mentalfloss.com/article/19727/how-toms-diner-tuned-mp3 https://www.mentalfloss.com/article/19727/how-toms-diner-tun...
- xnx 4y agoSurprised there was no mention of prior art like Lyra or Soundstream https://ai.googleblog.com/2021/08/soundstream-end-to-end-neural-audio.html https://ai.googleblog.com/2021/08/soundstream-end-to-end-neu...
- sdenton4 4y agoReading the paper, it's a reimplementation of Soundstream with a tweaked loss and layer norm added, and adds a 2-layer lstm to the encoder. These changes give a real quality boost over the quality of lyra v2 (which is also Soundstream, but lighter weight). Note that encodec is also using much more compute compared to Lyra v2. Additionally, LayerNorm is adding a lot of latency, which is why they advertise it as file compression instead of as a streaming codec. The entropy coding seems like a lightweight version of AudioLM.
- xnx 4y agoGreat summary. Sounds like nothing revolutionary, but rather a different configuration for a different use.
- Dwedit 4y agoNobody has mentioned the experimental LPCNet codec yet, so here's a link to this: https://jmvalin.ca/demo/lpcnet_codec/ https://jmvalin.ca/demo/lpcnet_codec/ 1.6kbit/sec for speech.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- qikInNdOutReply 4y agoCould such a AI be reversed, as in basically become a stable diffusion generation of music, with just keywords and a noise generator?
- rzzzt 4y agoJukebox is using a similar scheme: https://openai.com/blog/jukebox/#approach https://openai.com/blog/jukebox/#approach
- qikInNdOutReply 4y agoThanks.
- aljungberg 4y agoYes, I believe it had been done with Google’s Wavenet for example where normally it’s trained to generates speech conditioned on textual input. For fun they trained it on piano music and gave it no conditioning and it improvised piano pieces. Also see OpenAI’s jukebox.
- songeater 4y agoSo Jukebox can actually be broken down into 2 components / 3 steps: A) compression (take 44khz audio and compress into tokens using 3 VQVAE layers) B) generation (take the compressed tokens and use that on a transformer to generate sequences of those tokens) C) de-compresss (reverse the VQVAE) -> this is BY FAR the longest/most computationally expensive step Diffusion currently wont really help with (B) or even (A)... but there is a lot of new experiments going on with (3)... where the decompression using stable-diffusion is providing very interesting results using far less compute... check out JuicyJukebox notebooks (link to come... cant access github colab from my work computer).
- web007 4y agoI'll have to go and get the source and compare it to AAC. I'm really curious why that was left out of the comparison since it's the modern equivalent of MP3 (aka MP4 audio). It feels disingenuous.
- ofou 4y agoIt's a shame that most authors are French and didn't mention even a single time the work previously done by Fabrice Bellard. Plus, no lossless implementation? Maybe it's a good follow up. https://bellard.org/nncp/ https://bellard.org/nncp/
- ElijahLynn 4y agoPIED PIPER IS REAL!
- antisthenes 4y agoImpressive, but for all intents and purposes, Opus is just good enough for 99%+ of things related to audio. Hell, even the slightly outdated combo of LAME for personal audio library and AAC for multi-channel audio in movies is good enough.