8 ms·

I've just taken a minute to confirm what my ears told me in Audacity. Please have a look at this screenshot: https://cloudflare-ipfs.com/ipfs/Qma41RMzieQ6ZGdGem

I've just taken a minute to confirm what my ears told me in Audacity. Please have a look at this screenshot: https://cloudflare-ipfs.com/ipfs/Qma41RMzieQ6ZGdGem9rLxnxEL15oEs9tmiF288KmazfTs/ref_lyra_opus.png https://cloudflare-ipfs.com/ipfs/Qma41RMzieQ6ZGdGem9rLxnxEL1... The Lyra version is clearly much louder. This is a serious problem and it borders on being reasonable to call it "cheating". It's well known in the audio biz that if you ask people to compare two experiences, and one of them is a bit louder than the other, people will say that the louder one was better, or came through more clearly, or whatever it is you're trying to market for. For the purpose of comparing artifacts in two samples, it's absolutely crucial that they be the same volume. You might as well compare two image compression codecs where one of them "enhances" the colors of the original image. Note: I took the clips for this comparison from the "clean speech" examples at the original source on Googleblog, not the blogspam.

dkjaudyeqooe 6y ago
Isn't that due to audio (frequency) compression coming out of the generative model? I guess that can be tweaked either way but they're going to tend towards that exactly because it sounds louder and thus clearer.
- bscphil 6y ago
  There are a couple of effects here: 1. Lossy codecs will use a low-pass filter to get rid of hard to compress higher frequencies. This is often inaudible, but even when it is, it should lower the volume, unless you're applying some kind of compensation for it. 2. It's true that lossy codecs compress different frequencies differently, but that's not usually done in such a way that amounts to applying EQ to the frequencies. 3. Even if the relative balance of frequencies did shift as a result of applying lossy compression, this is still done in a way that the overall loudness of the audio does not change. In this case the Lyra output has changed significantly and in an easily audible way (about +6 dB). You could easily get the same effect in Opus just by amplifying (or applying compression to) the result, but Opus is doing things correctly.
rectang 6y ago
Excellent catch. To be precise you need a measure of perceptual loudness rather than raw waveform excursion, but I would expect the results to be in line with what you've found. > It's well known in the audio biz that if you ask people to compare two experiences, and one of them is a bit louder than the other, people will say that the louder one was better, or came through more clearly, or whatever it is you're trying to market for. As a former mastering engineer, you're absolutely right that this is well understood in the audio industry. I used to present my clients with level-matched comparisons of source audio vs. processed so they would understand exactly what was being done, aesthetically.
- bscphil 6y ago
  Here's an EBU R 128 measure using r128gain: File 'reference.flac': loudness = -25.7 LUFS, sample peak = -9.2 dBFS File 'lyra.flac': loudness = -19.9 LUFS, sample peak = -4.1 dBFS File 'opus.flac': loudness = -25.9 LUFS, sample peak = -9.7 dBFS So that also matches pretty closely what my ears heard.
  - perfmode 6y ago
    how do the samples compare when loudness is made constant/normalized?
    - 0-_-0 6y ago
      Just reduce the volume of the lyra one by hand. Doesn't change the fact that it sounds leagues above the others.
    - bscphil 6y ago
      I do still prefer Lyra overall, though not as much as some others (see sibling comment). To me, Lyra is cleaner and easier to understand, but the artifacts it introduces are more annoying and fatiguing than those introduced by Opus. Some people in this thread have reported trouble understanding Lyra, which I attribute to the strange artifacts it introduces.
  - 6y ago
fireattack 6y ago
Just as a reference Title RMS Peak Diff clean_p257_011_lyra -20.07 -1.13 18.93 clean_p257_011_opus -26.07 -6.65 19.41 clean_p257_011_refer -25.77 -6.15 19.63 PSD (Welch's method, window=213) https://i.imgur.com/Y8A4kkx.png https://i.imgur.com/Y8A4kkx.png
domoritz 6y ago
Off topic but how do you put images on ipfs and what’s the advantage over e.g. Imgur?
- bscphil 6y ago
  Cloudflare are kindly hosting [1] a free HTTP gateway for the IPFS [2] network. So I can host an image myself on a server with IPFS, and Cloudflare will cache it for me. This is better than Imgur because the latter has been redirecting users to annoying "social" pages with ads instead of showing them the actual image, at least in some cases. I also can't be sure whether Imgur recompresses your uploads or not - I assume they usually do. It's also more generally useful because I can host other files too, not just images. [1] https://www.cloudflare.com/distributed-web-gateway/ https://www.cloudflare.com/distributed-web-gateway/ [2] https://ipfs.io/ https://ipfs.io/
  - javajosh 6y ago
    Is hosting the image yourself, on like a $5 Digital Ocean Droplet and a $10 personal domain, out of the question? This would seem to be the ideal situation in terms of simple, decentralized file hosting solution. What are the downsides of this approach? (I can imagine a server package that can modify index.html sub-resource URLs depending on current server load, preferring private, locally hosted sub-resources but willing to use 3rd party solutions like Cloudflare, too, if required by a black swan event.)
    - bscphil 6y ago
      Out of the question? No. As convenient as running one command on a desktop computer? Also no. > the ideal situation in terms of simple, decentralized file hosting solution Not sure what you mean by "decentralized" if you are in fact hosting it yourself. > What are the downsides of this approach? Well, for the casual person it has the obvious downside that you have to have your own VPS. Most people don't have those. Even if you do, IPFS has a couple of advantages: you can host images anonymously, and anyone anywhere in the world can "pin" the image to make sure it stays live. If you're using a server and you forget to pay DO your $5 one month, all your images go poof into the ether.
adrianmonk 6y ago
Definitely a real effect, but it seems like Google accounted for that in their listening tests. The Google blog post links to the Lyra paper[1], and Section 5.2 of the paper says: > To evaluate the absolute quality of the different systems on different SNRs a Mean Opinion Score (MOS) listening test was performed. Except for data collection, we followed the ITU-T P.800 (ACR) recommendation. You can download those ITU test procedures[2], and skimming through that, it does mention making "the necessary gain adjustments, so as to bring each group of sentences to the standardized active speech level" and a 1000 Hz calibration test tone related to that. (See sections B.1.7 and B.1.8.) So, if I skimmed correctly, and if the ITU's method of distilling speech loudness into a single number is an effective way to match the volume levels[3], then it seems like they did what they could to avoid cheating at the listening tests. It is still interesting that Lyra makes things louder, though. --- [1] https://arxiv.org/pdf/2102.09660.pdf https://arxiv.org/pdf/2102.09660.pdf [2] https://www.itu.int/rec/T-REC-P.800-199608-I https://www.itu.int/rec/T-REC-P.800-199608-I [3] and even for speech that passes through different codecs before its loudness is determined
- bscphil 6y ago
  That's good information, thanks. My comment is mostly directed at the misleading blog post. I have no direct reason to believe that the study itself was compromised, though it would be great to have confirmation from the authors that it was not. The part about matching volume levels in the ITU recommendation seems to be talking about making sure the source recordings were balanced. All their clips might well have been exactly at the ITU recommended level of -26 dB, but if Lyra introduced a level mismatch this would have to have been corrected at a later stage, and it's at least possible that it might not have been. The Lyra paper does explicitly say that they didn't follow the ITU rec for "data collection". Interestingly, the Opus and Reference sources are almost exactly -26 dB relative to full scale (according to several measurements of loudness), but the Lyra clip is about 6 dB hotter. So the source (the reference clip) exactly follows the ITU rec. Did they remember to fix the levels on the Lyra clips? I hope so!
jhoechtl 6y ago
I wouldn't call this cheating though. Audio compression makes use of the mental way we perceive sound. If a sound artefact is perceived to sound more clear when it is louder compared to another one with the same compression bitrate yet less volume I would say this falls into the category psychoaccoustic compression
- rcthompson 6y ago
  If simply turning up the volume made it easier to understand the speech, then not turning up the volume on the other codecs would make for an unfair comparison.
notretarded 6y ago
If it sounds better who cares what methods are used?
dwild 6y ago
I lowered the volume on the Lyra one and the sound is still clearly WAY MORE clearer than the other two.

dkjaudyeqooe 6y ago

Isn't that due to audio (frequency) compression coming out of the generative model? I guess that can be tweaked either way but they're going to tend towards that exactly because it sounds louder and thus clearer.

bscphil 6y ago
There are a couple of effects here: 1. Lossy codecs will use a low-pass filter to get rid of hard to compress higher frequencies. This is often inaudible, but even when it is, it should lower the volume, unless you're applying some kind of compensation for it. 2. It's true that lossy codecs compress different frequencies differently, but that's not usually done in such a way that amounts to applying EQ to the frequencies. 3. Even if the relative balance of frequencies did shift as a result of applying lossy compression, this is still done in a way that the overall loudness of the audio does not change. In this case the Lyra output has changed significantly and in an easily audible way (about +6 dB). You could easily get the same effect in Opus just by amplifying (or applying compression to) the result, but Opus is doing things correctly.

rectang 6y ago

Excellent catch. To be precise you need a measure of perceptual loudness rather than raw waveform excursion, but I would expect the results to be in line with what you've found. > It's well known in the audio biz that if you ask people to compare two experiences, and one of them is a bit louder than the other, people will say that the louder one was better, or came through more clearly, or whatever it is you're trying to market for. As a former mastering engineer, you're absolutely right that this is well understood in the audio industry. I used to present my clients with level-matched comparisons of source audio vs. processed so they would understand exactly what was being done, aesthetically.

bscphil 6y ago
Here's an EBU R 128 measure using r128gain: File 'reference.flac': loudness = -25.7 LUFS, sample peak = -9.2 dBFS File 'lyra.flac': loudness = -19.9 LUFS, sample peak = -4.1 dBFS File 'opus.flac': loudness = -25.9 LUFS, sample peak = -9.7 dBFS So that also matches pretty closely what my ears heard.
- perfmode 6y ago
  how do the samples compare when loudness is made constant/normalized?
  - 0-_-0 6y ago
    Just reduce the volume of the lyra one by hand. Doesn't change the fact that it sounds leagues above the others.
  - bscphil 6y ago
    I do still prefer Lyra overall, though not as much as some others (see sibling comment). To me, Lyra is cleaner and easier to understand, but the artifacts it introduces are more annoying and fatiguing than those introduced by Opus. Some people in this thread have reported trouble understanding Lyra, which I attribute to the strange artifacts it introduces.
- 6y ago

fireattack 6y ago

Just as a reference Title RMS Peak Diff clean_p257_011_lyra -20.07 -1.13 18.93 clean_p257_011_opus -26.07 -6.65 19.41 clean_p257_011_refer -25.77 -6.15 19.63 PSD (Welch's method, window=213) https://i.imgur.com/Y8A4kkx.png https://i.imgur.com/Y8A4kkx.png

domoritz 6y ago

Off topic but how do you put images on ipfs and what’s the advantage over e.g. Imgur?

bscphil 6y ago
Cloudflare are kindly hosting [1] a free HTTP gateway for the IPFS [2] network. So I can host an image myself on a server with IPFS, and Cloudflare will cache it for me. This is better than Imgur because the latter has been redirecting users to annoying "social" pages with ads instead of showing them the actual image, at least in some cases. I also can't be sure whether Imgur recompresses your uploads or not - I assume they usually do. It's also more generally useful because I can host other files too, not just images. [1] https://www.cloudflare.com/distributed-web-gateway/ https://www.cloudflare.com/distributed-web-gateway/ [2] https://ipfs.io/ https://ipfs.io/
- javajosh 6y ago
  Is hosting the image yourself, on like a $5 Digital Ocean Droplet and a $10 personal domain, out of the question? This would seem to be the ideal situation in terms of simple, decentralized file hosting solution. What are the downsides of this approach? (I can imagine a server package that can modify index.html sub-resource URLs depending on current server load, preferring private, locally hosted sub-resources but willing to use 3rd party solutions like Cloudflare, too, if required by a black swan event.)
  - bscphil 6y ago
    Out of the question? No. As convenient as running one command on a desktop computer? Also no. > the ideal situation in terms of simple, decentralized file hosting solution Not sure what you mean by "decentralized" if you are in fact hosting it yourself. > What are the downsides of this approach? Well, for the casual person it has the obvious downside that you have to have your own VPS. Most people don't have those. Even if you do, IPFS has a couple of advantages: you can host images anonymously, and anyone anywhere in the world can "pin" the image to make sure it stays live. If you're using a server and you forget to pay DO your $5 one month, all your images go poof into the ether.

adrianmonk 6y ago

Definitely a real effect, but it seems like Google accounted for that in their listening tests. The Google blog post links to the Lyra paper[1], and Section 5.2 of the paper says: > To evaluate the absolute quality of the different systems on different SNRs a Mean Opinion Score (MOS) listening test was performed. Except for data collection, we followed the ITU-T P.800 (ACR) recommendation. You can download those ITU test procedures[2], and skimming through that, it does mention making "the necessary gain adjustments, so as to bring each group of sentences to the standardized active speech level" and a 1000 Hz calibration test tone related to that. (See sections B.1.7 and B.1.8.) So, if I skimmed correctly, and if the ITU's method of distilling speech loudness into a single number is an effective way to match the volume levels[3], then it seems like they did what they could to avoid cheating at the listening tests. It is still interesting that Lyra makes things louder, though. --- [1] https://arxiv.org/pdf/2102.09660.pdf https://arxiv.org/pdf/2102.09660.pdf [2] https://www.itu.int/rec/T-REC-P.800-199608-I https://www.itu.int/rec/T-REC-P.800-199608-I [3] and even for speech that passes through different codecs before its loudness is determined

bscphil 6y ago
That's good information, thanks. My comment is mostly directed at the misleading blog post. I have no direct reason to believe that the study itself was compromised, though it would be great to have confirmation from the authors that it was not. The part about matching volume levels in the ITU recommendation seems to be talking about making sure the source recordings were balanced. All their clips might well have been exactly at the ITU recommended level of -26 dB, but if Lyra introduced a level mismatch this would have to have been corrected at a later stage, and it's at least possible that it might not have been. The Lyra paper does explicitly say that they didn't follow the ITU rec for "data collection". Interestingly, the Opus and Reference sources are almost exactly -26 dB relative to full scale (according to several measurements of loudness), but the Lyra clip is about 6 dB hotter. So the source (the reference clip) exactly follows the ITU rec. Did they remember to fix the levels on the Lyra clips? I hope so!

jhoechtl 6y ago

I wouldn't call this cheating though. Audio compression makes use of the mental way we perceive sound. If a sound artefact is perceived to sound more clear when it is louder compared to another one with the same compression bitrate yet less volume I would say this falls into the category psychoaccoustic compression

rcthompson 6y ago
If simply turning up the volume made it easier to understand the speech, then not turning up the volume on the other codecs would make for an unfair comparison.

notretarded 6y ago

If it sounds better who cares what methods are used?

dwild 6y ago

I lowered the volume on the Lyra one and the sound is still clearly WAY MORE clearer than the other two.