4 ms·

Very interesting research that is 100% oversold in the popular media. Look at the original paper and listen to the reconstructed audio samples (at the end): ht

by titzer 3y ago

Very interesting research that is 100% oversold in the popular media. Look at the original paper and listen to the reconstructed audio samples (at the end): https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002176 https://journals.plos.org/plosbiology/article?id=10.1371/jou... Can you identify which song has been reconstructed? I could not. The basis for song "reconstruction" claims is the song-excerpt identification analysis (e.g. Fig 4) that divides the song into 38 5-second clips and then ranks how closely each identified excerpt is to the expected excerpt--effectively, how well does the reconstruction match 38 excerpts to 38 expected excerpts. This is not reconstructing the song, just matching puzzle parts. A nice piece of neural science, completely misrepresented in the media.

chewmieser 3y ago
I actually did not think it was that bad... It's pretty recognizable on the larger probe count & sample sizes. Pretty cool stuff even if it's early.
pohl 3y ago
I could. It's pretty obvious in journal.pbio.3002176.s007.wav
wnkrshm 3y ago
Isn't this the same issue as with the neuro-visual reconstruction techniques matching on a small collection of samples?
bondarchuk 3y ago
> The basis for song "reconstruction" claims [in the popular media] is the song-excerpt identification analysis Not correct for the Dutch state news, at least. They make the same "reconstruction" claims while showing the audio fragments that don't use any input data (for 1 and 29 patients), while mentioning that it's not very accurate but at any rate still recognizable. https://nos.nl/artikel/2486771-computer-herkent-pink-floyd-nummer-uit-brein-luisteraars https://nos.nl/artikel/2486771-computer-herkent-pink-floyd-n... Based on earlier neural decoding news items I was also expecting something completely overblown, but in this case I think the reporting is quite legit, and you might just be having a knee-jerk contrarian reaction.
aaroninsf 3y ago
As others are saying, my expectations were low based on this comment, so I found the reconstructions much better than expected. Impossible to say if I'd be able to ID them without any context... ...but that leads immediately to the thought that a useful metric for these things might well be whether Shazam and similar "landmark identification" tools can. Especially those which themselves are based on perceptual encoding models of some kind... the more human-perceptual-system based, the more interesting. I.e.: if our machine models of audio perception, can consume the output of machine models of the output of those models, and reliably track that back to the originals, the "fidelity" of the the lossy "encoding" ("recording") is becomes less interesting. Especially from a behavioralist/instrumentalist view. Which is the view underneath commercialization and e.g. surveillance. Does the <data blob> sound/look like the original? Who cares, if the machine knows it's <thing>, we're good. Vaguely related, in the long term it will be interesting to learn if the internal modeling (sic) of music in different people differs significantly as a function of e.g. musical training, proficiency, and the like. This would imply you could tell how good a musician someone is or might be, by inspecting the qualities of their internal model. Which then makes me wonder about grokking and eidetic memory for audio. Anecdote: Hollywood composer (and sometime pop musician and collaborator with Amy Mann) Jon Brion used to have a regular night at bar in LA, which we attended ~25 years ago. He was regularly joined by other then-famous musicians like Rickie Lee Jones. Part of his shtick was to pick up suggestions from the audience of free-associate musically, performing fragments of various popular music (e.g. 80s pop). It was truly remarkable, even uncanny and unsettling, how good his ability was to instantly riff out not just recognizable or faithful covers, but inflect them with a large number of the idiosyncratic micro-details of the original performance. At the time I understood this as the rare intersection of something like eidetic memory for music, and, a caricaturist's talent for emphasizing exactly those features which make the thing unique. Which I suspect are exactly the sorts of landmarks systems like Shazam tend to abstract out; and what I anticipate work like this will reveal are exploited by our own perceptual memory systems. Which, incidentally, seem to be distinct from "episodic memory" viz. the well established manner in which people far gone to various degraded memory and dementia conditions, can be "brought back" to themselves through flawless recall and performance/singing of songs they knew, even when they are otherwise "lost." C.f. https://www.openculture.com/2013/05/the_strange_day_when_bugs_bunny_saved_the_life_of_mel_blanc.html https://www.openculture.com/2013/05/the_strange_day_when_bug... as well

by titzer 3y ago

chewmieser 3y ago

I actually did not think it was that bad... It's pretty recognizable on the larger probe count & sample sizes. Pretty cool stuff even if it's early.

pohl 3y ago

I could. It's pretty obvious in journal.pbio.3002176.s007.wav

wnkrshm 3y ago

Isn't this the same issue as with the neuro-visual reconstruction techniques matching on a small collection of samples?

bondarchuk 3y ago

> The basis for song "reconstruction" claims [in the popular media] is the song-excerpt identification analysis Not correct for the Dutch state news, at least. They make the same "reconstruction" claims while showing the audio fragments that don't use any input data (for 1 and 29 patients), while mentioning that it's not very accurate but at any rate still recognizable. https://nos.nl/artikel/2486771-computer-herkent-pink-floyd-nummer-uit-brein-luisteraars https://nos.nl/artikel/2486771-computer-herkent-pink-floyd-n... Based on earlier neural decoding news items I was also expecting something completely overblown, but in this case I think the reporting is quite legit, and you might just be having a knee-jerk contrarian reaction.

aaroninsf 3y ago

As others are saying, my expectations were low based on this comment, so I found the reconstructions much better than expected. Impossible to say if I'd be able to ID them without any context... ...but that leads immediately to the thought that a useful metric for these things might well be whether Shazam and similar "landmark identification" tools can. Especially those which themselves are based on perceptual encoding models of some kind... the more human-perceptual-system based, the more interesting. I.e.: if our machine models of audio perception, can consume the output of machine models of the output of those models, and reliably track that back to the originals, the "fidelity" of the the lossy "encoding" ("recording") is becomes less interesting. Especially from a behavioralist/instrumentalist view. Which is the view underneath commercialization and e.g. surveillance. Does the <data blob> sound/look like the original? Who cares, if the machine knows it's <thing>, we're good. Vaguely related, in the long term it will be interesting to learn if the internal modeling (sic) of music in different people differs significantly as a function of e.g. musical training, proficiency, and the like. This would imply you could tell how good a musician someone is or might be, by inspecting the qualities of their internal model. Which then makes me wonder about grokking and eidetic memory for audio. Anecdote: Hollywood composer (and sometime pop musician and collaborator with Amy Mann) Jon Brion used to have a regular night at bar in LA, which we attended ~25 years ago. He was regularly joined by other then-famous musicians like Rickie Lee Jones. Part of his shtick was to pick up suggestions from the audience of free-associate musically, performing fragments of various popular music (e.g. 80s pop). It was truly remarkable, even uncanny and unsettling, how good his ability was to instantly riff out not just recognizable or faithful covers, but inflect them with a large number of the idiosyncratic micro-details of the original performance. At the time I understood this as the rare intersection of something like eidetic memory for music, and, a caricaturist's talent for emphasizing exactly those features which make the thing unique. Which I suspect are exactly the sorts of landmarks systems like Shazam tend to abstract out; and what I anticipate work like this will reveal are exploited by our own perceptual memory systems. Which, incidentally, seem to be distinct from "episodic memory" viz. the well established manner in which people far gone to various degraded memory and dementia conditions, can be "brought back" to themselves through flawless recall and performance/singing of songs they knew, even when they are otherwise "lost." C.f. https://www.openculture.com/2013/05/the_strange_day_when_bugs_bunny_saved_the_life_of_mel_blanc.html https://www.openculture.com/2013/05/the_strange_day_when_bug... as well