9 ms·
Our brains 'time-stamp' sounds to process the words we hear
- notacoward 4y agoSimilar things happen with vision BTW. When you start looking into it, what our eyes and ears and the connected parts of the brain actually do is pretty crazy. Our sensory systems are eventually consistent, and there are glitches, but amazingly it all mostly works.
- codeulike 4y agoWhen you start looking into it
- notacoward 4y agoHa! I wish I could say that was intentional. Thanks for pointing it out.
- ChrisClark 4y agoI think it could be almost described as a full on 3D rendering engine, we are getting the raw data from outside ourselves, but what we see is completely built up inside our own minds. Does that make sense?
- tornato7 4y agoI like that. We are always processing information in 3D, not just vision but you can imagine hearing someone behind you and your brain immediately places them in 3D space based on your prior knowledge of your surroundings. It's amazing sensor fusion because you can see/hear/smell/touch something all at the same time and associate it correctly with one object.
- kridsdale2 4y agoIt's much much more like our new AI image generators than a physical simulation of light and volume with mathematical precision. Imagination is a "good enough" fluctuation of seemingly random stimulation of neurons that we're able to judge as being quite approximate to the same stimulation pattern that we get from our physical sensors. Thinking about a scene or music and "hearing it" is like a unit test.
- notakio 4y agoIt has been illuminating for me over the past few weeks to have lost hearing in my right ear as a result of a lingering sinus infection; I understood that identifying the direction a sound might come from would be difficult with only one ear, but I had no idea how crucial it was for speech comprehension, particularly when any other noise is also present. I've found that I can barely even discern that speech is present within a mix of noises, much less am I able to comprehend the speech until such time as I can reduce the other noises to a minimal level, and directionally point my ear at the source of said speech. Conference calls were quite a delight there for a while.
- w_for_wumbo 4y agoEssentially our hardware isn't perfect, and there's delays so the software has a bunch of patches and workarounds to interpolate what it believes is occurring, which happens to be right a lot of the time. Normally only with illusions where this reality is confronted to us.
- drcongo 4y agoThere was an excellent PBS show called The Brain with Dr David Eagleman which, in one episode, went into detail on the way the brain re-syncs what it sees with what it hears, essentially doing a tiny bit of time travelling to ensure that your world makes sense. It's well worth seeking out.
- jvm___ 4y agoIs this why we turn down the car volume when looking for a house number? The resync process is taking too much bandwidth, or messing with our visual/temporal system, so reducing the audio input helps the system.
- spideymans 4y agoI couldn't find the show, but he has a related lecture on YouTube https://www.youtube.com/watch?v=vv_e99qbJ4U https://www.youtube.com/watch?v=vv_e99qbJ4U
- codeulike 4y agoThere's an interesting sortof thought-trap when thinking about these things - which this article doesn't fall into - but I'll mention it anyway cos its fun. Dennett calls it the Cartesian Theatre - and the idea is that when thinking about how the brain works, we may mistakenly imagine that once the brain has (say) processed these timestamped sounds, it then puts them all back together somewhere to 'play them back' to our consciousness. But thats a paradox because then you'd need another consciousness to interpret the reconstructed sound. Dennett likened it to imagining a little homunculus inside your brain that is watching a screen that plays back your conscious exprience. Of course it can't work like that, because it becomes recursive. So when the brain 'time stamps' these sounds (as they put it) it (probably) doesn't then need to ever put them back in the right order again. That bit of processing is done. A corollary to this idea is that consciousness is (most likely) spread throughout the brain so there is no 'one place' where things come back 'together'. That also means there is no one instant in time where we become 'conscious' of things. If its spread throughout the brain is must necessarily be smeared across a (fairly short) internal of time too. I think these days with neural nets being better understood perhaps we dont fall into this thought trap so much.
- officialjunk 4y agoby this logic, with humans having millions of neurons in other parts of the body, such as the gut, we may have consciousness spread out over the body, not just the brain.
- cscheid 4y agoAntonio Damasio is a neuroscientist who's done work in the area, and has decent popsci books about this idea. I read the older ones, "Descartes' Error" and "The Feeling of What Happens", and they're fun, good reads. Apparently he's written more on the subject as well.
- coldtea 4y agoNot sure about consciousness, but some of its processing is a given, the gut is even called a "second brain" by scientists in related fields.
- aix1 4y agoI wonder if Auditory Processing Disorder could be related to this mechanism. https://en.wikipedia.org/wiki/Auditory_processing_disorder https://en.wikipedia.org/wiki/Auditory_processing_disorder
- BWStearns 4y agoI wonder if this is part of why it's nearly impossible to form sentences when you hear an echo of your own voice on zoom or something, like your brain perceives it as a duplicate chunk that was mis-timestamped.
- kraquepype 4y agoI do wonder myself if there is some research on this, it is irrationally annoying when I hear an echo of myself while talking, so much so that I just can't proceed, it would be nice to know why that is.
- pessimizer 4y agoLook at research on stutters. There are treatments for stutter that involve replaying one's own voice back to stutterers with a delay. https://en.wikipedia.org/wiki/Delayed_Auditory_Feedback https://en.wikipedia.org/wiki/Delayed_Auditory_Feedback
- anon291 4y agoReminds me of positional encoding in attention based networks.
- johntb86 4y agoI was thinking that too, but this description makes it seem more like a type of windowing rather than the position encoding in a transformer (which is fixed): "the information [...] gets passed between different neural populations in a predictable way, which serves to time-stamp each sound with its relative order.”
- TheOtherHobbes 4y agoWe hugely underestimate how processed all of our senses are. Hearing doesn't listen to pressure waves. It does some very complex real time source separation to distinguish between different sound sources. Then it performs overtone and resonance matching to identify different speakers. Then it follows up with phoneme recognition to identify words - which somehow identifies phoneme invariants across a wide range of voice types, including kids, male/female, local/foreign and social register(class)/accent. Then it recognises emotional cues from intonation, which again are invariant across a huge range of sources. And then finally its labels all of that as linguistic metadata, converts the phonemes into words, and parses the word combinations. It's not until you try to listen to a foreign language that you hear the almost unprocessed audio for what it is. And even that still has elements of accent and intonation recognition.
- kridsdale2 4y agoNon linguistic elements of verbal communication are so universal nobody even really notices when fictional alien species in media communicate to human protagonists. It's noncontroversial to the audience that hostile/non-hostile, instruction, friendliness, cooperation, etc, are all embedded in the tones of all animals, robots, and even tree creatures throughout the universe.
- henrydark 4y agoThis isn't surprising given that it's well known that we share more than 90% of our DNA with tree creatures across the universe
- mahathu 4y ago>And then finally its labels all of that as linguistic metadata, converts the phonemes into words, and parses the word combinations. How did the scientific studies show this?
- gryson 4y agoYou're asking for a summary of the entire fields of speech perception and psycholinguistics. A good place to start is the groundbreaking speech perception experiments done at Haskins Laboratories in the 1950s by people such as Alvin Liberman.
- makz 4y agoProbably related, I’m not an native English speaker, and I have noticed that sometimes I don’t understand what someone is telling me in this language , just one or two words, and it kind of gets saved on a buffer in my mind and they keep talking and after a few seconds what was in the buffer gets “processed” and I understand it at the same time that whatever is being said to me currently. It feels weird.
- russdill 4y agoI have an oddly similar sensation when becoming aware of a sound that woke me.
- ncpa-cpl 4y agoI’ve had this happen in places were multiple languages are spoken. Hearing something in Language B, but not realizing, so the brain processes it in Language A. Then not being able to understand for a few seconds until the brain realizes that what I heard is actually Language B, and suddenly the first seconds of the speech make sense.
- w_for_wumbo 4y agoI'm a native English speaker and I have the same thing, where someone will say something I don't understand, and then in the middle of asking what they mean, I realise what they were trying to say (The processing just took longer than I expected)
- ianmcgowan 4y agoThis is something that also happens as you age - even if your hearing remains decent, Central Auditory Processing Disorder becomes an issue. I imagine it will be more pronounced with a second language, so you have to look forward to. https://www.rehab.research.va.gov/jour/05/42/4suppl2/martin.html https://www.rehab.research.va.gov/jour/05/42/4suppl2/martin....
- 1-more 4y agoI had this years later from The Simpsons of all things. In "Who Shot Mr. Burns (Part 2)" Tito Puente has a musical number "Señor Burns"[0] where the singer calls Burns "el diablo con dinero" which I remembered the sound of but not the meaning until I learned some Spanish like 10 years later. [0] https://www.youtube.com/watch?v=p9kdDet7G14 https://www.youtube.com/watch?v=p9kdDet7G14
- modeless 4y agoI'm surprised nobody has mentioned the connection to AI. Many models need "positional encodings" as part of their input in order to to understand the spatial or temporal relationship between different input tokens. It's not at all surprising to me that the brain would have an equivalent. It's not always clear what form these encodings should take. If we could figure out the actual encodings used by the brain, I bet we would find that they have big advantages over the ones we're currently using.
- carbocation 4y agoCame here looking for this comment. Even as someone who doesn't have feelings one way or another about any brain-neural network similarity, this one struck me the same way. Thanks for mentioning it.
- dogcomplex 4y agoOr the inverse - like "garbage" dna, our minds encode memories with "inefficient" baggage stamped to them... which are very likely how we tie emotion to memory. Perhaps a mind stamping with simple math could be much more "efficient".
- Sommer 4y agoThis reminds me of the correlogram modeling of auditory perception where the cochlea uses autocorrelation to encode temporal information of a sound signal. A neat idea that helps describe a lot of time encoded auditory processes. Short explanation: https://ccrma.stanford.edu/~malcolm/correlograms/index.html?About.html https://ccrma.stanford.edu/~malcolm/correlograms/index.html?... [stanford.edu]
- tibbydudeza 4y agoIt implies a single buffer so there is a delay in cognition which implies our consciousness is always lagging behind physical reality.
- thedudeabides5 4y agoThis is why (unmentioned company) runs on timeseries.
- jrnichols 4y ago
- jrnichols 4y agoIt's sad to see how many people in tech circles blindly bought into the "just wear a mask" bullying without ever considering that there are consequences for others. The fact my comment got voted down so quickly illustrates that. communication is complicated and when you remove the ability for a neurodivergent person to effectively communicate, you really change their entire life.
- sebringj 4y agoThat seems so efficient to do it all at once like that. The brain seems to be this massively parallel/buffered biological machine with an assembly pipeline to the consciousness center(s)? Like a mix of specific architecture layout (software 1.0) with a bunch of pattern matchers (software 2.0) maybe but I'm complete layman here obviously.
- lilyball 4y ago> The researchers found that the brain processes speech using a buffer This is not news to anyone with ADHD who regularly says "what?" before immediately figuring out what was already said and responding to it. It feels exactly like reprocessing an existing auditory buffer.
- smeej 4y agoI'm sure I'm missing some important nuance or something, but it sounds like what they're saying is, "You can understand spoken words because your brain processes the sounds in the order you hear them." This sounds like the most obvious thing in the world to me. Everything about me experiences time sequentially. If I remember things, I can cast my mind back in time, but I still remember things from beginning to end, not in some other order. I'd have assumed my brain processes my visual inputs sequentially too. I can't even think what other option there might be. Somehow everything I've seen or heard over some set period of time hits me all at once? What am I missing here?
- LocalH 4y ago> Everything about me experiences time sequentially. That's not entirely true. Ever look at a clock with a second hand and see the hand linger a little longer the first second? Congratulations, you just saw things out of order. Your brain did not receive input for the period of time when your eyes were in motion towards the clock, so it lied to you and filled in the missing data after the fact once it started getting visual input again.
- smeej 4y ago...but it still put it before the hand moved, so it's still sequential, even if the gap was filled in with something that didn't happen (the hand standing still) rather than my brain guessing that the hand moved once before the one I saw, isn't it?
- PaulHoule 4y agoI figured there had to be something like that.
- Waterluvian 4y agoMy wife jokingly describes what I do as “buffering” where I’m not paying attention and she will interrupt me and I’ll catch up, almost by re-hearing the last 2-4 seconds of what she said. It’s a bizarre sensation but I wonder if it’s actually rather common.
- sillysaurusx 4y agoIt’s a subtype of ADD. I have it too. Sluggish cognitive tempo. I don’t suppose you’re also a very visual person, who prefers to solve math problems geometrically rather than algebraically, etc. There may be a link between that buffering and the ability to turn abstract shapes in one’s mind.
- tro2102 4y agoI have both of these characteristics. Where did you learn it was a subtype of ADD?
- sillysaurusx 4y agoIntensive independent research. I learned of it around 2010 so I forget the exact trail. Eventually I hit on a medical journal that documented it, and I was like “that’s me.”
- gavinmckenzie 4y agoOh wow, you just managed to describe me completely. I'm inattentive-type ADHD (diagnosed as an adult), and all of my thought processes are super visual; constructing visual mental models in space is how I problem solve and a lifetime ago in school I was top of the math class when I could construct visual models of the problem but struggled when I couldn't. My diagnostic ADD testing was inconclusive because I was able to excel at working memory and shape tests because I am able to hold onto a ton of information if I can write it on my mental blackboard etc. When I'm listening to a conversation, what I'm hearing often produces a stream of involuntary visuals representing related information and subjects, like I have a mental background process showing me flash-cards of possibly useful information. I also have the issue where someone will talk to me and there's this delay where I often appear to be non-responsive for a moment or two until my brain spools-up and then I mentally lock into the conversation. Anyway, if you have any references or additional learnings you've found related to this topic, please share.
- eternalban 4y ago"Speech consists of a continuously-varying acoustic signal. Yet human listeners experience it as sequences of discrete speech sounds, which are used to recognise discrete words. To examine how the human brain appropriately sequences the speech signal, we recorded two-hour magnetoencephalograms from 21 participants listening to short narratives. Our analyses show that the brain continuously encodes the three most recently heard speech sounds in parallel, and maintains this information long past its dissipation from the sensory input. Each speech sound representation evolves over time, jointly encoding both its phonetic features and the amount of time elapsed since onset. As a result, this dynamic neural pattern encodes both the relative order and phonetic content of the speech sequence. These representations are active earlier when phonemes are more predictable, and are sustained longer when lexical identity is uncertain. Our results show how phonetic sequences in natural speech are represented at the level of populations of neurons, providing insight into what intermediary representations exist between the sensory input and sub-lexical units. The flexibility in the dynamics of these representations paves the way for further understanding of how such sequences may be used to interface with higher order structure such as lexical identity." Neural dynamics of phoneme sequences reveal position-invariant code for content and order | https://doi.org/10.1038/s41467-022-34326-1 https://doi.org/10.1038/s41467-022-34326-1
- sathley 4y agoI wonder if our brains do the same timestamping when we listen to someone sing a song. Our brain database will not have enough examples of those exact words being sung in that musical way. I guess that's why we need a little more focus to understand the lyrics of a song vs someone talking to us.
- aeternum 4y agoChildren however seem to prefer the musical version and find that easier to focus and understand, especially around 1-2 yrs old. Perhaps the tempo of music helps build establish this timestamping ability then is no longer needed.
- mooneater 4y agoI remember in grade 8 my geography teacher noticing I wasn't paying attention and asking: "What did I just say?" The sound of his last couple sentences were in my recent memory (timestamped?), but it seems I hadn't fully parsed them yet. When he asked, I took a brief moment to parse them, then to his disappointment I repeated them verbatim to him. I'll never forget the way my brain processed delayed audio that day!
- specialist 4y agoI experience something like audio dyslexia. At the phonetic level, jumbling syllables vs whole words. Reading the other comments, seems like many others learned to cope by consciously buffering, allowing parsing and comprehension to catch up. I'll often have multiple tentative interpretations. So then I intentionally replay the sounds I think I heard, as well as wait for more context (because a lot of speech and conversation is redundant), before settling on an answer. I eventually trained myself to not habitually say "What?" I eventually figured out that just pisses people off. It'll be hysterical if I've had something like ADHD all this time. Would explain so much.