11 ms·

Reconstructing images a person sees via non-invasive brain scans

179 points by yboris 4y ago

not-my-account 4y ago
This seems incredible! If we put aside questions on how to collect the data, does this not suggest that we would be able to "read minds" (intentionally left vague) with an FMRI?
- terminal_d 4y ago
  I think it will be similar to how diffusion-based image generation currently relies on what the model is trained on. This might work well for average folks, but for anyone sufficiently out of the norm / with expertise/mental hardware outside of the bounds of the trained model, the results will be subpar / useless.
  - stephc_int13 4y ago
    No, I think the model would have to be trained for every single individual.
- deleted 4y ago
  [deleted]
belkarx 4y ago
Preprint of a similar model (textual not visual stimuli): https://www.biorxiv.org/content/10.1101/2022.09.29.509744v1 https://www.biorxiv.org/content/10.1101/2022.09.29.509744v1
yboris 4y ago
Shortcut to the GitHub repository: https://github.com/zjc062/mind-vis https://github.com/zjc062/mind-vis
mdp2021 4y ago
It is not clear if those systems can just decode "seen" and not also "imagined".
- pxmpxm 4y ago
  Yeah - clearly they have some dimensionality reduced classifier for what's in the underlying mri data, but using stable diffusion to project it just seems odd. "The MRI of your brain tells us you're looking at a building with a tower of some sort, so here's an artist's impression of a building with a tower"
BWStearns 4y ago
In today's edition of "Don't build the torment nexus".... Really cool tech but I'm very curious if/worried that this will spawn some half-working interrogation tools.
- vbezhenar 4y ago
  Google glasses will ensure that you're paying attention to the ads.
- ReptileMan 4y ago
  Better than needles under the nails in any case...
giantdude 4y ago
Slightly more invasive attempts were made over 100 years ago. https://www.smithsonianmag.com/smart-news/how-forensic-scientists-once-tried-see-dead-persons-last-sight-180959157/ https://www.smithsonianmag.com/smart-news/how-forensic-scien...
bigyikes 4y ago
Is this for real? How is this not the biggest story in the world right now? This is absolutely incredible. I wish I had something intelligent to say. Just wow. It’s literally mind reading. Imagine plugging this tech into Neuralink. The brain-machine bandwidth problem seems way more tractable in light of this result.
- aquanext 4y ago
  I don't think being able to scan this is completely new. Perhaps this specific technique is, but I've heard of this being done at least once previously in the past decade.
- cma 4y ago
  2013 https://blog.ted.com/reading-minds-with-a-brain-scanner-its-happening-mary-lou-jepsen-at-ted2013/ https://blog.ted.com/reading-minds-with-a-brain-scanner-its-...
  - mdp2021 4y ago
    Mary Lou Jepsen. > That, of course, leads to very real concerns about privacy "The revenge of the tinfoil hat".
antegamisou 4y ago
For those of you that are overly concerned/excited about this keep in mind that fMRI scans are kind of looked down upon the neuroscience research community for the dubious misinterpretations from the findings (see dead Salmon fMRI study). It looks like PET imaging is what many are focusing their research into for better insight / “dynamic analysis” of the brain.
- mikeiz404 4y ago
  Dead Salmom fMRI Wired Article: https://www.wired.com/2009/09/fmrisalmon/ https://www.wired.com/2009/09/fmrisalmon/ Research Poster: http://www.prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://www.prefrontal.org/files/posters/Bennett-Salmon-2009....
- ramraj07 4y ago
  I know we haven’t proven fMRI is not truly measuring activity but have we proven it doesn’t even correlate?
  - mattkrause 4y ago
    The signal measured by fMRI is definitely related to neural activity, but the relationship is indirect and sometimes subtly complicated. Neurons compute and communicate electrochemically by controlling how ions (charged particles) flow across their membranes. At "rest", the inside of a neuron is slightly negative (- 70 mV) relative to its environment. However, its surface is studded with channels that let different types of ions into/out of the cell as well as "receptors" that can open or close these channels. Some are opened by specific chemicals, which can bring the cell's membrane potential closer to zero (e.g., by admitting sodium ions). Once it gets above about -55 mV, other channels that are sensitive to voltage take over, and produce a wave of activity that shoots down the axon and eventually causes the release of chemicals that affect other neurons, muscles, etc. This is what actually does work in the brain, but it's not what fMRI measures. Instead, fMRI exploits the fact this process is energetically expensive: neurons need oxygen and sugar (glucose) to do all that work, and so neurons need to plumbed into the vascular system so those can be delivered. The signal fMRI measures is called the BOLD signal, for Blood Oxygenation Level Dependent. The magnetic properties of a blood molecule carrying oxygen are a bit different from a deoxygenated one, and thus they can be told apart with some clever physics and signal processing. Based on what I said above, you might expect that fMRI thus measures the reduction in oxygenation, but it actually turns out to be a lot more complicated than that. Levels do dip when nearby neurons are active, but only briefly. After that, they shoot up, above baseline levels: more blood--and more oxygen--are delivered to the formerly-active area. This is mostly what fMRI measures. The neuromuscular coupling seems to vary between brain states and maybe even brain areas; it's affected by different health conditions and drugs too. It's also limited by the structure of the vascular system. People occasionally find that veins appear to be heavily involved in cognitive tasks, sometimes even more than the brain itself, but that's just because they're delivering the blood. Despite all this, the mechanisms are getting better understood and you can certainly interpret a BOLD change as suggesting that something is happening in that particular region, even if it's not clear exactly how it's implemented.
zactato 4y ago
This is super cool, but it seems pretty far from "mind reading". It's not reading what the person's brain is interpreting the image as, but rather what they're currently seeing. I wonder how different these signals are. It's a pretty big difference since the only signal they're getting is what the eyes are currently seeing. Essentially they're using a $1million fMRI to poorly do the job that a $5 dollar camera could do. This is still super cool, but not quite our wildest scifi dreams.
- IIAOPSW 4y ago
  disclaimer, what I'm about to say is just vague recollection of content that had already been pre-digested into layperson science news. There's probably a ton of loss in this signal. From what I understand, memory cells literally reproduce the original brain signal just as if it were coming from outside stimulus. Like playing the brain waves off a tape recorder.
  - badRNG 4y ago
    >From what I understand, memory cells literally reproduce the original brain signal just as if it were coming from outside stimulus. Like playing the brain waves off a tape recorder. I can't recall pictures of events that happened yesterday with anything resembling what it was like to be there and experience seeing the events in person. I can construct images of what things might've looked like, but the creative process of recalling that memory feels awfully different than experiencing events in real time.
  - mattkrause 4y ago
    Eh....Nobody really knows how memory works. There is certainly some evidence for "replay", but it's generally thought to be involved in "writing" it into longer-term storage rather than the storage mechanism per se. OTOH, there are some cool results with Hopfield-like networks where applying part of a pattern (somehow) cues the brain (or network) to produce the rest of it.
sithadmin 4y ago
Some of these results seem to be picking up on general abstract concepts moreso actual visual cues. for example, one sample has bright colored wooden park benches as input, and some generic brown wood furniture as an output. Perhaps a strange artifact of training biases, but a little disturbing at a glance. Extracting current visual data out of one's brain is one thing, but being able to potentially identify abstract thoughts is a whole different level of invasiveness.
- zwkrt 4y ago
  So there’s this philosopher Daniel Dennett. In broad strokes, he’s generally liked by philosophically minded scientists and disliked by the philosophy community. His position is that we will slowly chip away at the magic of consciousness as we get better and better tools to scientifically explore the brain. The consciousness is not some magical force that enters us but just a giant bag of tricks that our brain plays. Furthermore, that the “hard problem of consciousness”, i.e. “what it is like” to be something is not immune to the scientific process. I’ve always been defferential to his viewpoint, but this is the first time his arguments have felt justified. It’s surprising but in hindsight makes sense to me that when you look at a specific representation of an object that your brain internally kind of “draws a picture” of an abstract representation of that kind of object. You can probably think of the location of 100 trees, but it’s pretty ridiculous to think that your brain actually has pictures of 100 trees in 100 different drawers. You look at a tree and your brain draws a kindergarten picture of a tree. Here lay the platonic forms? Cool stuff.
  - goatlover 4y ago
    > The consciousness is not some magical force that enters us but just a giant bag of tricks that our brain plays. That's kind of a straw man of what Dennett's opponents argue. No philosopher seriously says that consciousness is a magical force entering the brain. Rather, they make arguments that materialism (physicalism) fails to account for consciousness, whatever that entails for metaphysics or epistemology. Some are naturalists, but not physicalists. Some are idealists. Some like David Chalmers propose a kind of property dualism. Others might be neutral monists or think we're cognitively closed to the solution. Or panpsychists were everything has a bit of consciousness. Materialism is a metaphysical assumption. We don't know the truth of what reality fundamentally is (it could be a simulation for all we know). We also don't know how to bridge the subjective/objective divide, as Thomas Nagel has argued. Another way of framing the argument is that the objective world is an abstraction across many subjective experiences, where consciousness is abstracted in favor of mathematical forms. That works well for describing/predicting patterns in nature we observe, including the brain, but then when you turn it around to explain the mind, it runs into difficulties, since the abstracted-away subjectivity has been left out. There's no need to muddy the waters with the word "magic". We don't know everything, and some things we might never know. We do know that we have experiences of color, sound, taste, smells, feels, dreams, inner dialog and what not. That's not magic, it's just how we experience being in the world, whatever the world and our experiences are. Here's a view opposing Dennett's from Jaron Lanier: http://www.jaronlanier.com/zombie.html http://www.jaronlanier.com/zombie.html
mrelectric 4y ago
I've seen research like this before. Does anyone have any other spicy links on this topic? Idea of recording dreams seems very interesting to me.
- cma 4y ago
  Mary Lou Jepsen gave a TED talk or something on it in 2013.
- mattkrause 4y ago
  Tons! Jack Gallant's group at UC Berkley has done a lot of interesting fMRI decoding. The approach in their earlier papers (e.g., Kay et al, 2008 [0]) is neat because it uses some knowledge of how early visual areas represent information, rather than just flinging a decoder at it. They later extended this to movies (Nishimoto et al., 2011 [1]), and have subsequently moved a bit away from sensory information to "semantic" decoding. Jack has a TED talk about this [2] and Kendrick Kay has written some News and Views/Perspective pieces [e.g., 3] that might be interesting too, as it links to other groups' work. I like their approach because it ties into neuroscience rather than just throwing ML at the wall and seeing what sticks, but there's also a huge cottage industry of "Blackbox ML" + MRI to do decoding. My sense is that this has moved slightly out of the limelight and into the stage where people incrementally improve it with newer ML techniques like GANs [4] and diffusion (this link), but it might just be me. For sleep specifically, you might be interested in Horikawa et al. (2013)[5], who could predict words associated with dream contents (e.g., "Car" but didn't try to reconstruct them). There's also a very short perspective on it [6]. [0] Kay, K., Naselaris, T., Prenger, R. et al. Identifying natural images from human brain activity. Nature 452, 352–355 (2008). https://doi.org/10.1038/nature06713 https://doi.org/10.1038/nature06713 [1] Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Current biology : CB, 21(19), 1641–1646. https://doi.org/10.1016/j.cub.2011.08.031 https://doi.org/10.1016/j.cub.2011.08.031 [2] https://www.youtube.com/watch?v=Ecvv-EvOj8M https://www.youtube.com/watch?v=Ecvv-EvOj8M [3] Kay, K. N., & Gallant, J. L. (2009). I can see what you see. Nature neuroscience, 12(3), 245. https://doi.org/10.1038/nn0309-245 https://doi.org/10.1038/nn0309-245 [4] Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., & van Gerven, M. A. J. (2018). Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage, 181, 775–785. https://doi.org/10.1016/j.neuroimage.2018.07.043 https://doi.org/10.1016/j.neuroimage.2018.07.043 [5] Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y. (2013). Neural decoding of visual imagery during sleep. Science (New York, N.Y.), 340(6132), 639–642. https://doi.org/10.1126/science.1234330 https://doi.org/10.1126/science.1234330 [6] https://www.nature.com/articles/nmeth.2504 https://www.nature.com/articles/nmeth.2504
stephc_int13 4y ago
This is not as advanced as some of you might think. You have to understand how diffusion models work to "reconstruct" images. You also have to understand that the model has to be trained to match pictures/brain scan for each individual. But this kind of approach can certainly be useful, especially with people with motor impairment after a stroke or accidental brain damage. Not sci-fi level yet.
- DontchaKnowit 4y ago
  For some reason having trouble getting the full study, anyway I'm no AI specialist so idk if reading it would answer my question, but : is this generalizeable? E.g. can they reconstruct and arbitrary visual stimulus, or just one that appeared as training data? If this is a case where they showed a patient an image of a giraffe 1000 times and mapped their brain, generated output,then backpropogated, etc and then now they can recreate a decent image of a giraffe when shown 1 of the images from training data : This is not that impressive. I mean it's cool and interesting but doesn't sound remotely ground breaking, because it's not actually mapping arbitrary inputs to arbitrary outputs. But if they can reproduce arbitrary visual stimuli this is pretty damn incredible, to me at least.
- tomxor 4y ago
  Yes, notice how much variance there is between the images. They are the same types of building, or similar sized animals, but the details and shapes can be wildly different. It's pretty clear what's happening is FMRI meaning -> stable diffusion image generation. Now this _is_ cool, but it's important to understand all that's happening is the natural language input to stable diffusion has been replaced with FMRI input. i.e the image is still generated by stable diffusion, not you, you are just inputting meaning by visualising a separate image, and that meaning is abstract enough to be coarsely learned through an FMRI... but ultimately the image is not produced by your brain, which will probably be more obvious to any of the test subjects.
  - not2b 4y ago
    Yes, how do we know that visual rather than verbal information is even being processed? I can get a decent image from Stable Diffusion or Dall-E from a pure text prompt, and if that's added to a very basic scribble and we do img2img a much better result can be produced, but it might have little to do with what the person is actually seeing.
IIAOPSW 4y ago
Assuming these results hold, I wonder what the legal implications would be. In a criminal prosecution or investigation, can you force someone to just think about the question while stealing the image straight out of their skull? Even worse, what happens when its only 85% accurate but 100% convincing to juries. In the US at least, I think 4th amendment protections obviously apply (though with this bench, don't count your precedents before they hatch). Elsewhere in the world, even fairly liberal freedom loving jurisdictions, the question is still up in the air imo. That should give everyone pause. The information age began with a single question the answer to which is still unfolding: .-- .... .- - / .... .- ... / --. --- -.. / .-- .-. --- ..- --. .... - ..--..
- antifa 4y ago
  > can you force someone to just think about the question while stealing the image straight out of their skull? Can the person being mind read, if aware of the technology, force their mind to only output unrelated content (e.g. shrek fighting zombies) or vividly imagine the accuser committing heinous crimes?
- mrozbarry 4y ago
  There is absolutely no way this sort of data would ever hold up in court.
- mathstuf 4y ago
  I'm curious what they'd do with someone with aphantasia. Can they get images out of my head? I certainly can't…
  - _ink_ 4y ago
    Exactly what I was thinking.
  - sheerun 4y ago
    TIL I'm in the 0.8% of population. I thought it was a rule, not an exception
BurningFrog 4y ago
Something I occasionally worry about: When/if mind reading does become possible, thought privacy will be gone, and humanity enters a brand new era of surveillance.
- necovek 4y ago
  We can finally put those tinfoil hats to good use, though.
- antegamisou 4y ago
  In my slightly uneducated opinion, this will only be possible once there’s solid understanding of how the human brain works, which if you think about it (ha!) has much lot more exciting implications apart from mind reading, it could even be a step closer to or the resolution to AGI. Besides, for a large part of the 8 billions I would suppose Google search history about personal struggles is pretty damn close to mind reading.
- EarthLaunch 4y ago
  I read an epic space opera [0] on Wattpad where the baddies are like this. > What if a thought could get you cancelled? What if a thought could get you killed? What if everyone could peer inside your mind and judge you? > Welcome to the galactic Torth Empire. Are you a thought leader? Or are you a slave? 0: https://www.wattpad.com/312043845-majority-sff-ooorahs-1-prologue-the-lion-within https://www.wattpad.com/312043845-majority-sff-ooorahs-1-pro...
- Tepix 4y ago
  Torture is possible yet here we are.
  - mdp2021 4y ago
    ...In a world in which trends are ghastly, also in the territories most advanced and old cultures. I witnessed some new latest attacks to Civilization and Dignity only in the last 24 hours (and they were also about intrusion). About torture: do check these pages: -- https://www.freedomfromtorture.org/news/where-does-torture-happen-around-the-world https://www.freedomfromtorture.org/news/where-does-torture-h... ¹ -- https://www.amnesty.org/en/latest/news/2015/06/torture-around-the-world/ https://www.amnesty.org/en/latest/news/2015/06/torture-aroun... > 141 – the number of countries in which Amnesty has reported on torture or other ill-treatment in the past five years. In some countries it’s a rarity, in others it’s widespread We hope that with «here we are» you did not mean "you personally do not see it in the street". -- ¹In which you will by the way immediately notice on the maps that India, 1.4 Bil ppl (current top populated country or reaching it), has not signed the Convention against Torture. In local legislation, it is punished as violence but «the offense attracts no particular relevance if the crime is committed by a police officer» (see http://www.humanrights.asia/tortures/torture-in-india/ http://www.humanrights.asia/tortures/torture-in-india/ ). Recent examples: 2020, Covid lockdowns: > By now, everyone has heard of the tragic deaths of P. Jayaraj and J. Benicks , a father-son duo in a small town in Thoothukudi. Jayaraj, 58, was arrested by the police following an altercation with them on keeping his son’s mobile phone shop open in violation of lockdown rules. After Benicks was also taken into custody, the two were mercilessly thrashed to death. // Being found guilty of the ‘offence’ of keeping a shop open during the lockdown [...] > The Tamil Nadu Police has acquired notoriety over the decades for employing torturous methods for law enforcement. During my tenure as Chief Justice of the Madras High Court, several cases in this regard were brought to the court. But this issue is not restricted to Tamil Nadu alone. Torture is, in fact, an integral part of police culture all over the country. Indeed, it would not be amiss to argue that this culture in India today is reminiscent of the brutality of the colonial police forces that we are so keen to forget [...] The data on torture show that it is not only an integral part of India’s policing culture; in some investigations (such as terror cases), it is treated as the centrepiece From https://www.thehindu.com/opinion/lead/indias-torture-culture-needs-to-end-now/article31973431.ece https://www.thehindu.com/opinion/lead/indias-torture-culture...
kazinator 4y ago
The images look like Stable Diffusion riffs based on an example image. Might it be that they are doing something like this: * presenting some images to a subject, converting them to some captured MRI signals. * training an A.I. image generation system on those same images, using the MRI signal as the associated text, instead of text. * then generating images using MRI patterns as prompts Still, scary what the method is able to infer.
- deleted 4y ago
  [deleted]
- lgats 4y ago
  > Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations. https://arxiv.org/pdf/2211.06956.pdf https://arxiv.org/pdf/2211.06956.pdf
basicplus2 4y ago
looks more like images/ combined images from an AI data base of previously collected images rather than just "reading the brain" outright
whimsicalism 4y ago
Wow, excited what this will mean for interfaces in the future.
galacticaactual 4y ago
All the people who celebrate Progress like this for Progress' sake never seem to be able to answer questions like "what will prevent the exploitation of this technology for authoritarian surveillance / capitalist profit / progressive thought policing." Your entire industry is built on ignoring that question.
- hamburglar 4y ago
  This fatalistic attitude leads to the sad conclusion that nobody should ever express new ideas for fear they might be misused. Or maybe you can express new ideas only after you have conclusively proven that they can’t be misused. So… never. Having an idea and suppressing it for fear of it being misused also doesn’t prevent anyone else from having the same idea. So what have you actually accomplished besides depriving yourself of your own ideas? Delaying the inevitable?
dqpb 4y ago
I want to record a dream.
jmugan 4y ago
This is pretty close to the joke blog post I wrote last April Fool's Day. Your thoughts are stored automatically in the cloud so you can share them with loved ones. https://deumbra.com/2022/04/introducing-thoughtlog-the-free-automatic-diary/ https://deumbra.com/2022/04/introducing-thoughtlog-the-free-...
- jmugan 4y ago
  And it's free. Supported by advertising. A few times a day you receive "sponsored thoughts."
graderjs 4y ago
Now we just need a way to encrypt our brain waves lol. I’m reminded of the book by Haruki Murakami: Hard boiled wonderland and the end of the world and the Keanu Reeves film, Johnny Mnemonic
roschdal 4y ago
Why? This has so many evil usecases.
sitkack 4y ago
Are there fMRI scans of people sleeping? Can we use this one people in a coma? What about non-human mammals? I feel like this is a hoax.
- aix1 4y ago
  There's existing literature on using fMRI on patients with a coma to detect conscious awareness. Here is one example (that references further papers): https://www.frontiersin.org/articles/10.3389/fnins.2019.00976/full https://www.frontiersin.org/articles/10.3389/fnins.2019.0097...
  - sitkack 4y ago
    This is amazing.
AussieWog93 4y ago
Did a PhD a few years back looking at the Applicaton of Machine Learning to Neurosignal Decoding (well, I started one anyway. Dropped out after a year for reasons the rest of this comment will explain). Turns out researchers in this field are terrible at actually turning brainwaves into actionable data, but brilliant at using statistical trickery + general misunderstanding about AI to bullshit their way into getting grants. Yes, there are a few actual exciting trials - like the work Shenoy did with the penetrative array and the robot arm, but the field as a whole is overwhelmingly bullshit. This research in particular is worse than bullshit - it's borderline fraud. There is no "reconstruction" taking place. They're using ML to map the response from a test stimulus to some interpolated response from 50 training samples, arbitrarily assigning new sample images to these points in space and then passing it off as mind reading. Same dumbfuck trick as the guy last year who pretended to be able to read what word people were thinking of using EEG, but with more layers of AI magic and crisp-looking images. I mean, seriously, look at those images. They've somehow managed to recreate them with more clarity than the human subject's actual memory/imagination possibly could. If there are any VCs or engineers here, please don't waste your time or money on this shit. If you do, I'll be sure to leave a rude comment on your HN article. :P
- mkasu 4y ago
  Yeah, I thought the same after seeing this. It's kind of a fun use-case of Diffusion models in this context, but as a scientific paper it seems too overselling. Well, it surely is the kind of clickbaity content to anticipate lots of retweets from. I only skimmed the paper, but from what I understood, it is essentially a diffusion model pre-trained on a handful classes. The brain information is then used to largely "pick which class to generate a random image from". The paper itself even picked the "better" examples. The supplemental materials show many more results, and many of them are just that, a randomly generated image of the same object class the person was seeing (or, the closest object class available in the training data). "Reconstruct" seems a pretty bad word choice. I think the results are presented in a way vastly overselling what they actually do. But that's a worrisome trend in most of AI research recently, unfortunately. (I have a PhD in a field of Applied Machine Learning. I work at a university in Computer Vision.)
- 4y ago
xqcgrek2 4y ago
If I understand the drawbacks to this correctly: 1) Each person's brain signals have to be trained separately -- there is no universal model 2) The set of trained images is small (e.g. 50) so five bits of information is enough to specify which image the person is thinking of... it is not really the AI imagining the image from scratch but an interpolated set of those 50 images diffused together. If my understanding is correct, the use case of this is very limited if the bits retrieved is really around five. Underwhelming.
ffssffss 4y ago
What is the meaning of this language in the "5. Results" section of the paper: Our main results are based on GOD which has no overlapping classes in the training and testing set. The training and testing were performed on the same subject, as individual differences remain a barrier when decoding at the group level [2, 16,21,32,34]. To compare with the literature, we report results from Subject 3 here and leave other subjects in the Appendix. What training and testing was performed on the same subject? Are they reading peoples minds or just overfitting a model on purpose? This papers seems to suggest an unbelievably implausible result, with absolutely stunning, society shattering implications, so I'm assuming it's either wrong or misleading, given how casually it's tossed on arxiv...
- AussieWog93 4y ago
  >Are they reading peoples minds or just overfitting a model on purpose? There's no question about it - they're overfitting a model on purpose.
  - hakuseki 4y ago
    They could be overfitting somehow, but I don't see how you get that from the quoted paragraph. Of course they need to use the same subject (i.e. human brain) for training and testing. Otherwise one would need to assume that different people's brains represent visual concepts in the same way.
hakuseki 4y ago
I'd really like to see the mouse (or other nonhuman animal) version of this experiment. In the human experiment we can see how the input and output represent the same categories of object. I think it would be really interesting to what extent a mouse's perceptions can be mapped onto human categories such as "fire truck" or "baseball game."

179 points by yboris 4y ago

This seems incredible! If we put aside questions on how to collect the data, does this not suggest that we would be able to "read minds" (intentionally left vague) with an FMRI?

terminal_d 4y ago
I think it will be similar to how diffusion-based image generation currently relies on what the model is trained on. This might work well for average folks, but for anyone sufficiently out of the norm / with expertise/mental hardware outside of the bounds of the trained model, the results will be subpar / useless.
- stephc_int13 4y ago
  No, I think the model would have to be trained for every single individual.
deleted 4y ago
[deleted]

belkarx 4y ago

Preprint of a similar model (textual not visual stimuli): https://www.biorxiv.org/content/10.1101/2022.09.29.509744v1 https://www.biorxiv.org/content/10.1101/2022.09.29.509744v1

yboris 4y ago

Shortcut to the GitHub repository: https://github.com/zjc062/mind-vis https://github.com/zjc062/mind-vis

mdp2021 4y ago

It is not clear if those systems can just decode "seen" and not also "imagined".

pxmpxm 4y ago
Yeah - clearly they have some dimensionality reduced classifier for what's in the underlying mri data, but using stable diffusion to project it just seems odd. "The MRI of your brain tells us you're looking at a building with a tower of some sort, so here's an artist's impression of a building with a tower"

BWStearns 4y ago

In today's edition of "Don't build the torment nexus".... Really cool tech but I'm very curious if/worried that this will spawn some half-working interrogation tools.

vbezhenar 4y ago
Google glasses will ensure that you're paying attention to the ads.
ReptileMan 4y ago
Better than needles under the nails in any case...

giantdude 4y ago

Slightly more invasive attempts were made over 100 years ago. https://www.smithsonianmag.com/smart-news/how-forensic-scientists-once-tried-see-dead-persons-last-sight-180959157/ https://www.smithsonianmag.com/smart-news/how-forensic-scien...

bigyikes 4y ago

Is this for real? How is this not the biggest story in the world right now? This is absolutely incredible. I wish I had something intelligent to say. Just wow. It’s literally mind reading. Imagine plugging this tech into Neuralink. The brain-machine bandwidth problem seems way more tractable in light of this result.

aquanext 4y ago
I don't think being able to scan this is completely new. Perhaps this specific technique is, but I've heard of this being done at least once previously in the past decade.
cma 4y ago
2013 https://blog.ted.com/reading-minds-with-a-brain-scanner-its-happening-mary-lou-jepsen-at-ted2013/ https://blog.ted.com/reading-minds-with-a-brain-scanner-its-...
- mdp2021 4y ago
  Mary Lou Jepsen. > That, of course, leads to very real concerns about privacy "The revenge of the tinfoil hat".

antegamisou 4y ago

For those of you that are overly concerned/excited about this keep in mind that fMRI scans are kind of looked down upon the neuroscience research community for the dubious misinterpretations from the findings (see dead Salmon fMRI study). It looks like PET imaging is what many are focusing their research into for better insight / “dynamic analysis” of the brain.

mikeiz404 4y ago
Dead Salmom fMRI Wired Article: https://www.wired.com/2009/09/fmrisalmon/ https://www.wired.com/2009/09/fmrisalmon/ Research Poster: http://www.prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://www.prefrontal.org/files/posters/Bennett-Salmon-2009....
ramraj07 4y ago
I know we haven’t proven fMRI is not truly measuring activity but have we proven it doesn’t even correlate?
- mattkrause 4y ago
  The signal measured by fMRI is definitely related to neural activity, but the relationship is indirect and sometimes subtly complicated. Neurons compute and communicate electrochemically by controlling how ions (charged particles) flow across their membranes. At "rest", the inside of a neuron is slightly negative (- 70 mV) relative to its environment. However, its surface is studded with channels that let different types of ions into/out of the cell as well as "receptors" that can open or close these channels. Some are opened by specific chemicals, which can bring the cell's membrane potential closer to zero (e.g., by admitting sodium ions). Once it gets above about -55 mV, other channels that are sensitive to voltage take over, and produce a wave of activity that shoots down the axon and eventually causes the release of chemicals that affect other neurons, muscles, etc. This is what actually does work in the brain, but it's not what fMRI measures. Instead, fMRI exploits the fact this process is energetically expensive: neurons need oxygen and sugar (glucose) to do all that work, and so neurons need to plumbed into the vascular system so those can be delivered. The signal fMRI measures is called the BOLD signal, for Blood Oxygenation Level Dependent. The magnetic properties of a blood molecule carrying oxygen are a bit different from a deoxygenated one, and thus they can be told apart with some clever physics and signal processing. Based on what I said above, you might expect that fMRI thus measures the reduction in oxygenation, but it actually turns out to be a lot more complicated than that. Levels do dip when nearby neurons are active, but only briefly. After that, they shoot up, above baseline levels: more blood--and more oxygen--are delivered to the formerly-active area. This is mostly what fMRI measures. The neuromuscular coupling seems to vary between brain states and maybe even brain areas; it's affected by different health conditions and drugs too. It's also limited by the structure of the vascular system. People occasionally find that veins appear to be heavily involved in cognitive tasks, sometimes even more than the brain itself, but that's just because they're delivering the blood. Despite all this, the mechanisms are getting better understood and you can certainly interpret a BOLD change as suggesting that something is happening in that particular region, even if it's not clear exactly how it's implemented.

zactato 4y ago

This is super cool, but it seems pretty far from "mind reading". It's not reading what the person's brain is interpreting the image as, but rather what they're currently seeing. I wonder how different these signals are. It's a pretty big difference since the only signal they're getting is what the eyes are currently seeing. Essentially they're using a $1million fMRI to poorly do the job that a $5 dollar camera could do. This is still super cool, but not quite our wildest scifi dreams.

IIAOPSW 4y ago
disclaimer, what I'm about to say is just vague recollection of content that had already been pre-digested into layperson science news. There's probably a ton of loss in this signal. From what I understand, memory cells literally reproduce the original brain signal just as if it were coming from outside stimulus. Like playing the brain waves off a tape recorder.
- badRNG 4y ago
  >From what I understand, memory cells literally reproduce the original brain signal just as if it were coming from outside stimulus. Like playing the brain waves off a tape recorder. I can't recall pictures of events that happened yesterday with anything resembling what it was like to be there and experience seeing the events in person. I can construct images of what things might've looked like, but the creative process of recalling that memory feels awfully different than experiencing events in real time.
- mattkrause 4y ago
  Eh....Nobody really knows how memory works. There is certainly some evidence for "replay", but it's generally thought to be involved in "writing" it into longer-term storage rather than the storage mechanism per se. OTOH, there are some cool results with Hopfield-like networks where applying part of a pattern (somehow) cues the brain (or network) to produce the rest of it.

sithadmin 4y ago

Some of these results seem to be picking up on general abstract concepts moreso actual visual cues. for example, one sample has bright colored wooden park benches as input, and some generic brown wood furniture as an output. Perhaps a strange artifact of training biases, but a little disturbing at a glance. Extracting current visual data out of one's brain is one thing, but being able to potentially identify abstract thoughts is a whole different level of invasiveness.

zwkrt 4y ago
So there’s this philosopher Daniel Dennett. In broad strokes, he’s generally liked by philosophically minded scientists and disliked by the philosophy community. His position is that we will slowly chip away at the magic of consciousness as we get better and better tools to scientifically explore the brain. The consciousness is not some magical force that enters us but just a giant bag of tricks that our brain plays. Furthermore, that the “hard problem of consciousness”, i.e. “what it is like” to be something is not immune to the scientific process. I’ve always been defferential to his viewpoint, but this is the first time his arguments have felt justified. It’s surprising but in hindsight makes sense to me that when you look at a specific representation of an object that your brain internally kind of “draws a picture” of an abstract representation of that kind of object. You can probably think of the location of 100 trees, but it’s pretty ridiculous to think that your brain actually has pictures of 100 trees in 100 different drawers. You look at a tree and your brain draws a kindergarten picture of a tree. Here lay the platonic forms? Cool stuff.
- goatlover 4y ago
  > The consciousness is not some magical force that enters us but just a giant bag of tricks that our brain plays. That's kind of a straw man of what Dennett's opponents argue. No philosopher seriously says that consciousness is a magical force entering the brain. Rather, they make arguments that materialism (physicalism) fails to account for consciousness, whatever that entails for metaphysics or epistemology. Some are naturalists, but not physicalists. Some are idealists. Some like David Chalmers propose a kind of property dualism. Others might be neutral monists or think we're cognitively closed to the solution. Or panpsychists were everything has a bit of consciousness. Materialism is a metaphysical assumption. We don't know the truth of what reality fundamentally is (it could be a simulation for all we know). We also don't know how to bridge the subjective/objective divide, as Thomas Nagel has argued. Another way of framing the argument is that the objective world is an abstraction across many subjective experiences, where consciousness is abstracted in favor of mathematical forms. That works well for describing/predicting patterns in nature we observe, including the brain, but then when you turn it around to explain the mind, it runs into difficulties, since the abstracted-away subjectivity has been left out. There's no need to muddy the waters with the word "magic". We don't know everything, and some things we might never know. We do know that we have experiences of color, sound, taste, smells, feels, dreams, inner dialog and what not. That's not magic, it's just how we experience being in the world, whatever the world and our experiences are. Here's a view opposing Dennett's from Jaron Lanier: http://www.jaronlanier.com/zombie.html http://www.jaronlanier.com/zombie.html

mrelectric 4y ago

I've seen research like this before. Does anyone have any other spicy links on this topic? Idea of recording dreams seems very interesting to me.

cma 4y ago
Mary Lou Jepsen gave a TED talk or something on it in 2013.
mattkrause 4y ago
Tons! Jack Gallant's group at UC Berkley has done a lot of interesting fMRI decoding. The approach in their earlier papers (e.g., Kay et al, 2008 [0]) is neat because it uses some knowledge of how early visual areas represent information, rather than just flinging a decoder at it. They later extended this to movies (Nishimoto et al., 2011 [1]), and have subsequently moved a bit away from sensory information to "semantic" decoding. Jack has a TED talk about this [2] and Kendrick Kay has written some News and Views/Perspective pieces [e.g., 3] that might be interesting too, as it links to other groups' work. I like their approach because it ties into neuroscience rather than just throwing ML at the wall and seeing what sticks, but there's also a huge cottage industry of "Blackbox ML" + MRI to do decoding. My sense is that this has moved slightly out of the limelight and into the stage where people incrementally improve it with newer ML techniques like GANs [4] and diffusion (this link), but it might just be me. For sleep specifically, you might be interested in Horikawa et al. (2013)[5], who could predict words associated with dream contents (e.g., "Car" but didn't try to reconstruct them). There's also a very short perspective on it [6]. [0] Kay, K., Naselaris, T., Prenger, R. et al. Identifying natural images from human brain activity. Nature 452, 352–355 (2008). https://doi.org/10.1038/nature06713 https://doi.org/10.1038/nature06713 [1] Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Current biology : CB, 21(19), 1641–1646. https://doi.org/10.1016/j.cub.2011.08.031 https://doi.org/10.1016/j.cub.2011.08.031 [2] https://www.youtube.com/watch?v=Ecvv-EvOj8M https://www.youtube.com/watch?v=Ecvv-EvOj8M [3] Kay, K. N., & Gallant, J. L. (2009). I can see what you see. Nature neuroscience, 12(3), 245. https://doi.org/10.1038/nn0309-245 https://doi.org/10.1038/nn0309-245 [4] Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., & van Gerven, M. A. J. (2018). Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage, 181, 775–785. https://doi.org/10.1016/j.neuroimage.2018.07.043 https://doi.org/10.1016/j.neuroimage.2018.07.043 [5] Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y. (2013). Neural decoding of visual imagery during sleep. Science (New York, N.Y.), 340(6132), 639–642. https://doi.org/10.1126/science.1234330 https://doi.org/10.1126/science.1234330 [6] https://www.nature.com/articles/nmeth.2504 https://www.nature.com/articles/nmeth.2504

stephc_int13 4y ago

This is not as advanced as some of you might think. You have to understand how diffusion models work to "reconstruct" images. You also have to understand that the model has to be trained to match pictures/brain scan for each individual. But this kind of approach can certainly be useful, especially with people with motor impairment after a stroke or accidental brain damage. Not sci-fi level yet.

DontchaKnowit 4y ago
For some reason having trouble getting the full study, anyway I'm no AI specialist so idk if reading it would answer my question, but : is this generalizeable? E.g. can they reconstruct and arbitrary visual stimulus, or just one that appeared as training data? If this is a case where they showed a patient an image of a giraffe 1000 times and mapped their brain, generated output,then backpropogated, etc and then now they can recreate a decent image of a giraffe when shown 1 of the images from training data : This is not that impressive. I mean it's cool and interesting but doesn't sound remotely ground breaking, because it's not actually mapping arbitrary inputs to arbitrary outputs. But if they can reproduce arbitrary visual stimuli this is pretty damn incredible, to me at least.
tomxor 4y ago
Yes, notice how much variance there is between the images. They are the same types of building, or similar sized animals, but the details and shapes can be wildly different. It's pretty clear what's happening is FMRI meaning -> stable diffusion image generation. Now this _is_ cool, but it's important to understand all that's happening is the natural language input to stable diffusion has been replaced with FMRI input. i.e the image is still generated by stable diffusion, not you, you are just inputting meaning by visualising a separate image, and that meaning is abstract enough to be coarsely learned through an FMRI... but ultimately the image is not produced by your brain, which will probably be more obvious to any of the test subjects.
- not2b 4y ago
  Yes, how do we know that visual rather than verbal information is even being processed? I can get a decent image from Stable Diffusion or Dall-E from a pure text prompt, and if that's added to a very basic scribble and we do img2img a much better result can be produced, but it might have little to do with what the person is actually seeing.

IIAOPSW 4y ago

Assuming these results hold, I wonder what the legal implications would be. In a criminal prosecution or investigation, can you force someone to just think about the question while stealing the image straight out of their skull? Even worse, what happens when its only 85% accurate but 100% convincing to juries. In the US at least, I think 4th amendment protections obviously apply (though with this bench, don't count your precedents before they hatch). Elsewhere in the world, even fairly liberal freedom loving jurisdictions, the question is still up in the air imo. That should give everyone pause. The information age began with a single question the answer to which is still unfolding: .-- .... .- - / .... .- ... / --. --- -.. / .-- .-. --- ..- --. .... - ..--..

antifa 4y ago
> can you force someone to just think about the question while stealing the image straight out of their skull? Can the person being mind read, if aware of the technology, force their mind to only output unrelated content (e.g. shrek fighting zombies) or vividly imagine the accuser committing heinous crimes?
mrozbarry 4y ago
There is absolutely no way this sort of data would ever hold up in court.
mathstuf 4y ago
I'm curious what they'd do with someone with aphantasia. Can they get images out of my head? I certainly can't…
- _ink_ 4y ago
  Exactly what I was thinking.
- sheerun 4y ago
  TIL I'm in the 0.8% of population. I thought it was a rule, not an exception

BurningFrog 4y ago

Something I occasionally worry about: When/if mind reading does become possible, thought privacy will be gone, and humanity enters a brand new era of surveillance.

necovek 4y ago
We can finally put those tinfoil hats to good use, though.
antegamisou 4y ago
In my slightly uneducated opinion, this will only be possible once there’s solid understanding of how the human brain works, which if you think about it (ha!) has much lot more exciting implications apart from mind reading, it could even be a step closer to or the resolution to AGI. Besides, for a large part of the 8 billions I would suppose Google search history about personal struggles is pretty damn close to mind reading.
EarthLaunch 4y ago
I read an epic space opera [0] on Wattpad where the baddies are like this. > What if a thought could get you cancelled? What if a thought could get you killed? What if everyone could peer inside your mind and judge you? > Welcome to the galactic Torth Empire. Are you a thought leader? Or are you a slave? 0: https://www.wattpad.com/312043845-majority-sff-ooorahs-1-prologue-the-lion-within https://www.wattpad.com/312043845-majority-sff-ooorahs-1-pro...
Tepix 4y ago
Torture is possible yet here we are.
- mdp2021 4y ago
  ...In a world in which trends are ghastly, also in the territories most advanced and old cultures. I witnessed some new latest attacks to Civilization and Dignity only in the last 24 hours (and they were also about intrusion). About torture: do check these pages: -- https://www.freedomfromtorture.org/news/where-does-torture-happen-around-the-world https://www.freedomfromtorture.org/news/where-does-torture-h... ¹ -- https://www.amnesty.org/en/latest/news/2015/06/torture-around-the-world/ https://www.amnesty.org/en/latest/news/2015/06/torture-aroun... > 141 – the number of countries in which Amnesty has reported on torture or other ill-treatment in the past five years. In some countries it’s a rarity, in others it’s widespread We hope that with «here we are» you did not mean "you personally do not see it in the street". -- ¹In which you will by the way immediately notice on the maps that India, 1.4 Bil ppl (current top populated country or reaching it), has not signed the Convention against Torture. In local legislation, it is punished as violence but «the offense attracts no particular relevance if the crime is committed by a police officer» (see http://www.humanrights.asia/tortures/torture-in-india/ http://www.humanrights.asia/tortures/torture-in-india/ ). Recent examples: 2020, Covid lockdowns: > By now, everyone has heard of the tragic deaths of P. Jayaraj and J. Benicks , a father-son duo in a small town in Thoothukudi. Jayaraj, 58, was arrested by the police following an altercation with them on keeping his son’s mobile phone shop open in violation of lockdown rules. After Benicks was also taken into custody, the two were mercilessly thrashed to death. // Being found guilty of the ‘offence’ of keeping a shop open during the lockdown [...] > The Tamil Nadu Police has acquired notoriety over the decades for employing torturous methods for law enforcement. During my tenure as Chief Justice of the Madras High Court, several cases in this regard were brought to the court. But this issue is not restricted to Tamil Nadu alone. Torture is, in fact, an integral part of police culture all over the country. Indeed, it would not be amiss to argue that this culture in India today is reminiscent of the brutality of the colonial police forces that we are so keen to forget [...] The data on torture show that it is not only an integral part of India’s policing culture; in some investigations (such as terror cases), it is treated as the centrepiece From https://www.thehindu.com/opinion/lead/indias-torture-culture-needs-to-end-now/article31973431.ece https://www.thehindu.com/opinion/lead/indias-torture-culture...

kazinator 4y ago

The images look like Stable Diffusion riffs based on an example image. Might it be that they are doing something like this: * presenting some images to a subject, converting them to some captured MRI signals. * training an A.I. image generation system on those same images, using the MRI signal as the associated text, instead of text. * then generating images using MRI patterns as prompts Still, scary what the method is able to infer.

deleted 4y ago
[deleted]
lgats 4y ago
> Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations. https://arxiv.org/pdf/2211.06956.pdf https://arxiv.org/pdf/2211.06956.pdf

basicplus2 4y ago

looks more like images/ combined images from an AI data base of previously collected images rather than just "reading the brain" outright

whimsicalism 4y ago

Wow, excited what this will mean for interfaces in the future.

galacticaactual 4y ago

All the people who celebrate Progress like this for Progress' sake never seem to be able to answer questions like "what will prevent the exploitation of this technology for authoritarian surveillance / capitalist profit / progressive thought policing." Your entire industry is built on ignoring that question.

hamburglar 4y ago
This fatalistic attitude leads to the sad conclusion that nobody should ever express new ideas for fear they might be misused. Or maybe you can express new ideas only after you have conclusively proven that they can’t be misused. So… never. Having an idea and suppressing it for fear of it being misused also doesn’t prevent anyone else from having the same idea. So what have you actually accomplished besides depriving yourself of your own ideas? Delaying the inevitable?

dqpb 4y ago

I want to record a dream.

jmugan 4y ago

This is pretty close to the joke blog post I wrote last April Fool's Day. Your thoughts are stored automatically in the cloud so you can share them with loved ones. https://deumbra.com/2022/04/introducing-thoughtlog-the-free-automatic-diary/ https://deumbra.com/2022/04/introducing-thoughtlog-the-free-...

jmugan 4y ago
And it's free. Supported by advertising. A few times a day you receive "sponsored thoughts."

graderjs 4y ago

Now we just need a way to encrypt our brain waves lol. I’m reminded of the book by Haruki Murakami: Hard boiled wonderland and the end of the world and the Keanu Reeves film, Johnny Mnemonic

roschdal 4y ago

Why? This has so many evil usecases.

sitkack 4y ago

Are there fMRI scans of people sleeping? Can we use this one people in a coma? What about non-human mammals? I feel like this is a hoax.

aix1 4y ago
There's existing literature on using fMRI on patients with a coma to detect conscious awareness. Here is one example (that references further papers): https://www.frontiersin.org/articles/10.3389/fnins.2019.00976/full https://www.frontiersin.org/articles/10.3389/fnins.2019.0097...
- sitkack 4y ago
  This is amazing.

AussieWog93 4y ago

Did a PhD a few years back looking at the Applicaton of Machine Learning to Neurosignal Decoding (well, I started one anyway. Dropped out after a year for reasons the rest of this comment will explain). Turns out researchers in this field are terrible at actually turning brainwaves into actionable data, but brilliant at using statistical trickery + general misunderstanding about AI to bullshit their way into getting grants. Yes, there are a few actual exciting trials - like the work Shenoy did with the penetrative array and the robot arm, but the field as a whole is overwhelmingly bullshit. This research in particular is worse than bullshit - it's borderline fraud. There is no "reconstruction" taking place. They're using ML to map the response from a test stimulus to some interpolated response from 50 training samples, arbitrarily assigning new sample images to these points in space and then passing it off as mind reading. Same dumbfuck trick as the guy last year who pretended to be able to read what word people were thinking of using EEG, but with more layers of AI magic and crisp-looking images. I mean, seriously, look at those images. They've somehow managed to recreate them with more clarity than the human subject's actual memory/imagination possibly could. If there are any VCs or engineers here, please don't waste your time or money on this shit. If you do, I'll be sure to leave a rude comment on your HN article. :P

mkasu 4y ago
Yeah, I thought the same after seeing this. It's kind of a fun use-case of Diffusion models in this context, but as a scientific paper it seems too overselling. Well, it surely is the kind of clickbaity content to anticipate lots of retweets from. I only skimmed the paper, but from what I understood, it is essentially a diffusion model pre-trained on a handful classes. The brain information is then used to largely "pick which class to generate a random image from". The paper itself even picked the "better" examples. The supplemental materials show many more results, and many of them are just that, a randomly generated image of the same object class the person was seeing (or, the closest object class available in the training data). "Reconstruct" seems a pretty bad word choice. I think the results are presented in a way vastly overselling what they actually do. But that's a worrisome trend in most of AI research recently, unfortunately. (I have a PhD in a field of Applied Machine Learning. I work at a university in Computer Vision.)
4y ago

xqcgrek2 4y ago

If I understand the drawbacks to this correctly: 1) Each person's brain signals have to be trained separately -- there is no universal model 2) The set of trained images is small (e.g. 50) so five bits of information is enough to specify which image the person is thinking of... it is not really the AI imagining the image from scratch but an interpolated set of those 50 images diffused together. If my understanding is correct, the use case of this is very limited if the bits retrieved is really around five. Underwhelming.

ffssffss 4y ago

What is the meaning of this language in the "5. Results" section of the paper: Our main results are based on GOD which has no overlapping classes in the training and testing set. The training and testing were performed on the same subject, as individual differences remain a barrier when decoding at the group level [2, 16,21,32,34]. To compare with the literature, we report results from Subject 3 here and leave other subjects in the Appendix. What training and testing was performed on the same subject? Are they reading peoples minds or just overfitting a model on purpose? This papers seems to suggest an unbelievably implausible result, with absolutely stunning, society shattering implications, so I'm assuming it's either wrong or misleading, given how casually it's tossed on arxiv...

AussieWog93 4y ago
>Are they reading peoples minds or just overfitting a model on purpose? There's no question about it - they're overfitting a model on purpose.
- hakuseki 4y ago
  They could be overfitting somehow, but I don't see how you get that from the quoted paragraph. Of course they need to use the same subject (i.e. human brain) for training and testing. Otherwise one would need to assume that different people's brains represent visual concepts in the same way.

hakuseki 4y ago

I'd really like to see the mouse (or other nonhuman animal) version of this experiment. In the human experiment we can see how the input and output represent the same categories of object. I think it would be really interesting to what extent a mouse's perceptions can be mapped onto human categories such as "fire truck" or "baseball game."