5 ms·

This is not as advanced as some of you might think. You have to understand how diffusion models work to "reconstruct" images. You also have to understand that

This is not as advanced as some of you might think. You have to understand how diffusion models work to "reconstruct" images. You also have to understand that the model has to be trained to match pictures/brain scan for each individual. But this kind of approach can certainly be useful, especially with people with motor impairment after a stroke or accidental brain damage. Not sci-fi level yet.

DontchaKnowit 4y ago
For some reason having trouble getting the full study, anyway I'm no AI specialist so idk if reading it would answer my question, but : is this generalizeable? E.g. can they reconstruct and arbitrary visual stimulus, or just one that appeared as training data? If this is a case where they showed a patient an image of a giraffe 1000 times and mapped their brain, generated output,then backpropogated, etc and then now they can recreate a decent image of a giraffe when shown 1 of the images from training data : This is not that impressive. I mean it's cool and interesting but doesn't sound remotely ground breaking, because it's not actually mapping arbitrary inputs to arbitrary outputs. But if they can reproduce arbitrary visual stimuli this is pretty damn incredible, to me at least.
tomxor 4y ago
Yes, notice how much variance there is between the images. They are the same types of building, or similar sized animals, but the details and shapes can be wildly different. It's pretty clear what's happening is FMRI meaning -> stable diffusion image generation. Now this _is_ cool, but it's important to understand all that's happening is the natural language input to stable diffusion has been replaced with FMRI input. i.e the image is still generated by stable diffusion, not you, you are just inputting meaning by visualising a separate image, and that meaning is abstract enough to be coarsely learned through an FMRI... but ultimately the image is not produced by your brain, which will probably be more obvious to any of the test subjects.
- not2b 4y ago
  Yes, how do we know that visual rather than verbal information is even being processed? I can get a decent image from Stable Diffusion or Dall-E from a pure text prompt, and if that's added to a very basic scribble and we do img2img a much better result can be produced, but it might have little to do with what the person is actually seeing.
  - tomxor 4y ago
    It's a good point, but actually a fairly simple one to test: try the same model on two different test subjects with no overlapping languages (preferably with very different languages that don't share many constructs).
zzzeek 4y ago
> You also have to understand that the model has to be trained to match pictures/brain scan for each individual. this was the impression I had and I think some of the commenters here should note this, assuming I am understanding correctly: this technique only works on a training set that comes from the same person. that is, you can't take model you trained on one person and apply it to someone else, their fMRI patterns would be completely different. Since it's just lining up pre-canned photos to various blotches and patterns seen in the scan. That's what it looked like to me, anyway, if I'm getting it wrong please clarify.

DontchaKnowit 4y ago

For some reason having trouble getting the full study, anyway I'm no AI specialist so idk if reading it would answer my question, but : is this generalizeable? E.g. can they reconstruct and arbitrary visual stimulus, or just one that appeared as training data? If this is a case where they showed a patient an image of a giraffe 1000 times and mapped their brain, generated output,then backpropogated, etc and then now they can recreate a decent image of a giraffe when shown 1 of the images from training data : This is not that impressive. I mean it's cool and interesting but doesn't sound remotely ground breaking, because it's not actually mapping arbitrary inputs to arbitrary outputs. But if they can reproduce arbitrary visual stimuli this is pretty damn incredible, to me at least.

tomxor 4y ago

Yes, notice how much variance there is between the images. They are the same types of building, or similar sized animals, but the details and shapes can be wildly different. It's pretty clear what's happening is FMRI meaning -> stable diffusion image generation. Now this _is_ cool, but it's important to understand all that's happening is the natural language input to stable diffusion has been replaced with FMRI input. i.e the image is still generated by stable diffusion, not you, you are just inputting meaning by visualising a separate image, and that meaning is abstract enough to be coarsely learned through an FMRI... but ultimately the image is not produced by your brain, which will probably be more obvious to any of the test subjects.

not2b 4y ago
Yes, how do we know that visual rather than verbal information is even being processed? I can get a decent image from Stable Diffusion or Dall-E from a pure text prompt, and if that's added to a very basic scribble and we do img2img a much better result can be produced, but it might have little to do with what the person is actually seeing.
- tomxor 4y ago
  It's a good point, but actually a fairly simple one to test: try the same model on two different test subjects with no overlapping languages (preferably with very different languages that don't share many constructs).

zzzeek 4y ago

> You also have to understand that the model has to be trained to match pictures/brain scan for each individual. this was the impression I had and I think some of the commenters here should note this, assuming I am understanding correctly: this technique only works on a training set that comes from the same person. that is, you can't take model you trained on one person and apply it to someone else, their fMRI patterns would be completely different. Since it's just lining up pre-canned photos to various blotches and patterns seen in the scan. That's what it looked like to me, anyway, if I'm getting it wrong please clarify.