17 ms·
High-res image reconstruction with latent diffusion models from human brain
- dheera 4y agoI wonder how well this would work with wearable brainwave detectors rather than MRI, seeing as MRI isn't really something I could have at home.
- deleted 4y ago[deleted]
- jejeyyy77 4y agoany idea what the best consumer brainwave detectors are on the market rn?
- mrtranscendence 4y agoNeed answer fast?
- jejeyyy77 4y ago[flagged]
- oth001 4y agoOpenBCI is one. Not cheap from what I've seen.
- babblingfish 4y agoBy brainwave detector I am going to assume you mean an EEG. An EEG measures electrical activity at the surface of the brain. A fMRI shows the activity of individual neurons of the entire brain in real time. It's sort of an apples to oranges comparison given the tools measure different things at vastly different resolutions.
- Madmallard 4y agoCouldn't we train an AI with FMRI or EEG with like billions of samples of people thinking and describing what they're thinking about and have it gradually train some level of accuracy?
- gus_massa 4y agoIn case someone miss it, there is a link to more info https://sites.google.com/view/stablediffusion-with-brain/ https://sites.google.com/view/stablediffusion-with-brain/ and to the preprint https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2 https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2
- rvnx 4y agoCreepy and cool at the same time. It goes into the bucket of things that are not ethically right, same ways as implanting chips to read monkeys brains. But technically interesting and well-executed.
- not-my-account 4y agoWhy is it unethical to put chips in monkey brains?
- Tomis02 4y agoHow did you get consent to put the chip in?
- xattt 4y agoYou ask for consent through the brain-computer interface. /s
- rvnx 4y agoConclusion from scientists: "If monkey drinks the smoothie, then it means he wants a surgery."
- deleted 4y ago[deleted]
- elil17 4y agoSound's like you're opposed to all animal research, not specifically brain-computer interface research. We also don't ask a monkey's consent before doing any other sort of experiment on it.
- deleted 4y ago[deleted]
- Tomis02 4y ago
- Aransentin 4y agoI immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2770 images, minus 982 of those used for validation. I posit that the system did not actually read any pictures from the brains, but simply overfitted all the training images into the network itself. For example, if one looks at a picture of a teddy bear, you'd get an overfitted picture of another teddy bear from the training dataset instead. The best evidence for this is a picture(1) from page 6 of the paper. Look at the second row. The building generated by 'mind reading' subject 2 and 4 look strikingly similar, but not very similar to the ground truth! From manually combing through the training dataset, I found a picture of a building that does look like that, and by scaling it down and cropping it exactly in the middle, it overlays rather closely(2) on the output that was ostensibly generated for an unrelated image. If so, at most they found that looking at similar subjects light up similar regions of the brain, putting Stable Diffusion on top of it serves no purpose. At worst it's entirely cherry-picked coincidences. 1. https://i.imgur.com/ILCD2Mu.png https://i.imgur.com/ILCD2Mu.png 2. https://i.imgur.com/ftMlGq8.png https://i.imgur.com/ftMlGq8.png
- kdma 4y agoGood find, when I read it I called bullshit but I got lost trying to understand the diagrams. Another gotcha is the semantic decoder, they are just looping the model on itself "A cozy teddy bear" + fMRI random input => A teddy bear!!!
- SubiculumCode 4y agoI feel like you might be moving the goal posts here a bit. Getting a reconstruction that is a bear, even if not the same bear, is impressive enough to be noteworthy.
- xmonkee 4y agoI think the point is that it's not a reconstruction. It's more like recognizing which letter of a thousand-letter alphabet is shown to the human after decoding their brain waves. Still impressive, but not really as impressive as visual reconstruction.
- ninesnines 4y agoI am suspicious of these results; if we blast a high frequency visual stimulus of a couple of letters and do quite a lot of post processing we can sometimes get a visual cortex map of those particular letters. However, these paper examples are very complex images and I’m very doubtful of the results - aransentin above made a couple of very valid points
- SubiculumCode 4y agoI mean in 2011, they were already able to do some basic reconstruction: see https://www.youtube.com/watch?v=nsjDnYxJ0bo https://www.youtube.com/watch?v=nsjDnYxJ0bo I could imagine improvements since then,especially with advances in image networks.
- 2bitencryption 4y agoAre any of the example images novel, i.e. new to the model? Or is the model only reconstructing images it has already seen before? Either way, if I'm understanding right, it's very impressive. If the only input to the model (after training) is a fMRI reading, and from that it can reconstruct an image, at the very least that shows it can strongly correlate brain patterns back to the original image. It'd be even cooler (and scarier?) if it works for novel images. I wonder what the output would look like for an image the model had never seen before? Would a person looking at a clock produce a roughly clock-like image, or would it be noise? All the usual skepticism to these models applies, of course. They are very good at hallucinating, and we are very good at applying our own meaning to their hallucinations.
- andai 4y agoThere was a video many years ago (early 2010s?) demoing a similar technology, which would overlay and blend many images on top of each other to make a fuzzy image approximating what was actually being viewed. Edit: found it! https://youtu.be/nsjDnYxJ0bo https://youtu.be/nsjDnYxJ0bo
- drzoltar 4y agoMy understanding is that we won’t get a “mind reader” model out of this, because visual stimulus vs your imagination happen in separate parts of the brain. In other words we won’t be reading the minds of suspected criminals anytime soon. Maybe someone with neurology experience can chime in here? Is it even theoretically possible to see what’s happening in the imagination?
- meindnoch 4y agoI have a hunch that any sort of mind-reading machine would have to be tailored uniquely to the individual you want to probe. The internal neural representations likely develop uniquely for each individual.
- soulofmischief 4y agoWe'll just train a model on training models.
- wongarsu 4y agoMaybe that's actually feasible, in the sense that you don't have to train a complete new model but can spit out a personalized model based on a calibration sequence of ten images.
- soulofmischief 4y agoYep! This process is known as fine-tuning.
- wongarsu 4y agoIn the best (worst?) case the method generalizes well, and you could just replace the training set of fMRI scans of people viewing images with fMRI scans of people asked to recall images they were shown previously, or fMRI scans of people told to imagine a scene based on a verbal description. It's rarely that easy though
- giantrobot 4y ago
- _448 4y agoSo this is like mind reading?
- donohoe 4y agoAs people and groups increasingly move this direction do we think about vectors for abuse in 10, 20 or 50+ years? The human mind is considered the only place where we have true privacy. All these efforts are taking that away. At this rate all notions of privacy will soon be dead.
- andai 4y agoThere was a guy at MIT about ten years ago (edit: 2018! Woah) who made a headset that would read electrical impulses from your face. Apparently when people think in words, the same nerves fire as when they speak, just at a lower activation level. Using those signals it is possible to reconstruct the words being thought. I'm surprised it didn't seem to go anywhere. Edit: found it https://youtu.be/RuUSc53Xpeg https://youtu.be/RuUSc53Xpeg
- judge2020 4y ago> I'm surprised it didn't seem to go anywhere. At least not publicly.
- danem 4y agoThere have been some corporate research groups that have tried to take this approach further, and they all have more or less failed as far as I know.
- nottathrowaway3 4y ago[dead]
- Filligree 4y agoPeople only think in words right before they say something, so I'm not sure how big a deal this is. I guess they'd be able to predict what I'm writing half a second before I write it? Would be useful if I lost the ability to write or speak, for whatever reason.
- 4y ago
- Lutzb 4y agoReminds me of this paper [1] from 2011. See it in action in [2] 1. https://www.cell.com/current-biology/fulltext/S0960-9822(11)00937-7?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0960982211009377%3Fshowall%3Dtrue https://www.cell.com/current-biology/fulltext/S0960-9822(11)... 2. https://www.youtube.com/watch?v=nsjDnYxJ0bo https://www.youtube.com/watch?v=nsjDnYxJ0bo Edit: Just realized the paper above is also from Shinji Nishimoto
- rvz 4y agoAnother small step into creating a worse dystopia than the one we are already living in. Please continue. /s Governments, three letter agencies and the like would be absolutely excited to see this. The future that no-one has asked for.
- bitL 4y agoCan't wait to this becoming one of individual performance metrics recording all brain states all the time (video/audio/etc.) and be a part of regular performance reviews...
- chrstphrknwtn 4y agoI don't see anything "high-res" about the reconstructed images.
- 00F_ 4y agohere we see, basically, a potential feedback loop. AI tools advance brain science -- more advanced brain science can then inform progress in AI. this is why the situation is dangerous: because people dont think about these feedback loops. people see AI and they move the goalposts and rationalize by saying that "cutting edge AI is still short of AGI so its ok." but most normal people dont think about how AI can be used to create AI or how AI could be used to revolutionize all kinds of fields that then plug back into AI. this is a very dangerous, non-linear space. its not the first non-linear space we have traversed but its certainly the least linear space we have ever entered into and it is the highest stakes humanity has ever or will ever deal with. even if this is just another bullshit article, im just making a point related to it. people need to be worried about this. for the first time in history, lots of people are now creeped out by AI. but they arent taking action or demanding change. we need regulation, grass-roots efforts to stop AI. even if the only way humanity could abort AI as a concept, or delay it for a significant amount of time, was to return to the iron age, and it certainly isnt the only way, it would be unambiguously worth it, in every way and from every angle. AI requires large compute. what we are doing now was impossible just 20 years ago. if not 20 then 30. you cant manufacture that kind of compute in your garage. global regulation would take care of it no problem. at the very least it would buy us an enormous amount of time that we could use to figure something else out. people always say that some hold-out country would defy global regulations. they wouldnt defy NATO, let alone a super-global coalition. and the idea of such a group or NATO enforcing compute regulations is not far-fetched whatsoever because the emergence of AGI or even advanced non-AGI goes against the interests of literally every human being. there is no group of humans that benefit from that ultimately. the problem is simply waking people up to this plain fact.
- TechBro8615 4y agoWhen in human history have we ever been able to stop technological advancement? > we need regulation No, we don't. Regulation doesn't stop technological progress - it puts it in the hands of an elite few. And besides, there are 130+ regulatory jurisdictions. For example, the US government doesn't fund human cloning research, but that doesn't mean China won't fund it. Or perhaps you'd also like a one world government that can jail anyone doing wrongthink on their GPU? Personally, I hope we get AGI (in the most Kurzweillian sense) as soon as possible. It will lead to a cambrian explosion of advancements across all fields of science. This is our best chance of cracking the secrets of the universe and answering fundamental questions, like whether FTL interstellar travel is truly impossible, or whether aging is really irreversible. Imagine an intelligence unencumbered by the "technical debt" we've accrued over centuries of building our scientific model of the world. AGI could simulate infinitely many novel paths through the "tech tree" of human history, replaying our scientific discoveries and trying different assumptions. What if we had 12 fingers and mathematics started from a base-12 system? What if we could see in infrared? We would have followed entirely different scientific paths; AGI will be able to find what we missed.
- ACV001 4y agoThis is big thing. Although this particular paper is not big thing, the many related quoted studies, set a trend.
- deleted 4y ago[deleted]
- samuelzxu 4y agoThere's also this paper with very similar methodology called Mind-Vis, and also accepted to CVPR 2023. https://mind-vis.github.io/ https://mind-vis.github.io/
- smusamashah 4y agoThere was this research where they reconstructed human face images from monkey brain scan. https://www.electronicproducts.com/scientists-reconstruct-images-of-human-faces-extracted-from-monkey-brain-scans/ https://www.electronicproducts.com/scientists-reconstruct-im... What's astonishing here is the quality of reconstruction. But I have not seen this research referenced a lot. Does someone how /why the reconstruction from monkey brain looks so perfect while we don't have anything close from human brain? Edit: better images here https://www.newscientist.com/article/2133343-photos-of-human-faces-reassembled-from-monkeys-brain-signals/ https://www.newscientist.com/article/2133343-photos-of-human...
- crispyambulance 4y agoIn 1990, there was a train-wreck Wim Wenders movie that I loved and still love called "Until the End of the World". It was about a scientist (played by Max Von Sydow) who created a machine that could record someone's dreams or visual experiences directly from the brain and play them back even to a blind person. https://youtu.be/gilzgbdk300?t=442 https://youtu.be/gilzgbdk300?t=442 Anyways, the images that were depicted in this work of fiction shot in 1990 about "the future" of 2000, had a very interesting look to them-- kind of distorted and dreamy like the images in the paper. Are the images in the paper just a case of overfitting? ¯\_(ツ)_/¯ but it still makes me giddy remembering the Wim Wenders film.
- dustractor 4y agoSuch a great soundtrack too! I rewatched it last week just for the jams. Also, for those into the glitch art genre, the dream sequences were WAY ahead of their time.
- exclipy 4y agoIn 2004, I wrote a short story about exactly this in high school. Using neural networks to "mind read" visual images from an fMRI scan of a brain. I thought it was farfetched, but look where we are now!
- 752963e64 4y ago[dead]
- politician 4y agoShow HN: Human Diffusion Hi everybody! We’re Joe and Ahmed and super thrilled to be launching Human Diffusion today! We’ve built an exciting new image generation system that supports economies in developing nations. Our product leverages the latent creativity of humanity by directly fitting employees with fMRI rigs and presenting them with text inquiries through our API (JavaScript SDK available, Python soon!). Unlike competing alternatives we preserve human jobs in an era of AI supremacy. I’d like to address rumors that our facilities amount to slaving brains to machines. This is a gross misunderstanding of the benefits we offer to our staff - they are family. Our 18 hour shifts are finely calibrated based on feedback collected through our API, and any suggestion of exploitation is flatly untrue. Send us an email (satire@humandiffusion.com) to get early access.
- convolvatron 4y agovery curious about the little 'semantic model' at the bottom of the brain. does anyone know how that gets constructed and how it gets fed into the results?
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- lazy_moderator1 4y agonot the first time something like this ended up on HN https://news.ycombinator.com/item?id=33632337 https://news.ycombinator.com/item?id=33632337
- fretime 4y agoI'm looking forward to it. When will the code be released, Thanks