11 ms·
Deep image reconstruction from human brain activity (2017)
- hardmaru 8y agoMore information on this thread: https://twitter.com/ykamit/status/1039672289413844993 https://twitter.com/ykamit/status/1039672289413844993
- NetOpWibby 8y agoThanks for this!
- lucidrains 8y agoWhen we build neural networks to decode our own neural networks...
- mycall 8y agoDeep^2 learning
- a-dub 8y agoneuroscientists have been using "ai" for basically ever. except to them it's not "ai," it's just mathematical tools for data analysis.
- lucidrains 8y agoNever mentioned AI. Simply noting that the mathematical tool used here is inspired by the brain.
- transpy 8y agoDo you remember that project that mapped where words are "stored" in the brain? I don't have it right now, but it was something like that. They used a neural approach to map the word vectors in the brain.
- jawarner 8y agoThat sounds like the brain atlas created with subjects listening to stories in an fMRI machine. Here's a Guardian article about it: https://www.theguardian.com/science/2016/apr/27/brain-atlas-showing-how-words-are-organised-neuroscience https://www.theguardian.com/science/2016/apr/27/brain-atlas-... And a link to the paper: https://www.nature.com/articles/nature17637 https://www.nature.com/articles/nature17637
- trashtester 8y agoWe already build neural networks to predict how our own neural networks react to input (marketing, ie facebook). The obvious next step is to combine that with the findings in the article. Ie: Create a VR (or AR) headset that can both present an audio-visual experience to the user, WHILE having enough sensors to accurately measure the users brain state (and other bodily states, such as heart rate, blood pressure, etc). That will place a game developer in a position to predict what events a game can create that would create the greatest possible engagement in a given gamer. They can time rewards, challenges, esthetical or sexual stimuli in the way that generates the greatest possible control of the player's future behaviour. In the hands of the greatest data scientists, this will allow them to create a hithertdo unprecedented level of addiction, which can then be monetetized through subscriptions, micro transactions, embedded adverticements or just by selling the data. In the hands of governments, this can be used to pacify any population, drive behaviour in desired directions, or create support (even demand) for given policies. In the hands of parents, it can greatly help them teach their children whatever they want to teach them. (Be it morality, religion, self-discipline, or whatever) In the hands of a great psychologist, it would be able to cure most mental disorders less severe than outright psychosis. How will we face this? Will anyone outside the elite be able to prevent this being used against them?
- randomdrake 8y agoThe code and data for these amazing results have been generously made available here: https://github.com/KamitaniLab/DeepImageReconstruction https://github.com/KamitaniLab/DeepImageReconstruction The code is well-commented Python and a pleasure to read.
- AlexCoventry 8y agoVery impressive! I was quite skeptical until I read your comment. Now I'm just a little skeptical.
- youwishboye 8y agoawesome! now I just need an fMRI machine...
- Elv13 8y agoNext: All you need to do to make the next billion $ movie is your imagination, literally (and a 100m$ helium cooled MRI machine, and 25$ worth of AWS credits).
- deleted 8y ago[deleted]
- mattkrause 8y agoMRI is crazy expensive, but you only really need $4M for the scanner, so doable for indie rom-come as well as blockbusters!
- snaky 8y agoMRI market is very small and regulated as of today. Wider adoption of reasonably simpler devices would change everything, just recall how crazy expensive computers were in 1950s. Among useful applications I would imagine, for example, the fast image-based information retrieval - e.g. useful for recommendation systems (in other words - Netflix could invest in a 'imagine anything and we find you a movie like that').
- mattkrause 8y agoMaybe, but I think if it happens at all, it's going to use some other form of neuroimaging. MRI needs a lot of power, a shielded room, strong magnetic fields (which are both annoying and dangerous). Many of these are physical constraints rather than just...where the market happens to be.
- trashtester 8y agoThe critical component is your imagination. If your imagination is at that level, you are likely to be successful in Holywood,even today. But I think the greatest utility of this tech is not in the movie industry, but in the gaming industry. Imagine a photo realistic version of sims where your family and friends know EXACTLY how to react to maximize your sense of meaning, happiness, enjoyment in precisely the way that is needed in order to prevent you from leaving the game?
- dreepDeam 8y agoWow, I'm calling a hoax. They produce images of dubious value, and for whatever reason, contaminate the results by mutating them with the Deep Dream stylization. This is as bad as the dead salmon results. EDIT: You can flag all you want, but look at this playback of the reconstruction: https://video.twimg.com/tweet_video/DSrWxhSVQAE2Xix.mp4 https://video.twimg.com/tweet_video/DSrWxhSVQAE2Xix.mp4 There's nothing useful happening here.
- LeoPanthera 8y agoThat's exactly what they say they're doing. That's not "contamination" - that's the whole point.
- dreepDeam 8y agoAnd that's exactly why this is garbage research. Point deep dream at an interference pattern of pure noise, after training it on a set of target images. Don't be surprised when it "finds" the image, pulled from raw static that contained no signal.
- emileokada 8y agoNot convinced this is as junk as you claim it to be. From the abstract: > While our model was solely trained with natural images, our method successfully generalized the reconstruction to artificial shapes, indicating that our model indeed reconstructs or generates images from brain activity, not simply matches to exemplars. If I am reading that correctly it is reconstructing images that the subjects had not seen before.
- sdinsn 8y agoThe code and data is here: https://github.com/KamitaniLab/DeepImageReconstruction https://github.com/KamitaniLab/DeepImageReconstruction Not a hoax. Your comparison to Deep Dream is quite apt, for good reason that you clearly don't realize- Deep Dream is an attempt to "enhance patterns in images via algorithmic pareidolia"- Pareidolia being a "psychological phenomenon in which the mind responds to a stimulus". So yes, this project is affected by a real life "deep dream".
- xamuel 8y agoReminds me of this project from 2011: https://www.youtube.com/watch?v=nsjDnYxJ0bo https://www.youtube.com/watch?v=nsjDnYxJ0bo If I were a billionaire, one of the projects I'd do would be to apply this sort of technology at scale, upon volunteers who were witness to poorly recorded historic events. WWII, Beatles concerts, JFK assassination, 9/11... all of them have been recorded from angles/perspectives that exist only in peoples' memories, and the longer we wait, the more of those memories will be lost.
- JHonaker 8y agoYou’re vastly overestimating the human brain’s ability to accurately recall first hand accounts. It’s actually kind of shocking how bad eyewitnesses are at remembering details.
- JoeAltmaier 8y agoIsn't it a Russian saying - "He lies like an eyewitness"
- sdinsn 8y agoFor those who don't see it, the full paper is linked on the right side (PDF): https://www.biorxiv.org/content/biorxiv/early/2017/12/28/240317.full.pdf https://www.biorxiv.org/content/biorxiv/early/2017/12/28/240...
- gamegoblin 8y agoAcknowledging that I know nothing about psychology or neuroscience... It's interesting that they were not able to extract good images from the test when they just asked the subjects to imagine the pictures. I speculate that is because the vast majority of people are able to recognize images of things, but not produce those images. For a simple example, see this article about people trying to draw a bicycle. Before clicking, try to draw a 2D representation of a bicycle -- the wheels, the frame, pedals, handlebars, etc. You'd be surprised how difficult it is. https://road.cc/content/blog/90885-science-cycology-can-you-draw-bicycle https://road.cc/content/blog/90885-science-cycology-can-you-... I would be very interested to see them repeat the experiment on e.g. artists who specialize in photorealism, or asking an expert on a very particular thing to imagine that particular thing (e.g. ask a violinist to imagine a violin). I wonder if that would yield more recognizable results than asking a random grad student to visualize a lion's face.
- PinkMilkshake 8y ago> It's interesting that they were not able to extract good images from the test when they just asked the subjects to imagine the pictures. I speculate that is because the vast majority of people are able to recognize images of things, but not produce those images. It seems reasonable to me. To use a computer as an analogy, the set of signals produced by a webcam recording a video to a hard drive are not going to be the same as the ones of the video playing from the hard drive on to the screen.
- NegativeLatency 8y agoOr another way: the data in a frame buffer is not the same as the data in a png file.
- jfoutz 8y agoFeynman wrote about this in one of his books. it's really hard to understand how someone else understands something. but you can say, well imagine it's orange and spherical and kind of hairy, and they'll have a picture in their mind you can kinda work with.
- lwansbrough 8y agoHmm. Do different humans produce relatively the same brain images for a given scene? If not, I’d imagine training this would be a real PITA.
- deleted 8y ago[deleted]
- fipple 8y agoWhy a PITA? Just show a known movie to the person while scanning.
- AlexCoventry 8y agofMRI data is very noisy, and that's reflected in the generated images. They're not at all recognizable, unless they're sampled from a very small class like simple geometric shapes. I doubt it's possible to do much better, with fMRI.
- mistermann 8y agoI was wondering the same thing, hopefully someone in the comments will know more about it.
- daodedickinson 8y agoBut how do I ice a data angel?
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- drcode 8y agoThis looks awesome, but the methodology and experimental setup look pretty complicated and just on that basis alone it seems like there'd be some risk of bias or overfitting to creep into the analysis... I'm not criticizing the researchers or the experimental design, it's just a danger in any complicated study like this. Bottom line, I'm definitely interested in this paper, but want to wait and see if others can replicate the results before getting overly excited.
- DoctorOetker 8y agoDoes the 50GB of data include the MRI measurements from both during training and testing?
- m3kw9 8y agoWouldn’t that training set be valuable!
- userbinator 8y agoThis is impressive, and certainly technology like this will probably find lots of very good uses, but I can't help imagining(!) dystopian societies where "thoughtcrime" can actually be monitored and "made actionable". A world in which "you have nothing to hide" turns into "you have nothing you can hide" is immensely disturbing.
- jeppz 8y agoWhen most people are connected to the cloud we will no longer need public security cameras, there will be someone on site automatically uploading what they saw.
- amelius 8y agoGoogle already knows what you will be thinking about in the next minutes; no fMRI needed. I bet they will soon replace the search bar by just a button.
- Agnosco 8y agoIt would be interesting to do this experiment with people who had high abilities in drawing lifelike pieces.
- icc97 8y agoFrom the paper: > We used the Caffe implementation (Jia et al., 2014) of the VGG19 deep neural network (DNN) model (Simonyan & Zisserman, 2015; available from https://github.com/BVLC/caffe/wiki/Model https://github.com/BVLC/caffe/wiki/Model - Zoo) . All visual images were resized to 224 × 224 pixels to compute outputs by the VGG19 model . The VGG19 model consisted of a total of sixteen convolutional layers and three fully connected layers. They used a VGG-19 network for the feature detection, so in my basic understanding the results of this could already be improved by switching to a ResNet.
- hardmaru 8y agoNot necessarily. While ResNets are optimized for the task of achieving the best image classification accuracy, it is no guarantee that the features extracted from it will transfer well to other tasks such as this one, especially when the task is to generate images. In fact, it has been shown that pre-trained VGG networks are a lot more useful than pre-trained ResNet for other tasks other than image classification, such as style transfer.
- icc97 8y agoVery interesting. For my curiosity I did some searching on VGG vs ResNet. * ResNets were better at feature extraction for image clustering [0] * One of the trade-offs of ResNets seems to be their relative complexity to VGG [1] * It surprises me that ResNets aren't significantly faster to train based on the large reduction of FLOPs (from the ResNet paper VGG-19 had 19.6 billion FLOPs vs ResNet-34 with 3.6 billion FLOPs) - I think people just train deeper ResNets e.g. ResNet-50 [0]: https://medium.com/@franky07724_57962/using-keras-pre-trained-models-for-feature-extraction-in-image-clustering-a142c6cdf5b1 https://medium.com/@franky07724_57962/using-keras-pre-traine... [1]: https://www.reddit.com/r/MachineLearning/comments/6e6mlf/d_is_vgg_common_in_newer_research_or_is_resnet/ https://www.reddit.com/r/MachineLearning/comments/6e6mlf/d_i...
- hardmaru 8y agoYou might enjoy this recent article on Distill: https://distill.pub/2018/differentiable-parameterizations/#section-styletransfer https://distill.pub/2018/differentiable-parameterizations/#s... They discuss VGG vs non-VGG architectures in the context of style transfer in Section 2, which was interesting to me.
- deleted 8y ago[deleted]
- DanielleMolloy 8y agoBlog post that has some info on the new GAN-based mind reading ideas from last year (including this one): https://www.mindcodec.com/using-gans-brain-reading/ https://www.mindcodec.com/using-gans-brain-reading/
- amelius 8y agoQuestion: does this setup allow to decode images from the brains of different people than the one(s) it was trained for? How well does that work compared to using the same person?
- amelius 8y agoAlso interesting to know how performance differs between e.g. twins, people in the same family, people of different cultures, occupations, etc.
- dontreact 8y agoIf you actually look at the images they are not very impressive especially if you know about retinotopy in the visual cortex. This means that there is a mapping between coordinates on the retina and coordinates in visual cortex. I'm not convinced this is doing much other than picking up on this long established fact to produce very vaguely similar images. Seriously look at how different the input image and the generated image are! I guess it makes sense that HN is skeptical about results more closely related to the community's expertise but it's a pretty crazy leap from being able to reconstruct something vaguely similar to what a person is currently seeing vs. trying to read someone's mind. I still have seen very little evidence that such a thing could at all be possible with FMRI. The spatial and temporal resolution are just far too low. The more everyone is impressed with results like this the more we delay the hard work of developing tools that actually have a shot at doing something like that.