3 ms·

Yes there was. However this is a different paper, describing a different method, applied to a different dataset, with different results. As the abstract says,

• satvikpendem 3y ago
  If it's still retrieving an image and not reconstructing it, if the dataset is large enough that's decently fine, but this is generally not how diffusion models work in general and I'd have expected the model to map the fMRI data to a wholly new image.
  • jph00 3y ago
    Please read the paper. Or at least the blog post. It's really quite readable. They explain that they've done both retrieval and reconstruction, and have lots of pictures showing examples of each. https://medarc-ai.github.io/mindeye/ https://medarc-ai.github.io/mindeye/
    • deleted 3y ago
      [deleted]
  • RC_ITR 3y ago
    If you can retrieve an image using a latent vector, it’s trivial to reconstruct it (decently well) with a diffusion model.
  • rocqua 3y ago
    They tested themselves both on retrieval and reconstruction.
• RC_ITR 3y ago
  > To achieve the goals of retrieval and reconstruction with a single model trained end-to-end, we adopt a novel approach of using two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). What you can think of contrastive learning as is: two separate models that take different inputs and make vectors of the same length as outputs. This is achieved by training both models on pairs of training data (in this case fMRI images and observed images). What the LAION-5B work shows is that they did a good enough job of this training that the models are really good at creating similar vectors for nearly any image and fMRI pair. Then, they make a prior model which basically says “our fMRI vectors are essentially image vectors with an arbitrary amount of randomness in them (representing the difference between the contrastive learning models). Let’s train a model to learn to remove that randomness, then we have image vectors.” So yes, this is an impressive result at first glance and not some overfitting trick. It’s also sort of bread and butter at this point (replace fMRI with “text” and that’s just what Stable Diffusion is). They’ll be lots of these sort of results coming out soon.
  • atom_101 3y ago
    This is mostly correct, except that there is only one model. This model takes an fMRI and predicts 2 outputs. The first is specialized for retrieval and the second can be fed into a diffusion model to reconstruct images. You can see the comparison in performance between LAION-5B retrieval and actual reconstructions in the paper. When retrieving from a large enough database like LAION-5B, we can get images that are quite similar to the seen images in terms of high level content, but not so similar in low-level details (relative position of objects, colors, texture, etc). Reconstruction with diffusion models does much better in terms of low-level metrics.
    • RC_ITR 3y ago
      How is contrastive learning done with one model, exactly? I agree only one is used in inference, but two are needed for training (otherwise how do you calculate a meaningful loss function?). Notice in the original CLIP paper, there's an image encoder and a text encoder, even though only the text encoder is used during inference. [0] [0] https://arxiv.org/pdf/2103.00020.pdf https://arxiv.org/pdf/2103.00020.pdf