7 ms·

Meta AI: Fist high-performance self-supervised algorithm for multiple modalities

89 points by leirbagarc 5y ago

silence48 5y ago
I don't believe it's the actual first but this is pretty awesome. too bad its facebook :/
- leirbagarc 5y ago
  I think the big achievement is how it surpassed in performance previous models for each individual modality.
  - gillesjacobs 5y ago
    That's not true for NLU at least. It is on par with 2018's RoBERTa on GLUE, many larger and advanced language models came after. It is still great work though, a robust masking representation architecture that works across modalities.
    - kuu 5y ago
      They mention that on the article: > We apply data2vec separately to speech, images and text and it outperformed the previous best single-purpose algorithms for computer vision and speech and it is competitive on NLP tasks.
macleginn 5y ago
Do they need a pre-trained modality-specific model for each modality to train this?
- WithinReason 5y ago
  "Our work does not perform multimodal training but aims to unifiy (sic) the learning objective for self-supervised learning in different modalities."
- lopuhin 5y ago
  No, they don't need a pre-trained model, they start from scratch. Also it's only the approach which is common between modalities, they train them separately.
drenei 5y ago
dang/mods: The title here has a small typo: it misspells algorithm.
- unwind 5y ago
  Before doing that, it also misspells "first".
  - abhaynayar 5y ago
    Also "algorithn".
    - leirbagarc 5y ago
      Guys, sorry for the typos. The original title was too long. I needed to replace "First" to "1st" (HN switch back automatically, lol) and try to abbreviate.
      - Kiro 5y ago
        Still doesn't explain "algorithn".
        dr_zoidberg 5y ago
        N is right next to M in QWERTY keyboards. Typing fast, you may hit one instead of the other.
        Kiro 5y ago
        Why would they type it manually at all? My understanding is that they copypasted the title, was prompted it was too long, tried to short First to 1st which HN automatically expanded upon which they removed some words instead. So how did algorithm become "algorithn"? This is not some random nitpicking. This is a great mystery worthy of its own detective TV show so I don't appreciate the downvotes. All my friends are extremely puzzled by this whole situation.
WithinReason 5y ago
Basically they cut out a part of the input and make the network predict the missing part. (edit: they actually predict the average of all features). This works for images, audio, text. This produces high quality feature representations for data which can be used to build specialised networks on. The two main tricks are: 1. Do the cutout in feature space, not the original input space. (edit: cutout is actually in input space) 2. The above would likely just collapse the features to 0, so they use the same network that does the reconstruction to produce the features (!). In their own words: "We first encode a masked version of the training sample (model in student mode) and then construct training targets by encoding the unmasked version of the input sample with the same model but when parameterized as an exponentially moving average of the model weights (model in teacher mode)"
- lopuhin 5y ago
  That's not quite how I understood it > Do the cutout in feature space, not the original input space. I think they do the cutout in the original input space, based on examples they show, e.g. they mask parts of text and grey out parts of an image. > and make the network predict the missing part I think predict the latent representation (as you say in (2)), but not of the missing part, but of the whole corrupted sample, and require it to be close to latent representation of the original input done by the teacher.
  - l-lousy 5y ago
    So they’re doing both masking for the inputs and knowledge distillation? Is it the combination of these two methods that’s novel?
  - WithinReason 5y ago
    You're right, they cut in the original space and average-pool the features before an L1/L2 loss
- dfgfhjkjlkhgjfh 5y ago
  I believe that your interpretation is not correct. Based on my brief reading of the paper, the model contains 1) some known architecture for embedding the modality, and 2) the feature reconstruction transformer network, the two being trained at the same time. If I am not mistaken, the masking occurs in the input modality, not the feature-space, even though it is the feature-space that is used for the reconstruction task. Regarding how the feature space is kept uncollapsed, it seems like a hyperparameter-tweaking (ie unsolved?) problem; quoting the paper: " Representation collapse. A common issue with algorithms which create and predict their own targets is representation collapse. This occurs when the model produces very similar representations for all masked segments making the problem trivial to solve. Different strategies have been proposed to address this issue, e.g., contrastive models such as wav2vec 2.0 (Baevski et al., 2020b) use the same target representation both as a positive and a negative example, preventing collapse. Algorithms such as BYOL (Grill et al., 2020) do not optimize the teacher parameters to minimize the loss. VicReg (Bardes et al., 2021) adds an explicit loss encouraging variance among different representations. In our experiments we found that collapse is most likely to happen in the following scenarios: First, the learning rate is too large or the learning rate warmup is too short which can often be solved by tuning the respective hyper-parameters. Second, the EMA decay rate is too low which leads to student model collapse which is propagated to the teacher due to parameter tracking. This can be addressed by carefully tuning τ0, τe and τn. Third, we found collapse to be more likely for modalities where adjacent targets are very correlated and where longer spans need to be masked, such as for speech. We address this by either explicitly penalizing the lack of variance (Bardes et al., 2021), or by promoting variance through normalizing target representations over the current sequence or batch (Grill et al., 2020). The former worked well for small models but is less reliable for larger models and it also requires tuning additional hyper-parameters. In contrast, we found applying instance or batch normalization before or after averaging targets to work well while being simpler. For models where targets are less correlated such as for vision and NLP, momentum tracking is sufficient to prevent representation collapse. "
alarak 5y ago
I don't understand how this is different from BYOL? I'd appreciate it if someone could give a small explanation.
- soraki_soladead 5y ago
  > Similar to our work, both BYOL (Grill et al., 2020) and DINO (Caron et al., 2021) regress neural network representations of a momentum encoder, but our work differs in that it uses a masked prediction task and we regress multiple neural network layer representations instead of just the top layer which we find to be more effective. Moreover, we demonstrate that our approach works for multiple modalities. From the related works section of the paper.
Bombthecat 5y ago
Crazy, and people think AI isn't moving forward anymore..
- sirk390 5y ago
  Crazy, but who thinks AI is not moving forward? I don't think anyone on HN.
  - plutonorm 5y ago
    Lots of people think it's essentially a dead end as it's missing some kind of 'secret sauce' that the brain has and which we are just too dumb to figure out.
    - zcw100 5y ago
      It's a lack of appreciation for how complicated the brain really is. The "secret sauce" is "it's really frickin' complex and we don't understand a fraction of it". It's like comparing an abacus to the latest AMD or Intel CPU and saying, "The abacus is a dead end because it's obviously missing that "secret sauce" that this magical computing oracle is doing"
  - T-A 5y ago
    https://zbigatron.com/artificial-intelligence-is-slowing-down/ https://zbigatron.com/artificial-intelligence-is-slowing-dow...
    - nothis 5y ago
      Sounds like the limit is computation speed, not conceptual. That should solve itself over the years.
  - rytill 5y ago
    Many people on HN think that, for instance, GPT-3 and that family of works does not represent any real advancement and continually disparage and poke holes in the output of the model. These threads are upvoted to the front page occasionally.
earth2mars 5y ago
Can someone explain this like I am 5. What are the use cases when it says works on images, text etc? Why is this a big deal? What's the human input here? And what output to expect? From what I understand, human validation (supervision) is not happening while algorithm is training on data. Is that right? Will this be open to the public via standard ML frameworks or proprietary?
- hahanbyul 5y ago
  It has already been public on their repository, fairseq.
  - kordlessagain 5y ago
    Except not the vision part, which is what they were discussing. Also, I don't see any way to run the examples as the code is missing those files: https://github.com/pytorch/fairseq/tree/main/examples/data2vec https://github.com/pytorch/fairseq/tree/main/examples/data2v... The code is what we're interested in here, not "hold onto your papers" talk that tells us how to be excited about it.
- iamstupidsimple 5y ago
  I imagine a multimodal platform like Facebook (with people using photos and text together in some way) would find a single model valuable.
- zcw100 5y ago
  The benefits are similar to when you consolidate tasks with a lot of similarities. Advancements in one area can be immediately applied to another area. You can consolidate your effort into a single point. Training a network is increasingly cost prohibitive, into the millions of dollars, and this would allow you to consolidate these expenses. EDIT: Looks like they still need to train for each modality so no benefit from that here. I'd imagine on the flip side the potential problems would be when gains in one modality are offset by a decrease in performance in another or you are prevented from trying new things because of the choices made to support a unified architecture.
- 5y ago
macilacilove 5y ago
It seems that they pass everything through an autoencoder first, and a different network tries to predict from a partially masked input the "correct" autoencoder latent space representation of the unmasked input. If it works, the decoder of the autoencoder can generate(guess) the unmasked data from the latent space.
algo_trader 5y ago
Is there progress on general structured/relational/graphed data modalities? In practice, you spend time and expertise to reform the data into previously-known-to-work form. FWIW, our datasets are huge, with dense data/noise ratio.
- zmgsabst 5y ago
  Can you clarify a little bit on what you mean by your question? One area of research is extracting an effective type theory from data that is viewed as a semantic model, eg sensor data of a phenomenon would lead to a type theory describing it. You’re essentially taking a TDA persistent homology/covering and reinterpreting that through the lens of homotopy type theory to “decompile” your data. There’s some early results, eg connecting convolutions and type division. But that’s overall at really early stages of research.
  - enchiridion 5y ago
    Any papers cover this?
    - zmgsabst 5y ago
      None published; I’m working on a white paper about type division in the abstract case this quarter, once I finish up the second one on shape algebras. Type division is like convolution from ML, which is why we can recognize “shapes of shapes” and undo the product structure. (And arguably, another avenue towards arriving at the manifold hypothesis.) I’m currently working on the “easy” direction of encoding the type statements to matrices, with the hope most steps are reversible. (So far, so good.) Still rough white paper: https://www.zmgsabstract.com/whitepapers/shapes-as-digital-images https://www.zmgsabstract.com/whitepapers/shapes-as-digital-i... GitHub repo for encoding type theory models, still VERY early: https://github.com/zmgsabstract/mathengine https://github.com/zmgsabstract/mathengine
  - algo_trader 5y ago
    > effective type theory from data that is viewed as a semantic model, Well, much simpler stuff. You have a logistic dataset of objects/GPS/time. Fine. Now you add historic truck data which is location time series. If is not obvious how you can learn the delivery times between 2 items. You need human expertise, and design a way to extract usable routes, and also solve multi-hop routes, and then maybe you can learn typical speed between 2 items. It is doable. But it not with a generic "multi-modal architecture".
deleted 5y ago
[deleted]
asix66 5y ago
https://archive.is/Cm81W https://archive.is/Cm81W
EZ-Cheeze 5y ago
Now do it with matrices GPT style Sheeeeeeeeeeeeeeeeeeit

89 points by leirbagarc 5y ago

silence48 5y ago

I don't believe it's the actual first but this is pretty awesome. too bad its facebook :/

leirbagarc 5y ago
I think the big achievement is how it surpassed in performance previous models for each individual modality.
- gillesjacobs 5y ago
  That's not true for NLU at least. It is on par with 2018's RoBERTa on GLUE, many larger and advanced language models came after. It is still great work though, a robust masking representation architecture that works across modalities.
  - kuu 5y ago
    They mention that on the article: > We apply data2vec separately to speech, images and text and it outperformed the previous best single-purpose algorithms for computer vision and speech and it is competitive on NLP tasks.

macleginn 5y ago

Do they need a pre-trained modality-specific model for each modality to train this?

WithinReason 5y ago
"Our work does not perform multimodal training but aims to unifiy (sic) the learning objective for self-supervised learning in different modalities."
lopuhin 5y ago
No, they don't need a pre-trained model, they start from scratch. Also it's only the approach which is common between modalities, they train them separately.

drenei 5y ago

dang/mods: The title here has a small typo: it misspells algorithm.

unwind 5y ago
Before doing that, it also misspells "first".
- abhaynayar 5y ago
  Also "algorithn".
  - leirbagarc 5y ago
    Guys, sorry for the typos. The original title was too long. I needed to replace "First" to "1st" (HN switch back automatically, lol) and try to abbreviate.
    - Kiro 5y ago
      Still doesn't explain "algorithn".
      - dr_zoidberg 5y ago
        N is right next to M in QWERTY keyboards. Typing fast, you may hit one instead of the other.
        Kiro 5y ago
        Why would they type it manually at all? My understanding is that they copypasted the title, was prompted it was too long, tried to short First to 1st which HN automatically expanded upon which they removed some words instead. So how did algorithm become "algorithn"? This is not some random nitpicking. This is a great mystery worthy of its own detective TV show so I don't appreciate the downvotes. All my friends are extremely puzzled by this whole situation.

WithinReason 5y ago

Basically they cut out a part of the input and make the network predict the missing part. (edit: they actually predict the average of all features). This works for images, audio, text. This produces high quality feature representations for data which can be used to build specialised networks on. The two main tricks are: 1. Do the cutout in feature space, not the original input space. (edit: cutout is actually in input space) 2. The above would likely just collapse the features to 0, so they use the same network that does the reconstruction to produce the features (!). In their own words: "We first encode a masked version of the training sample (model in student mode) and then construct training targets by encoding the unmasked version of the input sample with the same model but when parameterized as an exponentially moving average of the model weights (model in teacher mode)"

lopuhin 5y ago
That's not quite how I understood it > Do the cutout in feature space, not the original input space. I think they do the cutout in the original input space, based on examples they show, e.g. they mask parts of text and grey out parts of an image. > and make the network predict the missing part I think predict the latent representation (as you say in (2)), but not of the missing part, but of the whole corrupted sample, and require it to be close to latent representation of the original input done by the teacher.
- l-lousy 5y ago
  So they’re doing both masking for the inputs and knowledge distillation? Is it the combination of these two methods that’s novel?
- WithinReason 5y ago
  You're right, they cut in the original space and average-pool the features before an L1/L2 loss
dfgfhjkjlkhgjfh 5y ago
I believe that your interpretation is not correct. Based on my brief reading of the paper, the model contains 1) some known architecture for embedding the modality, and 2) the feature reconstruction transformer network, the two being trained at the same time. If I am not mistaken, the masking occurs in the input modality, not the feature-space, even though it is the feature-space that is used for the reconstruction task. Regarding how the feature space is kept uncollapsed, it seems like a hyperparameter-tweaking (ie unsolved?) problem; quoting the paper: " Representation collapse. A common issue with algorithms which create and predict their own targets is representation collapse. This occurs when the model produces very similar representations for all masked segments making the problem trivial to solve. Different strategies have been proposed to address this issue, e.g., contrastive models such as wav2vec 2.0 (Baevski et al., 2020b) use the same target representation both as a positive and a negative example, preventing collapse. Algorithms such as BYOL (Grill et al., 2020) do not optimize the teacher parameters to minimize the loss. VicReg (Bardes et al., 2021) adds an explicit loss encouraging variance among different representations. In our experiments we found that collapse is most likely to happen in the following scenarios: First, the learning rate is too large or the learning rate warmup is too short which can often be solved by tuning the respective hyper-parameters. Second, the EMA decay rate is too low which leads to student model collapse which is propagated to the teacher due to parameter tracking. This can be addressed by carefully tuning τ0, τe and τn. Third, we found collapse to be more likely for modalities where adjacent targets are very correlated and where longer spans need to be masked, such as for speech. We address this by either explicitly penalizing the lack of variance (Bardes et al., 2021), or by promoting variance through normalizing target representations over the current sequence or batch (Grill et al., 2020). The former worked well for small models but is less reliable for larger models and it also requires tuning additional hyper-parameters. In contrast, we found applying instance or batch normalization before or after averaging targets to work well while being simpler. For models where targets are less correlated such as for vision and NLP, momentum tracking is sufficient to prevent representation collapse. "

alarak 5y ago

I don't understand how this is different from BYOL? I'd appreciate it if someone could give a small explanation.

soraki_soladead 5y ago
> Similar to our work, both BYOL (Grill et al., 2020) and DINO (Caron et al., 2021) regress neural network representations of a momentum encoder, but our work differs in that it uses a masked prediction task and we regress multiple neural network layer representations instead of just the top layer which we find to be more effective. Moreover, we demonstrate that our approach works for multiple modalities. From the related works section of the paper.

Bombthecat 5y ago

Crazy, and people think AI isn't moving forward anymore..

sirk390 5y ago
Crazy, but who thinks AI is not moving forward? I don't think anyone on HN.
- plutonorm 5y ago
  Lots of people think it's essentially a dead end as it's missing some kind of 'secret sauce' that the brain has and which we are just too dumb to figure out.
  - zcw100 5y ago
    It's a lack of appreciation for how complicated the brain really is. The "secret sauce" is "it's really frickin' complex and we don't understand a fraction of it". It's like comparing an abacus to the latest AMD or Intel CPU and saying, "The abacus is a dead end because it's obviously missing that "secret sauce" that this magical computing oracle is doing"
- T-A 5y ago
  https://zbigatron.com/artificial-intelligence-is-slowing-down/ https://zbigatron.com/artificial-intelligence-is-slowing-dow...
  - nothis 5y ago
    Sounds like the limit is computation speed, not conceptual. That should solve itself over the years.
- rytill 5y ago
  Many people on HN think that, for instance, GPT-3 and that family of works does not represent any real advancement and continually disparage and poke holes in the output of the model. These threads are upvoted to the front page occasionally.

earth2mars 5y ago

Can someone explain this like I am 5. What are the use cases when it says works on images, text etc? Why is this a big deal? What's the human input here? And what output to expect? From what I understand, human validation (supervision) is not happening while algorithm is training on data. Is that right? Will this be open to the public via standard ML frameworks or proprietary?

hahanbyul 5y ago
It has already been public on their repository, fairseq.
- kordlessagain 5y ago
  Except not the vision part, which is what they were discussing. Also, I don't see any way to run the examples as the code is missing those files: https://github.com/pytorch/fairseq/tree/main/examples/data2vec https://github.com/pytorch/fairseq/tree/main/examples/data2v... The code is what we're interested in here, not "hold onto your papers" talk that tells us how to be excited about it.
iamstupidsimple 5y ago
I imagine a multimodal platform like Facebook (with people using photos and text together in some way) would find a single model valuable.
zcw100 5y ago
The benefits are similar to when you consolidate tasks with a lot of similarities. Advancements in one area can be immediately applied to another area. You can consolidate your effort into a single point. Training a network is increasingly cost prohibitive, into the millions of dollars, and this would allow you to consolidate these expenses. EDIT: Looks like they still need to train for each modality so no benefit from that here. I'd imagine on the flip side the potential problems would be when gains in one modality are offset by a decrease in performance in another or you are prevented from trying new things because of the choices made to support a unified architecture.
5y ago

macilacilove 5y ago

It seems that they pass everything through an autoencoder first, and a different network tries to predict from a partially masked input the "correct" autoencoder latent space representation of the unmasked input. If it works, the decoder of the autoencoder can generate(guess) the unmasked data from the latent space.

algo_trader 5y ago

Is there progress on general structured/relational/graphed data modalities? In practice, you spend time and expertise to reform the data into previously-known-to-work form. FWIW, our datasets are huge, with dense data/noise ratio.

zmgsabst 5y ago
Can you clarify a little bit on what you mean by your question? One area of research is extracting an effective type theory from data that is viewed as a semantic model, eg sensor data of a phenomenon would lead to a type theory describing it. You’re essentially taking a TDA persistent homology/covering and reinterpreting that through the lens of homotopy type theory to “decompile” your data. There’s some early results, eg connecting convolutions and type division. But that’s overall at really early stages of research.
- enchiridion 5y ago
  Any papers cover this?
  - zmgsabst 5y ago
    None published; I’m working on a white paper about type division in the abstract case this quarter, once I finish up the second one on shape algebras. Type division is like convolution from ML, which is why we can recognize “shapes of shapes” and undo the product structure. (And arguably, another avenue towards arriving at the manifold hypothesis.) I’m currently working on the “easy” direction of encoding the type statements to matrices, with the hope most steps are reversible. (So far, so good.) Still rough white paper: https://www.zmgsabstract.com/whitepapers/shapes-as-digital-images https://www.zmgsabstract.com/whitepapers/shapes-as-digital-i... GitHub repo for encoding type theory models, still VERY early: https://github.com/zmgsabstract/mathengine https://github.com/zmgsabstract/mathengine
- algo_trader 5y ago
  > effective type theory from data that is viewed as a semantic model, Well, much simpler stuff. You have a logistic dataset of objects/GPS/time. Fine. Now you add historic truck data which is location time series. If is not obvious how you can learn the delivery times between 2 items. You need human expertise, and design a way to extract usable routes, and also solve multi-hop routes, and then maybe you can learn typical speed between 2 items. It is doable. But it not with a generic "multi-modal architecture".

deleted 5y ago

[deleted]

asix66 5y ago

https://archive.is/Cm81W https://archive.is/Cm81W

EZ-Cheeze 5y ago

Now do it with matrices GPT style Sheeeeeeeeeeeeeeeeeeit