3 ms·

I've done some work with contrastive learning, and I see pros and cons. You are somehow trading off direct supervision for some other assumptions about invarian

I've done some work with contrastive learning, and I see pros and cons. You are somehow trading off direct supervision for some other assumptions about invariances in the data. This often works very well. I have also seen cases where contrastive learning fails to latch on to the features you want it to. And you end up effectively trying to do some feature engineering / preprocessing to highlight what you want the model to notice. So bottom line, I think CL is a specific instance of finding a simple rule or pattern that we can use to label select features that work for many tasks, but that's pretty much all it is. I think it's good progress to be able to find some of these simple core rules about what ML models are really noticing.

jszymborski 5y ago
I think our ability to consistently train good CL models will get a lot better if/when we better understand how to disentangle representations. There's already been great progress, but the better we're able to create meaningful latent spaces, the better we're going to get at CL (maybe that's a self-evident statement :P ).
- version_five 5y ago
  I think I agree, but do you think that in some way getting a more meaningful latent space will just take us back to classical kinds of models (my background is image processing so that's what I'm thinking of). Like if we can have a semantically relevant latent space, it is definitely a win, but it also sort of is a step back towards rules about what we expect to see, vs letting training figure it out. (And, the semantically relevant features may still themselves be found opaquely). I'm not sure how to think about all this, but I worry about a "turtles all the way down" situation where some higher level understanding is gained at the expense of lower level understanding.
  - jszymborski 5y ago
    Yah, I get that and it's a big shrug from me as to how far is too far. For example, "entangled" neurons have been shown to be a necessity to some degree when applying an informational bottleneck, and that they just tend to encode features that don't overlap very much. I'm currently of the mind that "encouraging" latent spaces to abide by certain expected distributions is often a nice way to do this (of course depending on the distribution). Historically, we've been limited to the rather vague and often inappropriate Gaussian (i.e. VAEs), but with adversarial VAEs and MMD, we're increasingly able to impose arbitrary distributions. Now of course, I'd agree that too narrow a distribution is probably not the right direction. But doing subtle things like half-continuous, half-discrete codes, or modelling rare entities with Poisson distributed codes could be interesting.
  - srean 5y ago
    I don't see anything wrong with that. Its about capturing and expressing structure. My view is that some structures are easy to specify using soft rules, others by connection architecture, yet another by functional priors, invariants etc etc. In the past we have been looking for one representation/specification language that works for all. I think that's not the most efficient approach even if these modes of representations have uniform approximation property. "Can represent" does not translate to "shortest representation" or "learnable representation" or "easy to manipulate". These properties are very useful and cannot be ignored.

jszymborski 5y ago

I think our ability to consistently train good CL models will get a lot better if/when we better understand how to disentangle representations. There's already been great progress, but the better we're able to create meaningful latent spaces, the better we're going to get at CL (maybe that's a self-evident statement :P ).

version_five 5y ago
I think I agree, but do you think that in some way getting a more meaningful latent space will just take us back to classical kinds of models (my background is image processing so that's what I'm thinking of). Like if we can have a semantically relevant latent space, it is definitely a win, but it also sort of is a step back towards rules about what we expect to see, vs letting training figure it out. (And, the semantically relevant features may still themselves be found opaquely). I'm not sure how to think about all this, but I worry about a "turtles all the way down" situation where some higher level understanding is gained at the expense of lower level understanding.
- jszymborski 5y ago
  Yah, I get that and it's a big shrug from me as to how far is too far. For example, "entangled" neurons have been shown to be a necessity to some degree when applying an informational bottleneck, and that they just tend to encode features that don't overlap very much. I'm currently of the mind that "encouraging" latent spaces to abide by certain expected distributions is often a nice way to do this (of course depending on the distribution). Historically, we've been limited to the rather vague and often inappropriate Gaussian (i.e. VAEs), but with adversarial VAEs and MMD, we're increasingly able to impose arbitrary distributions. Now of course, I'd agree that too narrow a distribution is probably not the right direction. But doing subtle things like half-continuous, half-discrete codes, or modelling rare entities with Poisson distributed codes could be interesting.
- srean 5y ago
  I don't see anything wrong with that. Its about capturing and expressing structure. My view is that some structures are easy to specify using soft rules, others by connection architecture, yet another by functional priors, invariants etc etc. In the past we have been looking for one representation/specification language that works for all. I think that's not the most efficient approach even if these modes of representations have uniform approximation property. "Can represent" does not translate to "shortest representation" or "learnable representation" or "easy to manipulate". These properties are very useful and cannot be ignored.