3 ms·

You can run triplet learning as unsupervised representation learning. Take let aug(X) be an augmentation of X (noise, translation, etc), and form the triple (X,

You can run triplet learning as unsupervised representation learning. Take let aug(X) be an augmentation of X (noise, translation, etc), and form the triple (X, aug(X), Y). Let f be your neural network (or whatevs). Then reduce the distance d(f(X), f(aug(X))) while increasing d(f(X), f(Y)). No labels needed.

dostres 8y ago
That’s supervised of a different sort. Also, it may end up learning the effects of the noise which may not reflect real data.
nabla9 8y ago
Has anyone done that and what is the advantage over modern autoencoders? I would really like to know, because I'm currently working with autoencoders. Relying only on synthetic samples can introduce all kinds of biased denoising and require endless tweaking. When you tie the weights and convolution kernels in the decoder to those of the encoder, you get relatively fast learning without excess number of variables and encoder-decoder can be much deeper.
- sdenton4 8y ago
  Here's the OG triplet metric learning paper: https://arxiv.org/abs/1412.6622 https://arxiv.org/abs/1412.6622 Their strategy is for labeled data; just take X and X' with the same labels, and Y from a different label. And here's an example of unlabeled triplet learning: https://arxiv.org/abs/1711.02209 https://arxiv.org/abs/1711.02209 In this case, aug(X) might be a slightly time or pitch shifted example. So whatever the label is for X, it will be the same as the label for aug(X). The advantage here is that as a framework it plays nicely with 'real' problems. Representation learning is rarely the actual problem; it's certainly not something an end-user cares about! If you are ultimately working on a classification problem and have a huge amount of unlabeled data and a tiny bit of labeled data, you can train the metric with both the Huff scheme (X and X' with same label) and the unsupervised scheme (X and aug(X)) simultaneously, and take advantage of the large dataset. A more general argument for metric learning is that, again, reconstruction error proooobably isn't what you actually care about. (Sure, you're sharing encoder and decoder weights, but why do you even need to decode?! How many parameters are being wasted on the need for the weight matrices to support decoding?) If clustering is what you want, metric learning gets at things being close or far from one another more directly.
  - nabla9 8y ago
    Thank's for the info. I look into it.

dostres 8y ago

That’s supervised of a different sort. Also, it may end up learning the effects of the noise which may not reflect real data.

nabla9 8y ago

Has anyone done that and what is the advantage over modern autoencoders? I would really like to know, because I'm currently working with autoencoders. Relying only on synthetic samples can introduce all kinds of biased denoising and require endless tweaking. When you tie the weights and convolution kernels in the decoder to those of the encoder, you get relatively fast learning without excess number of variables and encoder-decoder can be much deeper.

sdenton4 8y ago
Here's the OG triplet metric learning paper: https://arxiv.org/abs/1412.6622 https://arxiv.org/abs/1412.6622 Their strategy is for labeled data; just take X and X' with the same labels, and Y from a different label. And here's an example of unlabeled triplet learning: https://arxiv.org/abs/1711.02209 https://arxiv.org/abs/1711.02209 In this case, aug(X) might be a slightly time or pitch shifted example. So whatever the label is for X, it will be the same as the label for aug(X). The advantage here is that as a framework it plays nicely with 'real' problems. Representation learning is rarely the actual problem; it's certainly not something an end-user cares about! If you are ultimately working on a classification problem and have a huge amount of unlabeled data and a tiny bit of labeled data, you can train the metric with both the Huff scheme (X and X' with same label) and the unsupervised scheme (X and aug(X)) simultaneously, and take advantage of the large dataset. A more general argument for metric learning is that, again, reconstruction error proooobably isn't what you actually care about. (Sure, you're sharing encoder and decoder weights, but why do you even need to decode?! How many parameters are being wasted on the need for the weight matrices to support decoding?) If clustering is what you want, metric learning gets at things being close or far from one another more directly.
- nabla9 8y ago
  Thank's for the info. I look into it.