7 ms·

This doesn't even come close to the power of (not even particularly) modern collaborative filtering algorithms. CF doesn't need to assume that A talks to B wel

by Straw 5y ago

This doesn't even come close to the power of (not even particularly) modern collaborative filtering algorithms. CF doesn't need to assume that A talks to B well and B talks to C well implies A talks to C well- rather, it typically tries to learn an embedding for A, B, C that predicts their interactions effectively. In the simplest form, this could be applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation. More advanced forms would try to predict the interaction as some function of the two user embeddings, possibly even a neural network. This only shows that a particular feature based on 2-chains of interactions isn't useful, whereas the standard approaches I described can try to infer information across any number of connections, and don't necessarily assume that 2-chains as described are indicative of a particular result.

sillysaurusx 5y ago
Skip all that and cut to "a neural network." Presto. I wish I understood this simplicity when I first got into ML. I was always annoyed with claims like "a NN will solve your problems." Oh really? Yeah. Really. You can learn the theory later. Sure, applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation makes sense now. (I have to pause and think about it for a little bit, but I do indeed understand each part.) The cool thing was, I didn't need to. The theory helps inform my future ML training runs. But I was able to get most of the magic without knowing a damn thing. Literally just throw Pytorch at the problem and screw around until it starts giving results. You'll be amazed and shocked and delighted how far that gets you. (And all the professional mathematicians and statisticians will be disgusted, but who cares?)
- nestorD 5y ago
  Actually its one of those problems where it is worth not skipping. I have seen people applying neural networks to transform embeddings before doing a final dot-product where it could be proven (and indeed was confirmed in practice) that all of that was equivalent to a direct embedding. Furthermore, SVD type of algorithm are orders of magnitude faster than NN to train (which is why they become used to train word embeddings, see Glove).
  - sillysaurusx 5y ago
    If a NN is equivalent to $sophisticated_algorithm, that's a positive, not a negative. It means you can blindly let the NN do your thinking for you, and come out ahead. You're wrong about training times. You can get great results from far less than 117M params. My tiniest music model (similar to https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... but not the same one) was around 2M params. Trained it a couple days on my laptop's old CPU. Worked fine. I think people love to exaggerate their own importance, and in the importance of theory. There's a certain myth of The Smart Hero, where intelligence alone is the decisive factor that saves a project. Nope. It's determination. Bet on determination every time. And determined people can get very, very far with NNs.
    - Gimpei 5y ago
      This is totally counter to my experience. NNs are a real pain involving architectural searches, hyperparameter searches, etc. They take forever to train. Sometimes it's even difficult to get them to converge. The easy magical things are GBTs. If you have a classification problem that isn't super non-linear, try a GBT first. There's hardly any tweaking and the training is fast.
      - sillysaurusx 5y ago
        I'm surprised our experiences differ so greatly. It's true that you have to be particularly determined to get results, but it's not true that they take forever to train. At the height of my productivity, I was aiming for at least three completed experiments per day, and always made sure to have at least one going while I was sleeping. It's a bit like saying it takes forever to grow tomato plants. Well, yes. Yes it does. Welcome to gardening. ML is problem-space gardening. Yep, it's hyperparameter searches. Yep, it's architecture searches. No, it doesn't take nearly as long as you're implying, because you can quickly narrow the problem space in log(N) steps. (It would be foolish to do a hyperparam search in linear increments.) Resources have also never been more plentiful. TRC literally gives you 100 TPUs when you sign up. https://jaxtputest.vercel.app/ https://jaxtputest.vercel.app/ That means you can (and I did) run 100 training sessions simultaneously. https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf
- sesuximo 5y ago
  P-hacker news?
- richardw 5y ago
  You’re standing on the shoulders of a lot of giants, telling them you’re coming for them. Where do you thing the leverage is here? Those guys are driving cars and writing JavaScript automatically. Tuning NN knobs can also be done automatically. It often is. We could all do with a little more humility here. We’re surrounded by insane skills who write the simple knobs we get to turn.
- Straw 5y ago
  I'm a big NN proponent as well, and I generally agree, but as so many things can go wrong training I prefer to have a baseline (such as SVD in this case) first to make sure the NN is actually doing what you want. Now, you actually do have to understand how to apply the NN correctly for this problem- each user has a embedding, which is a trainable parameter, and you train it on pairs that have interacted. I don't think your average PyTorch MNIST tutorial will teach this.
phreeza 5y ago
If you use a standard distance metric for your embedding space, won't the triangle inequality still imply a slightly weaker version of these chains? I.e. if A-B and B-C are both smaller than epsilon, then A-C is smaller than 2*epsilon?
- zwaps 5y ago
  Yes The embedding needs to embed a social graph with similarity weights and predict ties. Of course, such partly cyclical graphs can not be guaranteed to embed in a low dimensional (aka far away from N) Euclidian or even Hyperbolic space. Which means, among other things, that throwing any old Pytorch NN at the problem will not work, similar to how you can’t just Word2Vec to GPT performance since the latter employs many parallel Euclidian embeddings (or, in other words, any selection or aggregation of layer embedding is contextual). Not that you can’t do it on Graphs, it’s just that understanding the issues is very helpful. For example, understanding that Euclidian embeddings imply a triangle inequality while social graphs of similarity do not, is a crucial thing to know before throwing stuff at pytorch.
  - phreeza 5y ago
    I like the analogy to multi-head attention. Is there work on something like it for collaborative filtering embeddings, eg splitting up the vector into multiple subvectors and calculating distances separately?
    - zwaps 5y ago
      Not sure about CF but in terms of representation learning there are, of course, transformer approaches to embedding graphs ( the article essentially talks about the predictive power of closed triads in a graph) So, a good starting point might be SDNE type graph embeddings with a transformer architecture, since these are already encoder/decoder setups. There’s a ton of ways to get node similarity in a graph, but it turns out that all of them have a substantive theory basis (walks versus neighborhoods, higher order structure vs proximity etc) and none of them are really just “NN, embedding and done“ Perhaps also because it is really simple to see ex ante that a flat Euclidian embedding can not work.
PaulDavisThe1st 5y ago
None of this solves the basic problem of what features in the world you want to put into the model. Heights of A, B, C? Age? Gender? Educational history? Reading or watching preferences? Handedness? Favorite color? You have absolute no idea which, if any, of the possible characteristics you could use to describe A, B or C, and so you only choices are to be vaguely selective and hope your guess is right, or throw everything you can think at the model and hope that there's still a clear signal. You may still get it wrong.
- Straw 5y ago
  These extra features could help it narrow things down more quickly, but CF doesn't need any of this to get good quality- what I described needs _no_ features, only the observation of interactions between users. It tries to find a pattern to explain the successful interactions, by generating a set of features for each user and optimizing it to fit the data.

by Straw 5y ago

sillysaurusx 5y ago

Skip all that and cut to "a neural network." Presto. I wish I understood this simplicity when I first got into ML. I was always annoyed with claims like "a NN will solve your problems." Oh really? Yeah. Really. You can learn the theory later. Sure, applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation makes sense now. (I have to pause and think about it for a little bit, but I do indeed understand each part.) The cool thing was, I didn't need to. The theory helps inform my future ML training runs. But I was able to get most of the magic without knowing a damn thing. Literally just throw Pytorch at the problem and screw around until it starts giving results. You'll be amazed and shocked and delighted how far that gets you. (And all the professional mathematicians and statisticians will be disgusted, but who cares?)

nestorD 5y ago
Actually its one of those problems where it is worth not skipping. I have seen people applying neural networks to transform embeddings before doing a final dot-product where it could be proven (and indeed was confirmed in practice) that all of that was equivalent to a direct embedding. Furthermore, SVD type of algorithm are orders of magnitude faster than NN to train (which is why they become used to train word embeddings, see Glove).
- sillysaurusx 5y ago
  If a NN is equivalent to $sophisticated_algorithm, that's a positive, not a negative. It means you can blindly let the NN do your thinking for you, and come out ahead. You're wrong about training times. You can get great results from far less than 117M params. My tiniest music model (similar to https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... but not the same one) was around 2M params. Trained it a couple days on my laptop's old CPU. Worked fine. I think people love to exaggerate their own importance, and in the importance of theory. There's a certain myth of The Smart Hero, where intelligence alone is the decisive factor that saves a project. Nope. It's determination. Bet on determination every time. And determined people can get very, very far with NNs.
  - Gimpei 5y ago
    This is totally counter to my experience. NNs are a real pain involving architectural searches, hyperparameter searches, etc. They take forever to train. Sometimes it's even difficult to get them to converge. The easy magical things are GBTs. If you have a classification problem that isn't super non-linear, try a GBT first. There's hardly any tweaking and the training is fast.
    - sillysaurusx 5y ago
      I'm surprised our experiences differ so greatly. It's true that you have to be particularly determined to get results, but it's not true that they take forever to train. At the height of my productivity, I was aiming for at least three completed experiments per day, and always made sure to have at least one going while I was sleeping. It's a bit like saying it takes forever to grow tomato plants. Well, yes. Yes it does. Welcome to gardening. ML is problem-space gardening. Yep, it's hyperparameter searches. Yep, it's architecture searches. No, it doesn't take nearly as long as you're implying, because you can quickly narrow the problem space in log(N) steps. (It would be foolish to do a hyperparam search in linear increments.) Resources have also never been more plentiful. TRC literally gives you 100 TPUs when you sign up. https://jaxtputest.vercel.app/ https://jaxtputest.vercel.app/ That means you can (and I did) run 100 training sessions simultaneously. https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf
sesuximo 5y ago
P-hacker news?
richardw 5y ago
You’re standing on the shoulders of a lot of giants, telling them you’re coming for them. Where do you thing the leverage is here? Those guys are driving cars and writing JavaScript automatically. Tuning NN knobs can also be done automatically. It often is. We could all do with a little more humility here. We’re surrounded by insane skills who write the simple knobs we get to turn.
Straw 5y ago
I'm a big NN proponent as well, and I generally agree, but as so many things can go wrong training I prefer to have a baseline (such as SVD in this case) first to make sure the NN is actually doing what you want. Now, you actually do have to understand how to apply the NN correctly for this problem- each user has a embedding, which is a trainable parameter, and you train it on pairs that have interacted. I don't think your average PyTorch MNIST tutorial will teach this.

phreeza 5y ago

If you use a standard distance metric for your embedding space, won't the triangle inequality still imply a slightly weaker version of these chains? I.e. if A-B and B-C are both smaller than epsilon, then A-C is smaller than 2*epsilon?

zwaps 5y ago
Yes The embedding needs to embed a social graph with similarity weights and predict ties. Of course, such partly cyclical graphs can not be guaranteed to embed in a low dimensional (aka far away from N) Euclidian or even Hyperbolic space. Which means, among other things, that throwing any old Pytorch NN at the problem will not work, similar to how you can’t just Word2Vec to GPT performance since the latter employs many parallel Euclidian embeddings (or, in other words, any selection or aggregation of layer embedding is contextual). Not that you can’t do it on Graphs, it’s just that understanding the issues is very helpful. For example, understanding that Euclidian embeddings imply a triangle inequality while social graphs of similarity do not, is a crucial thing to know before throwing stuff at pytorch.
- phreeza 5y ago
  I like the analogy to multi-head attention. Is there work on something like it for collaborative filtering embeddings, eg splitting up the vector into multiple subvectors and calculating distances separately?
  - zwaps 5y ago
    Not sure about CF but in terms of representation learning there are, of course, transformer approaches to embedding graphs ( the article essentially talks about the predictive power of closed triads in a graph) So, a good starting point might be SDNE type graph embeddings with a transformer architecture, since these are already encoder/decoder setups. There’s a ton of ways to get node similarity in a graph, but it turns out that all of them have a substantive theory basis (walks versus neighborhoods, higher order structure vs proximity etc) and none of them are really just “NN, embedding and done“ Perhaps also because it is really simple to see ex ante that a flat Euclidian embedding can not work.

PaulDavisThe1st 5y ago

None of this solves the basic problem of what features in the world you want to put into the model. Heights of A, B, C? Age? Gender? Educational history? Reading or watching preferences? Handedness? Favorite color? You have absolute no idea which, if any, of the possible characteristics you could use to describe A, B or C, and so you only choices are to be vaguely selective and hope your guess is right, or throw everything you can think at the model and hope that there's still a clear signal. You may still get it wrong.

Straw 5y ago
These extra features could help it narrow things down more quickly, but CF doesn't need any of this to get good quality- what I described needs _no_ features, only the observation of interactions between users. It tries to find a pattern to explain the successful interactions, by generating a set of features for each user and optimizing it to fit the data.