4 ms·

Skip all that and cut to "a neural network." Presto. I wish I understood this simplicity when I first got into ML. I was always annoyed with claims like "a NN

Skip all that and cut to "a neural network." Presto. I wish I understood this simplicity when I first got into ML. I was always annoyed with claims like "a NN will solve your problems." Oh really? Yeah. Really. You can learn the theory later. Sure, applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation makes sense now. (I have to pause and think about it for a little bit, but I do indeed understand each part.) The cool thing was, I didn't need to. The theory helps inform my future ML training runs. But I was able to get most of the magic without knowing a damn thing. Literally just throw Pytorch at the problem and screw around until it starts giving results. You'll be amazed and shocked and delighted how far that gets you. (And all the professional mathematicians and statisticians will be disgusted, but who cares?)

nestorD 5y ago
Actually its one of those problems where it is worth not skipping. I have seen people applying neural networks to transform embeddings before doing a final dot-product where it could be proven (and indeed was confirmed in practice) that all of that was equivalent to a direct embedding. Furthermore, SVD type of algorithm are orders of magnitude faster than NN to train (which is why they become used to train word embeddings, see Glove).
- sillysaurusx 5y ago
  If a NN is equivalent to $sophisticated_algorithm, that's a positive, not a negative. It means you can blindly let the NN do your thinking for you, and come out ahead. You're wrong about training times. You can get great results from far less than 117M params. My tiniest music model (similar to https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... but not the same one) was around 2M params. Trained it a couple days on my laptop's old CPU. Worked fine. I think people love to exaggerate their own importance, and in the importance of theory. There's a certain myth of The Smart Hero, where intelligence alone is the decisive factor that saves a project. Nope. It's determination. Bet on determination every time. And determined people can get very, very far with NNs.
  - Gimpei 5y ago
    This is totally counter to my experience. NNs are a real pain involving architectural searches, hyperparameter searches, etc. They take forever to train. Sometimes it's even difficult to get them to converge. The easy magical things are GBTs. If you have a classification problem that isn't super non-linear, try a GBT first. There's hardly any tweaking and the training is fast.
    - sillysaurusx 5y ago
      I'm surprised our experiences differ so greatly. It's true that you have to be particularly determined to get results, but it's not true that they take forever to train. At the height of my productivity, I was aiming for at least three completed experiments per day, and always made sure to have at least one going while I was sleeping. It's a bit like saying it takes forever to grow tomato plants. Well, yes. Yes it does. Welcome to gardening. ML is problem-space gardening. Yep, it's hyperparameter searches. Yep, it's architecture searches. No, it doesn't take nearly as long as you're implying, because you can quickly narrow the problem space in log(N) steps. (It would be foolish to do a hyperparam search in linear increments.) Resources have also never been more plentiful. TRC literally gives you 100 TPUs when you sign up. https://jaxtputest.vercel.app/ https://jaxtputest.vercel.app/ That means you can (and I did) run 100 training sessions simultaneously. https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf
      - lumost 5y ago
        Tbh it sounds like you got lucky on your first outing with ML and your first NN model worked well enough to deliver a success. Congratulations on achieving this rare milestone. While I’ve seen many data scientists fall into the trap of spending 1 week per iteration even when it’s clear the change produces negligible result. If you stick with the field long enough you’re likely to hit a dog of a problem that should have worked, but ultimately did not. The areas where ML is advancing fastest also suffer from elusive solutions. Language models and image recognition are only recently tractable problems - the folks trying to advance these fields do not experience magic NN improvements.
- caramelazimuth 5y ago
  How does using these embeddings (of sequences of interactional partners) compare with using embeddings of words/tokens?
  - kevin948 5y ago
    They should be quite similar. In the end you coax your embedding space to amount to some consistent measurement of what causes samples to diverge from one-another. You can do similarity search and all the sorts of things you do for word embeddings on embeddings generated for other scenarios.
sesuximo 5y ago
P-hacker news?
richardw 5y ago
You’re standing on the shoulders of a lot of giants, telling them you’re coming for them. Where do you thing the leverage is here? Those guys are driving cars and writing JavaScript automatically. Tuning NN knobs can also be done automatically. It often is. We could all do with a little more humility here. We’re surrounded by insane skills who write the simple knobs we get to turn.
Straw 5y ago
I'm a big NN proponent as well, and I generally agree, but as so many things can go wrong training I prefer to have a baseline (such as SVD in this case) first to make sure the NN is actually doing what you want. Now, you actually do have to understand how to apply the NN correctly for this problem- each user has a embedding, which is a trainable parameter, and you train it on pairs that have interacted. I don't think your average PyTorch MNIST tutorial will teach this.

nestorD 5y ago

Actually its one of those problems where it is worth not skipping. I have seen people applying neural networks to transform embeddings before doing a final dot-product where it could be proven (and indeed was confirmed in practice) that all of that was equivalent to a direct embedding. Furthermore, SVD type of algorithm are orders of magnitude faster than NN to train (which is why they become used to train word embeddings, see Glove).

sillysaurusx 5y ago
If a NN is equivalent to $sophisticated_algorithm, that's a positive, not a negative. It means you can blindly let the NN do your thinking for you, and come out ahead. You're wrong about training times. You can get great results from far less than 117M params. My tiniest music model (similar to https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... but not the same one) was around 2M params. Trained it a couple days on my laptop's old CPU. Worked fine. I think people love to exaggerate their own importance, and in the importance of theory. There's a certain myth of The Smart Hero, where intelligence alone is the decisive factor that saves a project. Nope. It's determination. Bet on determination every time. And determined people can get very, very far with NNs.
- Gimpei 5y ago
  This is totally counter to my experience. NNs are a real pain involving architectural searches, hyperparameter searches, etc. They take forever to train. Sometimes it's even difficult to get them to converge. The easy magical things are GBTs. If you have a classification problem that isn't super non-linear, try a GBT first. There's hardly any tweaking and the training is fast.
  - sillysaurusx 5y ago
    I'm surprised our experiences differ so greatly. It's true that you have to be particularly determined to get results, but it's not true that they take forever to train. At the height of my productivity, I was aiming for at least three completed experiments per day, and always made sure to have at least one going while I was sleeping. It's a bit like saying it takes forever to grow tomato plants. Well, yes. Yes it does. Welcome to gardening. ML is problem-space gardening. Yep, it's hyperparameter searches. Yep, it's architecture searches. No, it doesn't take nearly as long as you're implying, because you can quickly narrow the problem space in log(N) steps. (It would be foolish to do a hyperparam search in linear increments.) Resources have also never been more plentiful. TRC literally gives you 100 TPUs when you sign up. https://jaxtputest.vercel.app/ https://jaxtputest.vercel.app/ That means you can (and I did) run 100 training sessions simultaneously. https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf
    - lumost 5y ago
      Tbh it sounds like you got lucky on your first outing with ML and your first NN model worked well enough to deliver a success. Congratulations on achieving this rare milestone. While I’ve seen many data scientists fall into the trap of spending 1 week per iteration even when it’s clear the change produces negligible result. If you stick with the field long enough you’re likely to hit a dog of a problem that should have worked, but ultimately did not. The areas where ML is advancing fastest also suffer from elusive solutions. Language models and image recognition are only recently tractable problems - the folks trying to advance these fields do not experience magic NN improvements.
caramelazimuth 5y ago
How does using these embeddings (of sequences of interactional partners) compare with using embeddings of words/tokens?
- kevin948 5y ago
  They should be quite similar. In the end you coax your embedding space to amount to some consistent measurement of what causes samples to diverge from one-another. You can do similarity search and all the sorts of things you do for word embeddings on embeddings generated for other scenarios.

sesuximo 5y ago

P-hacker news?

richardw 5y ago

You’re standing on the shoulders of a lot of giants, telling them you’re coming for them. Where do you thing the leverage is here? Those guys are driving cars and writing JavaScript automatically. Tuning NN knobs can also be done automatically. It often is. We could all do with a little more humility here. We’re surrounded by insane skills who write the simple knobs we get to turn.

Straw 5y ago

I'm a big NN proponent as well, and I generally agree, but as so many things can go wrong training I prefer to have a baseline (such as SVD in this case) first to make sure the NN is actually doing what you want. Now, you actually do have to understand how to apply the NN correctly for this problem- each user has a embedding, which is a trainable parameter, and you train it on pairs that have interacted. I don't think your average PyTorch MNIST tutorial will teach this.