4 ms·

This is one of my favorite topics in all of AI. It was the most surprising and mysterious discovery for me. The answer is that the training process literally h

This is one of my favorite topics in all of AI. It was the most surprising and mysterious discovery for me. The answer is that the training process literally has to make the results smooth. That’s how training works. Imagine you have 100 photos. Your job is to classify them by color. You can place them however you want, but similar colors should be physically closer together. You can imagine the result would look a lot like a photoshop RGB picker, which is smooth. The surprise is, this works for any kind of input. Even text paired with images. The key is the loss function (a horrible name). In the color picker example, the loss function would be how similar two colors are. In the text to image example, it’s how dissimilar the input examples are from each other (Contrastive Loss). The brilliance of that is, pushing dissimilar pairs apart is the same thing as pulling similar pairs together, when you train for a long time on millions of examples. Electrons are all trying to push each other apart, but your body is still smooth. The reason it’s brilliant is because it’s far easier to measure dissimilar pairs than to come up with a good way of judging “does this text describe this image?” — you definitely know that it isn’t a bicycle, but you might not know whether a car is a corvette or a Tesla. But both the corvette and the Tesla will be pushed away from text that says it’s a bicycle, and toward text that says it’s a car. That means for a well-trained model, the input by definition is smooth with respect to the output, the same way that a small change in {latitude,longitude} in real life has a small change in the cultural difference of a given region of the world.

Michelangelo11 4y ago
Do you by any chance have a link to a paper or article that explains this in detail? I'd love to understand it better.
- jeabays 4y ago
- sillysaurusx 4y ago
  It doesn’t exist. The above explanation is the result of me spending almost all of my time immersing myself in ML for the last three years. gwern helped too. He has an intuition for ML that I’m still jealous of. Your best bet is to just start building things and worry about explanations later. It’s not far from the truth to say that even the most detailed explanation is still a longform way of saying “we don’t really know.” Some people get upset and refuse to believe that fundamental truth, but I’ve always been along for the ride more than the destination. It’s never been easier to dive in. I’ve always wanted to write detailed guides on how to start, and how to navigate the AI space, but somehow I wound up writing an ML fanfic instead: https://blog.gpt4.org/jaxtpu https://blog.gpt4.org/jaxtpu (Fun fact: my blog runs on a TPU.) I’m increasingly of the belief that all you need is a strong desire to create things, and some resources to play with. If you have both of those, it’s just a matter of time — especially putting in the time. That link explains how to get the resources. But I can’t help with how to get a desire to create things with ML. Mine was just a fascination with how strange computers can be when you wire them up with a small dose of calculus that I didn’t bother trying to understand until two years after I started. (If you mean contrastive loss specifically, https://openai.com/blog/clip/ https://openai.com/blog/clip/ is decent. But it’s just a droplet in the pond of all the wonderful things there are to learn about ML.)
  - Michelangelo11 4y ago
    Thanks! Really appreciate the response.
sizzle 4y ago
thanks for sharing your hard fought knowledge to us curious bystanders
hooande 4y ago
this is a very intuitive analysis. well done and thanks
snovv_crash 4y ago
IMO the term "cost function" is much more intuitive than "loss function" - it tells you the cost, which it attempts to minimize by some iterative process (in this case training)
NickNaraghi 4y ago
Fantastic, this helped me a lot! Thanks for taking the time to write this out.
deltaonefour 4y ago
I actually completely lost interest once I found this out. Simply taking some ML course like the old Andrew Ng courses online are enough for you to get the general idea. ML is simply curve fitting. It's a applied math problem that's quite common. In fact I lost a lot of interest in intelligence in general once I realized this was all that was going on. The implications really say that all of intelligence is really some form of curve fitting. The simplest form of this is linear regression which is used to derive an equation for a line from a set of 2D points. All ML is basically a 10,000 (or much more) dimensional extension of that. The magic is lost. Most of ML research is just to find the most efficient way to find the best fitting curve given the least amount of data points. A ML guys knowledge is centered around a bunch of tricks and techniques to achieve that goal with some N-D template equation. And the general template equation is all the same: A neural network. The answer to what intelligence is seems to be quite simple and not that profound at all... which makes sense given that we're able to create things like DALL-E in such a short time frame. One of the big mysteries of the universe (intelligence) and the thing I always wondered about was essentially answered within the last 2 decades which is pretty cool.. but it's like knowing the secret behind an amazing magic trick.
- sillysaurusx 4y ago
  It’s the other way around. ML is cool precisely because it’s a guitar for the mind, not a mind itself. https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... I made that by using ML as a guitar. I chose instruments and style the way a guitarist’s fingers chooses frets. And saying “give me this style with these instruments” is far easier than recording it yourself. For what it’s worth, I agree with you about AGI. https://twitter.com/theshawwn/status/1446076902607888385?s=21&t=jNnLLkDl2GlWQjorNlYMBw https://twitter.com/theshawwn/status/1446076902607888385?s=2... But for me, that means it’s far more interesting than AGI. Everyone has their eye on AGI, and no one seems to be taking ML at face value. That means the first companies to do it will stand to make a fortune.
  - deltaonefour 4y ago
    Why do people use analogies to prove a point? It doesn't prove anything. What was your point here? ML is like a guitar? What you said doesn't seem to contradict anything I said other then you find curve fitting interesting and I don't. Not trying to be offensive here, don't take it the wrong way. On a side note your music example also basically destroys the question of what is music? Well the answer to that question is that music is the set of all points on some sort of N-dimensional curve. The profoundness of the question is completely gone.

Michelangelo11 4y ago

Do you by any chance have a link to a paper or article that explains this in detail? I'd love to understand it better.

jeabays 4y ago
sillysaurusx 4y ago
It doesn’t exist. The above explanation is the result of me spending almost all of my time immersing myself in ML for the last three years. gwern helped too. He has an intuition for ML that I’m still jealous of. Your best bet is to just start building things and worry about explanations later. It’s not far from the truth to say that even the most detailed explanation is still a longform way of saying “we don’t really know.” Some people get upset and refuse to believe that fundamental truth, but I’ve always been along for the ride more than the destination. It’s never been easier to dive in. I’ve always wanted to write detailed guides on how to start, and how to navigate the AI space, but somehow I wound up writing an ML fanfic instead: https://blog.gpt4.org/jaxtpu https://blog.gpt4.org/jaxtpu (Fun fact: my blog runs on a TPU.) I’m increasingly of the belief that all you need is a strong desire to create things, and some resources to play with. If you have both of those, it’s just a matter of time — especially putting in the time. That link explains how to get the resources. But I can’t help with how to get a desire to create things with ML. Mine was just a fascination with how strange computers can be when you wire them up with a small dose of calculus that I didn’t bother trying to understand until two years after I started. (If you mean contrastive loss specifically, https://openai.com/blog/clip/ https://openai.com/blog/clip/ is decent. But it’s just a droplet in the pond of all the wonderful things there are to learn about ML.)
- Michelangelo11 4y ago
  Thanks! Really appreciate the response.

sizzle 4y ago

thanks for sharing your hard fought knowledge to us curious bystanders

hooande 4y ago

this is a very intuitive analysis. well done and thanks

snovv_crash 4y ago

IMO the term "cost function" is much more intuitive than "loss function" - it tells you the cost, which it attempts to minimize by some iterative process (in this case training)

NickNaraghi 4y ago

Fantastic, this helped me a lot! Thanks for taking the time to write this out.

deltaonefour 4y ago

I actually completely lost interest once I found this out. Simply taking some ML course like the old Andrew Ng courses online are enough for you to get the general idea. ML is simply curve fitting. It's a applied math problem that's quite common. In fact I lost a lot of interest in intelligence in general once I realized this was all that was going on. The implications really say that all of intelligence is really some form of curve fitting. The simplest form of this is linear regression which is used to derive an equation for a line from a set of 2D points. All ML is basically a 10,000 (or much more) dimensional extension of that. The magic is lost. Most of ML research is just to find the most efficient way to find the best fitting curve given the least amount of data points. A ML guys knowledge is centered around a bunch of tricks and techniques to achieve that goal with some N-D template equation. And the general template equation is all the same: A neural network. The answer to what intelligence is seems to be quite simple and not that profound at all... which makes sense given that we're able to create things like DALL-E in such a short time frame. One of the big mysteries of the universe (intelligence) and the thing I always wondered about was essentially answered within the last 2 decades which is pretty cool.. but it's like knowing the secret behind an amazing magic trick.

sillysaurusx 4y ago
It’s the other way around. ML is cool precisely because it’s a guitar for the mind, not a mind itself. https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... I made that by using ML as a guitar. I chose instruments and style the way a guitarist’s fingers chooses frets. And saying “give me this style with these instruments” is far easier than recording it yourself. For what it’s worth, I agree with you about AGI. https://twitter.com/theshawwn/status/1446076902607888385?s=21&t=jNnLLkDl2GlWQjorNlYMBw https://twitter.com/theshawwn/status/1446076902607888385?s=2... But for me, that means it’s far more interesting than AGI. Everyone has their eye on AGI, and no one seems to be taking ML at face value. That means the first companies to do it will stand to make a fortune.
- deltaonefour 4y ago
  Why do people use analogies to prove a point? It doesn't prove anything. What was your point here? ML is like a guitar? What you said doesn't seem to contradict anything I said other then you find curve fitting interesting and I don't. Not trying to be offensive here, don't take it the wrong way. On a side note your music example also basically destroys the question of what is music? Well the answer to that question is that music is the set of all points on some sort of N-dimensional curve. The profoundness of the question is completely gone.