5 ms·

To me, ML is tending towards religion. Religion in the way of blackbox processing. Seems no one can reason what happens within the network, the number of parame

by a_c 4y ago

To me, ML is tending towards religion. Religion in the way of blackbox processing. Seems no one can reason what happens within the network, the number of parameters, the number of layer, the choice of temperature, etc. Or it is just my ignorance. While the enlightenment nudges mankind toward science, could it be that the scientific breakthrough of ML is to knock us back into pre-enlightened age? Per stephen wolfram [1] > It’s worth emphasizing that there’s no “theory” being used here; it’s just a matter of what’s been found to work in practice. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

H8crilA 4y ago
The model knows not where it is, but it knows where it isn't. By subtracting where it is from where it isn't it obtains a difference, or gradient. Gradient is used to generate corrective commands to drive the model from a position where it was to a position where it wasn't.
- IshKebab 4y ago
  Yeah we know how SGD optimisers work. He was talking about network architecture.
  - H8crilA 4y ago
    I was making a joke in reference to this: https://www.youtube.com/watch?v=bZe5J8SVCYQ https://www.youtube.com/watch?v=bZe5J8SVCYQ
smartmic 4y ago
You might be interested in the writings of "Better Without AI", your comment immediately makes me think of this: https://betterwithout.ai/gradient-dissent#backpropaganda https://betterwithout.ai/gradient-dissent#backpropaganda
pixl97 4y ago
How different is this from medicine? When problems begin to contain irreducible complexity human reasoning starts to break. We can't test every possible state to determine which state is the best (in human reasonable energy use and timeframes). When this occurs we use heuristics and throw something at the wall to see what works. Sometimes there just may not be a simpler theory of why something works. Or the 'simple' theory is still massively complex. Reality doesn't owe us an easy explanation.
janalsncm 4y ago
We don’t use neural networks because they’re a priori the most accurate models of physical processes. We use them because they have two interesting properties: 1) They’re arbitrary function approximators. 2) We have a great way of fitting them to data (backpropagation). As an aside, it’s just an accident of nature (or is it?) that neural networks have passing similarity to biological systems. Anyways, it doesn’t make much sense to try to reason about what a neural net is doing inside. What it’s doing is trying to approximate the function it was trained to. That’s it.
efficientsticks 4y ago
It’s way past religion and it’s more like am explosion of science. GPT3.5-turbo has an IQ.. What will GPT 4’s IQ be? Even so, it’s escaped the lab already. Which model will make the better products?
alok-g 4y ago
This in my opinion is the route. Neurosciences are helping understand the brain better, but we don't much yet. AI is developing more capability, while we are losing understanding of how it is working. Convergence would happen at some point, where for a given feature/capability, we may have an equalish understanding of the brain and AI. Subsequently understanding of both may increase together. This may rinse and repeat for more desirable features of the brain. We may find that the brain does not have too much more to put into AI. What if the hard problem of consciousness is not actually is.
cycrutchfield 4y ago
In many fields the practice usually outruns the theory. Much of the history of Physics was like this, for example. The theory usually catches up, belatedly.
- deleted 4y ago
  [deleted]
spywaregorilla 4y ago
That quote is not about how the model works in general. It's saying that there's no theoretical or mathematical way to identify "the best" temperature. Which is obvious. What level of whimsy creates "the best" novel? It's subjective and noisy. It has nothing to do with a poor misunderstanding of the model. Temperature is not that hard of a concept. A strong understanding leaves you understanding that such a parameter will be kind of fluffy. That's the nature of creative processes. CSI is low temp. Twin peaks is high temp.
deleted 4y ago
[deleted]
charcircuit 4y ago
>no one can reason what happens within the network The formula of a neutral network is not hard to see. People would not be able to write code to do training or inference if they didn't know what is happening in the network. >the number of parameters, the number of layer You can measure the effectiveness of a neutral network to know how tweaking the number of parameters affects it. There are also restrictions like hardware or execution time that can place an upper bound on these aside from overfitting. >the choice of temperature Similarly, you can measure the quality of output to find the temperature that seems to work the best. The theory is that it's all one big optimization problem. Where the values of the parameters of the NN aren't the only thing you can tweak.
- a_c 4y ago
  > You can measure the effectiveness of a neutral network to know how tweaking the number of parameters affects it We can throw darts to the board and measure how close we are to the bullseye. Then we can tune parameters like throwing force, elbow angel, etc. These parameters have physical meaning. We can reason that by throwing harder, the trajectory of the dart can be so and so affected, hence causing the dart to fall closer to or further from the bullseye. Regarding "People would not be able to write code to do training or inference if they didn't know what is happening in the network", I understand from an operation perspective what the neurons are doing, but I also don't get what they are doing, if that makes sense. For example in each layer of a CNN, is it doing Fourier transform, low pass filtering, rotating, scaling or whatever? And even if they are, why does it work? Why does it give the result that we want? Can we reason that varying our throwing force will change the trajectory of the dart? Shouldn't we do that? Or we are happy with getting results of hot dog/not hot dog [1]. There lies my uneasiness with ML. I agree with cycrutchfield's comment that practice outrun theory though. We will learn more as we progress. [1] https://www.youtube.com/watch?v=vIci3C4JkL0 https://www.youtube.com/watch?v=vIci3C4JkL0
  - charcircuit 4y ago
    >We can throw darts to the board and measure how close we are to the bullseye. Then we can tune parameters like throwing force, elbow angel, etc. These parameters have physical meaning. We can reason that by throwing harder, the trajectory of the dart can be so and so affected, hence causing the dart to fall closer to or further from the bullseye. This is how Newtonian physics is derived. Similar to neural networks you create an approximation based off your observations. The actual reason why those approximations work is not necessary to be known to utilize Newtonian physics. >in each layer of a CNN, is it doing Fourier transform, low pass filtering, rotating, scaling or whatever? And even if they are, why does it work? Why does it give the result that we want? What the layers themselves are actually doing doesn't really matter and each one is just a part of a larger system. What is important is finding a set of weights that approximates a function that minimizes loss. People care about the ability to find this approximations. What exactly it is found is less of a concern. >Or we are happy with getting results of hot dog/not hot dog Neural networks are not limited to binary classification.

by a_c 4y ago

H8crilA 4y ago

The model knows not where it is, but it knows where it isn't. By subtracting where it is from where it isn't it obtains a difference, or gradient. Gradient is used to generate corrective commands to drive the model from a position where it was to a position where it wasn't.

IshKebab 4y ago
Yeah we know how SGD optimisers work. He was talking about network architecture.
- H8crilA 4y ago
  I was making a joke in reference to this: https://www.youtube.com/watch?v=bZe5J8SVCYQ https://www.youtube.com/watch?v=bZe5J8SVCYQ

smartmic 4y ago

You might be interested in the writings of "Better Without AI", your comment immediately makes me think of this: https://betterwithout.ai/gradient-dissent#backpropaganda https://betterwithout.ai/gradient-dissent#backpropaganda

pixl97 4y ago

How different is this from medicine? When problems begin to contain irreducible complexity human reasoning starts to break. We can't test every possible state to determine which state is the best (in human reasonable energy use and timeframes). When this occurs we use heuristics and throw something at the wall to see what works. Sometimes there just may not be a simpler theory of why something works. Or the 'simple' theory is still massively complex. Reality doesn't owe us an easy explanation.

janalsncm 4y ago

We don’t use neural networks because they’re a priori the most accurate models of physical processes. We use them because they have two interesting properties: 1) They’re arbitrary function approximators. 2) We have a great way of fitting them to data (backpropagation). As an aside, it’s just an accident of nature (or is it?) that neural networks have passing similarity to biological systems. Anyways, it doesn’t make much sense to try to reason about what a neural net is doing inside. What it’s doing is trying to approximate the function it was trained to. That’s it.

efficientsticks 4y ago

It’s way past religion and it’s more like am explosion of science. GPT3.5-turbo has an IQ.. What will GPT 4’s IQ be? Even so, it’s escaped the lab already. Which model will make the better products?

alok-g 4y ago

This in my opinion is the route. Neurosciences are helping understand the brain better, but we don't much yet. AI is developing more capability, while we are losing understanding of how it is working. Convergence would happen at some point, where for a given feature/capability, we may have an equalish understanding of the brain and AI. Subsequently understanding of both may increase together. This may rinse and repeat for more desirable features of the brain. We may find that the brain does not have too much more to put into AI. What if the hard problem of consciousness is not actually is.

cycrutchfield 4y ago

In many fields the practice usually outruns the theory. Much of the history of Physics was like this, for example. The theory usually catches up, belatedly.

deleted 4y ago
[deleted]

spywaregorilla 4y ago

That quote is not about how the model works in general. It's saying that there's no theoretical or mathematical way to identify "the best" temperature. Which is obvious. What level of whimsy creates "the best" novel? It's subjective and noisy. It has nothing to do with a poor misunderstanding of the model. Temperature is not that hard of a concept. A strong understanding leaves you understanding that such a parameter will be kind of fluffy. That's the nature of creative processes. CSI is low temp. Twin peaks is high temp.

deleted 4y ago

[deleted]

charcircuit 4y ago

>no one can reason what happens within the network The formula of a neutral network is not hard to see. People would not be able to write code to do training or inference if they didn't know what is happening in the network. >the number of parameters, the number of layer You can measure the effectiveness of a neutral network to know how tweaking the number of parameters affects it. There are also restrictions like hardware or execution time that can place an upper bound on these aside from overfitting. >the choice of temperature Similarly, you can measure the quality of output to find the temperature that seems to work the best. The theory is that it's all one big optimization problem. Where the values of the parameters of the NN aren't the only thing you can tweak.

a_c 4y ago
> You can measure the effectiveness of a neutral network to know how tweaking the number of parameters affects it We can throw darts to the board and measure how close we are to the bullseye. Then we can tune parameters like throwing force, elbow angel, etc. These parameters have physical meaning. We can reason that by throwing harder, the trajectory of the dart can be so and so affected, hence causing the dart to fall closer to or further from the bullseye. Regarding "People would not be able to write code to do training or inference if they didn't know what is happening in the network", I understand from an operation perspective what the neurons are doing, but I also don't get what they are doing, if that makes sense. For example in each layer of a CNN, is it doing Fourier transform, low pass filtering, rotating, scaling or whatever? And even if they are, why does it work? Why does it give the result that we want? Can we reason that varying our throwing force will change the trajectory of the dart? Shouldn't we do that? Or we are happy with getting results of hot dog/not hot dog [1]. There lies my uneasiness with ML. I agree with cycrutchfield's comment that practice outrun theory though. We will learn more as we progress. [1] https://www.youtube.com/watch?v=vIci3C4JkL0 https://www.youtube.com/watch?v=vIci3C4JkL0
- charcircuit 4y ago
  >We can throw darts to the board and measure how close we are to the bullseye. Then we can tune parameters like throwing force, elbow angel, etc. These parameters have physical meaning. We can reason that by throwing harder, the trajectory of the dart can be so and so affected, hence causing the dart to fall closer to or further from the bullseye. This is how Newtonian physics is derived. Similar to neural networks you create an approximation based off your observations. The actual reason why those approximations work is not necessary to be known to utilize Newtonian physics. >in each layer of a CNN, is it doing Fourier transform, low pass filtering, rotating, scaling or whatever? And even if they are, why does it work? Why does it give the result that we want? What the layers themselves are actually doing doesn't really matter and each one is just a part of a larger system. What is important is finding a set of weights that approximates a function that minimizes loss. People care about the ability to find this approximations. What exactly it is found is less of a concern. >Or we are happy with getting results of hot dog/not hot dog Neural networks are not limited to binary classification.