3 ms·

A lot of people who are skeptical about AI progress call it "statistical modeling" and point to large data sets involved and large amounts of hardware thrown at

A lot of people who are skeptical about AI progress call it "statistical modeling" and point to large data sets involved and large amounts of hardware thrown at it. Sort of implying it's some sort of a brute-force trick. I'm afraid they do not understand the size of problem/solution set. Suppose problem and solution are 1000 characters long and there's a set of 32 characters. Then a model is function F: X -> X where X is a set of 2^5000 elements. There are 2^10000 such functions, and the goal of a training process is to find the best one. Training set for a language model would be well under 1 PB. So a task here is to use a training set of size 2^50 to find a function from 2^10000 space. It's obvious that no amount of brute-forcing can possibly find this. An no classic mathematical statistical modeling can possibly help. A problem like this can be only approached via ANNs trained through backpropagation, and only because this training process is known to generalize.

runarberg 5y ago
We are still inferring a best fit based on available data, so “statistical modelling” is way more descriptive of what we are doing the “intelligence”. That doesn’t mean it isn’t impressive. These non-classical statistical inference are going beyond what we have been able to do with classic statistics, it is indeed impressive. Personally I only grow skeptical when claims are made about some vague Artificial General Intelligence or some other magic which no amount of inference is gonna give us.
- machiaweliczny 5y ago
  I think diff inference might give us that. Basically kinda like dreaming(generating) other samples and checking how they are different. This will help understand how thing is defined but maybe it works for all kind of things. Obviously it's tough to find such algorithm that works for all types of data/embeddings.
- killerstorm 5y ago
  Given that NN components are demonstrated to have universal computation properties, it is more like deriving a program from training data. Technically a program here is a 'curve' but it's insanely complex one. Do you consider programming math? Since technically you write a function which deals with 0s and 1s.
  - runarberg 5y ago
    There is no need to reduce programing to math (even though you can technically do so) just as there is no need to reduce ‘talking’ to math just because speech can be represented with numbers. In fact you can represent most things (probably all things) with numbers. I do think it is interesting philosophically how much powerful machine learning algorithms are over the traditional linear models. And perhaps it is a good description to say that “best fit program” is inferred. However these algorithms are still limited to inference. And—while useful—calling it intelligent is not very accurate—at least in the way most people use the word—no matter how complected of a construct they are inferring.
_kulang 5y ago
I would think of it as fitting a function which is over-parameterised. So there is intrinsically a model of how things are thought to behave, it’s just a relatively simple one that is over-parameterised so it captures patterns in the data. Get it wrong and it’s over-fit, get it wrong another way and it doesn’t generalise, etc. There can also be an internal representation which is not easily interpretable, an internal representation which is usually of a dimensionality much lower than the number of parameters. It is in this sense that it is a brute-force approach because we should like to know the underlying model, but instead we can throw a hugely over-parameterised but relatively simple model at a problem and it will learn (statistically) the underlying phenomenon. Like you say it does much better than a combinatorial brute-force approach.

runarberg 5y ago

We are still inferring a best fit based on available data, so “statistical modelling” is way more descriptive of what we are doing the “intelligence”. That doesn’t mean it isn’t impressive. These non-classical statistical inference are going beyond what we have been able to do with classic statistics, it is indeed impressive. Personally I only grow skeptical when claims are made about some vague Artificial General Intelligence or some other magic which no amount of inference is gonna give us.

machiaweliczny 5y ago
I think diff inference might give us that. Basically kinda like dreaming(generating) other samples and checking how they are different. This will help understand how thing is defined but maybe it works for all kind of things. Obviously it's tough to find such algorithm that works for all types of data/embeddings.
killerstorm 5y ago
Given that NN components are demonstrated to have universal computation properties, it is more like deriving a program from training data. Technically a program here is a 'curve' but it's insanely complex one. Do you consider programming math? Since technically you write a function which deals with 0s and 1s.
- runarberg 5y ago
  There is no need to reduce programing to math (even though you can technically do so) just as there is no need to reduce ‘talking’ to math just because speech can be represented with numbers. In fact you can represent most things (probably all things) with numbers. I do think it is interesting philosophically how much powerful machine learning algorithms are over the traditional linear models. And perhaps it is a good description to say that “best fit program” is inferred. However these algorithms are still limited to inference. And—while useful—calling it intelligent is not very accurate—at least in the way most people use the word—no matter how complected of a construct they are inferring.

_kulang 5y ago

I would think of it as fitting a function which is over-parameterised. So there is intrinsically a model of how things are thought to behave, it’s just a relatively simple one that is over-parameterised so it captures patterns in the data. Get it wrong and it’s over-fit, get it wrong another way and it doesn’t generalise, etc. There can also be an internal representation which is not easily interpretable, an internal representation which is usually of a dimensionality much lower than the number of parameters. It is in this sense that it is a brute-force approach because we should like to know the underlying model, but instead we can throw a hugely over-parameterised but relatively simple model at a problem and it will learn (statistically) the underlying phenomenon. Like you say it does much better than a combinatorial brute-force approach.