4 ms·

My interpretation of current ML solutions is that it's mostly brute force statistical models hoping to hit a satisfactory p-value (some implementations being mo

My interpretation of current ML solutions is that it's mostly brute force statistical models hoping to hit a satisfactory p-value (some implementations being more elegant in the brute force than others). The calculating power defines the limits of the finite domain you can apply the solution towards. A simplified analogy that I believe is applicable is the application of flocking behavior in birds. Current ML implementations would demand a large dataset of groups of birds in flight, both flocking and non-flocking, and curating the correct p-value of brute-forced models that allege to predict whether behavior is flocking or not (and in the case of an individual bird whether it is appropriate flocking behavior given the individual behaviors of birds in the dataset and their circumstances). But flocking behavior is easily modeled right now, and has been for decades, using simple rules for individuals and depending upon emergent phenomena within a group. I'm concerned that most of the ML efforts now are merely attaining the low-hanging fruits of brute force but will run into a wall that halts progress at the level of relatively "easy" things solved by worms and other comparable biological solutions since so many domains have a level of complexity that would exceed any realistically imaginable level of simple mathematical computing power.

jaredtn 6y ago
What do you mean by p-hacking? In statistics that refers to running a large number of experiments and taking the one with the best results, but ML trains for hundreds of thousands of steps on a single model.
- T-A 6y ago
  ML runs hundreds of thousands of experiments, each on a slightly different model, and takes the one with the best results.
  - ponker 6y ago
    That's not necessarily p-hacking. It becomes p-hacking when the model is hyperoptimized to the test set and thus fails when applied to new data from outside the test set.
panpanna 6y ago
This is one of my favorite quotes, which I think summerizes what you wrote in one sentence: "artificial intelligence is the second best way to solve any problem" (KJ Astrom, UC Santa Barbara)
- Upvoter33 6y ago
  eh, I get the point, but not really, at least for modern AI. Data-driven approaches just work better for large classes of hard problems. There's not going to be some simple algorithm to identify a kitten in a photo.
michannne 6y ago
Agreed. It's been stated time and time again that the next frontier of AI will come from a paradigm shift. Deep learning is great, but it is, at the end of the day statistical model fitting. Can it get us to the next frontier? Sure, as much as you can simulate a turing machine on a calculator and then simulate an R/B tree on that. Using ML to reach the next stage (of making algorithms capable of generic learning without being finely tuned) might be the hard way, and we may be approaching the mirror at mach speed. When, how, what, or why this paradigm shift will happen - no one knows, but we all know the alternative amounts to luck of the draw. I've largely been of the camp that ML/DL will be the herald of the next AI winter, and while GPT-3 is impressive, it has no new constraints that we haven't already seen before, and doesn't break any that needed to be broken to change the field (NLP notwithstanding). Soon enough, the limitations of ML/DL will be apparent, and while there will be a breadth of practical applications to explore and analyze, we will be able to see clearly the boundary of how far ML/DL can take us, and will have to be content with what we have until the shift occurs.
sriku 6y ago
There appears to be some hope in the (currently largely Julia based) SciML - scientific machine learning - ecosystem. The goal there is to combine scientific models and learn corrections from data. Pretty impressive set of libraries in this space.
rstuart4133 6y ago
It seems to me we have only solved one part of the puzzle. Solved is possibly too strong a word, but the basic method seems to be to take a guess at what sort of network works best (flat / rrn / convolution / ltsm), throw a lot of data at it and with luck you get a good result. Sometimes as happened with AlphaZero a better result than the planet has seen to from man or machine to date. I suspect we have a fair bit to learn on the "best network design for a particular task" front, but it looks like that will happen over time. I don't think that's too different to how nature does it in worms, as you say. However, our brains also have a different method of operation. We can see something just once, and learn from it. The small child scalds herself on the room heater, and never goes near it again. So the child has learnt from a single example. We have no idea how to do that. At some level it's probably a complex mechanism that involves memorising it, replaying the memory over and over again till, as you put it, the brain hits a satisfactory p-level. As far as I know no one has built such a mechanism yet. However, that's probably because even if you did build it, the fact remains the brain has learnt from a single example and we have no idea how to learn near perfectly from a single example. Let alone do as the child did and learn the lesson in the space of seconds or at most minutes.
- gatlin 6y ago
  > We have no idea how to do that. I agree. However, my response if I intentionally don't over-think it, is "well she learns from a single example because that experience was painful and she wants to minimize pain." Granted that's true as much as it is facile, but I do wonder if it hints at what the next frontier will consist of. A thermometer will happily keep reading out the temperature of a room heater until its physical form is damaged. A human child has a concept of pain to dissuade behaviors that lead to harm (in theory) (this is obviously simplified).

jaredtn 6y ago

What do you mean by p-hacking? In statistics that refers to running a large number of experiments and taking the one with the best results, but ML trains for hundreds of thousands of steps on a single model.

T-A 6y ago
ML runs hundreds of thousands of experiments, each on a slightly different model, and takes the one with the best results.
- ponker 6y ago
  That's not necessarily p-hacking. It becomes p-hacking when the model is hyperoptimized to the test set and thus fails when applied to new data from outside the test set.

panpanna 6y ago

This is one of my favorite quotes, which I think summerizes what you wrote in one sentence: "artificial intelligence is the second best way to solve any problem" (KJ Astrom, UC Santa Barbara)

Upvoter33 6y ago
eh, I get the point, but not really, at least for modern AI. Data-driven approaches just work better for large classes of hard problems. There's not going to be some simple algorithm to identify a kitten in a photo.

michannne 6y ago

Agreed. It's been stated time and time again that the next frontier of AI will come from a paradigm shift. Deep learning is great, but it is, at the end of the day statistical model fitting. Can it get us to the next frontier? Sure, as much as you can simulate a turing machine on a calculator and then simulate an R/B tree on that. Using ML to reach the next stage (of making algorithms capable of generic learning without being finely tuned) might be the hard way, and we may be approaching the mirror at mach speed. When, how, what, or why this paradigm shift will happen - no one knows, but we all know the alternative amounts to luck of the draw. I've largely been of the camp that ML/DL will be the herald of the next AI winter, and while GPT-3 is impressive, it has no new constraints that we haven't already seen before, and doesn't break any that needed to be broken to change the field (NLP notwithstanding). Soon enough, the limitations of ML/DL will be apparent, and while there will be a breadth of practical applications to explore and analyze, we will be able to see clearly the boundary of how far ML/DL can take us, and will have to be content with what we have until the shift occurs.

sriku 6y ago

There appears to be some hope in the (currently largely Julia based) SciML - scientific machine learning - ecosystem. The goal there is to combine scientific models and learn corrections from data. Pretty impressive set of libraries in this space.

rstuart4133 6y ago

It seems to me we have only solved one part of the puzzle. Solved is possibly too strong a word, but the basic method seems to be to take a guess at what sort of network works best (flat / rrn / convolution / ltsm), throw a lot of data at it and with luck you get a good result. Sometimes as happened with AlphaZero a better result than the planet has seen to from man or machine to date. I suspect we have a fair bit to learn on the "best network design for a particular task" front, but it looks like that will happen over time. I don't think that's too different to how nature does it in worms, as you say. However, our brains also have a different method of operation. We can see something just once, and learn from it. The small child scalds herself on the room heater, and never goes near it again. So the child has learnt from a single example. We have no idea how to do that. At some level it's probably a complex mechanism that involves memorising it, replaying the memory over and over again till, as you put it, the brain hits a satisfactory p-level. As far as I know no one has built such a mechanism yet. However, that's probably because even if you did build it, the fact remains the brain has learnt from a single example and we have no idea how to learn near perfectly from a single example. Let alone do as the child did and learn the lesson in the space of seconds or at most minutes.

gatlin 6y ago
> We have no idea how to do that. I agree. However, my response if I intentionally don't over-think it, is "well she learns from a single example because that experience was painful and she wants to minimize pain." Granted that's true as much as it is facile, but I do wonder if it hints at what the next frontier will consist of. A thermometer will happily keep reading out the temperature of a room heater until its physical form is damaged. A human child has a concept of pain to dissuade behaviors that lead to harm (in theory) (this is obviously simplified).