3 ms·

Aside from the irrational but probably inevitable over-hype, I find this whole AI/ML space quite fascinating and frustrating at the same time. Most of the succ

Aside from the irrational but probably inevitable over-hype, I find this whole AI/ML space quite fascinating and frustrating at the same time. Most of the successful algorithms are much less complex/sophisticated than you might believe from the outside, and I believe that's a very good sign because simplicity scales. But on the other hand, there is a huge amount of brute-forcing, you can't really do serious research in this field without a few dozen millions to spare, and I think that's always a limit on the innovation potential. Unfortunately writing good code/finding new algorithms is only a small part of the problem. I also think that many people don't completely understand how weak this model is to process real-world data when there no reliable way for the system to self-train on it.

visarga 6y ago
> I also think that many people don't completely understand how weak this model is to process real-world data when there no reliable way for the system to self-train on it. True, but probably any data can be used in self-training if you have captured enough of it. ML models can handle sequences, grids, trees, graphs, point clouds and sets as input formats. As for needing self-training in general, humans also need it even though we have such well attuned priors on the world based on billions of years of evolution. Even so, it takes years before a human can utter a phrase or accomplish a complex task. Basically today ML works by interpolation, if you have enough data points to cover your space you're good. But if you want to extrapolate outside, you're screwed. <rant>What we need is powerful simulators (physical or even neural based) to expose ML models to a variety of tasks inside the target domain. A simulator is like a dynamic dataset. A ML agent inside an environment (a simulator) is much more similar to humans than a model + dataset. Of course simulation is just as hard as AI.</>
cambalache 6y ago
I agree with you. I was in the Michael Jordan camp, as in: Machine Learning is just statistical inference rebranded and with better marketing. There are lots of cool applications for sure but there is no "magic sauce" beyond your usual models obtained by error minimization. But lately, especially seeing the progress with the Attention/Transformer models I have been slowly letting myself convinced that maybe that is all there is behind human intelligence: Lots of priors encoded by evolution in the brain, and general quasi-brute-force mechanism for inference, deduction and imagination. Of course this is probably a naive view, but I think the main point will remain: Intelligence can be created or simulated by giving an agent the capability to extrapolate in a way that "makes sense" in the environment the agent will live.
- ieatpies 6y ago
  Attention based models are still extremely data inefficient and fail to generalize. I recommend reading the "On the Measure of Intelligence" paper by Chollet if this topic interests you.
  - cambalache 6y ago
    I read the paper, I like Chollet, but he comes across as too pesimistic in that paper. I agree that current models are still very data inefficient, but, despite this, I think they are not that far away from the mark if you consider 2 things. 1)The brain probably has hardcoded a lot information. Things like an universal grammar a la Chomsky, the concept of continuity of time and space.Just add water and boom a 2 year old understands. The other thing is that humans are great extrapolating from few data points for a limited set of cognitive tasks, not all.
- tshoaib 6y ago
  Add that to the third Michael Jordan I've heard of
currymj 6y ago
most machine learning research does not require millions of dollars... if you’re thinking about GPT-3, it uses the transformer architecture. But the paper that introduced the transformer architecture trained for a few days on 8 GPUs — surely this was still serious research? OpenAI sure has a good PR team, but all their spectacular results were built on previous papers (sometimes from their own researchers) that introduced new techniques without spending millions of dollars.

visarga 6y ago

> I also think that many people don't completely understand how weak this model is to process real-world data when there no reliable way for the system to self-train on it. True, but probably any data can be used in self-training if you have captured enough of it. ML models can handle sequences, grids, trees, graphs, point clouds and sets as input formats. As for needing self-training in general, humans also need it even though we have such well attuned priors on the world based on billions of years of evolution. Even so, it takes years before a human can utter a phrase or accomplish a complex task. Basically today ML works by interpolation, if you have enough data points to cover your space you're good. But if you want to extrapolate outside, you're screwed. <rant>What we need is powerful simulators (physical or even neural based) to expose ML models to a variety of tasks inside the target domain. A simulator is like a dynamic dataset. A ML agent inside an environment (a simulator) is much more similar to humans than a model + dataset. Of course simulation is just as hard as AI.</>

cambalache 6y ago

I agree with you. I was in the Michael Jordan camp, as in: Machine Learning is just statistical inference rebranded and with better marketing. There are lots of cool applications for sure but there is no "magic sauce" beyond your usual models obtained by error minimization. But lately, especially seeing the progress with the Attention/Transformer models I have been slowly letting myself convinced that maybe that is all there is behind human intelligence: Lots of priors encoded by evolution in the brain, and general quasi-brute-force mechanism for inference, deduction and imagination. Of course this is probably a naive view, but I think the main point will remain: Intelligence can be created or simulated by giving an agent the capability to extrapolate in a way that "makes sense" in the environment the agent will live.

ieatpies 6y ago
Attention based models are still extremely data inefficient and fail to generalize. I recommend reading the "On the Measure of Intelligence" paper by Chollet if this topic interests you.
- cambalache 6y ago
  I read the paper, I like Chollet, but he comes across as too pesimistic in that paper. I agree that current models are still very data inefficient, but, despite this, I think they are not that far away from the mark if you consider 2 things. 1)The brain probably has hardcoded a lot information. Things like an universal grammar a la Chomsky, the concept of continuity of time and space.Just add water and boom a 2 year old understands. The other thing is that humans are great extrapolating from few data points for a limited set of cognitive tasks, not all.
tshoaib 6y ago
Add that to the third Michael Jordan I've heard of

currymj 6y ago

most machine learning research does not require millions of dollars... if you’re thinking about GPT-3, it uses the transformer architecture. But the paper that introduced the transformer architecture trained for a few days on 8 GPUs — surely this was still serious research? OpenAI sure has a good PR team, but all their spectacular results were built on previous papers (sometimes from their own researchers) that introduced new techniques without spending millions of dollars.