4 ms·

Yeah, I really don’t understand this recently popular viewpoint that the algorithm doesn’t matter, just how much data you throw at it. It doesn’t seem to be bas

Yeah, I really don’t understand this recently popular viewpoint that the algorithm doesn’t matter, just how much data you throw at it. It doesn’t seem to be based on anything more than wishful thinking. One can apply Hutter search to solve just about any problem conceivable given the data and guess what—you’ll approach the optimal solution! The only downside is that this process will take more time than available in our physical universe. I think people forget the time factor and how the entire field of computational complexity theory arose because the meta problem is not that we can’t solve the problem—it’s that we can’t solve it quickly enough on a timescale that matters to humans. Current NN architectures are missing something very fundamental related to the efficiency of problem solving, and I really don’t see how throwing more data at them is going to magically convert an EXPTIME algorithm into a PTIME one. (I’m not saying NNs are EXPTIME; I’m saying that they are incapable of solving entire classes of problems that have both PTIME and EXPTIME solutions, as the NN architecture is not able to “discover” PTIME solutions, thus rendering them incapable of solving those classes of problems in any practical sense).

advael 2y ago
Also, one of the major classes of problem that gets solved and we view as "progress" in machine learning is framing problems. Like we couldn't define "draw a good picture" in a way we could actually assess well, GANs and Diffusion turn out to be good ways to phrase problems like that. In the former case, it creates a way to define the problem as "make something indistinguishable from an example pulled from this dataset" and in the latter case, "I've randomized some of these pixels, undo that based on the description" The idea of "efficiency" and "progress" is this post-hoc rationalization that people who never understood the problem, pay people to understand the problem, apply to problems once they have a working solution in hand. It's a model that is inherently as dumb as a model can be, and the assumption it makes is that there is some general factor of progress on hard problems that can be dialed up and down. Sure, you can pay more scientists and probabilistically increase the rate at which problems are solved, but you can't predict how long it will take, how much failure it will involve, whether a particular scientist will solve a particular problem at all, whether that problem is even solvable in principle sometimes. Businesspeople and investors like models where you put in money and you get out growth at a predictable rate with a controllable timescale, and if this doesn't work you just kick it harder, and this ill fits most frontier research. Hell, it ill suits a lot of regular work.
- visarga 2y ago
  > Sure, you can pay more scientists and probabilistically increase the rate at which problems are solved, but you can't predict how long it will take, how much failure it will involve, whether a particular scientist will solve a particular problem at all, whether that problem is even solvable in principle sometimes. Fully agree, search is hard, unpredictable and expensive. Also a matter of luck, being at the right place and time, and observing something novel. That is why I put the emphasis of AI doing search, not just imitating humans.
  - advael 2y ago
    Okay, but what does that mean? AI is a search process. Do you mean you want the AI to formulate queries? Test hypotheses? Okay. How? What does that mean? What we know how to do is to mathematically define objective functions and tune against them. What objective function describes the behavior you want? Is there some structural paradigm we can use for this other than tuning the parameters on a neural network through optimization toward an objective function? If so, what is it? I'm sorry to be a little testy but what you've basically said is "We should go solve the hard problems in AI research". Dope. As an AI researcher I fully agree. Thanks. Am I supposed to clap or something?
visarga 2y ago
Not "throwing more data at them" but having the AI discover things by searching. AI needs to contribute to the search process to graduate the parrot label.

advael 2y ago

Also, one of the major classes of problem that gets solved and we view as "progress" in machine learning is framing problems. Like we couldn't define "draw a good picture" in a way we could actually assess well, GANs and Diffusion turn out to be good ways to phrase problems like that. In the former case, it creates a way to define the problem as "make something indistinguishable from an example pulled from this dataset" and in the latter case, "I've randomized some of these pixels, undo that based on the description" The idea of "efficiency" and "progress" is this post-hoc rationalization that people who never understood the problem, pay people to understand the problem, apply to problems once they have a working solution in hand. It's a model that is inherently as dumb as a model can be, and the assumption it makes is that there is some general factor of progress on hard problems that can be dialed up and down. Sure, you can pay more scientists and probabilistically increase the rate at which problems are solved, but you can't predict how long it will take, how much failure it will involve, whether a particular scientist will solve a particular problem at all, whether that problem is even solvable in principle sometimes. Businesspeople and investors like models where you put in money and you get out growth at a predictable rate with a controllable timescale, and if this doesn't work you just kick it harder, and this ill fits most frontier research. Hell, it ill suits a lot of regular work.

visarga 2y ago
> Sure, you can pay more scientists and probabilistically increase the rate at which problems are solved, but you can't predict how long it will take, how much failure it will involve, whether a particular scientist will solve a particular problem at all, whether that problem is even solvable in principle sometimes. Fully agree, search is hard, unpredictable and expensive. Also a matter of luck, being at the right place and time, and observing something novel. That is why I put the emphasis of AI doing search, not just imitating humans.
- advael 2y ago
  Okay, but what does that mean? AI is a search process. Do you mean you want the AI to formulate queries? Test hypotheses? Okay. How? What does that mean? What we know how to do is to mathematically define objective functions and tune against them. What objective function describes the behavior you want? Is there some structural paradigm we can use for this other than tuning the parameters on a neural network through optimization toward an objective function? If so, what is it? I'm sorry to be a little testy but what you've basically said is "We should go solve the hard problems in AI research". Dope. As an AI researcher I fully agree. Thanks. Am I supposed to clap or something?

visarga 2y ago

Not "throwing more data at them" but having the AI discover things by searching. AI needs to contribute to the search process to graduate the parrot label.