5 ms·

"It’s probably going to happen sooner than most people think. Hardware is improving at an exponential rate—the most surprising thing I’ve learned working on Ope

"It’s probably going to happen sooner than most people think. Hardware is improving at an exponential rate—the most surprising thing I’ve learned working on OpenAI is just how correlated increasing computing power and AI breakthroughs are—and the number of smart people working on AI is increasing exponentially as well. Double exponential functions get away from you fast." I'm not convinced that an exponential number of people working on AI produces exponential advancements. Wouldn't we see diminished returns with each new person, presumably each one is less capable and less expert? I see AI experiencing a hype-cycle like disillusionment before seeing "double exponential" returns.

tlb 9y ago
Diminishing returns on an exponential can still be exponential. Typically, diminishing returns means f(x) = x/(1+log(x)) where x is headcount and f is output. The overhead goes up with the log of the number of people, because a hierarchical organization will have log(N) layers of management that need to be traversed to make decisions. If we have exponentially increasing manpower, x=exp(t), then f = exp(t)/(1+log(exp(t)) which is O(exp(t)/t)
- krona 9y ago
  The Pareto distribution more applicable in this case, since we're talking about research and scientific discovery.
PaulHoule 9y ago
It is not just "less capable" or "less expert" but also the result of how "normal science" progresses by harvesting low hanging fruits.
- ericand 9y ago
  Well said. For example, with the hardware improvements of late (GPUs, TPUs, etc.), we are seeing lots of low hanging fruit as we haven't pushed in this direction before, but I imagine this diminishes soon.
LeoJiWoo 9y ago
It might just take one genius like Albert Einstein to create a new paradigm, so I could believe it happening sooner rather than later with the more people are working on it. I suppose it depends on your view of the theory of scientific revolutions.
- ericand 9y ago
  Good point. I tend to subscribe to Kuhn's Structure of Scientific Revolutions [0] which I think suggests that a paradigm shift comes as collective observations mount and it is eventually exhibited through a genius on the cutting edge. [0] https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...
- ilaksh 9y ago
  I don't think we need a genius or a new paradigm for AGI. I think the main ideas of a few different approaches to do it are already out there. There are a lot of tough problems to solve but to me it looks like it's a matter of hard work rather than any totally new approach. Look up for example Yann LeCun's AGI presentation. Now see projects like Ogmaneo or recent Deep Mind papers that do fast online learning or have ways to work around catastrophic forgetting and other issues raised by LeCun. I believe that one straightforward way to get there is to continue to apply neural network research towards attempting to mimic animals. And there are a bunch of people doing that and making progress.
  - p1esk 9y ago
    attempting to mimic animals That is harder than you might think. For example, the simplest animal with a nervous system, C. elegans worm has been simulated "only 20 to 30 percent of the way towards where we need to get" according to one of the OpenWorm project leaders. That's 302 neurons total. Using a computer analogy, it's like struggling to build an abacus when our ultimate goal is to build a Core i7 processor. rather than any totally new approach It's not clear to me that we have any approach to achieve AGI. People like Ben Goertzel have worked on that for decades without much to show for it. Current deep learning methods have little to do with AGI, they focus on very narrow applications, by design.
hobofan 9y ago
The AI research space is still a pretty green field. Right now you can basically take any random paper and combine it with another random paper and you will be able to find a use case where that succeeds and write a new paper about it. More experienced researchers will, of course, have a better intuition/knowledge of what techniques to combine to get more impressive results. As long as it stays like that (and by opening up new sub-areas, it could stay like that for quite some while), adding more somewhat qualified people to the field results in all the useful combinations being discovered faster and thus exposing new potential starting points. Probably no exponential growth in the strict sense, but quadratic growth isn't bad either.
- pg_bot 9y ago
  Perhaps we could build some sort of neural net that ingests AI research papers and then outputs the next big thing in AI. \s
  - hobofan 9y ago
    Half of the research papers titles already read like that, so why not? ¯\_(ツ)_/¯ ("Learning to learn by gradient descent by gradient descent")
- ericand 9y ago
  This is interesting. Good time to be a researcher!
fullshark 9y ago
I feel like AI is already on the tail end of that hype cycle and disillusionment.

tlb 9y ago

Diminishing returns on an exponential can still be exponential. Typically, diminishing returns means f(x) = x/(1+log(x)) where x is headcount and f is output. The overhead goes up with the log of the number of people, because a hierarchical organization will have log(N) layers of management that need to be traversed to make decisions. If we have exponentially increasing manpower, x=exp(t), then f = exp(t)/(1+log(exp(t)) which is O(exp(t)/t)

krona 9y ago
The Pareto distribution more applicable in this case, since we're talking about research and scientific discovery.

PaulHoule 9y ago

It is not just "less capable" or "less expert" but also the result of how "normal science" progresses by harvesting low hanging fruits.

ericand 9y ago
Well said. For example, with the hardware improvements of late (GPUs, TPUs, etc.), we are seeing lots of low hanging fruit as we haven't pushed in this direction before, but I imagine this diminishes soon.

LeoJiWoo 9y ago

It might just take one genius like Albert Einstein to create a new paradigm, so I could believe it happening sooner rather than later with the more people are working on it. I suppose it depends on your view of the theory of scientific revolutions.

ericand 9y ago
Good point. I tend to subscribe to Kuhn's Structure of Scientific Revolutions [0] which I think suggests that a paradigm shift comes as collective observations mount and it is eventually exhibited through a genius on the cutting edge. [0] https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...
ilaksh 9y ago
I don't think we need a genius or a new paradigm for AGI. I think the main ideas of a few different approaches to do it are already out there. There are a lot of tough problems to solve but to me it looks like it's a matter of hard work rather than any totally new approach. Look up for example Yann LeCun's AGI presentation. Now see projects like Ogmaneo or recent Deep Mind papers that do fast online learning or have ways to work around catastrophic forgetting and other issues raised by LeCun. I believe that one straightforward way to get there is to continue to apply neural network research towards attempting to mimic animals. And there are a bunch of people doing that and making progress.
- p1esk 9y ago
  attempting to mimic animals That is harder than you might think. For example, the simplest animal with a nervous system, C. elegans worm has been simulated "only 20 to 30 percent of the way towards where we need to get" according to one of the OpenWorm project leaders. That's 302 neurons total. Using a computer analogy, it's like struggling to build an abacus when our ultimate goal is to build a Core i7 processor. rather than any totally new approach It's not clear to me that we have any approach to achieve AGI. People like Ben Goertzel have worked on that for decades without much to show for it. Current deep learning methods have little to do with AGI, they focus on very narrow applications, by design.

hobofan 9y ago

The AI research space is still a pretty green field. Right now you can basically take any random paper and combine it with another random paper and you will be able to find a use case where that succeeds and write a new paper about it. More experienced researchers will, of course, have a better intuition/knowledge of what techniques to combine to get more impressive results. As long as it stays like that (and by opening up new sub-areas, it could stay like that for quite some while), adding more somewhat qualified people to the field results in all the useful combinations being discovered faster and thus exposing new potential starting points. Probably no exponential growth in the strict sense, but quadratic growth isn't bad either.

pg_bot 9y ago
Perhaps we could build some sort of neural net that ingests AI research papers and then outputs the next big thing in AI. \s
- hobofan 9y ago
  Half of the research papers titles already read like that, so why not? ¯\_(ツ)_/¯ ("Learning to learn by gradient descent by gradient descent")
ericand 9y ago
This is interesting. Good time to be a researcher!

fullshark 9y ago

I feel like AI is already on the tail end of that hype cycle and disillusionment.