5 ms·

While we're reaching the physical limits of classical chip design, do we have any ideas what the limits are on the algorithm side of things? As much speed up ha

While we're reaching the physical limits of classical chip design, do we have any ideas what the limits are on the algorithm side of things? As much speed up has come from software as hardware according to a few reports. http://www.johndcook.com/blog/2015/12/08/algorithms-vs-moores-law/ http://www.johndcook.com/blog/2015/12/08/algorithms-vs-moore...

ars 11y ago
> do we have any ideas what the limits are on the algorithm side of things? I've always believed that humans do not have the ability to program things smarter than themself, because we do not understand our own intelligence, so we have no way to reproduce it. At the time, I said the only alternative I can think of is make random permutations and pick the best one, and go from there. But I said this as a ridiculous suggestion that no one would actually do. But then Google actually did exactly that with their Go machine! So, that's what I think: The future of computing will be based on randomness and the job of the programmer will be to guide it, but not program it directly. (Can you imagine programming a webpage this way? Or writing a book this way?)
- danieltillett 11y ago
  Humans every day "program" things smart than themselves. Every parent who raises a child smarter than themselves does exactly that. Smarter than human level computers will be created the same way we create smart humans.
  - brainless 11y ago
    I do not think this is equal. Babies may actually be "brighter" than parents, without the parents having to actually "do" anything to produce the higher IQ baby. But building an artificial system means some humans actually have to design it.
    - danieltillett 11y ago
      Yes parents have to do something to end up with brighter children than themselves - if you put a child in a dark room from birth with no human interaction I can guarentee that they will not come out very bright 20 years later (there have been some very sad examples of this in history). The point I was making is that intelligent agents are built all the time by less intelligent agents. Humans will build more intellgent AI using exactly the same approach and these AI's can do the same. Of couse building an AI via evolutionary training opens up a whole lot of control issues. Do we really want to be creating entities vastly more intelligent than ourselves when we have no real idea of their interests and when we occupy something that is valuble (e.g. matter and energy) to this entity?
- shkkmo 11y ago
  While we may not have the ability to program things 'smarter' than themselves, we do have the ability to create tools that extend our intelligence. Machine learning, randomness, and even new javascript frameworks all fall under this umbrella.
- stochastician 11y ago
  (Totally self-promoting here) I did a lot of work along this line in my PhD -- stochastic architectures for probabilstic computation. http://ericjonas.com/pages/circuits.html http://ericjonas.com/pages/circuits.html There's an increasing amount of interest in this space.
  - abecedarius 11y ago
    Sounds like really neat work. It reminds me of an idea I've toyed with just a little: you know how chemical reaction networks can compute? E.g. https://johncarlosbaez.wordpress.com/2014/03/23/programming-with-chemical-reaction-networks/ https://johncarlosbaez.wordpress.com/2014/03/23/programming-... Last I heard, they'd used DNA strand displacement in vitro for simple neural network stuff. So I wondered how practical it'd be to encode a Bayes net as a reaction network, with the Gibbs sampling done directly by thermodynamics. This wouldn't compete with the sort of thing we do in silicon; maybe it'd work out for tiny diagnostic systems in synthetic bio? With less overhead than the neural nets. (Well, that took more space than I wanted. I downloaded one of your circuits papers for a gander.)
  - nickpsecurity 11y ago
    You're work is interesting. The concepts remind me of work in analog implementations of neural circuitry. See, the brain is likely a general-purpose, analog computer with digital-like parts. So, trying analog implementations was thought to have improvements. It did with one wafer-scale method I didn't see coming. http://yann.lecun.com/exdb/publis/pdf/boser-92.pdf http://yann.lecun.com/exdb/publis/pdf/boser-92.pdf http://www.kip.uni-heidelberg.de/Veroeffentlichungen/download.cgi/4713/ps/1856.pdf http://www.kip.uni-heidelberg.de/Veroeffentlichungen/downloa... It would be interesting to see someone combine the principles of your work with analog implementations on a decent process node. Yours is kind of like a hybrid between properties of analog and digital cells. The real thing might be even more effective albeit harder to automate. There's some analog EDA but it's almost always custom work.
- TrevorJ 11y ago
  >But then Google actually did exactly that with their Go machine! I would argue that this is not what they did. What they did was take something a human player does (study games to learn how to play). And then parallelized this to a level that a human can't match. Basically the equivalent of having one go player playing against an entire team of go players who are all experts.
- b1449835 11y ago
  The point of machines is that they do a single task better than us. A dishwasher is superhuman at washing plates but nothing else. If we model machines to be more like us they will end up with the same disadvantages of a normal human. At that point why not employ real humans in the first place?
  - noir_lord 11y ago
    > At that point why not employ real humans in the first place? Cost mostly, If you can replicate a human-level intelligence for 10,000 (or 100,000) and have it work 24/7 with no time off and scale out to thousands of them then you'd have something absolutely terrifying in it's capability. Image an AWS of a 1000 von-neumann level intelligences working co-operatively 24/7 on a problem. The stuff of sci-fi right now but maybe one day, we know it's physically possible to build human level intelligences since we prove that the rest is 'just' engineering.
gizmo686 11y ago
It is worth remembering that just because we are reaching some physical limits of "classical" chip design. For example, both x86 and arm date back to the 80s. While I have no doubt that the implementation of these architectures has improved to reflect modern manufacturing capabilities, this still suggests that there is room for architectural improvements in performance. Beyond simple architectural improvements, we could still move beyond basic transistor based computing. The most common example is quantum computing (which offers an asymptotic improvement in some cases), however I can imagine there beyond other classical devices that can compute certain functions more efficiently than a pure transistor based solution can.
- kevinnk 11y ago
  > For example, both x86 and arm date back to the 80s. Modern x86 and high performance ARM cores are almost unrecognizable compared to processors in the late 90s, much less the 80s. (Also, ARM was founded in 1990, not the 80s). > While I have no doubt that the implementation of these architectures has improved to reflect modern manufacturing capabilities, this still suggests that there is room for architectural improvements in performance. There are still performance improvements to be had, but it's not going to be anywhere near the performance scaling of Moore's law. The rate of architectural improvements is also slowing down as well (and increasingly only applicable for a smaller and smaller fraction of workloads). > I can imagine there beyond other classical devices that can compute certain functions more efficiently than a pure transistor based solution can. Like...? Quantum is 10+ years away right now and unlikely to get fast any time soon. CMOS has had decades and billions of dollars invested in scaling; it's going to be a long time before any of the current "CMOS killers" (virtually all of which are still transistors) reach parity.
  - sien 11y ago
    From wikipedia on ARM: The official Acorn RISC Machine project started in October 1983. But everything else you say is bang on.
- jessriedel 11y ago
  > Beyond simple architectural improvements, we could still move beyond basic transistor based computing. The most common example is quantum computing (which offers an asymptotic improvement in some cases) The number of problems for which quantum computing offers a speedup is very limited. It is absolutely not a general, all-purpose computational architecture.
nickpsecurity 11y ago
A ton of it did. There's many subtopics for algorithms, microarchitecture tricks, hardware accelerators, I/O schemes, improvements for RTL/transistor optimization, and so on. Each have enough papers it can be hard to fibd stuff. Most of best stuff gets patented and controlled by dominant companies.

ars 11y ago

> do we have any ideas what the limits are on the algorithm side of things? I've always believed that humans do not have the ability to program things smarter than themself, because we do not understand our own intelligence, so we have no way to reproduce it. At the time, I said the only alternative I can think of is make random permutations and pick the best one, and go from there. But I said this as a ridiculous suggestion that no one would actually do. But then Google actually did exactly that with their Go machine! So, that's what I think: The future of computing will be based on randomness and the job of the programmer will be to guide it, but not program it directly. (Can you imagine programming a webpage this way? Or writing a book this way?)

danieltillett 11y ago
Humans every day "program" things smart than themselves. Every parent who raises a child smarter than themselves does exactly that. Smarter than human level computers will be created the same way we create smart humans.
- brainless 11y ago
  I do not think this is equal. Babies may actually be "brighter" than parents, without the parents having to actually "do" anything to produce the higher IQ baby. But building an artificial system means some humans actually have to design it.
  - danieltillett 11y ago
    Yes parents have to do something to end up with brighter children than themselves - if you put a child in a dark room from birth with no human interaction I can guarentee that they will not come out very bright 20 years later (there have been some very sad examples of this in history). The point I was making is that intelligent agents are built all the time by less intelligent agents. Humans will build more intellgent AI using exactly the same approach and these AI's can do the same. Of couse building an AI via evolutionary training opens up a whole lot of control issues. Do we really want to be creating entities vastly more intelligent than ourselves when we have no real idea of their interests and when we occupy something that is valuble (e.g. matter and energy) to this entity?
shkkmo 11y ago
While we may not have the ability to program things 'smarter' than themselves, we do have the ability to create tools that extend our intelligence. Machine learning, randomness, and even new javascript frameworks all fall under this umbrella.
stochastician 11y ago
(Totally self-promoting here) I did a lot of work along this line in my PhD -- stochastic architectures for probabilstic computation. http://ericjonas.com/pages/circuits.html http://ericjonas.com/pages/circuits.html There's an increasing amount of interest in this space.
- abecedarius 11y ago
  Sounds like really neat work. It reminds me of an idea I've toyed with just a little: you know how chemical reaction networks can compute? E.g. https://johncarlosbaez.wordpress.com/2014/03/23/programming-with-chemical-reaction-networks/ https://johncarlosbaez.wordpress.com/2014/03/23/programming-... Last I heard, they'd used DNA strand displacement in vitro for simple neural network stuff. So I wondered how practical it'd be to encode a Bayes net as a reaction network, with the Gibbs sampling done directly by thermodynamics. This wouldn't compete with the sort of thing we do in silicon; maybe it'd work out for tiny diagnostic systems in synthetic bio? With less overhead than the neural nets. (Well, that took more space than I wanted. I downloaded one of your circuits papers for a gander.)
- nickpsecurity 11y ago
  You're work is interesting. The concepts remind me of work in analog implementations of neural circuitry. See, the brain is likely a general-purpose, analog computer with digital-like parts. So, trying analog implementations was thought to have improvements. It did with one wafer-scale method I didn't see coming. http://yann.lecun.com/exdb/publis/pdf/boser-92.pdf http://yann.lecun.com/exdb/publis/pdf/boser-92.pdf http://www.kip.uni-heidelberg.de/Veroeffentlichungen/download.cgi/4713/ps/1856.pdf http://www.kip.uni-heidelberg.de/Veroeffentlichungen/downloa... It would be interesting to see someone combine the principles of your work with analog implementations on a decent process node. Yours is kind of like a hybrid between properties of analog and digital cells. The real thing might be even more effective albeit harder to automate. There's some analog EDA but it's almost always custom work.
TrevorJ 11y ago
>But then Google actually did exactly that with their Go machine! I would argue that this is not what they did. What they did was take something a human player does (study games to learn how to play). And then parallelized this to a level that a human can't match. Basically the equivalent of having one go player playing against an entire team of go players who are all experts.
b1449835 11y ago
The point of machines is that they do a single task better than us. A dishwasher is superhuman at washing plates but nothing else. If we model machines to be more like us they will end up with the same disadvantages of a normal human. At that point why not employ real humans in the first place?
- noir_lord 11y ago
  > At that point why not employ real humans in the first place? Cost mostly, If you can replicate a human-level intelligence for 10,000 (or 100,000) and have it work 24/7 with no time off and scale out to thousands of them then you'd have something absolutely terrifying in it's capability. Image an AWS of a 1000 von-neumann level intelligences working co-operatively 24/7 on a problem. The stuff of sci-fi right now but maybe one day, we know it's physically possible to build human level intelligences since we prove that the rest is 'just' engineering.

gizmo686 11y ago

It is worth remembering that just because we are reaching some physical limits of "classical" chip design. For example, both x86 and arm date back to the 80s. While I have no doubt that the implementation of these architectures has improved to reflect modern manufacturing capabilities, this still suggests that there is room for architectural improvements in performance. Beyond simple architectural improvements, we could still move beyond basic transistor based computing. The most common example is quantum computing (which offers an asymptotic improvement in some cases), however I can imagine there beyond other classical devices that can compute certain functions more efficiently than a pure transistor based solution can.

kevinnk 11y ago
> For example, both x86 and arm date back to the 80s. Modern x86 and high performance ARM cores are almost unrecognizable compared to processors in the late 90s, much less the 80s. (Also, ARM was founded in 1990, not the 80s). > While I have no doubt that the implementation of these architectures has improved to reflect modern manufacturing capabilities, this still suggests that there is room for architectural improvements in performance. There are still performance improvements to be had, but it's not going to be anywhere near the performance scaling of Moore's law. The rate of architectural improvements is also slowing down as well (and increasingly only applicable for a smaller and smaller fraction of workloads). > I can imagine there beyond other classical devices that can compute certain functions more efficiently than a pure transistor based solution can. Like...? Quantum is 10+ years away right now and unlikely to get fast any time soon. CMOS has had decades and billions of dollars invested in scaling; it's going to be a long time before any of the current "CMOS killers" (virtually all of which are still transistors) reach parity.
- sien 11y ago
  From wikipedia on ARM: The official Acorn RISC Machine project started in October 1983. But everything else you say is bang on.
jessriedel 11y ago
> Beyond simple architectural improvements, we could still move beyond basic transistor based computing. The most common example is quantum computing (which offers an asymptotic improvement in some cases) The number of problems for which quantum computing offers a speedup is very limited. It is absolutely not a general, all-purpose computational architecture.

nickpsecurity 11y ago

A ton of it did. There's many subtopics for algorithms, microarchitecture tricks, hardware accelerators, I/O schemes, improvements for RTL/transistor optimization, and so on. Each have enough papers it can be hard to fibd stuff. Most of best stuff gets patented and controlled by dominant companies.