11 ms·

I don't think it's inertia. I think RAM is fundamentally very useful, and that's why it's not going away. You want to store data somewhere if you access it more

by yosefk 10y ago

I don't think it's inertia. I think RAM is fundamentally very useful, and that's why it's not going away. You want to store data somewhere if you access it more than once, it has to be a large array of cells if it's a lot of data, and accessing data in the array by address seems to me more useful than accessing it by something else (such as by key/content.) Similarly, a file of registers addressable by number is very useful, etc. I don't mind new and original ideas - let's hear new and original ideas, if someone invents a non-round wheel that's better than a round one, great! I do mind a general assumption that wheels are round due to inertia without explaining what if not round wheels we're gonna use once we're past inertia.

p1esk 10y ago
I agreed with your previous comment, but here I think you're missing an elephant in the room: a brain does not use "RAM", as it's implemented in von Neumann computers. There seems to be a long term storage of some sort, sure, but that's more like a hard drive storage than RAM. And in the brain, memory is integrated in the compute elements themselves. So the obvious way to copy such architecture is to use analog computing elements, which are at the same time memory elements - neural network weights. When NN algorithm development reaches the point where they work well enough for commercial applications (which seems to be rather soon), the next step will be specialized hardware to run those applications, and that's where we can start building analog chips with huge weight arrays, programmed as analog values on transistors/capacitors/memristors/etc. That's when a true departure from von Neumann might take place.
- SilasX 10y ago
  Human brains are also very inefficient at computation (per unit energy, time, mass, volume, or some combination of the four), unless it's a problem they were tuned for over a really long evolutionary time (e.g. facial recognition, bipedal locomotion, etc). Sure, if you find a problem that a NN algorithm is really good at, then mapping the hardware to the NN algorithm will get an efficiency speedup, but that's because of the application-specific hardware, not because the vNA is a bad general computation model in terms of what we can physically instantiate.
  - p1esk 10y ago
    You can say the same thing about our computers: they are very inefficient at tasks they were not designed to do. Actually they are very inefficient even at tasks they were designed to do, but since they do them very fast, we've been ignoring it (at least until Moore's law started to slow down). Just think about all layers of abstraction present in current computing platforms, and how much computation really happens when you type 2+2 in a Python console. Also, there are a lot more problems a brain solves better than computers. Software NNs slowly learn to solve more and more of those, primarily by better copying how brain does it. So, in summary, vNA might not be a "bad" general computational model, but it surely is very inefficient compared to human brain, when it comes to "computing" vast majority of tasks.
    - taneq 10y ago
      > Just think about all layers of abstraction present in current computing platforms, and how much computation really happens when you type 2+2 in a Python console. That's not a limitation of von Neumann architecture. That's a consequence of us using the vast and cheap computational resources available to us to make things easy.
    - sushirain 10y ago
      Your arguments are compelling. I hope others will embrace them more.
    - SilasX 10y ago
      >You can say the same thing about our computers: they are very inefficient at tasks they were not designed to do. Hold on -- we need to be consistent about what part of them was designed for what purpose. In the context of the discussion, it's the hardware. Computer hardware is only "designed" to be able to maintain a small state while performing a huge number of Boolean circuits (from some large set of possible circuits) per second, writing to external stores. They were not designed to e.g. make graphs, host discussion forums, etc, and yet do that very well --because we can apply software that converts those problems into "do these Boolean circuits/writes really fast". Considering how handily they beat the alternatives at those tasks, I think it would be fair to say that they're good at things they're not designed for! It's just that there remain a few problems that aren't tractable to this approach so far; but focusing on those problems is a misleading picture. >Actually they are very inefficient even at tasks they were designed to do, but since they do them very fast, we've been ignoring it (at least until Moore's law started to slow down) As above, they were only designed to do those circuits and they're good at that.
- _yosefk 10y ago
  You can't program a brain the way you can program computers. You can't program NNs either, im that sense. They have their place, I work for a company shipping them now with hw acceleration. Let's ignore the role of RAM here amd the fact that NNs are not like real neurons and let me grant the point that it's not a vNA, programming model-wise. Most of computing however will not be NN-based because you want to program things precisely, and the brain is terrible at executing commands precisely.
  - p1esk 10y ago
    But that's the thing - I don't want to program things precisely! Most people don't. I want to tell my computer what I want in plain English, and I want it to understand me, translate my wish into actions, and perform those actions. Just like a human servant would do. That's the future of programming (or so we should hope, at least).
    - kbenson 10y ago
      Well, I would say it sounds like you want a computer that understands you imprecisely so it can account for the idiosyncrasies of human communication, but I think you still want it to carry out the action it's decided you want in a precise manner. Protocol negotiation does not lend itself to imprecise ordering of actions or commands.
      - p1esk 10y ago
        Of course. If I hire a human servant, I will expect him to understand me even when I'm not very precise in my commands, and perform the action I wanted precisely. In fact, the best servant is the one who does what you want before you have to ask him, right? That's exactly what I expect from computers in the future.
        imtringued 10y ago
        I don't know why you're assuming the AI is smart enough to perfectly understand what you want even with incomplete knowledge but is stupid enough to need your help even though it's effectively capable of doing everything by itself.
- dharma1 10y ago
  I don't think von Neumann computing will go away but agree that analog computing will massively boost perf of neural nets in the near future. There is a really good in depth talk about the subject here from Microsoft Research a couple of years ago - still extremely relevant "Will Analog Computing and Neural Nets Power Innovation after Moore's Law?" https://youtu.be/dkIuIIp6bl0 https://youtu.be/dkIuIIp6bl0
adrianratnapala 10y ago
While I think you are right that the von Neuman architecture was not an arbitrary choice. But I think big computers are evolving into something else. In theory, a von Neuman machine asks for a word of data and gets it back from RAM. But nowadays we need caches which communicate with the real RAM using something vaguely like network packets. The hardware then emulates a von Neuman addressing scheme on top of that. But look at what software does with all that RAM: good programming isolates objects from each other and forces them to communicate through well defined interfaces. Bitter experience with threads teaches us to pass values around more than to share state. It seems there is a room for the software abstractions to map more directly onto the hardware than they do now. In which case computers will no longer be giant von Neuman machines, but networks of little ones.

by yosefk 10y ago

p1esk 10y ago

I agreed with your previous comment, but here I think you're missing an elephant in the room: a brain does not use "RAM", as it's implemented in von Neumann computers. There seems to be a long term storage of some sort, sure, but that's more like a hard drive storage than RAM. And in the brain, memory is integrated in the compute elements themselves. So the obvious way to copy such architecture is to use analog computing elements, which are at the same time memory elements - neural network weights. When NN algorithm development reaches the point where they work well enough for commercial applications (which seems to be rather soon), the next step will be specialized hardware to run those applications, and that's where we can start building analog chips with huge weight arrays, programmed as analog values on transistors/capacitors/memristors/etc. That's when a true departure from von Neumann might take place.

SilasX 10y ago
Human brains are also very inefficient at computation (per unit energy, time, mass, volume, or some combination of the four), unless it's a problem they were tuned for over a really long evolutionary time (e.g. facial recognition, bipedal locomotion, etc). Sure, if you find a problem that a NN algorithm is really good at, then mapping the hardware to the NN algorithm will get an efficiency speedup, but that's because of the application-specific hardware, not because the vNA is a bad general computation model in terms of what we can physically instantiate.
- p1esk 10y ago
  You can say the same thing about our computers: they are very inefficient at tasks they were not designed to do. Actually they are very inefficient even at tasks they were designed to do, but since they do them very fast, we've been ignoring it (at least until Moore's law started to slow down). Just think about all layers of abstraction present in current computing platforms, and how much computation really happens when you type 2+2 in a Python console. Also, there are a lot more problems a brain solves better than computers. Software NNs slowly learn to solve more and more of those, primarily by better copying how brain does it. So, in summary, vNA might not be a "bad" general computational model, but it surely is very inefficient compared to human brain, when it comes to "computing" vast majority of tasks.
  - taneq 10y ago
    > Just think about all layers of abstraction present in current computing platforms, and how much computation really happens when you type 2+2 in a Python console. That's not a limitation of von Neumann architecture. That's a consequence of us using the vast and cheap computational resources available to us to make things easy.
  - sushirain 10y ago
    Your arguments are compelling. I hope others will embrace them more.
  - SilasX 10y ago
    >You can say the same thing about our computers: they are very inefficient at tasks they were not designed to do. Hold on -- we need to be consistent about what part of them was designed for what purpose. In the context of the discussion, it's the hardware. Computer hardware is only "designed" to be able to maintain a small state while performing a huge number of Boolean circuits (from some large set of possible circuits) per second, writing to external stores. They were not designed to e.g. make graphs, host discussion forums, etc, and yet do that very well --because we can apply software that converts those problems into "do these Boolean circuits/writes really fast". Considering how handily they beat the alternatives at those tasks, I think it would be fair to say that they're good at things they're not designed for! It's just that there remain a few problems that aren't tractable to this approach so far; but focusing on those problems is a misleading picture. >Actually they are very inefficient even at tasks they were designed to do, but since they do them very fast, we've been ignoring it (at least until Moore's law started to slow down) As above, they were only designed to do those circuits and they're good at that.
_yosefk 10y ago
You can't program a brain the way you can program computers. You can't program NNs either, im that sense. They have their place, I work for a company shipping them now with hw acceleration. Let's ignore the role of RAM here amd the fact that NNs are not like real neurons and let me grant the point that it's not a vNA, programming model-wise. Most of computing however will not be NN-based because you want to program things precisely, and the brain is terrible at executing commands precisely.
- p1esk 10y ago
  But that's the thing - I don't want to program things precisely! Most people don't. I want to tell my computer what I want in plain English, and I want it to understand me, translate my wish into actions, and perform those actions. Just like a human servant would do. That's the future of programming (or so we should hope, at least).
  - kbenson 10y ago
    Well, I would say it sounds like you want a computer that understands you imprecisely so it can account for the idiosyncrasies of human communication, but I think you still want it to carry out the action it's decided you want in a precise manner. Protocol negotiation does not lend itself to imprecise ordering of actions or commands.
    - p1esk 10y ago
      Of course. If I hire a human servant, I will expect him to understand me even when I'm not very precise in my commands, and perform the action I wanted precisely. In fact, the best servant is the one who does what you want before you have to ask him, right? That's exactly what I expect from computers in the future.
      - imtringued 10y ago
        I don't know why you're assuming the AI is smart enough to perfectly understand what you want even with incomplete knowledge but is stupid enough to need your help even though it's effectively capable of doing everything by itself.
dharma1 10y ago
I don't think von Neumann computing will go away but agree that analog computing will massively boost perf of neural nets in the near future. There is a really good in depth talk about the subject here from Microsoft Research a couple of years ago - still extremely relevant "Will Analog Computing and Neural Nets Power Innovation after Moore's Law?" https://youtu.be/dkIuIIp6bl0 https://youtu.be/dkIuIIp6bl0

adrianratnapala 10y ago

While I think you are right that the von Neuman architecture was not an arbitrary choice. But I think big computers are evolving into something else. In theory, a von Neuman machine asks for a word of data and gets it back from RAM. But nowadays we need caches which communicate with the real RAM using something vaguely like network packets. The hardware then emulates a von Neuman addressing scheme on top of that. But look at what software does with all that RAM: good programming isolates objects from each other and forces them to communicate through well defined interfaces. Bitter experience with threads teaches us to pass values around more than to share state. It seems there is a room for the software abstractions to map more directly onto the hardware than they do now. In which case computers will no longer be giant von Neuman machines, but networks of little ones.