4 ms·

So Moore's Law really is dead this time. And TPU's are only faster than GPU's through lower precision. Will compute at least keep getting cheaper, perhaps thro

So Moore's Law really is dead this time. And TPU's are only faster than GPU's through lower precision. Will compute at least keep getting cheaper, perhaps through economies of scale? Is Kurzweil's magical next information technology, to carry on the exponential, anywhere in sight?

nootropicat 9y ago
Imagine asics for everything. I mean everything, like implementing a javascript engine directly. The energy efficiency could go up 2-3 orders of magnitude (looking at the difference in bitcoin mining between gpus and asics). Old gaming consoles had cartridges (with memory); I can imagine a future in which complex software is transported in the same manner, except cartridges contain specialized asics. Or perhaps a step forward - a chip making device in every home, an equivalent of sorts of burning music to cd.
- iainmerrick 9y ago
  I mean everything, like implementing a javascript engine directly. In that case, rather than a Javascript engine, wouldn't you have an ASIC for the script itself?
  - yjftsjthsd-h 9y ago
    You wouldn't want a new chip per website. Maybe hardware handling of a standard library, though.
    - AstralStorm 9y ago
      Or even just of the expensive operations like synchronization, context switches, DMA to cache from network interface... Wait. We have most of that already. :)
- _chris_ 9y ago
  I assume you meant to reference this, but just to make it clear, those old gaming console cartridges sometimes DID have special chips in them. Much to the horror of emulator writers a decade later, hehe. Wiki: https://en.wikipedia.org/wiki/List_of_Super_NES_enhancement_chips https://en.wikipedia.org/wiki/List_of_Super_NES_enhancement_...
- hyperpallium 9y ago
  Energy efficiency is important (in a server/bitminer and in mobile), but do you have figures for (1) performance; (2) cost? Custom asics would be expensive now, but perhaps... exponential... cost decrease with sufficient demand. A related approach is programmable asics. I've heard this was researched decades ago, but (I presume), silicon was so cheap, so to speak, it wasn't worth it then. Today, with "peak silicon", perhaps all these discarded tailings will be picked over. To mix mining metaphors.
  - nootropicat 9y ago
    The only relevant figures I know of are from mining bitcoin. [0]. >A related approach is programmable asics. I've heard this was researched decades ago What do you mean, if not fpgas? [0] https://en.bitcoin.it/wiki/Mining_hardware_comparison https://en.bitcoin.it/wiki/Mining_hardware_comparison https://en.bitcoin.it/wiki/Non-specialized_hardware_comparison https://en.bitcoin.it/wiki/Non-specialized_hardware_comparis...
    - hyperpallium 9y ago
      thanks. wait, we're looking at 18,000,000 Mhash/s vs 2568 Mhash/s Like... 4 orders of magnitude? And, 11,000 Mhash/J vs 3 Mhash/J, over 3 ordera of mag, just like you said. yah, fpgas
- majewsky 9y ago
  > Old gaming consoles had cartridges (with memory); I can imagine a future in which complex software is transported in the same manner, except cartridges contain specialized asics. I think if that scenario happens, the user won't notice because those ASICs will be deployed in "the cloud".
- Tobba_ 9y ago
  Personally I'd predict the opposite (and current) pattern, creating generic chips which can replace many ASICs / less generic chips. If you can produce a chip which can replace 10 other low/medium-volume designs, economical scaling will win out as long as you're not adding too much overhead. This has been driving FPGAs forward for quite some time, although they're inherently pretty inefficient in terms of die size. Plus, it also provides some logistical advantages in terms of supply chain fragility. See also: how ridiculously cheap microcontrollers have gotten, and the current messy DRAM pricing (high-capacity chips used in phones sometimes ending up cheaper than low-performance/capacity chips).
tzahola 9y ago
Terahertz graphene CPUs!
nthompson 9y ago
At APS March Meeting in LA, there were groups fabricating Josephson junctions. They appeared to be succeeding at ~15mK and individual logic gates, not ICs. I got the impression that their field was advancing steadily, but there was no excitement in their section, unlike anything with buzzwords like "machine learning", or "topological".
tim333 9y ago
Nvidia is cracking along. "Nvidia’s GPUs today are 25 times faster than five years ago. If they were advancing according to Moore’s law, he said, they only would have increased their speed by a factor of 10." https://spectrum.ieee.org/view-from-the-valley/computing/hardware/move-over-moores-law-make-way-for-huangs-law https://spectrum.ieee.org/view-from-the-valley/computing/har... I find it interesting that they are crossing the levels proposed for human equivalence around now depending on how you figure it - Moravec figured about 100 terraflops and you can now get "118.5 teraflops tensor-based deep learning performance" from "The Quadro GV100 is available now on nvidia.com for $8,900" https://www.hpcwire.com/2018/03/27/nvidia-brings-the-power-of-volta-to-workstations-with-new-quadro/ https://www.hpcwire.com/2018/03/27/nvidia-brings-the-power-o... https://www.jetpress.org/volume1/moravec.htm https://www.jetpress.org/volume1/moravec.htm Just need a bit of software.
- Dylan16807 9y ago
  Any improvement like that is from the GPU being given different kinds of circuits entirely. The raw performance difference is not very impressive. In late 2012 you'd get about 8 gigaflops per dollar. Late 2017 gets you almost double that. They're being used somewhat more effectively, but a normal non-neural-net benchmark shows a total performance improvement of... 3x. They're doing better than CPUs but they're still falling behind Moore's law.
  - marcofatica 9y ago
    > a normal non-neural-net benchmark shows a total performance improvement of... 3x because they aren't intended to run non-neural-net workloads. you don't judge a dishwasher by it's ability to wash your clothes, do you?
    - Dylan16807 9y ago
      When you're judging whether Moore's law is dead, you need to separate out the improvements from changing the FPU layout and the improvements from lithography. And since the consumer chips don't dedicate a lot of space to neural net units, they're the best baseline. It's cool that they made some neural net things go hugely faster. But they could make the same fundamental chip on 6-year-old foundries, only slower by a factor of 2 or 3. That really impressive number has nothing to do with Moore's law.

nootropicat 9y ago

Imagine asics for everything. I mean everything, like implementing a javascript engine directly. The energy efficiency could go up 2-3 orders of magnitude (looking at the difference in bitcoin mining between gpus and asics). Old gaming consoles had cartridges (with memory); I can imagine a future in which complex software is transported in the same manner, except cartridges contain specialized asics. Or perhaps a step forward - a chip making device in every home, an equivalent of sorts of burning music to cd.

iainmerrick 9y ago
I mean everything, like implementing a javascript engine directly. In that case, rather than a Javascript engine, wouldn't you have an ASIC for the script itself?
- yjftsjthsd-h 9y ago
  You wouldn't want a new chip per website. Maybe hardware handling of a standard library, though.
  - AstralStorm 9y ago
    Or even just of the expensive operations like synchronization, context switches, DMA to cache from network interface... Wait. We have most of that already. :)
_chris_ 9y ago
I assume you meant to reference this, but just to make it clear, those old gaming console cartridges sometimes DID have special chips in them. Much to the horror of emulator writers a decade later, hehe. Wiki: https://en.wikipedia.org/wiki/List_of_Super_NES_enhancement_chips https://en.wikipedia.org/wiki/List_of_Super_NES_enhancement_...
hyperpallium 9y ago
Energy efficiency is important (in a server/bitminer and in mobile), but do you have figures for (1) performance; (2) cost? Custom asics would be expensive now, but perhaps... exponential... cost decrease with sufficient demand. A related approach is programmable asics. I've heard this was researched decades ago, but (I presume), silicon was so cheap, so to speak, it wasn't worth it then. Today, with "peak silicon", perhaps all these discarded tailings will be picked over. To mix mining metaphors.
- nootropicat 9y ago
  The only relevant figures I know of are from mining bitcoin. [0]. >A related approach is programmable asics. I've heard this was researched decades ago What do you mean, if not fpgas? [0] https://en.bitcoin.it/wiki/Mining_hardware_comparison https://en.bitcoin.it/wiki/Mining_hardware_comparison https://en.bitcoin.it/wiki/Non-specialized_hardware_comparison https://en.bitcoin.it/wiki/Non-specialized_hardware_comparis...
  - hyperpallium 9y ago
    thanks. wait, we're looking at 18,000,000 Mhash/s vs 2568 Mhash/s Like... 4 orders of magnitude? And, 11,000 Mhash/J vs 3 Mhash/J, over 3 ordera of mag, just like you said. yah, fpgas
majewsky 9y ago
> Old gaming consoles had cartridges (with memory); I can imagine a future in which complex software is transported in the same manner, except cartridges contain specialized asics. I think if that scenario happens, the user won't notice because those ASICs will be deployed in "the cloud".
Tobba_ 9y ago
Personally I'd predict the opposite (and current) pattern, creating generic chips which can replace many ASICs / less generic chips. If you can produce a chip which can replace 10 other low/medium-volume designs, economical scaling will win out as long as you're not adding too much overhead. This has been driving FPGAs forward for quite some time, although they're inherently pretty inefficient in terms of die size. Plus, it also provides some logistical advantages in terms of supply chain fragility. See also: how ridiculously cheap microcontrollers have gotten, and the current messy DRAM pricing (high-capacity chips used in phones sometimes ending up cheaper than low-performance/capacity chips).

tzahola 9y ago

Terahertz graphene CPUs!

nthompson 9y ago

At APS March Meeting in LA, there were groups fabricating Josephson junctions. They appeared to be succeeding at ~15mK and individual logic gates, not ICs. I got the impression that their field was advancing steadily, but there was no excitement in their section, unlike anything with buzzwords like "machine learning", or "topological".

tim333 9y ago

Nvidia is cracking along. "Nvidia’s GPUs today are 25 times faster than five years ago. If they were advancing according to Moore’s law, he said, they only would have increased their speed by a factor of 10." https://spectrum.ieee.org/view-from-the-valley/computing/hardware/move-over-moores-law-make-way-for-huangs-law https://spectrum.ieee.org/view-from-the-valley/computing/har... I find it interesting that they are crossing the levels proposed for human equivalence around now depending on how you figure it - Moravec figured about 100 terraflops and you can now get "118.5 teraflops tensor-based deep learning performance" from "The Quadro GV100 is available now on nvidia.com for $8,900" https://www.hpcwire.com/2018/03/27/nvidia-brings-the-power-of-volta-to-workstations-with-new-quadro/ https://www.hpcwire.com/2018/03/27/nvidia-brings-the-power-o... https://www.jetpress.org/volume1/moravec.htm https://www.jetpress.org/volume1/moravec.htm Just need a bit of software.

Dylan16807 9y ago
Any improvement like that is from the GPU being given different kinds of circuits entirely. The raw performance difference is not very impressive. In late 2012 you'd get about 8 gigaflops per dollar. Late 2017 gets you almost double that. They're being used somewhat more effectively, but a normal non-neural-net benchmark shows a total performance improvement of... 3x. They're doing better than CPUs but they're still falling behind Moore's law.
- marcofatica 9y ago
  > a normal non-neural-net benchmark shows a total performance improvement of... 3x because they aren't intended to run non-neural-net workloads. you don't judge a dishwasher by it's ability to wash your clothes, do you?
  - Dylan16807 9y ago
    When you're judging whether Moore's law is dead, you need to separate out the improvements from changing the FPU layout and the improvements from lithography. And since the consumer chips don't dedicate a lot of space to neural net units, they're the best baseline. It's cool that they made some neural net things go hugely faster. But they could make the same fundamental chip on 6-year-old foundries, only slower by a factor of 2 or 3. That really impressive number has nothing to do with Moore's law.