11 ms·
ARM Pioneer Sophie Wilson Also Thinks Moore’s Law is Coming to an End
- Symmetry 9y agoYup, we won't be able to keep shrinking MOSFETs forever. There's like to be an interregnum of some sort before a new computing substrate is developed that give us substantially faster gates. And possibly fewer but higher frequency gates at first, which would be interesting. In the mean time we might see a new golden age of computer architecture where the only way to increase performance is to question assumptions about how we design computers.
- static_noise 9y agoMoores law has been the driving force of chip development? Prophet Moore predicted the future and now engineers start breaking the law? Isn't it the other way around that Moore made an observation about some effect that arose naturally? The formula was then called Moores law and its extrapolation had great predictive power for a long time. Similar effects occur all through industries when you start scaling things up. Quality will go up and cost per unit will go down. Often following a simple mathematical formula which describes the learning curve. In many technologies there is something called maturity where the straight line in the diagram starts to bend and approaches a technical limit. Markets overcome this a few times by changing the technological approach of solving a problem to an approach that has a better limit. This makes the general trend continue for decades... until the point where the next technology is so expensive that noone can afford it anymore. Thus far Silicon has won every round and chip manufacturing plants cost many billions of dollars.
- wtallis 9y agoMoore's law started out purely observational, but over the years it became a driving force of its own. Chip manufacturer's performance is judged against Moore's law. Roadmaps and timelines are drafted with Moore's law in mind. Massive research investments are undertaken with the expectation that they will enable a company to keep pace with Moore's law. Even if you're not doing chip design, anyone planning more than one chip product cycle into the future needs to take into account Moore's law. If you're building a hardware system or even a software project, ignoring Moore's law means that by the time you ship, your product might be cheaper than expected but also missing features that are now cost-effective.
- parrellel 9y agoChip manufactures aren't going to be able to keep it up though. Look at all the problems getting a 10nm chip working, look at how those issues are going to get much worse at 7 and 3nm scales. Advances now seem more putting all the optimizations back in that they ignored in the quest for the physical bottom.
- kurthr 9y agoThe death of Moore's Law will have as much to do with CFOs deciding that the investment isn't worth the return as it will with technological innovation. When Intel decided to layoff 12k last year, it seemed like the writing was on the wall. ITRS seemed to think so, anyway: https://www.hpcwire.com/2016/07/28/transistors-wont-shrink-beyond-2021-says-final-itrs-report/ https://www.hpcwire.com/2016/07/28/transistors-wont-shrink-b... Going from Tick-Tock to Tick-Tock-Tweak... and this year to Tick-Tock-Tweak-Tuck the fourth year of 14nm (still as compact as other companies 10nm) makes the slowdown palpable. Perhaps they will manage a 2.7x shrink at their "10nm node" with or without EUV, but it's not the straight scaling of yesteryear.
- deepnotderp 9y agoIt's partially that, but EUV litho is REALLY hard to make at a low enough resolution.
- kurthr 9y agoI'm not sure what you mean. EUV at 13nm (Extreme UV, after marketing with the name X-Ray Lithography failed) is much easier to make work at 10nm than the current multi-patterned immersion technology based on 193nm ArF excimer. http://www.anandtech.com/show/10097/euv-lithography-makes-good-progress-still-not-ready-for-prime-time http://www.anandtech.com/show/10097/euv-lithography-makes-go... The problem with EUV is that the source power (laser excitation of plasma) is too low, making exposure times too long for the expense of the equipment.
- dbcooper 9y ago"Shot noise" is a problem too, due to the high energy of the EUV photons.
- deepnotderp 9y agoYes that's true, but shot noise is a real problem.
- 9y ago
- deepnotderp 9y agoI think a key point that's ignored is that data movement is the new problem. For example, according to Lawrence Livermore National Laboratories, the cost of moving a 64-bit word 1mm ON CHIP on the 10nm projection is approximately equal to doing a 64-bit FLOP. And the cost of DRAM is outrageous... It's what's holding back exascale and will hold back general purpose compute as well. Architectures MUST change radically to adapt to this or there can be no progress.
- 0xCMP 9y agoAny links? This would be interesting to read/watch more about
- deepnotderp 9y agoI wasn't able to find the exact slideshow, but here's something that gets the point across more or less. Page 37 is what you're looking for. https://fas.org/irp/agency/dod/jason/exascale.pdf https://fas.org/irp/agency/dod/jason/exascale.pdf
- wtallis 9y ago> the cost of moving a 64-bit word 1mm ON CHIP on the 10nm projection is approximately equal to doing a 64-bit FLOP Is this the cost in Joules or nanoseconds?
- deepnotderp 9y agoJoules, but the cost of memory access in terms of speed has an even worse disparity.
- lottin 9y agoThis is just another law at work, namely the law of diminishing returns. At some point, each additional transistor that you add on a chip will increase its computational power less and less.
- paulsutter 9y agoAI processor speedups will advance faster than Moore's law in the next 2-3 years, mostly due to lower precision (12/8/4 bits instead of 64/32 bits), massive parallelism, and a different programming paradigm. Google's TPU's for example are close to hardwired matrix multiplication. Maybe speedups for traditional scalar-oriented code matters less now. Intel Lake Crest: "will enable training of neural networks at 100 times the performance on today’s GPUs, said Diane Bryant, executive vice president and general manager of Intel’s data center group" https://venturebeat.com/2016/11/17/intel-will-test-nervanas-lake-crest-silicon-in-first-half-of-2017-knights-crest-also-coming/ https://venturebeat.com/2016/11/17/intel-will-test-nervanas-... Google TPU: "The TPU...used 8-bit integer math...process 92 TOPS" (trillion operations per second) https://www.nextplatform.com/2017/04/05/first-depth-look-googles-tpu-architecture/ https://www.nextplatform.com/2017/04/05/first-depth-look-goo... Generally: http://www.moorinsightsstrategy.com/what-to-expect-in-2017-from-amd-intel-nvidia-xilinx-and-others-for-machine-learning/ http://www.moorinsightsstrategy.com/what-to-expect-in-2017-f...
- zitterbewegung 9y agoI really don't think so. The current graphic cards are heading torward the same issues Intel is doing. It's getting longer for improvement in GPUs to occur .
- paulsutter 9y agoGPUs are more complex than what's needed for AI, take a close look at Google TPU for example.
- jacquesm 9y agoThe TPU is used in inference, not in training, GPUs can be used for both. https://www.extremetech.com/computing/247199-googles-dedicated-tensorflow-processor-tpu-makes-hash-intel-nvidia-inference-workloads https://www.extremetech.com/computing/247199-googles-dedicat...
- throwaway2048 9y ago
- 0xCMP 9y agoI imagine this will begin to put some pressure back to making things faster again as speed-ups that were previously expected fail to appear. (i.e. JS performance on mobile) These days it's not a big deal to most developers, but I think over the next few years if there aren't major advances in speed we will want to get that extra battery life and speed out of our applications and devices. Independent Developers hopefully will have a good financial reason to do that, unlike today.
- justinbaker84 9y agoVery sad to see this ending.
- Animats 9y agoSome limits were hit a decade ago. The Pentium 4 (2004) clocked at 3.8GHz max. Most Intel processors today are slower than that. Intel's fastest offering is a little over 4GHz. The article says that 28nm will dominate for another decade, even though 14nm fabs exist. Having to use extreme ultraviolet (really soft X-rays) for lithography runs costs way up. EUV "light sources" are insanely complex, involving heating falling droplets of metal to plasma levels with lasers. It's amazing that works as a production technology. The equipment looks like something from a high energy physics lab. It's interesting that we hit the limit of photons before the limits of atoms or electrons. Another problem with all this downsizing is electromigration. Every once in a while, an atom gets pulled out of position by the electric field across a gap. Higher temperatures make it worse. Narrower wires make it more of a problem. This is now a major reason ICs wear out in use. Getting rid of the heat is another problem. High performance CPUs are already cooling-limited. This is also why 3D IC schemes aren't too useful for active components like CPUs. Getting heat out of the middle of the stack is hard. Memory can be stacked, if it's not used too hard. There's no problem making lots of CPUs on a chip, if the application can use them. Things look better server-side; you can use vast numbers of CPUs in a server farm, but it's hard to see what 20 or 100 CPUs would do for a laptop. Drastically different architectures may help on specialized problems. GPUs have turned out to be more generally useful than expected. There will probably be "deep learning" ICs; that's a problem where the basic operation is simple and there's massive parallelism. For ordinary CPU power per CPU, we're close to done.
- adventured 9y ago> but it's hard to see what 20 or 100 CPUs would do for a laptop. It requires a few assumptions, but here it goes: 1) Assume applications increase their CPU demands significantly over the coming decades. Why? Who knows. Maybe (very plausibly) extremely advanced (compared to today) AI use & integration. 2) Move to, essentially, a CPU per process model I have a couple dozen processes running on my system. Make the CPUs cheap enough and give me 30 of them. Perhaps the average system will have four or five major AI agents running locally on it, that are particularly good at various things, and those agents will be constantly running processing intensive tasks (I'm assuming here that ~95% of all AI tasks will be performed in the cloud; which is to say I expect the computing power consumed daily by an individual in ... 30 years to be a hundred plus times what the average user is consuming per day now (doing things like watching YouTube or checking Facebook or running Snapchat or WhatsApp)).
- buzzybee 9y agoIf one believes Ray Kurzweil(among others), this is just a phase shift where the focus of change moves away from this technology towards a new one. But then the question is: which one? We do have some options floating around.
- rhaps0dy 9y ago>Even for highly parallel workloads like ray tracing, the performance increase levels off at about 20 times. “No matter how many processors I apply, ray tracing ain’t going to go any faster than 20 times faster,” What? That's just not true. Matrix multiplication is one such embarrassingly parallel workload that can go much faster than 20 times. Ray tracing very probably too.
- randcraw 9y agoI suspect Wilson was saying "Since 5% of the runtime of the raytracing algorithm cannot be sped up through parallelism, even if the time needed to run the other 95% were reduced to zero due to parallelism (or some other magic), the speedup of raytracing could never exceed 100/(100-95), or 20X." In essence, Amdahl's Law trumps Moore's Law.
- marcosdumay 9y agoMoore's Law trumps Amdahl's Law when you decide to simply solve two problems at the same time. Instead of a 19X speedup on rendering, you get an 18X speedup on both rendering, physics, and AI, and gain some extra responsivity on the mean-time.
- Dylan16807 9y agoAnd that's true with some workloads, and maybe some systems that use raytracing, but not raytracing itself. The only overhead is combining the final data from each processor, and that's log(n) in the number of processors with a very small constant. A system with a million independent processors can raytrace very nearly a million times faster.
- Qantourisc 9y agoThis can simple not be true: proof: take 2 computers render in parallel, combine images at the end when happy with the result. The combined images will look better. The 2 computers shared no resources. As such, ray-tracing scales. Luxrender for example has a problem here: all threads are writing to the same output buffer, causing bottlenecks and non perfect scaling. (Might be corrected by now.) The advise then was: run 2 or more Luxrenders, and combine the output image (luxrenders flm file).
- rini17 9y agoMemory did not go faster so much. You can cram bazillions of transistors on a chip, even do clever tricks to fix power consumption/dissipation...but no trick will feed them data fast enough.
- unlmtd 9y agoThe law that isn't.
- to3m 9y ago> In 1975, Wilson was part of the team that developed the 6502 You can get a better summary of her early career from her computer history museum oral history interview: http://www.computerhistory.org/collections/catalog/102746190 http://www.computerhistory.org/collections/catalog/102746190 - worth your time.
- lsllc 9y agoFantastic read, thank you!
- api 9y agoI disagree about the limitations of software parallelism. The article is correct that many existing algorithms like ray tracing or apps like web rendering have inherent limits to parallelization, but there exist a large number of "embarrassingly parallel" things that simply are not done on small PCs and phones right now because they're too costly. This includes things like neural networks, genetic algorithms, all kinds of optimization algorithms, etc. This article is from 2007 so it predates the AI renaissance. Lots of AI, ML, and optimization stuff can happily eat as many cores as you want to throw at it. Then there's the multitasking angle. On a desktop at least I often run dozens of applications, developer VMS, etc. I could definitely use 20 cores in a desktop/laptop right now. We have tests that easily max out a 24 core server that I'd love to run on my own box.
- framebit 9y agoInteresting and relevant paper on the end of Moore's Law: ftp://ftp.cs.utexas.edu/pub/dburger/papers/ISCA11.pdf
- deepnotderp 9y agoWe also always tend to neglect the equally important counterpart to Moore's Law,Dennard scaling. Dennard scaling is on its deathbed, and has been plateauing from around 40/28nm. Since power consumption is now the problem for everyone, including supercomputers, this problem will compound the almost impossible to solve data movement wail m
- kutkloon7 9y agoIs this even news? I have heard the dramatic "Oh no Moore's law is coming to an end" a dozen times during computer engineering courses. Professors are usually slow to adapt new information and it is already a couple years ago that I took those courses. I think that the transistor count has been slowing down for about a decade already.
- visarga 9y agoOn the other hand, many computer functions have reached the "good enough" level. A normal laptop can handle web browsing and document editing just fine. Resolution over Retina level and digital cameras over 10 megapixels are not necessary. Also, sound fidelity over 44khz is not useful. Video over 4K also is on a diminishing curve of returns. We have little extra improvement to get from many domains. Where do you think more processing power would add a large benefit?