12 ms·

The post-exponential era of AI and Moore’s Law

114 points by ChuckMcM 7y ago

jeremydeanlakey 7y ago
I wonder does the compute power per dollar have to level off so soon?
ChuckMcM 7y ago
The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially. The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. A lot of frameworks and abstractions will be ripped out and thrown away as 'taking too much time for not enough benefit.'
- cft 7y ago
  Rust will fare well. Ruby won't.
  - fouc 7y ago
    Ruby isn't in the critical path, so that's a bad example. In general, the real issue is the layers of abstractions between the hardware and the end user. The over complicated architectures. The constant re-invention of databases/operating systems/virtual machines at each layer.
    - oblio 7y ago
      > The constant re-invention of databases/operating systems/virtual machines at each layer. Those are because of the human struggle, the political and economic fights. They're not because of purely technical problems. Each layer is usually backed by a party with a very clear interest, usually control, and each layer on top tries to escape that control. It's been that way since the times of p-code.
    - clarry 7y ago
      > The constant re-invention of databases/operating systems/virtual machines I wish that were the case but operating systems and systems software in general seems to be the most stagnant field. Everyone just buys the same FLOSS stack for $0 and compatibility is king, so there's very little research going in this field and much less of it ends up in any product you're ever going to see in use. I would love to see a vibrant scene of competing operating systems and databases with fresh new ideas and research.
      - fouc 7y ago
        I meant that it is re-implemented at every layer.. Like a browser is practically an OS on its own.
  - ivalm 7y ago
    I don't have a hat in the game but why won't Ruby do well or not well based on this? It's not like basic webapps need to be much more efficient, and no one was writing high performance compute in ruby anyways.
    - DennisP 7y ago
      Looking over the techempower benchmarks, the fastest web frameworks are over fifty times faster than ruby/rails. If you can replace a hundred web servers with two, that matters.
      - ghettoimp 7y ago
        Sure. But an awful lot of folks don't have a hundred web servers. If you're running something simple, you can pretty much use whatever you want.
      - ativzzz 7y ago
        The vast majority of web apps are CRUD apps, which can achieve sub 100ms responses on any language with caching, background workers and minor optimizations. There will always be problems where you need faster responses, and the solutions to those problems are already not using Ruby. You don't use a hammer when a screwdriver is appropriate, but nails work pretty well in most places and screws are unnecessary.
        DennisP 7y ago
        100ms with how many responses per second? Throughput matters too. And techempower does some pretty realistic-looking benchmarks. For example: > In this test, the framework's ORM is used to fetch all rows from a database table containing an unknown number of Unix fortune cookie messages (the table has 12 rows, but the code cannot have foreknowledge of the table's size). An additional fortune cookie message is inserted into the list at runtime and then the list is sorted by the message text. Finally, the list is delivered to the client using a server-side HTML template. The message text must be considered untrusted and properly escaped and the UTF-8 fortune messages must be rendered properly. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
  - buzzkillington 7y ago
    FPGAs will fare well, general processors won't. You know we've reach the end of Moore's law when balanced ternary computers become the norm.
  - csomar 7y ago
    That has a lot to do with why I took on learning Rust. My last year Macbook Pro has a 6-core CPU. At this rate, we should have 32-64-core CPUs for consumers in a few years. Not using all the processors means your program only makes use of 3-1.5% of the available power. My prediction is that, in a few years, lots of programming languages and software developers will simply become obsolete because what the other guys can produce is heaps of performance, security and manageability atop. But I might be wrong, so make your career decisions carefully.
    - oblio 7y ago
      I hope you're right. However Intel Core Duo was released in January 2006 and it had 2 cores. So in a bit more than 13 years we only have 3x as many cores. Core 2 Quad was released in January 2007, so if you count from that one, a bit more than 12 years for 1.5x as many cores. I wouldn't bet on widespread 64 core CPUs for consumers in the next 10 years or so. Most commercial software just doesn't use it efficiently and consumers don't need it for existing software. We need a software breakthrough, a killer app, something that people really want and that really needs a ton of cores. For now, I don't see it. But then again, that's how killer apps are, nobody sees them until they're there.
      - csomar 7y ago
        > However Intel Core Duo was released in January 2006 and it had 2 cores. So in a bit more than 13 years we only have 3x as many cores. Core 2 Quad was released in January 2007, so if you count from that one, a bit more than 12 years for 1.5x as many cores. https://xkcd.com/605/ https://xkcd.com/605/ If you are going to extrapolate, don't pick the date. Take all the numbers you have and trace the average value. If you are picking the numbers, you can extrapolate to any point you want. > I wouldn't bet on widespread 64 core CPUs for consumers in the next 10 years or so. We already have 32-core processors for servers. Imo, the biggest challenge will be the laptop and heat management rather than putting a bunch of processors together. > We need a software breakthrough, a killer app, something that people really want and that really needs a ton of cores. We need that now. Most software I'm using is slow. Part of why I picked the iPhone (and I bet many people did) was speed. It is fast. People care about speed and if you introduce them to something faster then they'll get hooked. Also, you are already using all of your cores because of multi-tasking. It makes your computer experience overall better. At 6 cores, multi-threading for an application doesn't give much noticeable experience, since you probably have 5-6 things running on your computer doing other stuff like network or an audio-player. > For now, I don't see it. But then again, that's how killer apps are, nobody sees them until they're there. > I hope you're right. Don't worry, just hang in there. The start of multi-cores was slow but then that's all there is right now. Seems intel/amd have mastered how to add more cores but the CPU frequency is pretty much capped going forward!
- carlmr 7y ago
  >Because it is going to demand engineers who can write fast code over engineers who write code fast. I surely hope so, the waste mentality has to end.
  - brazzy 7y ago
    Premature optimization will never become the right thing to do. Most code has absolutely no reason to be particularly fast because it doesn't get executed very frequently.
    - carlmr 7y ago
      Nobody said that. Just that you should think about which algorithms you use, profile, and not waste so much. The wasted power at this point is significant.
      - brazzy 7y ago
        That is sensible, but I've also heard people basically call any code that is not micro-optimized wasteful and harp on about how back in the day it was second nature for programmers to care about bytes and cycles in every line of code they wrote, and modern day programmers are just lazy and incompetent.
        carlmr 7y ago
        Yeah, I get where you're coming from. There's as always a sweet spot between the two extremes. But needing a new computer for running Word or Excel smoothly just seems like there's a lot of low hanging fruit left.
- foota 7y ago
  I'm not sure that's a given, who's to say we won't be happy with the current performance we have?
  - skohan 7y ago
    There are still problems, like protein folding, or training larger and larger deep learning models, which have nowhere near the performance you'd actually want in a perfect world. There's always going to be a bigger target in some of these domains, and it would be a shame to imagine we'll reach a ceiling in some areas of scientific inquiry because of hardware limitations.
    - foota 7y ago
      This is true, but imo most the gains outside of traditional CPU improvements will be either in specialized hardware or algorithms, I don't see much room for improvement in like optimizing code there.
- pmoriarty 7y ago
  It's not just training AI models that's demanding exponentially increasing compute power, but also streaming, storing, and processing the enormous amounts of scientific data coming from (for example) high-end physics experiments at world-class laboratories like CERN or the LHC, and there's also the tremendous amounts of surveillance data and data processing that's sprung up in recent decades, with an insatiable appetite of various agencies and companies to track, record, and analyze everyone and everything.
  - ben_w 7y ago
    True, but from what people were telling me before I left university, scientific processing always produced more data than could be analysed, even after accounting for the effort put into efficient algorithms.
- WizardAustralis 7y ago
  We will probably start to see this a lot in the video game consoles field. On the hardware front, looking at the early specs of the PS5 for instance show them emphasis faster storage access speed more so than added CPU/GPU/RAM. Consoles were always more likely to see speed optimizations, but this had slid aside a bit in the last two decades.
- coldtea 7y ago
  >Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. Or just making slower things more orthogonally parallelized and throwing more hardware units at the problem (as opposed to more powerful hardware or faster code). Most code by engineers "who write code fast" is not in performance critical domains...
  - brudgers 7y ago
    I agree mostly. But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket?
    - oblio 7y ago
      If my experience with the IT world has taught me anything, you'd be surprised for how long :-)
    - coldtea 7y ago
      >But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket? Why would they need to? They can sell improved cameras and other additions instead of improved CPUs...
      - brudgers 7y ago
        I agree there's no need. What phone camera improvements do you envision that don't involve more transistors on the CMOS sensor, GPU, CPU etc? Microfilm?
  - jerf 7y ago
    A lot of the ways programmers write slow code ends up with multiplicative factors you can't practically recover with simply throwing more hardware at it, because it's really easy for us to write something 100x or 1000x slower (even in the same big-O class) by using a slow language with poor cache locality, that makes it easy to do more work than was necessary, by poorly using threading resources, etc. There's a lot of multiplicative factors that can stack together fairly quickly. On the plus side, it turns out recovering most of that isn't that hard with a bit of careful language selection and just a bit of care in writing code. I think this is one of the somewhat subtle reasons for Go's success is that it trims away most of the slow things that a lot of "scripting" languages do without throwing away too much of the power, and I anticipate the continued entry and success of other languages like Nim into this space on the language chart in the next few years. To get to screaming fast may take a lot of effort, but "pretty fast, let's parallelize what we have instead of hyperoptimizing it" isn't too hard. Because of all of these multiplicative slowdowns, even at very small scales it's still generally a better idea to start by optimizing and removing some of your multiplicative slowdowns before you just throw more hardware at it. Compute is merely cheap, not free. It's the rare bit of code that nobody's spent any time optimizing, but is also free of these multiplicative slowdown problems.
- caseymarquis 7y ago
  In my experience, poor performance is only very occasionally the fault of abstractions and frameworks. The problem is near universally between the chair and the keyboard. I don't think we'll be throwing away the old abstractions, but adding new ones in beside them. As an example, I made a demo a couple years back which processed and rendered about 64 MB of CNC G Code (a programming language for telling robots how to move and spin) in a web browser to match CNC toolpaths (the path of the robots motion) to their realtime run statistics (show me a realtime 3d map of where we can speed up the robot). To do this, you need to simulate the movements of the machine to build a 3d model, and then pair the model to collected data. You then need to cull/average data which is too fine to be visible, and ship this processed data to the browser as arrays of floating point numbers, where it gets rendered as -effectively- thousands of colored 3d arcs. The naive implementation took minutes to run. The final version took about 1 second. I didn't have to throw away any abstractions or frameworks to make things faster. I had to use my language's new abstractions for massively parallel processing and stack allocated data to speed up server side processing, and then use libraries for decoding base64 binary into floating point arrays on the browser side to speed up data transfer and parsing. I guess my point is that the end of Moore's law doesn't mean throwing old things away, it means using existing solutions when appropriate, and approaching new problems in different ways.
- Shorel 7y ago
  "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." The end of Moore's law is the end of passing up opportunities in that critical 3%. There's a lot of performance debt accumulated. It's only 3%. Optimize when it matters!
- streetcat1 7y ago
  Correct. As long as your bottleneck is the CPU.
- fulafel 7y ago
  Is there basis for thinking that the relative demand of fast-code programmers will increase vs code-fast programmers? There doesn't seem to be anything in the horizon that would make performance a more pressing concern in the work of the 99% of developers who do make business software and consumer web/mobile apps. They just sometimes call into libraries that have had performance work put into them and get by with rudimentary understanding of what seems to cause slowdowns. In the other dimension, there does seem to be a continued focus and resulting improvements in velocity. The recent ML popularity is all about leveraging packaged optimized code (Tensorflow etc) with vast majority of programmers freed from the engine level details and detailed understanding of GPU perf work or parallel/distributed programming.
narrator 7y ago
It'd be really weird if we could keep expanding Moore's law and surpass the brain on a performance per watt ratio.
- hyko 7y ago
  Why?
_bxg1 7y ago
Honestly, this is a bit of a relief. All of the AI nightmare scenarios (be they the Terminator kind, or the more realistic hyper-empowerment-of-a-few-elites kind) rely on that exponential growth continuing unimpeded. If there's no exponential growth, there's no runaway AI that swiftly outpaces human understanding.
- npo9 7y ago
  Let me fuel your paranoia a little. It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Quantum computers might play a role in increasing computational power in the next decade. Right now a lot of AI research is low hanging fruit recently made available with advances in CPU and GPU. After the advances dry up maybe more fundamental research will produce much better results with our current technology.
  - _bxg1 7y ago
    Thanks :P
  - martinpw 7y ago
    > It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Why would cloud computing continue to see significant decreases in costs once Moore's law stops? Clearly there will be some ongoing decline in infrastructure costs, but if your main requirement is for lots of compute (for AI) then it seems hard to maintain an exponential rate of improvement overall if the silicon is no longer getting faster and cheaper, and the exponential is what you need, else your costs get out of control quickly.
    - npo9 7y ago
      I was speaking about linear decreases. Cheaper power, more refined manufacturing processes, and economy of scale benefits are where I expect some of the linear benefits to come from. We can also expect CPU manufactures to compete for awhile, getting increasingly marginal improvements. Linear improvements will get us there in the end.
jacobcammack 7y ago
I don’t agree fundamentally with this article. Correct details, but missing the Forest for the trees. There are too many elements at work evolutionarily speaking to ignore the emergence of God knows what. Plus our AI (not to mention our most basic computing axioms) are absolutely juvenile as we are brand new as a species to be developing anything at all that has to do with computing. That doubly goes for “AI”... whatever that means.
m3at 7y ago
The "Bitter Lesson" post from Rich Sutton from earlier this year [1] seems a very good complement to this article: he explain how all of the big improvements in the field came from new methods that leveraged the much larger compute available from Moore's law, instead of progressive buildup over existing methods. A great quote from McCarthy also regularly referenced by Sutton is "Intelligence is the computational part of the ability to achieve goals", which (IMO) help picture the tight link between compute growth and AI. It's only a few minutes read, I highly recommend it: [1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- _bxg1 7y ago
  It could be that the slowing of Moore's Law results in a more diverse array of specialized computing hardware. After all, the resurgence of ML didn't happen because x86 got fast enough, but because video games - of all things - funded the maturation of a whole new category of massively-parallel chips.
  - codesushi42 7y ago
    the resurgence of ML ... because video games - of all things - funded the maturation of a whole new category of massively-parallel chips Fake news. It has far more to do with the rise of distributed computing than the existence of GPUs.
    - jchw 7y ago
      I don’t really see this; All of the currently popular machine learning frameworks either support GPU or are oriented around GPU based execution. For most developers working on AI it is still the defacto standard.
      - sgt101 7y ago
        I think.. both. Distributed computing (Hadoop) was why a lot of data got collected and made available(ish). GPU's are the engines that made CNN's (in particular) tractable, and opened up a bunch of applications for many companies, and opened up a reasonable route to results for a generation of researchers.
sdoken123 7y ago
Could it be that with Quantum Computing computational power will continue to increase exponentially?
- yarg 7y ago
  For some problems, perhaps - but it could also be the case maintaining entanglement becomes exponentially harder with an increasing number of qubits.
- npo9 7y ago
  Is Quantum Computing computational power increasing exponentially now?
  - yarg 7y ago
    Yes - and on two fronts. The number of qubits that can be entangled is currently going up at an exponential rate, and with each additional qubit the power of the machine (in a certain sense) doubles.
  - buzzkillington 7y ago
    Yes, we have gone from being able to factor the number 1 to being able to factor the number 2 in the last 20 years.
orasis 7y ago
One factor: the computational cost of training any extremely expensive model is amortized across all uses of that model.
e_carra 7y ago
Modern hardware has lot to improve, memories and transmission lines have been the main bottlenecks for more than a decade now, bigger caches helped but are not enough. Solutions are being developed, like on-circuit optical fiber transmission lines, but it takes time.
boyadjian 7y ago
Maybe it is a good thing, so the singularity will be avoided. There is so much wrong usages of AI.
- mikorym 7y ago
  Mathematician here, I don't think the singularity as popularised through people like Ray Kurzweil would happen. In particular, their metaphor of an "infinity" point seems to me like the incorrect application of mathematical ideas to social contexts. I think the question of artificial general intelligence is a different question than a singularity, much like how the Chinese room addresses a different question than the Turing test.
  - TomMarius 7y ago
    Why is it a different question? AFAIK the assumption is that if the computer can improve itself (which it can if it has IQ 100), there is nothing holding the singularity back.
    - TheOtherHobbes 7y ago
      Which is dead wrong, because not only is there no usable definition of "improve itself", but there isn't even any understanding of the kinds of skills required to create a usable definition. It's the difference between a computer that is taught how to compose okay-ish music, and a computer that learns spontaneously how to compose really really great music and do all of the social, cultural, and financial things required to create a career for itself as a notable composer and then does something entirely new and surprising given that starting point. They're completely different problem classes, operating on completely different levels of sophistication and insight. A lot of "real" AI problems are cultural, social, psychological, and semantic, and are going to need entirely new forms of meta-computation. You're not going to get there with any current form of ML, no matter how fast it runs, because no current form of ML can represent the problems that need to be solved to operate in those domains - never mind spontaneously generate effective solutions for those problems.
ilaksh 7y ago
General purpose AI is not waiting for more compute power. Its waiting for algorithms that can do it. I'm not 100% sure deep learning can get all the way there. We might need a new approach. But the deep learning leaders are taking it very seriously and making some progress. For example, Yann LeCun is talking about self-supervised, learning models of the world. MILA (Bengio's group) is talking about "state representation learning, or the ability to capture latent generative factors of an environment". Hinton now has capsule networks. In my opinion these types of approaches are very promising for general purpose AI.
- jononor 7y ago
  Yeah, many ML applications are bottlenecked on availability of labels, not by compute. Especially once outside very well-defined and established task. I think self-supervised has large potential, at least in expanding ML applications towards "more general".
sprash 7y ago
I predict there will be much more assembly programming required in the future to squeeze out as much as performance as possible because the end of Moore's Law is already very apparent for several applications that can not be easily parallelized. It is a complete myth that you "can't beat the C-compiler" as it is claimed so often. The compiler can't know many things you know about the problem at hand. So far I have been able to beat gcc on a regular basis while just relying on x86_64 (which means using no instructions/registers beyond SSE2.0). Right now I'm looking for a C/Shader-like language for x86_64/Linux which allows you to be much closer to the metal without requiring you to go down to the cumbersome level of ASM syntax and at the same time shedding libc and use syscalls instead (E.g. if you have specific knowledge about the data structures you want to map you can use the heap much more efficient than plain old malloc). So far I found nothing.
- Shorel 7y ago
  "Right now I'm looking for a C/Shader-like language for x86_64/Linux" You seem like the ideal candidate for writing such a language. And the book about it.
  - deleted 7y ago
    [deleted]
- deleted 7y ago
  [deleted]
skohan 7y ago
I predict that the slowing of Moore's law is going to make HPC and optimization in general a much more valuable skillset in the next few decades. In the past 15 years or so, we've been more or less happy to treat CPU cycles as a limitless resource, and as a result modern software stacks have a lot of fat in them. At the end of the era of free speed increases, trimming the fat is going to be a lot more important.
- pjmlp 7y ago
  You already see that ongoing with Java and .NET, refocusing on having AOT as part of the toolchain instead of some third party toolchain, more language features for lowlevel control and GPGPU access as well. Or the pendulum going back to compiled languages after the scripting craziness of the last decade.
- oblio 7y ago
  It depends where this is going to happen. Libraries, OSes, RDBMSes, etc. will have to be optimized for sure. But actual apps? There's way more app writers than library writes. And most line-of-business apps don't need to be faster, they need to be easier to use and more stable.
- streetcat1 7y ago
  CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.
  - zozbot234 7y ago
    True enough, but modern software stacks also have a lot of wasteful memory use.
    - streetcat1 7y ago
      Right. Hence Moore's law is irrelevant. I.e. the CPU is NOT the bottleneck. For example, who cares if you have 64 core , 5Ghz cpu, if the still doing I/O from an HDD.
  - 7y ago
fyp 7y ago
Since AI is heavily parallelizable, it only matters that cost(as in dollars) will keep exponentially decreasing. It doesn't matter if you can't double the transistor density of a single cpu if you can just double the number of machines. At the end of the day you still managed to double performance for the same price. See https://en.wikipedia.org/wiki/FLOPS#Hardware_costs https://en.wikipedia.org/wiki/FLOPS#Hardware_costs (note: in another thread someone noted that this wiki is outdated/inaccurate. If anyone have the relevant expertise they should help edit it)
- 7thaccount 7y ago
  Some problems don't parallelize well (some mixed integer programming problems) and some matrix operations. We look at the end of Moore's law with horror.
- 4NDR10D 7y ago
  Hardware Acceleration/Parallization is the next frontier. We've already seen the benefit of some pretty simple ASICs (TPU was built to be simple) as well as more general purpose accelerators. Hardware architects used to have a hard time, because often the best option was to simply wait for CPUs to get faster. Now that we've seen CPU power begin to stall it makes economic sense not only to invest in more parallel software but more appliciation specific accelerators. CPUs/GPUs are beasts of hardware architecture, being complex mostly due to their flexibility. We can achieve higher performance with dedicated hardware (or FPGAs), and it looks like the economic reasons to do so are slowly becoming more certain.
shusson 7y ago
> The takeaway is that, even if we assume great efficiency breakthroughs and performance improvements to reduce the rate of doubling, AI progress seems to be increasingly compute-limited at a time when our collective growth in computing power is beginning to falter A similar conclusion can be made for genomics too.
Symmetry 7y ago
For a long time we had a situation where transistors got smaller and cheaper and faster and more power efficient all at the same through the wonders of Denard scaling and we gestured broadly at the whole thing and called it all "Moore's Law" without needing to distinguish which exponential improvement the term referred to. But in the mid 2000s Denard scaling broke down and now it looks like transistors are still getting smaller and cheaper and more power efficient but they aren't getting exponentially faster any more. So we've mostly settled on the smaller bit as being the true "Moore's Law" and that's more or less kept going but might be running out of steam. It's still very nice from the perspective of very parallel tasks but as consumers we don't see as much benefit from it in our computers except in graphics. But sooner or later Moore's Law will run out too, transistors can only get so small as long as they're made of atoms. Luckily the laws of physics have precise limits for just how efficient computation can be, Landauer's principle, that are just as binding as the laws for how efficient a heat engine can be. Progress in engine efficiency did indeed form a converging sigmoid on what's allowed by Carnot. But we're still quite a distance away from Landauer's limit and if transistors can't get us there we have every reason to believe that other computational substrates are possible and can. In the meantime we might be looking at an interregnum. But while AI is gobbling up computational cycles because they're available there's no reason to think that efficiency gains aren't possible - just look at the orders of magnitude improvement in training resources from AlphaGo to AlphaZero. I don't see that this should stop progress, though it might slow it. [1]https://en.wikipedia.org/wiki/Landauer%27s_principle https://en.wikipedia.org/wiki/Landauer%27s_principle
- rrss 7y ago
  Transistors haven't really been getting smaller for the last few nodes, they've mostly just been getting denser. AFAIK nobody currently knows how make finfets much smaller without making them awful.
- marcosdumay 7y ago
  Transistors haven't been getting cheaper either. New processes have a higher cost per transistor for a while. Or better, as old processes get improved, transistors are getting either more compactly packed or cheaper, but not both. The expensive ones are getting a little bit faster too, but not in proportion to the reduced die size.
lachlan-sneff 7y ago
The topic is quite controversial, but there is a path forward. There are several theoretical computing technologies that can get very close to the theoretical maximum allowed by physics (as well as being revisible), but we can't build them yet because nanofactories/molecular assemblers/whatever-you-call-them don't exist yet.
hyperpallium 7y ago
Why can't we just have bigger dies? Same density, more transistors. If yields are too low, connect smaller dies somehow.
- clarry 7y ago
  If you stop scaling node size down and start scaling die size up, the result will be higher prices and power consumption. If you don't mind, go buy a Threadripper or EPYC. Might consider getting multiple sockets while at it. Caveat: won't fit in your pocket. I'm sure there'll be additional interconnect issues to worry about as the physical cluster of dies gets larger and larger.
hyperpallium 7y ago
I am optimistic that Moore's Law will eventually recover, with a new technology, perhaps silicon-based, perhaps not. Information processing is not intrinsically limited by silicon - for example, mammalian brains are more powerful. But that's lomg-term, big-picture. Technologies can remain stagnant longer than you can remain alive.
- giacaglia 7y ago
  It's funny that the article mentions that processors have not increase their speed, and therefore Moore's law is dead. In fact, as many people know Intel has been struggling with their 10nm chip, and now with a new CEO, this might change. All other processor manufacturers are catching up and some are moving ahead of Intel. That's the whole reason Apple is trying to move away from X86 with their MACs
jobseeker990 7y ago
We could make AI more efficient by switching to analog. Otherwise we're encoding the real world into digital and then back into analog again.
CRUDite 7y ago
Moore's law may be dead, but it is still mind boggling to project if forwards.. Seth lloyd in his paper [1] on the limits of computation, mentions in 250 years computational density will equal that of a black hole (kilogram sized). [1] https://cds.cern.ch/record/396654/files/9908043.pdf https://cds.cern.ch/record/396654/files/9908043.pdf
vagab0nd 7y ago
> A couple of years ago I was talking to the CEO of an AI company who argued that AI progress was basically an S-curve, and we had already reached its top for sound processing, were nearing it for image and video, but were only halfway up the curve for text. No prize for guessing which one his company specialized in — but he seems to have been entirely correct. This feels a little off. To me, it feels like image has made the most progress, then text, then sound, then video. Anyone knows which company they were referring to?

114 points by ChuckMcM 7y ago

I wonder does the compute power per dollar have to level off so soon?

The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially. The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. A lot of frameworks and abstractions will be ripped out and thrown away as 'taking too much time for not enough benefit.'

cft 7y ago
Rust will fare well. Ruby won't.
- fouc 7y ago
  Ruby isn't in the critical path, so that's a bad example. In general, the real issue is the layers of abstractions between the hardware and the end user. The over complicated architectures. The constant re-invention of databases/operating systems/virtual machines at each layer.
  - oblio 7y ago
    > The constant re-invention of databases/operating systems/virtual machines at each layer. Those are because of the human struggle, the political and economic fights. They're not because of purely technical problems. Each layer is usually backed by a party with a very clear interest, usually control, and each layer on top tries to escape that control. It's been that way since the times of p-code.
  - clarry 7y ago
    > The constant re-invention of databases/operating systems/virtual machines I wish that were the case but operating systems and systems software in general seems to be the most stagnant field. Everyone just buys the same FLOSS stack for $0 and compatibility is king, so there's very little research going in this field and much less of it ends up in any product you're ever going to see in use. I would love to see a vibrant scene of competing operating systems and databases with fresh new ideas and research.
    - fouc 7y ago
      I meant that it is re-implemented at every layer.. Like a browser is practically an OS on its own.
- ivalm 7y ago
  I don't have a hat in the game but why won't Ruby do well or not well based on this? It's not like basic webapps need to be much more efficient, and no one was writing high performance compute in ruby anyways.
  - DennisP 7y ago
    Looking over the techempower benchmarks, the fastest web frameworks are over fifty times faster than ruby/rails. If you can replace a hundred web servers with two, that matters.
    - ghettoimp 7y ago
      Sure. But an awful lot of folks don't have a hundred web servers. If you're running something simple, you can pretty much use whatever you want.
    - ativzzz 7y ago
      The vast majority of web apps are CRUD apps, which can achieve sub 100ms responses on any language with caching, background workers and minor optimizations. There will always be problems where you need faster responses, and the solutions to those problems are already not using Ruby. You don't use a hammer when a screwdriver is appropriate, but nails work pretty well in most places and screws are unnecessary.
      - DennisP 7y ago
        100ms with how many responses per second? Throughput matters too. And techempower does some pretty realistic-looking benchmarks. For example: > In this test, the framework's ORM is used to fetch all rows from a database table containing an unknown number of Unix fortune cookie messages (the table has 12 rows, but the code cannot have foreknowledge of the table's size). An additional fortune cookie message is inserted into the list at runtime and then the list is sorted by the message text. Finally, the list is delivered to the client using a server-side HTML template. The message text must be considered untrusted and properly escaped and the UTF-8 fortune messages must be rendered properly. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
- buzzkillington 7y ago
  FPGAs will fare well, general processors won't. You know we've reach the end of Moore's law when balanced ternary computers become the norm.
- csomar 7y ago
  That has a lot to do with why I took on learning Rust. My last year Macbook Pro has a 6-core CPU. At this rate, we should have 32-64-core CPUs for consumers in a few years. Not using all the processors means your program only makes use of 3-1.5% of the available power. My prediction is that, in a few years, lots of programming languages and software developers will simply become obsolete because what the other guys can produce is heaps of performance, security and manageability atop. But I might be wrong, so make your career decisions carefully.
  - oblio 7y ago
    I hope you're right. However Intel Core Duo was released in January 2006 and it had 2 cores. So in a bit more than 13 years we only have 3x as many cores. Core 2 Quad was released in January 2007, so if you count from that one, a bit more than 12 years for 1.5x as many cores. I wouldn't bet on widespread 64 core CPUs for consumers in the next 10 years or so. Most commercial software just doesn't use it efficiently and consumers don't need it for existing software. We need a software breakthrough, a killer app, something that people really want and that really needs a ton of cores. For now, I don't see it. But then again, that's how killer apps are, nobody sees them until they're there.
    - csomar 7y ago
      > However Intel Core Duo was released in January 2006 and it had 2 cores. So in a bit more than 13 years we only have 3x as many cores. Core 2 Quad was released in January 2007, so if you count from that one, a bit more than 12 years for 1.5x as many cores. https://xkcd.com/605/ https://xkcd.com/605/ If you are going to extrapolate, don't pick the date. Take all the numbers you have and trace the average value. If you are picking the numbers, you can extrapolate to any point you want. > I wouldn't bet on widespread 64 core CPUs for consumers in the next 10 years or so. We already have 32-core processors for servers. Imo, the biggest challenge will be the laptop and heat management rather than putting a bunch of processors together. > We need a software breakthrough, a killer app, something that people really want and that really needs a ton of cores. We need that now. Most software I'm using is slow. Part of why I picked the iPhone (and I bet many people did) was speed. It is fast. People care about speed and if you introduce them to something faster then they'll get hooked. Also, you are already using all of your cores because of multi-tasking. It makes your computer experience overall better. At 6 cores, multi-threading for an application doesn't give much noticeable experience, since you probably have 5-6 things running on your computer doing other stuff like network or an audio-player. > For now, I don't see it. But then again, that's how killer apps are, nobody sees them until they're there. > I hope you're right. Don't worry, just hang in there. The start of multi-cores was slow but then that's all there is right now. Seems intel/amd have mastered how to add more cores but the CPU frequency is pretty much capped going forward!
carlmr 7y ago
>Because it is going to demand engineers who can write fast code over engineers who write code fast. I surely hope so, the waste mentality has to end.
- brazzy 7y ago
  Premature optimization will never become the right thing to do. Most code has absolutely no reason to be particularly fast because it doesn't get executed very frequently.
  - carlmr 7y ago
    Nobody said that. Just that you should think about which algorithms you use, profile, and not waste so much. The wasted power at this point is significant.
    - brazzy 7y ago
      That is sensible, but I've also heard people basically call any code that is not micro-optimized wasteful and harp on about how back in the day it was second nature for programmers to care about bytes and cycles in every line of code they wrote, and modern day programmers are just lazy and incompetent.
      - carlmr 7y ago
        Yeah, I get where you're coming from. There's as always a sweet spot between the two extremes. But needing a new computer for running Word or Excel smoothly just seems like there's a lot of low hanging fruit left.
foota 7y ago
I'm not sure that's a given, who's to say we won't be happy with the current performance we have?
- skohan 7y ago
  There are still problems, like protein folding, or training larger and larger deep learning models, which have nowhere near the performance you'd actually want in a perfect world. There's always going to be a bigger target in some of these domains, and it would be a shame to imagine we'll reach a ceiling in some areas of scientific inquiry because of hardware limitations.
  - foota 7y ago
    This is true, but imo most the gains outside of traditional CPU improvements will be either in specialized hardware or algorithms, I don't see much room for improvement in like optimizing code there.
pmoriarty 7y ago
It's not just training AI models that's demanding exponentially increasing compute power, but also streaming, storing, and processing the enormous amounts of scientific data coming from (for example) high-end physics experiments at world-class laboratories like CERN or the LHC, and there's also the tremendous amounts of surveillance data and data processing that's sprung up in recent decades, with an insatiable appetite of various agencies and companies to track, record, and analyze everyone and everything.
- ben_w 7y ago
  True, but from what people were telling me before I left university, scientific processing always produced more data than could be analysed, even after accounting for the effort put into efficient algorithms.
WizardAustralis 7y ago
We will probably start to see this a lot in the video game consoles field. On the hardware front, looking at the early specs of the PS5 for instance show them emphasis faster storage access speed more so than added CPU/GPU/RAM. Consoles were always more likely to see speed optimizations, but this had slid aside a bit in the last two decades.
coldtea 7y ago
>Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. Or just making slower things more orthogonally parallelized and throwing more hardware units at the problem (as opposed to more powerful hardware or faster code). Most code by engineers "who write code fast" is not in performance critical domains...
- brudgers 7y ago
  I agree mostly. But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket?
  - oblio 7y ago
    If my experience with the IT world has taught me anything, you'd be surprised for how long :-)
  - coldtea 7y ago
    >But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket? Why would they need to? They can sell improved cameras and other additions instead of improved CPUs...
    - brudgers 7y ago
      I agree there's no need. What phone camera improvements do you envision that don't involve more transistors on the CMOS sensor, GPU, CPU etc? Microfilm?
- jerf 7y ago
  A lot of the ways programmers write slow code ends up with multiplicative factors you can't practically recover with simply throwing more hardware at it, because it's really easy for us to write something 100x or 1000x slower (even in the same big-O class) by using a slow language with poor cache locality, that makes it easy to do more work than was necessary, by poorly using threading resources, etc. There's a lot of multiplicative factors that can stack together fairly quickly. On the plus side, it turns out recovering most of that isn't that hard with a bit of careful language selection and just a bit of care in writing code. I think this is one of the somewhat subtle reasons for Go's success is that it trims away most of the slow things that a lot of "scripting" languages do without throwing away too much of the power, and I anticipate the continued entry and success of other languages like Nim into this space on the language chart in the next few years. To get to screaming fast may take a lot of effort, but "pretty fast, let's parallelize what we have instead of hyperoptimizing it" isn't too hard. Because of all of these multiplicative slowdowns, even at very small scales it's still generally a better idea to start by optimizing and removing some of your multiplicative slowdowns before you just throw more hardware at it. Compute is merely cheap, not free. It's the rare bit of code that nobody's spent any time optimizing, but is also free of these multiplicative slowdown problems.
caseymarquis 7y ago
In my experience, poor performance is only very occasionally the fault of abstractions and frameworks. The problem is near universally between the chair and the keyboard. I don't think we'll be throwing away the old abstractions, but adding new ones in beside them. As an example, I made a demo a couple years back which processed and rendered about 64 MB of CNC G Code (a programming language for telling robots how to move and spin) in a web browser to match CNC toolpaths (the path of the robots motion) to their realtime run statistics (show me a realtime 3d map of where we can speed up the robot). To do this, you need to simulate the movements of the machine to build a 3d model, and then pair the model to collected data. You then need to cull/average data which is too fine to be visible, and ship this processed data to the browser as arrays of floating point numbers, where it gets rendered as -effectively- thousands of colored 3d arcs. The naive implementation took minutes to run. The final version took about 1 second. I didn't have to throw away any abstractions or frameworks to make things faster. I had to use my language's new abstractions for massively parallel processing and stack allocated data to speed up server side processing, and then use libraries for decoding base64 binary into floating point arrays on the browser side to speed up data transfer and parsing. I guess my point is that the end of Moore's law doesn't mean throwing old things away, it means using existing solutions when appropriate, and approaching new problems in different ways.
Shorel 7y ago
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." The end of Moore's law is the end of passing up opportunities in that critical 3%. There's a lot of performance debt accumulated. It's only 3%. Optimize when it matters!
streetcat1 7y ago
Correct. As long as your bottleneck is the CPU.
fulafel 7y ago
Is there basis for thinking that the relative demand of fast-code programmers will increase vs code-fast programmers? There doesn't seem to be anything in the horizon that would make performance a more pressing concern in the work of the 99% of developers who do make business software and consumer web/mobile apps. They just sometimes call into libraries that have had performance work put into them and get by with rudimentary understanding of what seems to cause slowdowns. In the other dimension, there does seem to be a continued focus and resulting improvements in velocity. The recent ML popularity is all about leveraging packaged optimized code (Tensorflow etc) with vast majority of programmers freed from the engine level details and detailed understanding of GPU perf work or parallel/distributed programming.

narrator 7y ago

It'd be really weird if we could keep expanding Moore's law and surpass the brain on a performance per watt ratio.

hyko 7y ago
Why?

_bxg1 7y ago

Honestly, this is a bit of a relief. All of the AI nightmare scenarios (be they the Terminator kind, or the more realistic hyper-empowerment-of-a-few-elites kind) rely on that exponential growth continuing unimpeded. If there's no exponential growth, there's no runaway AI that swiftly outpaces human understanding.

npo9 7y ago
Let me fuel your paranoia a little. It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Quantum computers might play a role in increasing computational power in the next decade. Right now a lot of AI research is low hanging fruit recently made available with advances in CPU and GPU. After the advances dry up maybe more fundamental research will produce much better results with our current technology.
- _bxg1 7y ago
  Thanks :P
- martinpw 7y ago
  > It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Why would cloud computing continue to see significant decreases in costs once Moore's law stops? Clearly there will be some ongoing decline in infrastructure costs, but if your main requirement is for lots of compute (for AI) then it seems hard to maintain an exponential rate of improvement overall if the silicon is no longer getting faster and cheaper, and the exponential is what you need, else your costs get out of control quickly.
  - npo9 7y ago
    I was speaking about linear decreases. Cheaper power, more refined manufacturing processes, and economy of scale benefits are where I expect some of the linear benefits to come from. We can also expect CPU manufactures to compete for awhile, getting increasingly marginal improvements. Linear improvements will get us there in the end.

jacobcammack 7y ago

I don’t agree fundamentally with this article. Correct details, but missing the Forest for the trees. There are too many elements at work evolutionarily speaking to ignore the emergence of God knows what. Plus our AI (not to mention our most basic computing axioms) are absolutely juvenile as we are brand new as a species to be developing anything at all that has to do with computing. That doubly goes for “AI”... whatever that means.

m3at 7y ago

The "Bitter Lesson" post from Rich Sutton from earlier this year [1] seems a very good complement to this article: he explain how all of the big improvements in the field came from new methods that leveraged the much larger compute available from Moore's law, instead of progressive buildup over existing methods. A great quote from McCarthy also regularly referenced by Sutton is "Intelligence is the computational part of the ability to achieve goals", which (IMO) help picture the tight link between compute growth and AI. It's only a few minutes read, I highly recommend it: [1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

_bxg1 7y ago
It could be that the slowing of Moore's Law results in a more diverse array of specialized computing hardware. After all, the resurgence of ML didn't happen because x86 got fast enough, but because video games - of all things - funded the maturation of a whole new category of massively-parallel chips.
- codesushi42 7y ago
  the resurgence of ML ... because video games - of all things - funded the maturation of a whole new category of massively-parallel chips Fake news. It has far more to do with the rise of distributed computing than the existence of GPUs.
  - jchw 7y ago
    I don’t really see this; All of the currently popular machine learning frameworks either support GPU or are oriented around GPU based execution. For most developers working on AI it is still the defacto standard.
    - sgt101 7y ago
      I think.. both. Distributed computing (Hadoop) was why a lot of data got collected and made available(ish). GPU's are the engines that made CNN's (in particular) tractable, and opened up a bunch of applications for many companies, and opened up a reasonable route to results for a generation of researchers.

sdoken123 7y ago

Could it be that with Quantum Computing computational power will continue to increase exponentially?

yarg 7y ago
For some problems, perhaps - but it could also be the case maintaining entanglement becomes exponentially harder with an increasing number of qubits.
npo9 7y ago
Is Quantum Computing computational power increasing exponentially now?
- yarg 7y ago
  Yes - and on two fronts. The number of qubits that can be entangled is currently going up at an exponential rate, and with each additional qubit the power of the machine (in a certain sense) doubles.
- buzzkillington 7y ago
  Yes, we have gone from being able to factor the number 1 to being able to factor the number 2 in the last 20 years.

orasis 7y ago

One factor: the computational cost of training any extremely expensive model is amortized across all uses of that model.

e_carra 7y ago

Modern hardware has lot to improve, memories and transmission lines have been the main bottlenecks for more than a decade now, bigger caches helped but are not enough. Solutions are being developed, like on-circuit optical fiber transmission lines, but it takes time.

boyadjian 7y ago

Maybe it is a good thing, so the singularity will be avoided. There is so much wrong usages of AI.

mikorym 7y ago
Mathematician here, I don't think the singularity as popularised through people like Ray Kurzweil would happen. In particular, their metaphor of an "infinity" point seems to me like the incorrect application of mathematical ideas to social contexts. I think the question of artificial general intelligence is a different question than a singularity, much like how the Chinese room addresses a different question than the Turing test.
- TomMarius 7y ago
  Why is it a different question? AFAIK the assumption is that if the computer can improve itself (which it can if it has IQ 100), there is nothing holding the singularity back.
  - TheOtherHobbes 7y ago
    Which is dead wrong, because not only is there no usable definition of "improve itself", but there isn't even any understanding of the kinds of skills required to create a usable definition. It's the difference between a computer that is taught how to compose okay-ish music, and a computer that learns spontaneously how to compose really really great music and do all of the social, cultural, and financial things required to create a career for itself as a notable composer and then does something entirely new and surprising given that starting point. They're completely different problem classes, operating on completely different levels of sophistication and insight. A lot of "real" AI problems are cultural, social, psychological, and semantic, and are going to need entirely new forms of meta-computation. You're not going to get there with any current form of ML, no matter how fast it runs, because no current form of ML can represent the problems that need to be solved to operate in those domains - never mind spontaneously generate effective solutions for those problems.

ilaksh 7y ago

General purpose AI is not waiting for more compute power. Its waiting for algorithms that can do it. I'm not 100% sure deep learning can get all the way there. We might need a new approach. But the deep learning leaders are taking it very seriously and making some progress. For example, Yann LeCun is talking about self-supervised, learning models of the world. MILA (Bengio's group) is talking about "state representation learning, or the ability to capture latent generative factors of an environment". Hinton now has capsule networks. In my opinion these types of approaches are very promising for general purpose AI.

jononor 7y ago
Yeah, many ML applications are bottlenecked on availability of labels, not by compute. Especially once outside very well-defined and established task. I think self-supervised has large potential, at least in expanding ML applications towards "more general".

sprash 7y ago

I predict there will be much more assembly programming required in the future to squeeze out as much as performance as possible because the end of Moore's Law is already very apparent for several applications that can not be easily parallelized. It is a complete myth that you "can't beat the C-compiler" as it is claimed so often. The compiler can't know many things you know about the problem at hand. So far I have been able to beat gcc on a regular basis while just relying on x86_64 (which means using no instructions/registers beyond SSE2.0). Right now I'm looking for a C/Shader-like language for x86_64/Linux which allows you to be much closer to the metal without requiring you to go down to the cumbersome level of ASM syntax and at the same time shedding libc and use syscalls instead (E.g. if you have specific knowledge about the data structures you want to map you can use the heap much more efficient than plain old malloc). So far I found nothing.

Shorel 7y ago
"Right now I'm looking for a C/Shader-like language for x86_64/Linux" You seem like the ideal candidate for writing such a language. And the book about it.
- deleted 7y ago
  [deleted]
deleted 7y ago
[deleted]

skohan 7y ago

I predict that the slowing of Moore's law is going to make HPC and optimization in general a much more valuable skillset in the next few decades. In the past 15 years or so, we've been more or less happy to treat CPU cycles as a limitless resource, and as a result modern software stacks have a lot of fat in them. At the end of the era of free speed increases, trimming the fat is going to be a lot more important.

pjmlp 7y ago
You already see that ongoing with Java and .NET, refocusing on having AOT as part of the toolchain instead of some third party toolchain, more language features for lowlevel control and GPGPU access as well. Or the pendulum going back to compiled languages after the scripting craziness of the last decade.
oblio 7y ago
It depends where this is going to happen. Libraries, OSes, RDBMSes, etc. will have to be optimized for sure. But actual apps? There's way more app writers than library writes. And most line-of-business apps don't need to be faster, they need to be easier to use and more stable.
streetcat1 7y ago
CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.
- zozbot234 7y ago
  True enough, but modern software stacks also have a lot of wasteful memory use.
  - streetcat1 7y ago
    Right. Hence Moore's law is irrelevant. I.e. the CPU is NOT the bottleneck. For example, who cares if you have 64 core , 5Ghz cpu, if the still doing I/O from an HDD.
- 7y ago

fyp 7y ago

Since AI is heavily parallelizable, it only matters that cost(as in dollars) will keep exponentially decreasing. It doesn't matter if you can't double the transistor density of a single cpu if you can just double the number of machines. At the end of the day you still managed to double performance for the same price. See https://en.wikipedia.org/wiki/FLOPS#Hardware_costs https://en.wikipedia.org/wiki/FLOPS#Hardware_costs (note: in another thread someone noted that this wiki is outdated/inaccurate. If anyone have the relevant expertise they should help edit it)

7thaccount 7y ago
Some problems don't parallelize well (some mixed integer programming problems) and some matrix operations. We look at the end of Moore's law with horror.
4NDR10D 7y ago
Hardware Acceleration/Parallization is the next frontier. We've already seen the benefit of some pretty simple ASICs (TPU was built to be simple) as well as more general purpose accelerators. Hardware architects used to have a hard time, because often the best option was to simply wait for CPUs to get faster. Now that we've seen CPU power begin to stall it makes economic sense not only to invest in more parallel software but more appliciation specific accelerators. CPUs/GPUs are beasts of hardware architecture, being complex mostly due to their flexibility. We can achieve higher performance with dedicated hardware (or FPGAs), and it looks like the economic reasons to do so are slowly becoming more certain.

shusson 7y ago

> The takeaway is that, even if we assume great efficiency breakthroughs and performance improvements to reduce the rate of doubling, AI progress seems to be increasingly compute-limited at a time when our collective growth in computing power is beginning to falter A similar conclusion can be made for genomics too.

Symmetry 7y ago

For a long time we had a situation where transistors got smaller and cheaper and faster and more power efficient all at the same through the wonders of Denard scaling and we gestured broadly at the whole thing and called it all "Moore's Law" without needing to distinguish which exponential improvement the term referred to. But in the mid 2000s Denard scaling broke down and now it looks like transistors are still getting smaller and cheaper and more power efficient but they aren't getting exponentially faster any more. So we've mostly settled on the smaller bit as being the true "Moore's Law" and that's more or less kept going but might be running out of steam. It's still very nice from the perspective of very parallel tasks but as consumers we don't see as much benefit from it in our computers except in graphics. But sooner or later Moore's Law will run out too, transistors can only get so small as long as they're made of atoms. Luckily the laws of physics have precise limits for just how efficient computation can be, Landauer's principle, that are just as binding as the laws for how efficient a heat engine can be. Progress in engine efficiency did indeed form a converging sigmoid on what's allowed by Carnot. But we're still quite a distance away from Landauer's limit and if transistors can't get us there we have every reason to believe that other computational substrates are possible and can. In the meantime we might be looking at an interregnum. But while AI is gobbling up computational cycles because they're available there's no reason to think that efficiency gains aren't possible - just look at the orders of magnitude improvement in training resources from AlphaGo to AlphaZero. I don't see that this should stop progress, though it might slow it. [1]https://en.wikipedia.org/wiki/Landauer%27s_principle https://en.wikipedia.org/wiki/Landauer%27s_principle

rrss 7y ago
Transistors haven't really been getting smaller for the last few nodes, they've mostly just been getting denser. AFAIK nobody currently knows how make finfets much smaller without making them awful.
marcosdumay 7y ago
Transistors haven't been getting cheaper either. New processes have a higher cost per transistor for a while. Or better, as old processes get improved, transistors are getting either more compactly packed or cheaper, but not both. The expensive ones are getting a little bit faster too, but not in proportion to the reduced die size.

lachlan-sneff 7y ago

The topic is quite controversial, but there is a path forward. There are several theoretical computing technologies that can get very close to the theoretical maximum allowed by physics (as well as being revisible), but we can't build them yet because nanofactories/molecular assemblers/whatever-you-call-them don't exist yet.

hyperpallium 7y ago

Why can't we just have bigger dies? Same density, more transistors. If yields are too low, connect smaller dies somehow.

clarry 7y ago
If you stop scaling node size down and start scaling die size up, the result will be higher prices and power consumption. If you don't mind, go buy a Threadripper or EPYC. Might consider getting multiple sockets while at it. Caveat: won't fit in your pocket. I'm sure there'll be additional interconnect issues to worry about as the physical cluster of dies gets larger and larger.

hyperpallium 7y ago

I am optimistic that Moore's Law will eventually recover, with a new technology, perhaps silicon-based, perhaps not. Information processing is not intrinsically limited by silicon - for example, mammalian brains are more powerful. But that's lomg-term, big-picture. Technologies can remain stagnant longer than you can remain alive.

giacaglia 7y ago
It's funny that the article mentions that processors have not increase their speed, and therefore Moore's law is dead. In fact, as many people know Intel has been struggling with their 10nm chip, and now with a new CEO, this might change. All other processor manufacturers are catching up and some are moving ahead of Intel. That's the whole reason Apple is trying to move away from X86 with their MACs

jobseeker990 7y ago

We could make AI more efficient by switching to analog. Otherwise we're encoding the real world into digital and then back into analog again.

CRUDite 7y ago

Moore's law may be dead, but it is still mind boggling to project if forwards.. Seth lloyd in his paper [1] on the limits of computation, mentions in 250 years computational density will equal that of a black hole (kilogram sized). [1] https://cds.cern.ch/record/396654/files/9908043.pdf https://cds.cern.ch/record/396654/files/9908043.pdf

vagab0nd 7y ago

> A couple of years ago I was talking to the CEO of an AI company who argued that AI progress was basically an S-curve, and we had already reached its top for sound processing, were nearing it for image and video, but were only halfway up the curve for text. No prize for guessing which one his company specialized in — but he seems to have been entirely correct. This feels a little off. To me, it feels like image has made the most progress, then text, then sound, then video. Anyone knows which company they were referring to?