4 ms·

Spoiler: the “Huang's law” this author is trying to coin is that AI processors (surely this means ML) are still following something like Moore's Law in terms of

Spoiler: the “Huang's law” this author is trying to coin is that AI processors (surely this means ML) are still following something like Moore's Law in terms of rate of improvement of processing speed per year. Which is cool I suppose but comparing it to Moore's Law seems a bit unfair.

CyberDildonics 6y ago
I feel that I took the days where headlines just said what an article was about for granted. Also it is extra silly because Moore's law is about transistor density and transistor cost.
- colejohnson66 6y ago
  I have heard Moore’s law generalized to “computing power doubles every two years” which could then apply to “AI”
  - DiogenesKynikos 6y ago
    There's a subtle difference, though. Moore's law benefits all applications, by increasing performance across the board. This new "Huang's law" increases performance for certain critical applications, while presumably decreasing performance of other applications (at fixed cost). The difference is generally increasing transistor density vs. specializing hardware to perform critical tasks (like graphics rendering or tensor algebra).
  - CyberDildonics 6y ago
    That's not a generalization, it's a misunderstanding. Not every program can benefit from doubling transistors because not every program is written to minimize serial computations and the impact of memory latency. The reason Moore's law was insightful was because he didn't make the simplistic generalization of computing power (which is more difficult to quantify anyway) and looked at what was really happening on a fundamental level from an engineering perspective.
- dsjoerg 6y ago
  Yes, the competition for attention used to be at the level of the whole newspaper, or the whole magazine. Now the attention competition is at the article level, and the headline is near the top of the funnel. To not be subjected to (as much of) this insanity, you can use a smart aggregator like HN or a well-moderated subreddit.
ghaff 6y ago
Whether you call it Huang's Law or recast Moore's Law to mean continuing performance improvements on the workloads where it most matters, there's definitely a widespread industry interest in promoting "Moore's Law is continuing." You saw it in full display at Hot Chips this year. And it's not even unreasonable. We collectively used transistors in general purpose CPUs in a way that wasn't most efficient but made it easy from a software and operational perspective. And that's changing now.
- computerphage 6y ago
  Do you mean that we clung to single threaded compute for too long? Like we spent too many transistors on just making one fast core because it's easy to write code to use one core?
  - ghaff 6y ago
    Intel at least absolutely clung to clock speed and one core for software reasons--in large part because of pressure from a certain very important software partner. But multicore on a single (mostly) general purpose architecture (leaving aside vector instructions) is still easier from a software perspective than the very heterogeneous architecture--vector, array, FPGA, etc.--we're increasingly adopting.
    - epmaybe 6y ago
      > very important software partner are you referring to microsoft?
      - ghaff 6y ago
        Yes. A senior Intel exec told me around the time multicore was coming in something to the effect of "Of course we knew we had to switch from frequency to multicore but Microsoft was terrified of the software implications of moving away from pure instruction level parallelism." This was of course somewhat self-serving given all the criticism Intel caught for being slow to adopt multicore. But there were certainly concerns about software support at the time, especially on the desktop. (In the end, it ended up not being as big a problem as some were fearing for various reasons. But it was a real fear at the time.)

CyberDildonics 6y ago

I feel that I took the days where headlines just said what an article was about for granted. Also it is extra silly because Moore's law is about transistor density and transistor cost.

colejohnson66 6y ago
I have heard Moore’s law generalized to “computing power doubles every two years” which could then apply to “AI”
- DiogenesKynikos 6y ago
  There's a subtle difference, though. Moore's law benefits all applications, by increasing performance across the board. This new "Huang's law" increases performance for certain critical applications, while presumably decreasing performance of other applications (at fixed cost). The difference is generally increasing transistor density vs. specializing hardware to perform critical tasks (like graphics rendering or tensor algebra).
- CyberDildonics 6y ago
  That's not a generalization, it's a misunderstanding. Not every program can benefit from doubling transistors because not every program is written to minimize serial computations and the impact of memory latency. The reason Moore's law was insightful was because he didn't make the simplistic generalization of computing power (which is more difficult to quantify anyway) and looked at what was really happening on a fundamental level from an engineering perspective.
dsjoerg 6y ago
Yes, the competition for attention used to be at the level of the whole newspaper, or the whole magazine. Now the attention competition is at the article level, and the headline is near the top of the funnel. To not be subjected to (as much of) this insanity, you can use a smart aggregator like HN or a well-moderated subreddit.

ghaff 6y ago

Whether you call it Huang's Law or recast Moore's Law to mean continuing performance improvements on the workloads where it most matters, there's definitely a widespread industry interest in promoting "Moore's Law is continuing." You saw it in full display at Hot Chips this year. And it's not even unreasonable. We collectively used transistors in general purpose CPUs in a way that wasn't most efficient but made it easy from a software and operational perspective. And that's changing now.

computerphage 6y ago
Do you mean that we clung to single threaded compute for too long? Like we spent too many transistors on just making one fast core because it's easy to write code to use one core?
- ghaff 6y ago
  Intel at least absolutely clung to clock speed and one core for software reasons--in large part because of pressure from a certain very important software partner. But multicore on a single (mostly) general purpose architecture (leaving aside vector instructions) is still easier from a software perspective than the very heterogeneous architecture--vector, array, FPGA, etc.--we're increasingly adopting.
  - epmaybe 6y ago
    > very important software partner are you referring to microsoft?
    - ghaff 6y ago
      Yes. A senior Intel exec told me around the time multicore was coming in something to the effect of "Of course we knew we had to switch from frequency to multicore but Microsoft was terrified of the software implications of moving away from pure instruction level parallelism." This was of course somewhat self-serving given all the criticism Intel caught for being slow to adopt multicore. But there were certainly concerns about software support at the time, especially on the desktop. (In the end, it ended up not being as big a problem as some were fearing for various reasons. But it was a real fear at the time.)