3 ms·

1- the chips are not efficient currently (graphic card reused as neural net) 10X-100X gain 2- Moore’s Laws 3- Algorithm/architecture improvement

by alfor 4y ago

DannyBee 4y ago
1. Im not sure who you believe will produce these chips, or who will use them. You are correct that specialized inference chips will get you a 10x gain. So what? If you want several million of them in a datacenter, that's a tall order. That's on top of A. the hundreds of millions it will take to get a design to production. B. The complete and total lack of allocation at anybody who could make you chips, except at very very very high cost, if at all - have you forgotten that automakers still can't get cheap chips made on older processes? Most allocation of newer processes is bought out for years. While there is some ability to get things made at newer process, building a chip for 7nm is 10-100x as expensive as say 45nm. C. The fact that someone has to be willing to build, plan, and execute putting them in datacenters. This will all happen, but like, everyone just assumes the hard part is the chip inefficiency. We already can make designs that are ~10x more efficient at inference (though it depends on if you mean power or speed or what). The fact that there are not millions in datacenters should tell you something about the difficulty and economics of accomplishing this. People aren't sitting around twiddling their thumbs. If Microsoft or Google or anyone could make themselves a "100x better cloud for AI", they would do it. 2. Dead. Dennard scaling went out the window years ago. Other scaling has mostly followed. The move to specialization you see and push to higher frequencies is because its all dead. 3. Will take a while. The economics of this kind of thing sucks. It will not change instantly, there isn't the capability to make it happen.
- ddalex 4y ago
  I think you missed the TPU, which is a Google chip that gets you the 10x in inference, and there are millions of them ALREADY in the datacenters, designed, fabricated and installed. You can use one, for free, with Colab
  - DannyBee 4y ago
    I know a surprising amount about of Google and TPU's :) This is not accurate - they are neither cheap nor really built for inference. I'd have to go look hard at what is public vs not if you want me to go into more.
- MobiusHorizons 4y ago
  TSMCs earnings show a significant decrease is demand this past quarter. AMD, Nvidia, and Intel all report falling demand. There will likely be allocation of 7nm and 5nm opening up even in the near future, especially as 4nm and 3nm come online in the next few years. Shortages of 28nm and older nodes are not indicative of other nodes, because 28nm is (or at least was) the most cost effective node per transistor, (so plenty of demand) but no new fabs are being built for that node.
- alfor 4y ago
  I am thinking about Tesla with Dojo and Tenstorrent. Both have a similar architecture (different scale) where they dich most of the vram for a fabric of identical cores. Instead of being limited by the vram bandwidth they run at the chip speed. Nvidia/Intel/AMD/Apple/Google and others surely have plans underway. As the demand for AI grow (now clear that there is a huge market) I think we will see more players enter this field. The landscape of software will have a dramatic shift, how much of the current cpu running in datacenter will be chips for AI in the future, I think it will be most of them. Jim Keller has a few good interviews about it.

1- the chips are not efficient currently (graphic card reused as neural net) 10X-100X gain 2- Moore’s Laws 3- Algorithm/architecture improvement

by alfor 4y ago

1- the chips are not efficient currently (graphic card reused as neural net) 10X-100X gain 2- Moore’s Laws 3- Algorithm/architecture improvement

DannyBee 4y ago

1. Im not sure who you believe will produce these chips, or who will use them. You are correct that specialized inference chips will get you a 10x gain. So what? If you want several million of them in a datacenter, that's a tall order. That's on top of A. the hundreds of millions it will take to get a design to production. B. The complete and total lack of allocation at anybody who could make you chips, except at very very very high cost, if at all - have you forgotten that automakers still can't get cheap chips made on older processes? Most allocation of newer processes is bought out for years. While there is some ability to get things made at newer process, building a chip for 7nm is 10-100x as expensive as say 45nm. C. The fact that someone has to be willing to build, plan, and execute putting them in datacenters. This will all happen, but like, everyone just assumes the hard part is the chip inefficiency. We already can make designs that are ~10x more efficient at inference (though it depends on if you mean power or speed or what). The fact that there are not millions in datacenters should tell you something about the difficulty and economics of accomplishing this. People aren't sitting around twiddling their thumbs. If Microsoft or Google or anyone could make themselves a "100x better cloud for AI", they would do it. 2. Dead. Dennard scaling went out the window years ago. Other scaling has mostly followed. The move to specialization you see and push to higher frequencies is because its all dead. 3. Will take a while. The economics of this kind of thing sucks. It will not change instantly, there isn't the capability to make it happen.

ddalex 4y ago
I think you missed the TPU, which is a Google chip that gets you the 10x in inference, and there are millions of them ALREADY in the datacenters, designed, fabricated and installed. You can use one, for free, with Colab
- DannyBee 4y ago
  I know a surprising amount about of Google and TPU's :) This is not accurate - they are neither cheap nor really built for inference. I'd have to go look hard at what is public vs not if you want me to go into more.
MobiusHorizons 4y ago
TSMCs earnings show a significant decrease is demand this past quarter. AMD, Nvidia, and Intel all report falling demand. There will likely be allocation of 7nm and 5nm opening up even in the near future, especially as 4nm and 3nm come online in the next few years. Shortages of 28nm and older nodes are not indicative of other nodes, because 28nm is (or at least was) the most cost effective node per transistor, (so plenty of demand) but no new fabs are being built for that node.
alfor 4y ago
I am thinking about Tesla with Dojo and Tenstorrent. Both have a similar architecture (different scale) where they dich most of the vram for a fabric of identical cores. Instead of being limited by the vram bandwidth they run at the chip speed. Nvidia/Intel/AMD/Apple/Google and others surely have plans underway. As the demand for AI grow (now clear that there is a huge market) I think we will see more players enter this field. The landscape of software will have a dramatic shift, how much of the current cpu running in datacenter will be chips for AI in the future, I think it will be most of them. Jim Keller has a few good interviews about it.