5 ms·

At this rate they'll be manufacturing 0nm chips soon, and in a decade they'll be on -1nm. But despite the weird naming scheme, it's clear from transistor densi

by lwneal 4y ago

At this rate they'll be manufacturing 0nm chips soon, and in a decade they'll be on -1nm. But despite the weird naming scheme, it's clear from transistor density [1] and GPU prices [2] that foundries are still making progress in transistors per dollar. That progress is just barely beginning to make large neural networks (Stable Diffusion, vision and speech systems, language model AIs) deployable in consumer applications. It might not matter whether your cell phone renders this page in 1ms or 10ms, but the difference between talking to a 20B parameter language model and a 200B net is night and day [3]. If TSMC/Samsung/Intel can squeeze out just one or two more nodes, then by the middle of the century we might have limited general-purpose AI in every home and office. [1] https://en.wikipedia.org/wiki/Transistor_count#GPUs https://en.wikipedia.org/wiki/Transistor_count#GPUs [2] https://pcpartpicker.com/trends/price/video-card/ https://pcpartpicker.com/trends/price/video-card/ [3] https://textsynth.com/playground.html https://textsynth.com/playground.html

Ptchd 4y ago
They aren't even really 1nm.... but either way, they could switch to pico-meter.....
simpsond 4y ago
Intel chose Angstroms for their roadmap nodes: 20A = 2nm.
algo_trader 4y ago
> clear from .. that foundries are still making progress in transistors per dollar. Are they? Home GPU price have been elevated due to crypto. Latest node CPUs are mostly opaque long-term contracts. The xxBN transistors chips are priced at $xx,000. If you have a (better) reference I would love to see it.
- oezi 4y ago
  Just look at the latest Apple silicon. Transistor per dollar and TDP is showing great strides. Intel and Nvidia just got too lazy or fumbled their node progress.
  - Siira 4y ago
    Nvidia uses TSMC for fabrication.
alexose 4y ago
As a traditional "web dev" kind of hacker, I feel like I'm just sitting idly by while a massive transition happens underneath me. I understand roughly why this shift is happening (machine language proving to solve a whole raft of hard problems) and how it's happening (specialized chip designs for matrix math). But I don't understand where it's all going, or how I can plug into it. It feels like a fundamentally different landscape than what I'm used to. There's more alchemy, perhaps. Or maybe it's that the truly important models are trained using tools and data that are out of reach for individuals. Does anyone else feel this way? Better yet, has anyone felt this way and overcome it by getting up to speed in the ML world?
- largbae 4y ago
  Just like every other technological advance, there is a sort of "food chain" that builds on top of these foundational technologies. You didn't have to work in cryptography to play a role in the massive proliferation of online commerce and banking. There were and are many, many conventional tasks and non-PhD-making innovations between cryptography existing and the wonderful low-friction commerce we now enjoy. Don't know how language translation models work? No problem, use one that someone else made to make a web framework that self-internationalizes to the user's browser default language without the site creator even knowing that language exists!
  - alexose 4y ago
    That's certainly true! I can use my current skillset to help connect users with new tech. And there have been many minor revolutions during the course of my career, many of which have been incorporated into the sort of work I do. I guess the difference (for me, anyway) is that this change isn't incremental. It's a fundamentally new type of computing-- One that comes with a totally new way of approaching problems. Listening to Andrej Karpathy talk about Software 2.0, for instance, it seems probable that ML has a place in many parts of the stack. It's possible I'm just projecting my insecurities, here, but my experience has been that changes to computing hardware usually result in changes across the entire industry. And this feels like a pretty meaningful change.
- 4y ago
deleted 4y ago
[deleted]
bogomipz 4y ago
>"That progress is just barely beginning to make large neural networks (Stable Diffusion, vision and speech systems, language model AIs) deployable in consumer applications." Interesting. Is Stable Diffusion a product of neural network size then? Is the size of the network a function of chips density? Also is there a Stable Diffusion app that currently works on edge devices?
- emikulic 4y ago
  If edge devices includes my gamer PC then yes. For apps I'd recommend https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui
IIAOPSW 4y ago
I'll be bold enough to make the contrarian prediction. The approach of throwing ever more parameters in the model and ever more transistors on the chip is at best a brute force approach to AI and will likely plateau in effectiveness long before we get to "general purpose AI". We do not need 1nm neurons running at GHZ rates and training on a corpus of everything ever said just to comprehend language. There needs to be an algorithmic breakthrough. There is likely already more than enough processing power. Even bolder prediction: When we finally understand how the brain actually does it, the algorithmic improvement will be so enormous that the machine learning tasks which run on massive servers today will be able to run on the phone currently in your pocket.
- zarzavat 4y ago
  I feel the same way. What if there’s a better way to use those transistors? The semiconductor researchers spend a lot of effort to make ever-smaller transistors. What is a transistor? It’s a tiny switch. The ML researchers meanwhile use the language of linear algebra to define mathematical transformations of real numbers with nice differentiability properties. The chipmakers are then tasked with reconciling the two. So they use transistors to make gates. And gates to make adders. And adders to make integer multipliers. And integer multipliers to make floating point multipliers. And fp multipliers to implement matrix multiplication. And now you can run your cat diffuser model on those transistors. But what is the chance that the configuration of transistors in a floating point multiplier is anywhere close to the most efficient transistor configuration for learning? The only reason we’re using multiplication of real numbers is because the math people said so.
  - IIAOPSW 4y ago
    Since we are openly speculating, I think the missing ingredient is feedback loops. There is no explicit input side and output side of the brain. Its all just a ball of neurons. There is propagation delay between the neurons. This makes it possible to have self sustaining loops of neurons firing. The longer the loop, the longer the amount of time it takes to go full circle. We call this phenomena "brain waves". I think what we get wrong is that individual neurons rarely represent anything. They are a medium for the waves. The waves are the currency of thought. A brain is a series of electro-mechanical oscillators that resonates with abstract concepts and patterns. AFAIK, most research is still using the old "neurons represent single things" paradigm. Someone needs to tell them, there's no such thing as a "grandmother neuron".
ksec 4y ago
>At this rate they'll be manufacturing 0nm chips soon, and in a decade they'll be on -1nm. The industry has chosen angstroms as the next unit. Where 1.4nm will becomes 14A. ( Intel for now, but TSMC has uses the term in a few of its presentation ) >If TSMC/Samsung/Intel can squeeze out just one or two more nodes, We have a very solid roadmap all the way to 1nm, or 10A by the end of this decade. As long as the market is willing to pay for it. At least TSMC 3nm and 2nm is pretty much done. >limited general-purpose AI in every home Like the comment below, I am not convinced the brute force approach works. You can already build a 800mm2 NPU today that is equivalent to a chip used in "in every home and office." by the end of this decade. But we are still no where near it.

by lwneal 4y ago

Ptchd 4y ago

They aren't even really 1nm.... but either way, they could switch to pico-meter.....

simpsond 4y ago

Intel chose Angstroms for their roadmap nodes: 20A = 2nm.

algo_trader 4y ago

> clear from .. that foundries are still making progress in transistors per dollar. Are they? Home GPU price have been elevated due to crypto. Latest node CPUs are mostly opaque long-term contracts. The xxBN transistors chips are priced at $xx,000. If you have a (better) reference I would love to see it.

oezi 4y ago
Just look at the latest Apple silicon. Transistor per dollar and TDP is showing great strides. Intel and Nvidia just got too lazy or fumbled their node progress.
- Siira 4y ago
  Nvidia uses TSMC for fabrication.

alexose 4y ago

As a traditional "web dev" kind of hacker, I feel like I'm just sitting idly by while a massive transition happens underneath me. I understand roughly why this shift is happening (machine language proving to solve a whole raft of hard problems) and how it's happening (specialized chip designs for matrix math). But I don't understand where it's all going, or how I can plug into it. It feels like a fundamentally different landscape than what I'm used to. There's more alchemy, perhaps. Or maybe it's that the truly important models are trained using tools and data that are out of reach for individuals. Does anyone else feel this way? Better yet, has anyone felt this way and overcome it by getting up to speed in the ML world?

largbae 4y ago
Just like every other technological advance, there is a sort of "food chain" that builds on top of these foundational technologies. You didn't have to work in cryptography to play a role in the massive proliferation of online commerce and banking. There were and are many, many conventional tasks and non-PhD-making innovations between cryptography existing and the wonderful low-friction commerce we now enjoy. Don't know how language translation models work? No problem, use one that someone else made to make a web framework that self-internationalizes to the user's browser default language without the site creator even knowing that language exists!
- alexose 4y ago
  That's certainly true! I can use my current skillset to help connect users with new tech. And there have been many minor revolutions during the course of my career, many of which have been incorporated into the sort of work I do. I guess the difference (for me, anyway) is that this change isn't incremental. It's a fundamentally new type of computing-- One that comes with a totally new way of approaching problems. Listening to Andrej Karpathy talk about Software 2.0, for instance, it seems probable that ML has a place in many parts of the stack. It's possible I'm just projecting my insecurities, here, but my experience has been that changes to computing hardware usually result in changes across the entire industry. And this feels like a pretty meaningful change.
4y ago

deleted 4y ago

[deleted]

bogomipz 4y ago

>"That progress is just barely beginning to make large neural networks (Stable Diffusion, vision and speech systems, language model AIs) deployable in consumer applications." Interesting. Is Stable Diffusion a product of neural network size then? Is the size of the network a function of chips density? Also is there a Stable Diffusion app that currently works on edge devices?

emikulic 4y ago
If edge devices includes my gamer PC then yes. For apps I'd recommend https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui

IIAOPSW 4y ago

I'll be bold enough to make the contrarian prediction. The approach of throwing ever more parameters in the model and ever more transistors on the chip is at best a brute force approach to AI and will likely plateau in effectiveness long before we get to "general purpose AI". We do not need 1nm neurons running at GHZ rates and training on a corpus of everything ever said just to comprehend language. There needs to be an algorithmic breakthrough. There is likely already more than enough processing power. Even bolder prediction: When we finally understand how the brain actually does it, the algorithmic improvement will be so enormous that the machine learning tasks which run on massive servers today will be able to run on the phone currently in your pocket.

zarzavat 4y ago
I feel the same way. What if there’s a better way to use those transistors? The semiconductor researchers spend a lot of effort to make ever-smaller transistors. What is a transistor? It’s a tiny switch. The ML researchers meanwhile use the language of linear algebra to define mathematical transformations of real numbers with nice differentiability properties. The chipmakers are then tasked with reconciling the two. So they use transistors to make gates. And gates to make adders. And adders to make integer multipliers. And integer multipliers to make floating point multipliers. And fp multipliers to implement matrix multiplication. And now you can run your cat diffuser model on those transistors. But what is the chance that the configuration of transistors in a floating point multiplier is anywhere close to the most efficient transistor configuration for learning? The only reason we’re using multiplication of real numbers is because the math people said so.
- IIAOPSW 4y ago
  Since we are openly speculating, I think the missing ingredient is feedback loops. There is no explicit input side and output side of the brain. Its all just a ball of neurons. There is propagation delay between the neurons. This makes it possible to have self sustaining loops of neurons firing. The longer the loop, the longer the amount of time it takes to go full circle. We call this phenomena "brain waves". I think what we get wrong is that individual neurons rarely represent anything. They are a medium for the waves. The waves are the currency of thought. A brain is a series of electro-mechanical oscillators that resonates with abstract concepts and patterns. AFAIK, most research is still using the old "neurons represent single things" paradigm. Someone needs to tell them, there's no such thing as a "grandmother neuron".

ksec 4y ago

>At this rate they'll be manufacturing 0nm chips soon, and in a decade they'll be on -1nm. The industry has chosen angstroms as the next unit. Where 1.4nm will becomes 14A. ( Intel for now, but TSMC has uses the term in a few of its presentation ) >If TSMC/Samsung/Intel can squeeze out just one or two more nodes, We have a very solid roadmap all the way to 1nm, or 10A by the end of this decade. As long as the market is willing to pay for it. At least TSMC 3nm and 2nm is pretty much done. >limited general-purpose AI in every home Like the comment below, I am not convinced the brute force approach works. You can already build a 800mm2 NPU today that is equivalent to a chip used in "in every home and office." by the end of this decade. But we are still no where near it.