5 ms·

Prediction: sometime by 2033 most consumer CPUs will have the Apple Mx chip architecture with unified memory and the price for 2/4/16 TB of RAM will be similar

Prediction: sometime by 2033 most consumer CPUs will have the Apple Mx chip architecture with unified memory and the price for 2/4/16 TB of RAM will be similar with today's price for 2/4/16 TB of SSD. Today's price for 1TB of consumer RAM is around 3,000 USD; since it's paramount to be able to run these models locally, especially as they get better the privacy concerns will become even greater, it's time for hardware to step up to the challenge: x10 price reduction in 10 years, a new curve for Gordon Moore's observation.

brucethemoose2 4y ago
AMD"s 7000 laptops already have an Apple-like NPU, and Intel is following shortly after that. IDK about DRAM density breakthroughs though, in fact I kinda worry scaling will slow down over the next decade.
ketzo 4y ago
Question for someone smarter than me: What would it take for existing RAM to get that much better? I have a dim sense of the hardware industry, but I know that disk storage has gotten 10-100x cheaper, CPU/GPU power increases quite steadily, but RAM is still quite expensive. Is it even possible for RAM to get much, much better than it is currently? Are there physical limitations, or is there just not the same amount of market interest in more RAM?
- ly3xqhl8g9 4y ago
  Until now there wasn't much incentive for a consumer to have 1TB+ of RAM, for any CPU or GPU task: what were you going to do with that much RAM? render 1,000 more Chrome tabs? Beyond the supply/demand economics and also corporate greed [1], there are fabrication issues, just look at the current mess [2], but as new US-based fabs come online from 2025 and beyond [3], they will be solved to some extent. Theoretically, the maximum amount of RAM a 64-bit machine can address is around 16.8 million terabytes [4]: as always with computing, the journey hasn't even begun. [1] Easier and more money in manufacturing RAM for smartphones, https://fossbytes.com/flagship-phones-ram-vs-pc-ram https://fossbytes.com/flagship-phones-ram-vs-pc-ram [2] "EU tells Taiwan to forget about a bilateral investment pact even as bloc seeks more chips", https://www.scmp.com/news/china/article/3213163/eu-tells-taiwan-forget-about-bilateral-investment-pact-even-bloc-seeks-more-chips https://www.scmp.com/news/china/article/3213163/eu-tells-tai... [3] "U.S. Semiconductor Renaissance: All the Upcoming Fabs" (2022), https://www.tomshardware.com/news/new-us-fabs-everything-we-know https://www.tomshardware.com/news/new-us-fabs-everything-we-... [4] https://superuser.com/questions/65191/what-is-the-maximum-amount-of-ram-a-64bit-machine-can-theoretically-address https://superuser.com/questions/65191/what-is-the-maximum-am...
  - senttoschool 4y ago
    There isn't much incentive for a consumer to have 1TB of RAM in 2033 either. AI models that require a huge amount of memory will be done in the cloud. Will memory demand increase by 2033? Of course. Will demand reach 1TB of RAM for a consumer level computer/mobile device? No.
    - ketzo 4y ago
      Yeah, but I think it might legitimately 5 or 10x, if running models on-device is as helpful as it sounds.
    - visarga 4y ago
      > Will demand reach 1TB of RAM for a consumer level computer/mobile device? No. Speculative Sampling allows to run a smaller model locally and only check up with a larger model in the cloud from time to time, once every few tokens.
    - ikekkdcjkfke 4y ago
      There is incentive to use RAM as planned obselence though. As can be seen when updating old devices, forcing apps to swap frequently in and out of disk
      - _5uxp 4y ago
        I think nowadays Intel CPUs are more planned obselence that any other hardware, so behind.
    - dr_zoidberg 4y ago
      Today the upper range of RAM in devices is in the hundred-GB. 10 years ago it was about 16GB or so, and making a (probably bad) interpolation I think it wouldn't be crazy to have the upper range in the TB mark in 10 years time. We could get there faster too, for specific use cases (compiling, rendering, etc). On the other hand, yes, everyday use (web, mail, video/media consumption) don't require today much more than 8 or 16GB of RAM. If we go a bit creative, a PC with Linux can run very smoothly on 4GB alone, and surely someone here can point to their one anecdote of a machine with 2 GB or even 1 GB sporting a nice desktop environment, or 128mb CLI-only machine. Edit: also memory has to improve its bandwidth and data transfer rates to keep up with faster processors, so it could also improve over 10 years time without much focus on storage capacity. Or maybe they focus on latency instead, or a mix of all three. Point is that it's not a single metric to improve.
- magicalhippo 4y ago
  > Is it even possible for RAM to get much, much better than it is currently? DRAM currently consists of a small capacitor per bit (cell), which can hold a small amount of charge (1) or not (0). The amount of charge it can hold, its capacitance, depends on the area and the distance between the conductors[1]. So, as you try to cram more cells into the same area, the amount of capacitance goes down. This is problematic because of leakage[2], causing the charge to leave the capacitor. This is why DRAM cells are refreshed periodically, hence the "dynamic" in DRAM. Too much leakage, and you spend all the time refreshing rather than accessing the memory. To raise the capacitance back up, you can move the conductors closer. However, if the insulating material stays the same, this will lead to increased leakage. Thus a better insulating material is needed. While the current DRAM cells are quite impressive, it seems they're starting to hit fundamental road blocks with the current technology[3]. There's a limit to just how little amount of charge a cell can hold and still be reliably read, and finding better insulators is getting harder and harder. It may well be a new approach is the solution to continued scaling. Moving away from individual capacitors as the basis for a memory cell might be a way forward[4]. [1]: https://en.wikipedia.org/wiki/Capacitance#Capacitors https://en.wikipedia.org/wiki/Capacitance#Capacitors [2]: https://en.wikipedia.org/wiki/Leakage_(electronics)#In_capacitors https://en.wikipedia.org/wiki/Leakage_(electronics)#In_capac... [3]: https://www.techinsights.com/blog/dram-scaling-trend-and-beyond https://www.techinsights.com/blog/dram-scaling-trend-and-bey... [4]: https://spectrum.ieee.org/new-type-of-dram-could-accelerate-ai https://spectrum.ieee.org/new-type-of-dram-could-accelerate-...
  - ketzo 4y ago
    Exactly the kind of response I was hoping for. This is super interesting, thanks for this.
senttoschool 4y ago
Hm... The unified memory model doesn't mean it's faster for inference. Nvidia's GPUs run circles around Apple Silicon in training and inference right now. Though a part of that is software optimization. If anything, the reason to have unified memory SoC is to increase general efficiency, not to provide faster AI inference. I think chip makers will instead start making their NPU (such as Apple's neural engine) really massive. It'll start to take more transistors away from the CPU and GPU. By 2033, perhaps you're buying an NPU with a CPU and GPU attached to it, not the other way around like today. Also, 2/4/16TB of RAM will not have a 10x price reduction in 10 years. As far as I know, there's nothing in the horizon to do this. Prices will go down but it might stay relatively flat for the next 10 years. See this chart for $/GB: https://aiimpacts.org/wp-content/uploads/2020/04/memorytrend.png https://aiimpacts.org/wp-content/uploads/2020/04/memorytrend... Memory stopped getting exponentially cheaper some time around 2011. This is why the MacBook Air still starts with 8GB of RAM. The one thing that maybe could have helped was Intel's Optane tech, which they abandoned last year. It's memory that sits in between the speed of DRAM and SSDs.
- notfed 4y ago
  Eh, I'm not sure Optane was any faster than modern m.2 SSDs, was it?
  - senttoschool 4y ago
    Intel stopped innovating Optaine years ago. If Intel kept improving it, it would be much faster than SSDs today. Even today, an Optane drive is much faster in latency and important metrics like random reads.
- ly3xqhl8g9 4y ago
  "an NPU with a CPU and GPU attached to it" Sure, plenty of room for optimizations. "there's nothing in the horizon to do this" Sure, the prediction is that a new curve is laying just a bit beyond the horizon. MacBook Air still starts with 8GB of RAM for the same reason they still start with a 256 GB disk and for the same reason an 8 TB SSD from Apple means +$2,200†: corporate greed since it's good enough. The argument is it won't be good enough in the near future. † whereas a Samsung 870 8TB SSD is around 700 USD
colordrops 4y ago
I'm not clear oh what is unique about the privacy concerns of AI that will drive it off of the cloud unlike other services.
- ly3xqhl8g9 4y ago
  First of all, the issues of today are also of extreme importance [1], but there is a difference between some queries on a search engine or some numbers around your life (caloric intake, period regularity, etc.) and the deep and in-depth conversations people will have with the future chatbots (from personal issues bordering on psychological counselling to financial advice exposing desires and choices in a way a simple query for "restaurants near me" never could). [1] "Telehealth startup Cerebral shared millions of patients’ data with advertisers", https://news.ycombinator.com/item?id=35094866 https://news.ycombinator.com/item?id=35094866
  - colordrops 4y ago
    We aren't just talking about search. In fact you gave an example of something that is completely in the cloud - health services. Health, banking, education, and pretty much everything else are already in the cloud, and are as sensitive as anything that could happen using AI interactions.
PartiallyTyped 4y ago
There’s no way that TBs of ram will be accessible to the average consumer. Even in the previous decade we did not see any improvement. In memory availability. There has not been any need to go further than 32 GB for the general public for a long time now and even the average consumer does not have more than 16. There must be a real necessity for that to happen, and I do not see how LLMs provide substantial benefits to push for a change.
- ly3xqhl8g9 4y ago
  Running locally your own personal assistant that knows everything about you on a deeply personal, conversational level seems a rather real necessity. But since people seem not to care today who has their files, their emails, their health records, maybe I am wrong and the next 10 years will actually only bring us the first $100 trillion market cap company. LLMs are merely a technical detail, a stepping stone, the major breakthrough will be the first algorithm of a synthetic agent. Do we really want the boring dystopia HAL, Rehoboam, and so forth, or our own personal Star Trek Data.

brucethemoose2 4y ago

AMD"s 7000 laptops already have an Apple-like NPU, and Intel is following shortly after that. IDK about DRAM density breakthroughs though, in fact I kinda worry scaling will slow down over the next decade.

ketzo 4y ago

Question for someone smarter than me: What would it take for existing RAM to get that much better? I have a dim sense of the hardware industry, but I know that disk storage has gotten 10-100x cheaper, CPU/GPU power increases quite steadily, but RAM is still quite expensive. Is it even possible for RAM to get much, much better than it is currently? Are there physical limitations, or is there just not the same amount of market interest in more RAM?

ly3xqhl8g9 4y ago
Until now there wasn't much incentive for a consumer to have 1TB+ of RAM, for any CPU or GPU task: what were you going to do with that much RAM? render 1,000 more Chrome tabs? Beyond the supply/demand economics and also corporate greed [1], there are fabrication issues, just look at the current mess [2], but as new US-based fabs come online from 2025 and beyond [3], they will be solved to some extent. Theoretically, the maximum amount of RAM a 64-bit machine can address is around 16.8 million terabytes [4]: as always with computing, the journey hasn't even begun. [1] Easier and more money in manufacturing RAM for smartphones, https://fossbytes.com/flagship-phones-ram-vs-pc-ram https://fossbytes.com/flagship-phones-ram-vs-pc-ram [2] "EU tells Taiwan to forget about a bilateral investment pact even as bloc seeks more chips", https://www.scmp.com/news/china/article/3213163/eu-tells-taiwan-forget-about-bilateral-investment-pact-even-bloc-seeks-more-chips https://www.scmp.com/news/china/article/3213163/eu-tells-tai... [3] "U.S. Semiconductor Renaissance: All the Upcoming Fabs" (2022), https://www.tomshardware.com/news/new-us-fabs-everything-we-know https://www.tomshardware.com/news/new-us-fabs-everything-we-... [4] https://superuser.com/questions/65191/what-is-the-maximum-amount-of-ram-a-64bit-machine-can-theoretically-address https://superuser.com/questions/65191/what-is-the-maximum-am...
- senttoschool 4y ago
  There isn't much incentive for a consumer to have 1TB of RAM in 2033 either. AI models that require a huge amount of memory will be done in the cloud. Will memory demand increase by 2033? Of course. Will demand reach 1TB of RAM for a consumer level computer/mobile device? No.
  - ketzo 4y ago
    Yeah, but I think it might legitimately 5 or 10x, if running models on-device is as helpful as it sounds.
  - visarga 4y ago
    > Will demand reach 1TB of RAM for a consumer level computer/mobile device? No. Speculative Sampling allows to run a smaller model locally and only check up with a larger model in the cloud from time to time, once every few tokens.
  - ikekkdcjkfke 4y ago
    There is incentive to use RAM as planned obselence though. As can be seen when updating old devices, forcing apps to swap frequently in and out of disk
    - _5uxp 4y ago
      I think nowadays Intel CPUs are more planned obselence that any other hardware, so behind.
  - dr_zoidberg 4y ago
    Today the upper range of RAM in devices is in the hundred-GB. 10 years ago it was about 16GB or so, and making a (probably bad) interpolation I think it wouldn't be crazy to have the upper range in the TB mark in 10 years time. We could get there faster too, for specific use cases (compiling, rendering, etc). On the other hand, yes, everyday use (web, mail, video/media consumption) don't require today much more than 8 or 16GB of RAM. If we go a bit creative, a PC with Linux can run very smoothly on 4GB alone, and surely someone here can point to their one anecdote of a machine with 2 GB or even 1 GB sporting a nice desktop environment, or 128mb CLI-only machine. Edit: also memory has to improve its bandwidth and data transfer rates to keep up with faster processors, so it could also improve over 10 years time without much focus on storage capacity. Or maybe they focus on latency instead, or a mix of all three. Point is that it's not a single metric to improve.
magicalhippo 4y ago
> Is it even possible for RAM to get much, much better than it is currently? DRAM currently consists of a small capacitor per bit (cell), which can hold a small amount of charge (1) or not (0). The amount of charge it can hold, its capacitance, depends on the area and the distance between the conductors[1]. So, as you try to cram more cells into the same area, the amount of capacitance goes down. This is problematic because of leakage[2], causing the charge to leave the capacitor. This is why DRAM cells are refreshed periodically, hence the "dynamic" in DRAM. Too much leakage, and you spend all the time refreshing rather than accessing the memory. To raise the capacitance back up, you can move the conductors closer. However, if the insulating material stays the same, this will lead to increased leakage. Thus a better insulating material is needed. While the current DRAM cells are quite impressive, it seems they're starting to hit fundamental road blocks with the current technology[3]. There's a limit to just how little amount of charge a cell can hold and still be reliably read, and finding better insulators is getting harder and harder. It may well be a new approach is the solution to continued scaling. Moving away from individual capacitors as the basis for a memory cell might be a way forward[4]. [1]: https://en.wikipedia.org/wiki/Capacitance#Capacitors https://en.wikipedia.org/wiki/Capacitance#Capacitors [2]: https://en.wikipedia.org/wiki/Leakage_(electronics)#In_capacitors https://en.wikipedia.org/wiki/Leakage_(electronics)#In_capac... [3]: https://www.techinsights.com/blog/dram-scaling-trend-and-beyond https://www.techinsights.com/blog/dram-scaling-trend-and-bey... [4]: https://spectrum.ieee.org/new-type-of-dram-could-accelerate-ai https://spectrum.ieee.org/new-type-of-dram-could-accelerate-...
- ketzo 4y ago
  Exactly the kind of response I was hoping for. This is super interesting, thanks for this.

senttoschool 4y ago

Hm... The unified memory model doesn't mean it's faster for inference. Nvidia's GPUs run circles around Apple Silicon in training and inference right now. Though a part of that is software optimization. If anything, the reason to have unified memory SoC is to increase general efficiency, not to provide faster AI inference. I think chip makers will instead start making their NPU (such as Apple's neural engine) really massive. It'll start to take more transistors away from the CPU and GPU. By 2033, perhaps you're buying an NPU with a CPU and GPU attached to it, not the other way around like today. Also, 2/4/16TB of RAM will not have a 10x price reduction in 10 years. As far as I know, there's nothing in the horizon to do this. Prices will go down but it might stay relatively flat for the next 10 years. See this chart for $/GB: https://aiimpacts.org/wp-content/uploads/2020/04/memorytrend.png https://aiimpacts.org/wp-content/uploads/2020/04/memorytrend... Memory stopped getting exponentially cheaper some time around 2011. This is why the MacBook Air still starts with 8GB of RAM. The one thing that maybe could have helped was Intel's Optane tech, which they abandoned last year. It's memory that sits in between the speed of DRAM and SSDs.

notfed 4y ago
Eh, I'm not sure Optane was any faster than modern m.2 SSDs, was it?
- senttoschool 4y ago
  Intel stopped innovating Optaine years ago. If Intel kept improving it, it would be much faster than SSDs today. Even today, an Optane drive is much faster in latency and important metrics like random reads.
ly3xqhl8g9 4y ago
"an NPU with a CPU and GPU attached to it" Sure, plenty of room for optimizations. "there's nothing in the horizon to do this" Sure, the prediction is that a new curve is laying just a bit beyond the horizon. MacBook Air still starts with 8GB of RAM for the same reason they still start with a 256 GB disk and for the same reason an 8 TB SSD from Apple means +$2,200†: corporate greed since it's good enough. The argument is it won't be good enough in the near future. † whereas a Samsung 870 8TB SSD is around 700 USD

colordrops 4y ago

I'm not clear oh what is unique about the privacy concerns of AI that will drive it off of the cloud unlike other services.

ly3xqhl8g9 4y ago
First of all, the issues of today are also of extreme importance [1], but there is a difference between some queries on a search engine or some numbers around your life (caloric intake, period regularity, etc.) and the deep and in-depth conversations people will have with the future chatbots (from personal issues bordering on psychological counselling to financial advice exposing desires and choices in a way a simple query for "restaurants near me" never could). [1] "Telehealth startup Cerebral shared millions of patients’ data with advertisers", https://news.ycombinator.com/item?id=35094866 https://news.ycombinator.com/item?id=35094866
- colordrops 4y ago
  We aren't just talking about search. In fact you gave an example of something that is completely in the cloud - health services. Health, banking, education, and pretty much everything else are already in the cloud, and are as sensitive as anything that could happen using AI interactions.

PartiallyTyped 4y ago

There’s no way that TBs of ram will be accessible to the average consumer. Even in the previous decade we did not see any improvement. In memory availability. There has not been any need to go further than 32 GB for the general public for a long time now and even the average consumer does not have more than 16. There must be a real necessity for that to happen, and I do not see how LLMs provide substantial benefits to push for a change.

ly3xqhl8g9 4y ago
Running locally your own personal assistant that knows everything about you on a deeply personal, conversational level seems a rather real necessity. But since people seem not to care today who has their files, their emails, their health records, maybe I am wrong and the next 10 years will actually only bring us the first $100 trillion market cap company. LLMs are merely a technical detail, a stepping stone, the major breakthrough will be the first algorithm of a synthetic agent. Do we really want the boring dystopia HAL, Rehoboam, and so forth, or our own personal Star Trek Data.