4 ms·

Does not look like very fancy or innovative. In the end, they just put a dedicated coprocessor directly with their memory chip. They named it AI because buzzwo

Does not look like very fancy or innovative. In the end, they just put a dedicated coprocessor directly with their memory chip. They named it AI because buzzword and marketing bullshit...

captain_price7 6y ago
AI is likely to be the biggest beneficiary of this architecture. The on-memory processing chips are likely to be simpler than the CPU ones (i.e. more akin to GPU cores), and allows parallelism- both of which point to numerical processing and AI.
- jabberwcky 6y ago
  I like the idea of flipping DIMMs to get capacity and processing improvements, and also the thought that mass-produced memory with this tech could potentially significantly reduce the cost of AI hardware through commoditization
Tuna-Fish 6y ago
AI is one of the very few domains where processing-in-memory makes sense. This particular system is a little slapdash, but I feel strongly that in the very near future, this is the only architecture that will be used for AI. (Meaning, relatively small chiplet AI processors with ram stacked on top of them.) The reason for that is that as the precision used for the coefficients has gone down, the relative energy cost of doing computation on them has turned into a rounding error when compared to the cost of moving data to the alus, and in AI there is very little cost of distributing the processing power into many small chips, which are relatively far away from each other.
- krona 6y ago
  By AI you mean matrix multiplication on single/half-precision floats? You could be a bit more specific.
BenoitP 6y ago
> They named it AI because buzzword and marketing bullshit... They named it AI because it massively boosts embarrassingly parallel workloads. You can think of Processing In Memory as rendering the mapPartitions() operation free in Spark's MapReduce ML workloads. Some algorithms like DNA sequencing have a tradeoff between map and reduce [1]: you spend more time generating higher quality matches between the short sequences (map), before sending them for global matching (reduce). And PIM lets you exploit that. For an order of magnitude: the average Intel has about 60GB/s of RAM bandwidth per socket. 256 GB of UPMEM's RAM let you have 2.5TB/s of local bandwidth to a computation unit (to 2560 'dumb' cores @400Mhz) [2]. [1] https://www.researchgate.net/publication/346703874_Variant_Calling_Parallelization_on_Processor-in-Memory_Architecture https://www.researchgate.net/publication/346703874_Variant_C... [2] https://www.upmem.com/technology/ https://www.upmem.com/technology/
- choppaface 6y ago
  I feel like push-down operations might be a better analogy from the mapreduce world? It strikes me these processors would be most helpful in pre-multiplies, filter operations, and perhaps for scatters. All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. While I’m sure the “AI” labeling is pure marketing, I’d imagine Samsung would love to target workloads beyond deep learning training and inference.
  - BenoitP 6y ago
    > All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. Yes! and that's the beauty of it. It is not an accelerator, these are fully generic cores. Not equivalent to 'smart' Intel cores with all the branch prediction, prefetching and caching magic; but with massive computation capabilities nonetheless. GPUs do have massive amounts of memory (both in RAM and registers), but you have to have preloaded your stuff into it beforehand. And what you can actually do efficiently are SIMD operations. I'd liken PIM to a better GPU-CPU blend: you get to keep your CPU doing its things with massive parallel operations concurrently. Also, these seem to be mostly independent cores, so you would not be limited to SIMD. Let's bet: in 10 years, AWS will have a new offering: the 'nano lambda'. You get a PIM core share, with 10 MB local 'persistent' RAM (keeping your data + a continuation of your code when it is not running), running your tiny Loom thread [1], at the edge, billed at 1us granularity, only when it is running, and for 0.0000000000000001 USD per us. [1] https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht...
jjcon 6y ago
I don’t think many people outside of AI will have a use for this (at least at first) so why would you market it any other way?
- smolder 6y ago
  Many computing tasks are very parallel as well as depending on memory bandwidth. I think it'd be useful for almost any of them.

Does not look like very fancy or innovative. In the end, they just put a dedicated coprocessor directly with their memory chip. They named it AI because buzzword and marketing bullshit...

captain_price7 6y ago

AI is likely to be the biggest beneficiary of this architecture. The on-memory processing chips are likely to be simpler than the CPU ones (i.e. more akin to GPU cores), and allows parallelism- both of which point to numerical processing and AI.

jabberwcky 6y ago
I like the idea of flipping DIMMs to get capacity and processing improvements, and also the thought that mass-produced memory with this tech could potentially significantly reduce the cost of AI hardware through commoditization

Tuna-Fish 6y ago

AI is one of the very few domains where processing-in-memory makes sense. This particular system is a little slapdash, but I feel strongly that in the very near future, this is the only architecture that will be used for AI. (Meaning, relatively small chiplet AI processors with ram stacked on top of them.) The reason for that is that as the precision used for the coefficients has gone down, the relative energy cost of doing computation on them has turned into a rounding error when compared to the cost of moving data to the alus, and in AI there is very little cost of distributing the processing power into many small chips, which are relatively far away from each other.

krona 6y ago
By AI you mean matrix multiplication on single/half-precision floats? You could be a bit more specific.

BenoitP 6y ago

> They named it AI because buzzword and marketing bullshit... They named it AI because it massively boosts embarrassingly parallel workloads. You can think of Processing In Memory as rendering the mapPartitions() operation free in Spark's MapReduce ML workloads. Some algorithms like DNA sequencing have a tradeoff between map and reduce [1]: you spend more time generating higher quality matches between the short sequences (map), before sending them for global matching (reduce). And PIM lets you exploit that. For an order of magnitude: the average Intel has about 60GB/s of RAM bandwidth per socket. 256 GB of UPMEM's RAM let you have 2.5TB/s of local bandwidth to a computation unit (to 2560 'dumb' cores @400Mhz) [2]. [1] https://www.researchgate.net/publication/346703874_Variant_Calling_Parallelization_on_Processor-in-Memory_Architecture https://www.researchgate.net/publication/346703874_Variant_C... [2] https://www.upmem.com/technology/ https://www.upmem.com/technology/

choppaface 6y ago
I feel like push-down operations might be a better analogy from the mapreduce world? It strikes me these processors would be most helpful in pre-multiplies, filter operations, and perhaps for scatters. All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. While I’m sure the “AI” labeling is pure marketing, I’d imagine Samsung would love to target workloads beyond deep learning training and inference.
- BenoitP 6y ago
  > All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. Yes! and that's the beauty of it. It is not an accelerator, these are fully generic cores. Not equivalent to 'smart' Intel cores with all the branch prediction, prefetching and caching magic; but with massive computation capabilities nonetheless. GPUs do have massive amounts of memory (both in RAM and registers), but you have to have preloaded your stuff into it beforehand. And what you can actually do efficiently are SIMD operations. I'd liken PIM to a better GPU-CPU blend: you get to keep your CPU doing its things with massive parallel operations concurrently. Also, these seem to be mostly independent cores, so you would not be limited to SIMD. Let's bet: in 10 years, AWS will have a new offering: the 'nano lambda'. You get a PIM core share, with 10 MB local 'persistent' RAM (keeping your data + a continuation of your code when it is not running), running your tiny Loom thread [1], at the edge, billed at 1us granularity, only when it is running, and for 0.0000000000000001 USD per us. [1] https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht...

jjcon 6y ago

I don’t think many people outside of AI will have a use for this (at least at first) so why would you market it any other way?

smolder 6y ago
Many computing tasks are very parallel as well as depending on memory bandwidth. I think it'd be useful for almost any of them.