4 ms·

> They named it AI because buzzword and marketing bullshit... They named it AI because it massively boosts embarrassingly parallel workloads. You can think of

> They named it AI because buzzword and marketing bullshit... They named it AI because it massively boosts embarrassingly parallel workloads. You can think of Processing In Memory as rendering the mapPartitions() operation free in Spark's MapReduce ML workloads. Some algorithms like DNA sequencing have a tradeoff between map and reduce [1]: you spend more time generating higher quality matches between the short sequences (map), before sending them for global matching (reduce). And PIM lets you exploit that. For an order of magnitude: the average Intel has about 60GB/s of RAM bandwidth per socket. 256 GB of UPMEM's RAM let you have 2.5TB/s of local bandwidth to a computation unit (to 2560 'dumb' cores @400Mhz) [2]. [1] https://www.researchgate.net/publication/346703874_Variant_Calling_Parallelization_on_Processor-in-Memory_Architecture https://www.researchgate.net/publication/346703874_Variant_C... [2] https://www.upmem.com/technology/ https://www.upmem.com/technology/

choppaface 6y ago
I feel like push-down operations might be a better analogy from the mapreduce world? It strikes me these processors would be most helpful in pre-multiplies, filter operations, and perhaps for scatters. All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. While I’m sure the “AI” labeling is pure marketing, I’d imagine Samsung would love to target workloads beyond deep learning training and inference.
- BenoitP 6y ago
  > All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. Yes! and that's the beauty of it. It is not an accelerator, these are fully generic cores. Not equivalent to 'smart' Intel cores with all the branch prediction, prefetching and caching magic; but with massive computation capabilities nonetheless. GPUs do have massive amounts of memory (both in RAM and registers), but you have to have preloaded your stuff into it beforehand. And what you can actually do efficiently are SIMD operations. I'd liken PIM to a better GPU-CPU blend: you get to keep your CPU doing its things with massive parallel operations concurrently. Also, these seem to be mostly independent cores, so you would not be limited to SIMD. Let's bet: in 10 years, AWS will have a new offering: the 'nano lambda'. You get a PIM core share, with 10 MB local 'persistent' RAM (keeping your data + a continuation of your code when it is not running), running your tiny Loom thread [1], at the edge, billed at 1us granularity, only when it is running, and for 0.0000000000000001 USD per us. [1] https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht...
  - choppaface 6y ago
    Huh I don't know about 'nano lambda', but I could see it being an add-on to S3. Customers already have petabytes upon petabytes there and if you could push down a filter or pre-multiply op and save a million bucks then sure people would do that. If EMR does it automagically then even better. When could it happen? My guess is S3 favors very old compute hardware but I could be wrong. If it does, though, that could also mean it's just very cheap to replace ;)
    - BenoitP 6y ago
      For predicate pushdown in S3, I'd see that operation in the SSD controller. You wouldn't even have to load the date in RAM before filtering it.
why_Mr_Anderson 6y ago
embarrassingly parallel workloads - so...GPU?
- BenoitP 6y ago
  GPU mostly does SIMD. This would enable to do independent workloads. If your simple single threaded webserver only requires 20MB of RAM, you could just run it in its tiny chunk of RAM; and it would sometimes ask the CPU to coordinate with the NIC. Also, a GPU requires transport of data to it. PIM lets the CPU do concurrent work at very low latencies.
ksec 6y ago
Well PIM or Processing In Memory or Computational Memory [1] isn't new. The Question is what exactly did they put in those memory or what are the target performance speed up in that specific domain. This PR provides embarrassingly little details. [1] https://en.wikipedia.org/wiki/Computational_RAM https://en.wikipedia.org/wiki/Computational_RAM
- BenoitP 6y ago
  Yep, this is indeed very little detail. To me this PR is the PR you send when you just discovered you're late, but try to retain the clients' attention. They felt the wind turning, saw the market ask for it, saw companies like UPMEM having a lead. If you want more details, UPMEM has more. If I recall correctly they etched their own core right into the same silicon as the DRAM. Here is a link that I think will satisfy your curiosity: https://www.upmem.com/nextplatform-com-2019-10-03-accelerating-compute-by-cramming-it-into-dram/ https://www.upmem.com/nextplatform-com-2019-10-03-accelerati...
  - tmotwu 6y ago
    If it interests you - stated at the bottom of the press release, they appear to be presenting a conference accepted paper on this. Looking at the ISSCC program, I presume this is what they're talking about: "A 20nm 6GB Function-In-Memory DRAM, Based on HBM2 with a 1.2TFLOPS Programmable Computing Unit Using Bank-Level Parallelism, for Machine Learning Applications" [1]. I suspect this is where there will be a deep dive into the gory technical details of the architecture and performance. It's tomorrow so it won't be a long wait. [1] https://submissions2.mirasmart.com/ISSCC2021/PDF/ISSCC2021AdvanceProgram.pdf https://submissions2.mirasmart.com/ISSCC2021/PDF/ISSCC2021Ad... (p 38)

choppaface 6y ago

I feel like push-down operations might be a better analogy from the mapreduce world? It strikes me these processors would be most helpful in pre-multiplies, filter operations, and perhaps for scatters. All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. While I’m sure the “AI” labeling is pure marketing, I’d imagine Samsung would love to target workloads beyond deep learning training and inference.

BenoitP 6y ago
> All that stuff is not just relevant to tensorflow / pytorch stuff but also databases. Yes! and that's the beauty of it. It is not an accelerator, these are fully generic cores. Not equivalent to 'smart' Intel cores with all the branch prediction, prefetching and caching magic; but with massive computation capabilities nonetheless. GPUs do have massive amounts of memory (both in RAM and registers), but you have to have preloaded your stuff into it beforehand. And what you can actually do efficiently are SIMD operations. I'd liken PIM to a better GPU-CPU blend: you get to keep your CPU doing its things with massive parallel operations concurrently. Also, these seem to be mostly independent cores, so you would not be limited to SIMD. Let's bet: in 10 years, AWS will have a new offering: the 'nano lambda'. You get a PIM core share, with 10 MB local 'persistent' RAM (keeping your data + a continuation of your code when it is not running), running your tiny Loom thread [1], at the edge, billed at 1us granularity, only when it is running, and for 0.0000000000000001 USD per us. [1] https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht...
- choppaface 6y ago
  Huh I don't know about 'nano lambda', but I could see it being an add-on to S3. Customers already have petabytes upon petabytes there and if you could push down a filter or pre-multiply op and save a million bucks then sure people would do that. If EMR does it automagically then even better. When could it happen? My guess is S3 favors very old compute hardware but I could be wrong. If it does, though, that could also mean it's just very cheap to replace ;)
  - BenoitP 6y ago
    For predicate pushdown in S3, I'd see that operation in the SSD controller. You wouldn't even have to load the date in RAM before filtering it.

why_Mr_Anderson 6y ago

embarrassingly parallel workloads - so...GPU?

BenoitP 6y ago
GPU mostly does SIMD. This would enable to do independent workloads. If your simple single threaded webserver only requires 20MB of RAM, you could just run it in its tiny chunk of RAM; and it would sometimes ask the CPU to coordinate with the NIC. Also, a GPU requires transport of data to it. PIM lets the CPU do concurrent work at very low latencies.

ksec 6y ago

Well PIM or Processing In Memory or Computational Memory [1] isn't new. The Question is what exactly did they put in those memory or what are the target performance speed up in that specific domain. This PR provides embarrassingly little details. [1] https://en.wikipedia.org/wiki/Computational_RAM https://en.wikipedia.org/wiki/Computational_RAM

BenoitP 6y ago
Yep, this is indeed very little detail. To me this PR is the PR you send when you just discovered you're late, but try to retain the clients' attention. They felt the wind turning, saw the market ask for it, saw companies like UPMEM having a lead. If you want more details, UPMEM has more. If I recall correctly they etched their own core right into the same silicon as the DRAM. Here is a link that I think will satisfy your curiosity: https://www.upmem.com/nextplatform-com-2019-10-03-accelerating-compute-by-cramming-it-into-dram/ https://www.upmem.com/nextplatform-com-2019-10-03-accelerati...
- tmotwu 6y ago
  If it interests you - stated at the bottom of the press release, they appear to be presenting a conference accepted paper on this. Looking at the ISSCC program, I presume this is what they're talking about: "A 20nm 6GB Function-In-Memory DRAM, Based on HBM2 with a 1.2TFLOPS Programmable Computing Unit Using Bank-Level Parallelism, for Machine Learning Applications" [1]. I suspect this is where there will be a deep dive into the gory technical details of the architecture and performance. It's tomorrow so it won't be a long wait. [1] https://submissions2.mirasmart.com/ISSCC2021/PDF/ISSCC2021AdvanceProgram.pdf https://submissions2.mirasmart.com/ISSCC2021/PDF/ISSCC2021Ad... (p 38)