4 ms·

Telum is attached to the L2/L3 so it's that bandwidth as long as the model fits into it. Afterwards you go to memory and you really don't want to compare DDR4

Telum is attached to the L2/L3 so it's that bandwidth as long as the model fits into it. Afterwards you go to memory and you really don't want to compare DDR4 modules with RAIM against HBM3 or the likes that a current compute GPU uses. Latency might be closer but you mentioned bandwidth.

rbanffy 3y ago
IIRC, with 32 MB per core (8 per die - a Telum package has two of them), one socket has 256MB of cache for 16 cores. L1 and L2 are in-core, but L3 is on-die unused L2 from other core and L4 is unused L3 on the other die. I am not sure if that continues off-package, but if it does, a drawer with 4 chips of 16 cores each will have 2 GB of off-chip cache and a full-sized Z with 5 drawers would have 10 GB of off-drawer cache (at this level it's probably not that much faster than same-drawer memory). As for the RAIM, I think it's safe to assume it has a very wide path (n-1 modules) to the sockets and that aggregate bandwidth will not leave the 4 sockets starved (even if latencies suffer because of the redundancy, but you can replace a defective memory module with a running computer without it having to pause.
- zwaps 3y ago
  That's not a lot compared to 8xA100
  - cbsmith 3y ago
    A100's are definitely still the best, but IBM's offering isn't a joke. Depending on what you're trying to do, it might be a good way to go.
    - justinclift 3y ago
      We'd probably need to see pricing for IBMs offering, because it's possible it'll be eye watering-ly high compared to buying even A100's.
    - jabl 3y ago
      TBH, I don't really think they compete in anything like similar markets. You buy a DGX A100, or a cluster of them, for training and running large deep learning models (or for doing "traditional" HPC). IBM's solution is more a small inference engine that is part of the CPU, so you don't need to move you data off-chip when doing a little bit of inferencing as part of some other workflow. I don't work with mainframes so I could be talking out of my behind, but maybe something like DL-assisted fraud detection as part of processing bank transactions?
      - cbsmith 3y ago
        > TBH, I don't really think they compete in anything like similar markets. Yes.
- sillywalk 3y ago
  Some more details for people who know about these things, emphasis mine: 2 chips make a dual-chip module with 512MB cache, 4 dual-chip modules make a drawer with 2GB cache, and a 4 drawer system with 32 chips makes 8GB cache. "The L3 and L4 virtual design across the fabric provides 50% more cache per core, with improved latencies. The idea is that software and microcode still see two separate caches. These caches are shared and distributed across all eight cores via a 320 GB/s ring and across the integrated fabric. Horizontal cache persistence should further reduce cache misses as well. Specifically, when a cache line is ejected, the system looks for available cache capacity on other caches, first on the chip and then even across the 32-chip fabric."[0] "The accelerator itself delivers an aggregate of over 6 TFLOPS of 16-bit floating-point throughput per chip to scale up to roughly 200 TFLOPS per system. 1024 processor tiles in a systolic array make up the matrix array, and 256 fp16/32 tiles make up the accelerator for computing activations and include built-in functions for RELU, tanH, and log. The platform also provides enterprise-class availability and security, as one should expect in a Z, with virtualization, error checking/recovery, and memory protection mechanisms. While 6 TFLOPS does not sound impressive, keep in mind that this accelerator is optimized for transaction processing. Most data are in floating-point and are highly structured, unlike in voice or image processing. Consequently, we believe this accelerator will deliver adequate performance and is undoubtedly much faster than offloading to another GPU-equipped server or running the inference on a Z core. The latency of off-platform inference can cause transactions to time out, and inference does not complete"[0] "Intelligent Prefetcher and Write-Back – 120+ GB/s read bandwidth to internal scratchpad – 80+ GB/s store bandwidth – Multi-zone scratchpad for concurrent data load, execution and write-back Intelligent Data Mover and Formatter – 600+ GB/s bandwidth – Format and prepare data on the fly for compute and write-back"[1] [0] https://cambrian-ai.com/wp-content/uploads/edd/2021/08/IBM-Telum-Research-Paper-Cambrian-AI.pdf https://cambrian-ai.com/wp-content/uploads/edd/2021/08/IBM-T... [1] https://hc33.hotchips.org/assets/program/conference/day1/HC2021.C1.3%20IBM%20Cristian%20Jacobi%20Final.pdf https://hc33.hotchips.org/assets/program/conference/day1/HC2...

rbanffy 3y ago

IIRC, with 32 MB per core (8 per die - a Telum package has two of them), one socket has 256MB of cache for 16 cores. L1 and L2 are in-core, but L3 is on-die unused L2 from other core and L4 is unused L3 on the other die. I am not sure if that continues off-package, but if it does, a drawer with 4 chips of 16 cores each will have 2 GB of off-chip cache and a full-sized Z with 5 drawers would have 10 GB of off-drawer cache (at this level it's probably not that much faster than same-drawer memory). As for the RAIM, I think it's safe to assume it has a very wide path (n-1 modules) to the sockets and that aggregate bandwidth will not leave the 4 sockets starved (even if latencies suffer because of the redundancy, but you can replace a defective memory module with a running computer without it having to pause.

zwaps 3y ago
That's not a lot compared to 8xA100
- cbsmith 3y ago
  A100's are definitely still the best, but IBM's offering isn't a joke. Depending on what you're trying to do, it might be a good way to go.
  - justinclift 3y ago
    We'd probably need to see pricing for IBMs offering, because it's possible it'll be eye watering-ly high compared to buying even A100's.
  - jabl 3y ago
    TBH, I don't really think they compete in anything like similar markets. You buy a DGX A100, or a cluster of them, for training and running large deep learning models (or for doing "traditional" HPC). IBM's solution is more a small inference engine that is part of the CPU, so you don't need to move you data off-chip when doing a little bit of inferencing as part of some other workflow. I don't work with mainframes so I could be talking out of my behind, but maybe something like DL-assisted fraud detection as part of processing bank transactions?
    - cbsmith 3y ago
      > TBH, I don't really think they compete in anything like similar markets. Yes.
sillywalk 3y ago
Some more details for people who know about these things, emphasis mine: 2 chips make a dual-chip module with 512MB cache, 4 dual-chip modules make a drawer with 2GB cache, and a 4 drawer system with 32 chips makes 8GB cache. "The L3 and L4 virtual design across the fabric provides 50% more cache per core, with improved latencies. The idea is that software and microcode still see two separate caches. These caches are shared and distributed across all eight cores via a 320 GB/s ring and across the integrated fabric. Horizontal cache persistence should further reduce cache misses as well. Specifically, when a cache line is ejected, the system looks for available cache capacity on other caches, first on the chip and then even across the 32-chip fabric."[0] "The accelerator itself delivers an aggregate of over 6 TFLOPS of 16-bit floating-point throughput per chip to scale up to roughly 200 TFLOPS per system. 1024 processor tiles in a systolic array make up the matrix array, and 256 fp16/32 tiles make up the accelerator for computing activations and include built-in functions for RELU, tanH, and log. The platform also provides enterprise-class availability and security, as one should expect in a Z, with virtualization, error checking/recovery, and memory protection mechanisms. While 6 TFLOPS does not sound impressive, keep in mind that this accelerator is optimized for transaction processing. Most data are in floating-point and are highly structured, unlike in voice or image processing. Consequently, we believe this accelerator will deliver adequate performance and is undoubtedly much faster than offloading to another GPU-equipped server or running the inference on a Z core. The latency of off-platform inference can cause transactions to time out, and inference does not complete"[0] "Intelligent Prefetcher and Write-Back – 120+ GB/s read bandwidth to internal scratchpad – 80+ GB/s store bandwidth – Multi-zone scratchpad for concurrent data load, execution and write-back Intelligent Data Mover and Formatter – 600+ GB/s bandwidth – Format and prepare data on the fly for compute and write-back"[1] [0] https://cambrian-ai.com/wp-content/uploads/edd/2021/08/IBM-Telum-Research-Paper-Cambrian-AI.pdf https://cambrian-ai.com/wp-content/uploads/edd/2021/08/IBM-T... [1] https://hc33.hotchips.org/assets/program/conference/day1/HC2021.C1.3%20IBM%20Cristian%20Jacobi%20Final.pdf https://hc33.hotchips.org/assets/program/conference/day1/HC2...