4 ms·

Just want to flag a potential unit issue: 900gbps vs 300GB/s? Also worth noting - TPUv4 uses a 6-way 3D torus interconnect vs the 3-way "multi ToR" NVLINK topo

Just want to flag a potential unit issue: 900gbps vs 300GB/s? Also worth noting - TPUv4 uses a 6-way 3D torus interconnect vs the 3-way "multi ToR" NVLINK topology; the total bisection bandwidth of the TPUv4 pod is over 1PB/s! Can't wait to see what TPUv5 looks like. As you say, it's probably already chugging away with v6 on track to tape out in a year. That said, I think NVidia has nailed bringing the ecosystem along, and I think making the whole setup look more like "one huge GPU" could simplify a lot of ML programming. I am actually disappointed I haven't seen more of that style in CPU programming. Where's my 20,000 core 100TB RAM VM instance?

llm_nerd 3y ago
>900gbps vs 300GB/s? The nvidia device uses a fabric with 900 GBps switched fabric between any of the 256 nodes in the system. The TPUv4 3d torus network is basically a ring network of 56 GBps connections creating separate rings. From a raw perspective, the nvidia solution is the overwhelming winner. There is absolutely no contest.
- deleted 3y ago
  [deleted]
mike_hearn 3y ago
> Where's my 20,000 core 100TB RAM VM instance? Machines with terabytes of RAM do exist and get used - working well on such setups is a goal of modern JVM GCs for instance - but making a single machine that large which acts like a single machine isn't easy, nor especially desirable. One machine is a unified failure domains outside of mainframe-land, so if you had a 20k core machine with 100TB of RAM you could never reboot it to apply OS updates and it'd die all the time from failed parts. Even once you get beyond that most software stacks use locking and stop scaling beyond a few hundred cores at best and that's assuming very heavily optimized stacks. AI workloads are easier because they're designed from scratch to be inherently parallel without lots of little locks and custom data structures all over the place like a regular computer has. Disk storage is one of the places where you can parallelize and scale out relatively easily and you do see datacenter sized disks there.
londons_explore 3y ago
> Where's my 20,000 core 100TB RAM VM instance? You could simulate this with a bunch of regular machines and a networked hypervisor. You could do some kind of smart caching so that processes rarely need to wait to access RAM stored on a remote machine. Combined that with a big lock eliding/speculation scheme (ie. When a process reads memory that might have been written by a remote CPU, you continue as if it hadn't, and if you later find out that data was written then you rollback). These rollbacks 'undo' all work done in however many microseconds it takes for data to travel from one side of the machine cluster to another. Reads of RAM that aren't cached yet on the local node can also be speculated - you just assume that RAM contained null bytes and continue execution, rolling back and replaying when the actual data arrives. So if you can make sure that processes are contending for locks and writing conflicting data less often than once per system-roundtrip-latency, then you should get a high performance system.
- mochomocha 3y ago
  This is certainly a very interesting thought to entertain and your ideas make sense. One thing that makes things harder on the CPU side in this hypothetical scenario is that CPUs tend to execute much more diverse instructions/computations than GPUs. So all the caching & speculation you mention is probably all the more important.
  - londons_explore 3y ago
    After writing the comment, I considered writing a little toy example just to try out the idea... It would be neat to see Linux boot with 1000 CPU's... But upon further thought, a lot of things such a system would need are actually rather inefficient to implement in software (ie. rollbackable RAM), yet quite cheap in hardware (for example rollbackable RAM can be implemented with regular RAM plus either a buffer of 'overwritten data' or a write queue)
    - samstave 3y ago
      A write queue with a dupe-back-end to say a blob on S3 or whatever would be interesting for mirrors of outcomes could be stored. The biggest issue it seems is bandwidth and humans' patience for a response...
inhumantsar 3y ago
Nvidia's is 900GB/s, not gbps
- dweekly 3y ago
  You're totally right. My bad. I went back to double check the docs and indeed the chips are 900GB/s total over 18 NVLink connections. https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/
whatusername 3y ago
If I'm reading the docs right (TBH - I'm probably not) it looks like on a z16 you can get 200 cores and 40TB of Memory on a single "VM" (LPAR). So 1/100th of the CPU and 40% of the RAM. (I suspect the RAM comparison is reasonable - I'm not sure about how to compare the CPU's).

llm_nerd 3y ago

>900gbps vs 300GB/s? The nvidia device uses a fabric with 900 GBps switched fabric between any of the 256 nodes in the system. The TPUv4 3d torus network is basically a ring network of 56 GBps connections creating separate rings. From a raw perspective, the nvidia solution is the overwhelming winner. There is absolutely no contest.

deleted 3y ago
[deleted]

mike_hearn 3y ago

> Where's my 20,000 core 100TB RAM VM instance? Machines with terabytes of RAM do exist and get used - working well on such setups is a goal of modern JVM GCs for instance - but making a single machine that large which acts like a single machine isn't easy, nor especially desirable. One machine is a unified failure domains outside of mainframe-land, so if you had a 20k core machine with 100TB of RAM you could never reboot it to apply OS updates and it'd die all the time from failed parts. Even once you get beyond that most software stacks use locking and stop scaling beyond a few hundred cores at best and that's assuming very heavily optimized stacks. AI workloads are easier because they're designed from scratch to be inherently parallel without lots of little locks and custom data structures all over the place like a regular computer has. Disk storage is one of the places where you can parallelize and scale out relatively easily and you do see datacenter sized disks there.

londons_explore 3y ago

> Where's my 20,000 core 100TB RAM VM instance? You could simulate this with a bunch of regular machines and a networked hypervisor. You could do some kind of smart caching so that processes rarely need to wait to access RAM stored on a remote machine. Combined that with a big lock eliding/speculation scheme (ie. When a process reads memory that might have been written by a remote CPU, you continue as if it hadn't, and if you later find out that data was written then you rollback). These rollbacks 'undo' all work done in however many microseconds it takes for data to travel from one side of the machine cluster to another. Reads of RAM that aren't cached yet on the local node can also be speculated - you just assume that RAM contained null bytes and continue execution, rolling back and replaying when the actual data arrives. So if you can make sure that processes are contending for locks and writing conflicting data less often than once per system-roundtrip-latency, then you should get a high performance system.

mochomocha 3y ago
This is certainly a very interesting thought to entertain and your ideas make sense. One thing that makes things harder on the CPU side in this hypothetical scenario is that CPUs tend to execute much more diverse instructions/computations than GPUs. So all the caching & speculation you mention is probably all the more important.
- londons_explore 3y ago
  After writing the comment, I considered writing a little toy example just to try out the idea... It would be neat to see Linux boot with 1000 CPU's... But upon further thought, a lot of things such a system would need are actually rather inefficient to implement in software (ie. rollbackable RAM), yet quite cheap in hardware (for example rollbackable RAM can be implemented with regular RAM plus either a buffer of 'overwritten data' or a write queue)
  - samstave 3y ago
    A write queue with a dupe-back-end to say a blob on S3 or whatever would be interesting for mirrors of outcomes could be stored. The biggest issue it seems is bandwidth and humans' patience for a response...

inhumantsar 3y ago

Nvidia's is 900GB/s, not gbps

dweekly 3y ago
You're totally right. My bad. I went back to double check the docs and indeed the chips are 900GB/s total over 18 NVLink connections. https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/

whatusername 3y ago

If I'm reading the docs right (TBH - I'm probably not) it looks like on a z16 you can get 200 cores and 40TB of Memory on a single "VM" (LPAR). So 1/100th of the CPU and 40% of the RAM. (I suspect the RAM comparison is reasonable - I'm not sure about how to compare the CPU's).