5 ms·

By SIMD style do you mean really SIMD (hardware threads) or some kind of programming constructs? I don't think you need a software scheduler for the first case,

By SIMD style do you mean really SIMD (hardware threads) or some kind of programming constructs? I don't think you need a software scheduler for the first case, and there shouldn't be thousands of threads on a single machine for the second case. (you can probably use a cluster, but the scheduler probably won't be scheduling those thousands of threads directly)

synergy20 4y ago
I mean SIMT(CUDA style), or SPMD, and also Tensorflow host|device runtime scheduling, it's a mystery to me how Nvidia etc schedule the AI workload(along with intrinsic) to achieve huge volume parallelism.
- pca006132 4y ago
  For CUDA, iirc these tasks (kernel launches) are put into streams (similar to threads) and scheduled for execution by the driver. Within each kernel launch, each 32 threads, called a warp, are executed together in a single unit, skipping some instructions when they have different control flow. I think the driver perhaps schedule in a warp level, and these warps are executing similar things so can be scheduled together. I am not an expert in this so I am not sure if this is how they do it.
  - synergy20 4y ago
    That's pretty much how that works, and they sync threads inside warp, I have been groping in the dark for a while, and always hoped someone can write up some details to help me to get the idea straight. Intel, AMD, Google(TPU) all have their own way to schedule 'kernel's which are very different from CUDA, there are no details about them, I was just curious like 'how do they work across CPU|GPU'? Thanks for the reply.
    - maxgio92 4y ago
      Thank you all for this. Scheduling on GPUs is a topic in the dark for me, to be discovered
      - astrange 4y ago
        GPU "threads" aren't exactly CPU threads, and GPU "cores" aren't really CPU cores. It's better to think of threads as SIMD instructions and cores as ALUs. GPU execution order is typically either "immediate mode" or "tile mode", where tiles are more common on mobile GPUs, but Nvidia has also used them. https://www.realworldtech.com/tile-based-rasterization-nvidia-gpus/ https://www.realworldtech.com/tile-based-rasterization-nvidi...
        maxgio92 4y ago
        thank you for the details!

synergy20 4y ago

I mean SIMT(CUDA style), or SPMD, and also Tensorflow host|device runtime scheduling, it's a mystery to me how Nvidia etc schedule the AI workload(along with intrinsic) to achieve huge volume parallelism.

pca006132 4y ago
For CUDA, iirc these tasks (kernel launches) are put into streams (similar to threads) and scheduled for execution by the driver. Within each kernel launch, each 32 threads, called a warp, are executed together in a single unit, skipping some instructions when they have different control flow. I think the driver perhaps schedule in a warp level, and these warps are executing similar things so can be scheduled together. I am not an expert in this so I am not sure if this is how they do it.
- synergy20 4y ago
  That's pretty much how that works, and they sync threads inside warp, I have been groping in the dark for a while, and always hoped someone can write up some details to help me to get the idea straight. Intel, AMD, Google(TPU) all have their own way to schedule 'kernel's which are very different from CUDA, there are no details about them, I was just curious like 'how do they work across CPU|GPU'? Thanks for the reply.
  - maxgio92 4y ago
    Thank you all for this. Scheduling on GPUs is a topic in the dark for me, to be discovered
    - astrange 4y ago
      GPU "threads" aren't exactly CPU threads, and GPU "cores" aren't really CPU cores. It's better to think of threads as SIMD instructions and cores as ALUs. GPU execution order is typically either "immediate mode" or "tile mode", where tiles are more common on mobile GPUs, but Nvidia has also used them. https://www.realworldtech.com/tile-based-rasterization-nvidia-gpus/ https://www.realworldtech.com/tile-based-rasterization-nvidi...
      - maxgio92 4y ago
        thank you for the details!