4 ms·

Thanks, good to know. Perhaps it is different for diffusion; with llms, layers are generally split across gpus, meaning inference has to happen on one gpu befor

by fzzzy 2y ago

Thanks, good to know. Perhaps it is different for diffusion; with llms, layers are generally split across gpus, meaning inference has to happen on one gpu before the values can be passed between the layer split.

Y_Y 2y ago
That's only if your model is too big for a single GPU and you're not batching.
- fzzzy 2y ago
  Yes, that's what I was doing. Thanks for the info.