4 ms·

Thanks for the input. Before I start throwing += into my PyTorch code can you explain what you mean here: > Generally, it's best to have this optimization be d

Thanks for the input. Before I start throwing += into my PyTorch code can you explain what you mean here: > Generally, it's best to have this optimization be done automatically by an optimizing compiler. What compiler should be optimizing this operation? There are comments on the commit reporting errors under certain conditions.

chillee 4y ago
To clarify, by "compilers" I mean "deep learning compilers". There's many different paths to optimizing compilers folks use with PyTorch. One with close integration is NVFuser (see https://www.reddit.com/r/MachineLearning/comments/xa75km/p_pytorchs_newest_nvfuser_on_stable_diffusion_to/ https://www.reddit.com/r/MachineLearning/comments/xa75km/p_p...), although there are other compilers like ONNXRuntime. Yes, handling autograd (during training) is a whole different thing, and not all compilers support that.