3 ms·
If this is the way forward, I'd love for a better developer experience. I'm currently wrangling with pytorch.compile (flex attention doesn't like bias terms -
by Scene_Cast2 2y ago
If this is the way forward, I'd love for a better developer experience.
I'm currently wrangling with pytorch.compile (flex attention doesn't like bias terms - see issues 145869 and 144511 if curious). As much as I love speed and optimization, JIT (at least the pytorch flavor) currently has weird return types that break VS Code's intellisense, weird stack traces, limitations around printing, and random issues like the bias term, and limitations such as not supporting sparsity.
Speaking of pytorch JIT workflows - what's a nice way of having a flag to turn off compilation?
- dleeftink 2y ago> random issues like the bias term I'd like to know more about this!
- Scene_Cast2 2y agoBasically this style of code - https://github.com/pytorch-labs/attention-gym/pull/84/files https://github.com/pytorch-labs/attention-gym/pull/84/files - has issues like this - https://github.com/pytorch/pytorch/pull/137452 https://github.com/pytorch/pytorch/pull/137452 https://github.com/pytorch/pytorch/issues/144511 https://github.com/pytorch/pytorch/issues/144511 https://github.com/pytorch/pytorch/issues/145869 https://github.com/pytorch/pytorch/issues/145869 For some higher level context, see https://pytorch.org/blog/flexattention/ https://pytorch.org/blog/flexattention/
- singhrac 2y agoYou can decorate a function with @torch.compiler.disable() to disable a specific function (and anything further down the call stack).