4 ms·
To give another take, Python's dynamic nature allows you to easily change class properties, methods, and so on. However, what is difficult with Python is to act
by chillee 5y ago
To give another take, Python's dynamic nature allows you to easily change class properties, methods, and so on. However, what is difficult with Python is to actually modify the executed code.
For example, say you had
def f(x):
return F.relu(x)
Now, you want to change all the activations in your network from a relu to gelu. This... is not so easy generically. You might be able to do it for your personal code, but what if you're importing a model from torchvision?
With FX though, you can simply trace out the graph, substitute the F.relu with a F.gelu, and be done in <10 lines of code!
Essentially, it gives you the freedom to perform transformations on your code (although it places limitations on what your code can contain, like no control flow).
- Kalanos 5y agoIsn't that why function arguments exist? `if act=='relu'`
- chillee 5y agohaha, yes, but that requires you to modify existing code to do so (which isn't always possible!). There might also be other things you want to do (like add profiling after each op) that would be tedious to do manually, but can easily automated with FX (https://pytorch.org/tutorials/intermediate/fx_profiling_tutorial.html https://pytorch.org/tutorials/intermediate/fx_profiling_tuto...). Another example is the recent support from torchvision for extracting intermediate feature activations (https://github.com/pytorch/vision/releases/tag/v0.11.0 https://github.com/pytorch/vision/releases/tag/v0.11.0). Like, sure, it was probably possible to refactor all of their code to enable users to specify extracting an intermediate feature, but it's much cleaner to do with FX.
- davidatbu 5y agoThanks @chillee! Your comments are helpful.