4 ms·

This is a great article. Regarding > While these projects are performant, they often come with tradeoffs in ease of use, such as requiring model conversion to

by andy99 3y ago

This is a great article. Regarding > While these projects are performant, they often come with tradeoffs in ease of use, such as requiring model conversion to specific formats or building and shipping new dependencies. I think it should be acknowledged that (at least IMO) pytorch model formats are not very portable and this is a big part of the problem. It would be nice to see industry move towards a better format (gguf?) that can easily be ported between frameworks and not leave you stuck using torch to load it. Likewise, pytorch is a massive dependency to include with a project, especially for simple inference, so while other projects have new dependencies, they can often be a lot lighter than for a pytorch model, again particularly for inference code.

chillee 3y ago
Yeah, for sure. I think for deployment purposes, many times these model conversions are necessary (such as if you don't want to use Python). However, I do think these model conversions are often a significant pain for users. So, in some sense, the goal here is to show that the performance component and the "convert your model for deployment" component can be disentangled. We also have work on allowing you to "export" an AOT-compiled version of your model with torch.compile, and that should allow you to deploy your models to run in other settings.
- andy99 3y ago
  Thanks for the reply. "show that the performance component and the "convert your model for deployment" component can be disentangled" makes sense. Also, I liked the part of the article about torch.compile producing faster matrix-vector multiplication than cublas. I've seen the same thing on CPU, that it's way faster to just write and manually optimize a loop over a bunch of dot products than it is to use BLAS routines because of how simple the "matmul" actually is. I don't know how widely known that is.

by andy99 3y ago

chillee 3y ago

Yeah, for sure. I think for deployment purposes, many times these model conversions are necessary (such as if you don't want to use Python). However, I do think these model conversions are often a significant pain for users. So, in some sense, the goal here is to show that the performance component and the "convert your model for deployment" component can be disentangled. We also have work on allowing you to "export" an AOT-compiled version of your model with torch.compile, and that should allow you to deploy your models to run in other settings.

andy99 3y ago
Thanks for the reply. "show that the performance component and the "convert your model for deployment" component can be disentangled" makes sense. Also, I liked the part of the article about torch.compile producing faster matrix-vector multiplication than cublas. I've seen the same thing on CPU, that it's way faster to just write and manually optimize a loop over a bunch of dot products than it is to use BLAS routines because of how simple the "matmul" actually is. I don't know how widely known that is.