13 ms·
This is very cool! I really hope the ONNX project gets much more adoption in the next months and years and help reduce the fragmentation in the ML ecosystem.
by hashtag-til 3y ago
This is very cool! I really hope the ONNX project gets much more adoption in the next months and years and help reduce the fragmentation in the ML ecosystem.
- brucethemoose2 3y agoEh... I have seen ONNX demos for years, and they tend to stay barebones and slow, kinda like this. NCNN, MLIR and TVM based ports have been far more impressive.
- lostmsu 3y agoHave you used TVM much? Is it only good for inference?
- brucethemoose2 3y agoInference only, as far as I know. TBH the mlc projects were my first exposure, but they are all impressive, as are the bits of source I poked through. And the autotuning makes a huge difference in my quick testing.
- mathisfun123 3y agoLololol show me an "MLIR" port. Do you mean tensorflow port or jax port or torch port (that uses torch-mlir)? Or do you really mean llama implemented in linalg/tosa/tendor?
- brucethemoose2 3y agoI wasn't talking about Llama specifically. I was thinking of the SHARK Stable Diffusion port (which uses MLIR/IREE), as it considerably outpaced the ONNX runtime. But apparently the performance of llama on torch-mlir is progressing: https://github.com/nod-ai/SHARK/issues/1707 https://github.com/nod-ai/SHARK/issues/1707
- claytonjy 3y agoI'm not sure there's much chance of that happening. ONNX seems to be the broadest in coverage, but for basically any model ONNX supports, there's a faster alternative. For the latest generative/transformer stuff (whisper, llama, etc) it's often specialized C(++) stuff, but torch 2.0 compilation keeps geting better, BetterTransformers, TensorRT, etc.