28 ms·
Eh... I have seen ONNX demos for years, and they tend to stay barebones and slow, kinda like this. NCNN, MLIR and TVM based ports have been far more impressive
by brucethemoose2 3y ago
Eh... I have seen ONNX demos for years, and they tend to stay barebones and slow, kinda like this.
NCNN, MLIR and TVM based ports have been far more impressive.
- lostmsu 3y agoHave you used TVM much? Is it only good for inference?
- brucethemoose2 3y agoInference only, as far as I know. TBH the mlc projects were my first exposure, but they are all impressive, as are the bits of source I poked through. And the autotuning makes a huge difference in my quick testing.
- mathisfun123 3y agoLololol show me an "MLIR" port. Do you mean tensorflow port or jax port or torch port (that uses torch-mlir)? Or do you really mean llama implemented in linalg/tosa/tendor?
- brucethemoose2 3y agoI wasn't talking about Llama specifically. I was thinking of the SHARK Stable Diffusion port (which uses MLIR/IREE), as it considerably outpaced the ONNX runtime. But apparently the performance of llama on torch-mlir is progressing: https://github.com/nod-ai/SHARK/issues/1707 https://github.com/nod-ai/SHARK/issues/1707