3 ms·
I more relevant benchmark would be against TensorRT, the current kind of model serving performance on NVIDIA GPUs.
by kookamamie 3y ago
I more relevant benchmark would be against TensorRT, the current kind of model serving performance on NVIDIA GPUs.
- lamchob 3y agoIn the paper (https://dl.acm.org/doi/10.1145/3575693.3575702 https://dl.acm.org/doi/10.1145/3575693.3575702) there is also a TensorRT comparison. There, they are pretty much neck and neck. Sometimes faster, sometimes slower.
- gpekhimenko 3y ago+1. There are multiple comparisons in the paper, and we plan to provide a few more soon. The typical story that if the model is at least a bit unusual, not overly optimized manually, Hidet can provide some speedup even over TensorRT.