5 ms·

Hidet: A Deep Learning Compiler for Efficient Model Serving

123 points by ashvardanian 4y ago

lamchob 4y ago
If you read into the paper (https://dl.acm.org/doi/10.1145/3575693.3575702 https://dl.acm.org/doi/10.1145/3575693.3575702), one can find more performance comparisons. There, from a latency/throughput PoV they are en par with existing tools like TVM/Ansor. Sometimes faster, sometimes slower. What is more interesting is this: They have very GPU-specific auto-tuning routine that drastically reduces the optimzation space, compared to TVM/Ansor. They go from ~10^6 possible implementations for an operator to a "few hundred", which enabled much faster time-to-solution. This is achieved with a GPU-centric problem formulation and search space. In essence, they trade how widely applicable their approach is (from "any" kind of hardware to only GPU-style architectures) for retrieval speed.
LeanderK 4y ago
> USING HIDET SCRIPT TO WRITE CUSTOM OPERATORS this is really cool. I sometimes would like to have custom operators that are more or less trivial but the amount of work to create the operators by hand is just not worth it.
pavelstoev 4y ago
Generally, Hidet outperforms other inference compilers - PyTorch Eager, ORT, TRT, TVM. For example, PyTorch Eager - too much framework overhead. ORT -doesn't do operator fusion. TRT - close-sourced and hard to fix if a model can not run. TVM - tuning time is too long, also limited expressiveness in optimization. Additionally Hidet comes with Hidet Script, a brand new domain-specific language to write tensor programs in Python with high flexibility to express optimizations that can only be done in C++ CUDA C code. Hidet Script also supports operator tuning and automatic fusion.
brucethemoose2 4y ago
Hmm, does not work for Stable Diffusion yet :( But inductor/triton didn't work in the 2.0 nightlies either, and now it works fine for SD.
- akbarnur 4y ago
  Hey @bructhemoose2 can you file an issue, we will try to fix it ASAP: https://github.com/hidet-org/hidet/issues https://github.com/hidet-org/hidet/issues
- mlazos 4y ago
  This bug was in the front end (dynamo) so I’m not surprised it happens with hidet too. I fixed this for the 2.0.1 release.
  - brucethemoose2 4y ago
    On 2.1, I am getting backend errors atm. For instance, with the unet: > torch._dynamo.exc.BackendCompilerFailed: backend='hidet_backend' raised: NotImplementedError: hidet: Tensor.to(..., device=...) is not supported for symbolic tensors... Early torch 2.0 was full of backend unimplemented/unsupported errors too, but I can't even remember when that was all implemented and the dynamo issues popped up (or even when everything started working) since I stayed on torch nightly.
    - mlazos 4y ago
      Ah ok this wasn’t what I thought then. I haven’t taken a look at Hidet yet so I won’t be much help here :( I thought you were talking about 2.0.0
    - yyding 4y ago
      Hi, do you mind submitting an issue to the hidet github repo so we can have a look? Thank you!
vrglvrglvrgl 4y ago
[dead]
kookamamie 4y ago
I more relevant benchmark would be against TensorRT, the current kind of model serving performance on NVIDIA GPUs.
- lamchob 4y ago
  In the paper (https://dl.acm.org/doi/10.1145/3575693.3575702 https://dl.acm.org/doi/10.1145/3575693.3575702) there is also a TensorRT comparison. There, they are pretty much neck and neck. Sometimes faster, sometimes slower.
  - gpekhimenko 4y ago
    +1. There are multiple comparisons in the paper, and we plan to provide a few more soon. The typical story that if the model is at least a bit unusual, not overly optimized manually, Hidet can provide some speedup even over TensorRT.
Maclennan 4y ago
[dead]
psuedo_uuh 4y ago
Hidet: a deep learning model to detect when you’re ready for your bidet to activate
junrushao1994 4y ago
Nice work! This is interesting to read the comparison between Hidet and Triton in this blog: > Hidet Script vs. Triton: Triton greatly simplifies the CUDA programming by introducing the tile-based programming model where the parallel execution unit is thread blocks instead of threads. However, this simplification also prevents the tensor program developers from manipulating the fine-grained computation and memory resources (e.g., warps, shared memory) in their preferred ways. It would be challenging to implement an optimization that requires fine-grained control of these resources using Triton if it has not been implemented by the Triton compiler itself. Hidet Script, on the other hand, simplifies tensor programming while still enabling users to implement their own optimizations with extensive flexibility. It’s worth noting that the more granular control of Hidet Script also brings added complexity compared to Triton.

123 points by ashvardanian 4y ago

If you read into the paper (https://dl.acm.org/doi/10.1145/3575693.3575702 https://dl.acm.org/doi/10.1145/3575693.3575702), one can find more performance comparisons. There, from a latency/throughput PoV they are en par with existing tools like TVM/Ansor. Sometimes faster, sometimes slower. What is more interesting is this: They have very GPU-specific auto-tuning routine that drastically reduces the optimzation space, compared to TVM/Ansor. They go from ~10^6 possible implementations for an operator to a "few hundred", which enabled much faster time-to-solution. This is achieved with a GPU-centric problem formulation and search space. In essence, they trade how widely applicable their approach is (from "any" kind of hardware to only GPU-style architectures) for retrieval speed.

LeanderK 4y ago

> USING HIDET SCRIPT TO WRITE CUSTOM OPERATORS this is really cool. I sometimes would like to have custom operators that are more or less trivial but the amount of work to create the operators by hand is just not worth it.

pavelstoev 4y ago

Generally, Hidet outperforms other inference compilers - PyTorch Eager, ORT, TRT, TVM. For example, PyTorch Eager - too much framework overhead. ORT -doesn't do operator fusion. TRT - close-sourced and hard to fix if a model can not run. TVM - tuning time is too long, also limited expressiveness in optimization. Additionally Hidet comes with Hidet Script, a brand new domain-specific language to write tensor programs in Python with high flexibility to express optimizations that can only be done in C++ CUDA C code. Hidet Script also supports operator tuning and automatic fusion.

brucethemoose2 4y ago

Hmm, does not work for Stable Diffusion yet :( But inductor/triton didn't work in the 2.0 nightlies either, and now it works fine for SD.

akbarnur 4y ago
Hey @bructhemoose2 can you file an issue, we will try to fix it ASAP: https://github.com/hidet-org/hidet/issues https://github.com/hidet-org/hidet/issues
mlazos 4y ago
This bug was in the front end (dynamo) so I’m not surprised it happens with hidet too. I fixed this for the 2.0.1 release.
- brucethemoose2 4y ago
  On 2.1, I am getting backend errors atm. For instance, with the unet: > torch._dynamo.exc.BackendCompilerFailed: backend='hidet_backend' raised: NotImplementedError: hidet: Tensor.to(..., device=...) is not supported for symbolic tensors... Early torch 2.0 was full of backend unimplemented/unsupported errors too, but I can't even remember when that was all implemented and the dynamo issues popped up (or even when everything started working) since I stayed on torch nightly.
  - mlazos 4y ago
    Ah ok this wasn’t what I thought then. I haven’t taken a look at Hidet yet so I won’t be much help here :( I thought you were talking about 2.0.0
  - yyding 4y ago
    Hi, do you mind submitting an issue to the hidet github repo so we can have a look? Thank you!

vrglvrglvrgl 4y ago

[dead]

kookamamie 4y ago

I more relevant benchmark would be against TensorRT, the current kind of model serving performance on NVIDIA GPUs.

lamchob 4y ago
In the paper (https://dl.acm.org/doi/10.1145/3575693.3575702 https://dl.acm.org/doi/10.1145/3575693.3575702) there is also a TensorRT comparison. There, they are pretty much neck and neck. Sometimes faster, sometimes slower.
- gpekhimenko 4y ago
  +1. There are multiple comparisons in the paper, and we plan to provide a few more soon. The typical story that if the model is at least a bit unusual, not overly optimized manually, Hidet can provide some speedup even over TensorRT.

Maclennan 4y ago

[dead]

psuedo_uuh 4y ago

Hidet: a deep learning model to detect when you’re ready for your bidet to activate

junrushao1994 4y ago

Nice work! This is interesting to read the comparison between Hidet and Triton in this blog: > Hidet Script vs. Triton: Triton greatly simplifies the CUDA programming by introducing the tile-based programming model where the parallel execution unit is thread blocks instead of threads. However, this simplification also prevents the tensor program developers from manipulating the fine-grained computation and memory resources (e.g., warps, shared memory) in their preferred ways. It would be challenging to implement an optimization that requires fine-grained control of these resources using Triton if it has not been implemented by the Triton compiler itself. Hidet Script, on the other hand, simplifies tensor programming while still enabling users to implement their own optimizations with extensive flexibility. It’s worth noting that the more granular control of Hidet Script also brings added complexity compared to Triton.