zhye

4 ms·

1. 1.▲
   by zhye 2y ago

   Glad to see MLC is becoming more mature :) I can imagine the unified engine could help build agents on multiple devices. Any ideas on how those edge and cloud models collaborate on compound tasks (e.g. the compound ai systems: https:/
2. 2.▲
   Cascade Inference: Memory Bandwidth Efficient Shared Prefix Batch Decoding(flashinfer.ai)

   2 points by zhye 3y ago|0 comments
3. 3.▲
   by zhye 3y ago

   It will take some effort to implement operators but not too much (cutlass's group gemm already support different mnk's), however the performance benefit is marginal compared to padding all LoRA ranks to the same rank because all
4. 4.▲
   by zhye 3y ago

   Good question, in general implementing kernels on page tables is tricky in Tensor Compilers because integer set analysis might fail sometimes (but can be fixed with some tweaks). I think using compilers like TVM can help deploy serving syst
5. 5.▲
   by zhye 3y ago

   Serving LLM with AMD GPUs to serve LLM looks impressive, MLC is evolving fast! Any results on NVLink/xGMI instead of PCIe?