5 ms·
Universal LLM Deployment Engine with ML Compilation
- ruihangl 2y agoA unified efficient open-source LLM deployment engine for both cloud server and local use cases. It comes with full OpenAI-compatible API that runs directly with Python, iOS, Android, browsers. Supporting deploying latest large language models such as Qwen2, Phi3, and more.
- crowwork 2y agoruns on qwen2 on iphone with 26 tok/sec and a OpenAI style swift API
- CharlieRuan 2y agoFrom first-hand experience, the all-in-one framework really helps reduce engineering effort!
- cyx6 2y agoAI ALL IN ONE! Super universal and performant!
- yongwww 2y agoThe MLCEngine presents an approach to universal LLM deployment, glad to know it works for both local servers and cloud devices with competitive performance. Looking forward to exploring it further!
- neetnestor 2y agoLooks cool. I'm looking forward to trying building some interesting apps using the SDKs.
- zhye 2y agoGlad to see MLC is becoming more mature :) I can imagine the unified engine could help build agents on multiple devices. Any ideas on how those edge and cloud models collaborate on compound tasks (e.g. the compound ai systems: https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/ https://bair.berkeley.edu/blog/2024/02/18/compound-ai-system...)