MDK8888

4 ms·

1. 1.▲
   A Minimal Implementation of Vllm(github.com)

   3 points by MDK8888 2y ago|1 comments
2. 2.▲
   by MDK8888 2y ago

   I built vllm from the ground up. Starting from the same kernels, I integrate them into GPT2, build the KV cache manager, the scheduler, and finally the FastAPI server on top. This is purely educational for now, but in the future it will bec
3. 3.▲
   by MDK8888 3y ago

   This has been added to the Readme under the 'Background' section.
4. 4.▲
   by MDK8888 3y ago

   Hey, hope you're doing well! I've updated the repo with instructions on how to get started and documentation!
5. 5.▲
   A Python Library to 6-7x the inference speed of your HF models(github.com)

   75 points by MDK8888 3y ago|15 comments
6. 6.▲
   by MDK8888 3y ago

   A few months back, the PyTorch team released GPT, Fast ( https://news.ycombinator.com/item?id=38477197 ), a collection of techniques to 10x the inference speed of Llama-2-7b. This library generalizes those techniques (quantiz
7. 7.▲
   by MDK8888 3y ago

   SageMode is an open source python library for deploying, scaling, and monitoring your LLMs at scale. It's designed to be highly flexible but also standardized. You can deploy any Huggingface Model that exists in version 4.26 or earlier
8. 8.▲
   Show HN: SageMode - A Python library for deploying, scaling, and monitoring LLMs(github.com)

   3 points by MDK8888 3y ago|1 comments