2 ms·
TinyChat: Large Language Model on the Edge
- enduku 3y agoTinyChat is an efficient, lightweight, Python-native serving framework for 4-bit LLMs by AWQ. It delivers 2.3x generation speed up on RTX4090. Code: https://github.com/mit-han-lab/llm-awq/tree/main/tinychat https://github.com/mit-han-lab/llm-awq/tree/main/tinychat