4 ms·
A less hyped inference engine with INT8/FP16 inference supports on both CPU / GPU (cuda). Model supports list: GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, LLAMA, T5, W
by wsxiaoys 3y ago
A less hyped inference engine with INT8/FP16 inference supports on both CPU / GPU (cuda).
Model supports list:
GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, LLAMA, T5, WHISPER
( Found this library during my research on alternatives to triton/FasterTransformer in Tabby https://github.com/TabbyML/tabby https://github.com/TabbyML/tabby)