3 ms·
CTranslate2: An efficient inference engine for Transformer models
- wsxiaoys 3y agoA less hyped inference engine with INT8/FP16 inference supports on both CPU / GPU (cuda). Model supports list: GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, LLAMA, T5, WHISPER ( Found this library during my research on alternatives to triton/FasterTransformer in Tabby https://github.com/TabbyML/tabby https://github.com/TabbyML/tabby)