3 ms·
https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals Inference only: https://lite.koboldai.net/ https://lite.koboldai.ne
by brucethemoose2 3y ago
https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals
Inference only: https://lite.koboldai.net/ https://lite.koboldai.net/
- tayo42 3y agowow didnt know that, thats really cool, ill have to play with it
- brucethemoose2 3y agoIf you want to host for the AI Horde, Aphrodite has the most GPU throughput, and Koboldcpp is the easiest and quite flexible: https://github.com/PygmalionAI/aphrodite-engine https://github.com/PygmalionAI/aphrodite-engine https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp Picking a model to host is quite a rabbit hole, but I am very interested in SciPhi and Zephyr at the moment. I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.