If you want to host for the AI Horde, Aphrodite has the most GPU throughput, and Koboldcpp is the easiest and quite flexible: https://github.com/PygmalionAI/ap

If you want to host for the AI Horde, Aphrodite has the most GPU throughput, and Koboldcpp is the easiest and quite flexible: https://github.com/PygmalionAI/aphrodite-engine https://github.com/PygmalionAI/aphrodite-engine https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp Picking a model to host is quite a rabbit hole, but I am very interested in SciPhi and Zephyr at the moment. I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.