Have you tried llama.cpp? It doesn't need a GPU so it's generally cheaper to run, and inference speed is decent (1-10 tokens per second depending on model size

Have you tried llama.cpp? It doesn't need a GPU so it's generally cheaper to run, and inference speed is decent (1-10 tokens per second depending on model size and hardware specs). Not sure if it's been set up to work with the open assistant stuff yet, but should be soon given how fast things are moving.