3 ms·
Service to auto route LLM/Model traffic
I suggest creating a service that monitors all the LLMs/AI model deployment services on Azure, AWS, Google Cloud, Groq, Krutrim, and other cloud deployments and routes your traffic based on availability, latency, rate-limiting, and other parameters. Manage everything at the backend, just pay a single unified bill, with no deployments or accounts on any other service. Sounds cool?
- yorwba 2y agoSounds similar to SkyPilot: https://docs.skypilot.co https://docs.skypilot.co But you still need to create your own accounts for each service.