4 ms·
This problem sounds like an excellent opportunity. We need a race to the bottom for hosting LLMs to democratize the tech and lower costs. I cheer on anyone who
by pokstad 6mo ago
This problem sounds like an excellent opportunity. We need a race to the bottom for hosting LLMs to democratize the tech and lower costs. I cheer on anyone who figures this out.
- mememememememo 6mo agoThis is classic queuing theory, rate limits etc. I don't have an answer but I would look there.
- taraindara 6mo agoWhat if you could group multiple of them. Long queries run on the group that’s commonly doing those. Shorter queries que faster because they’ll execute faster.