3 ms·
That sounds really difficult to use to me. I'd need to figure out how to connect to a custom messaging protocol, and how to keep that connection online (maybe r
by simonw 3y ago
That sounds really difficult to use to me. I'd need to figure out how to connect to a custom messaging protocol, and how to keep that connection online (maybe routing through firewalls etc), and handle retries and connection failures and suchlike.
An HTTP polling API may be less efficient but I know how to use that from all of the programming languages I work with already.
- alexchamberlain 3y agoFWIW, AMQP is the same protocol as RabbitMQ- there's a library for pretty much every language/ecosystem.
- simonw 3y agoYeah, I've used that in the past and found it significantly harder than HTTP.
- mike_hearn 3y agoAMQP isn't a custom messaging protocol and MQ libraries do all of that for you already (handling reconnects and retries etc). Connection interruptions aren't an issue. You're actually worrying about low level details of the sort MQ systems are specifically designed to solve for you, but HTTP stacks do not. For example, someone below asks "how do I see my submitted jobs". MQ protocols solve that for you out of the box. You can browse the queues that you own. It's all already standardized and implemented. Because they rolled their own protocol, OpenAI lacks this function and will have to develop it themselves. Then because it's all oriented around batches (when the underlying problem does not actually need batching), the users will have to run a local DB to keep track of what prompts map to what batches or they'll find themselves redownloading the whole batch to find out what's in it. As for firewalls, you open the port. The only issue there is if you're behind a firewall you don't have any control over and where the people who do don't want you using AI, but that's not an issue in the datacenter deployments that are presumably generating inferencing batch workloads, and such firewalls or proxies can break HTTP APIs too anyway.