4 ms·
Have you guys run into the issue of gRPC being used for ML inference engines/platforms, and having the message size be an issue? Recently, I was working on bui
by sjnair96 5y ago
Have you guys run into the issue of gRPC being used for ML inference engines/platforms, and having the message size be an issue?
Recently, I was working on building the backend for our Speech Recognition engine and it seems like the message size limit came in the way of being able to perform batch/offline inference.
I might be lacking some of the intuition behind designing and deploying such services, and am curious if you guys have run into this issue. The platform doesn't seem to accept the channel options from the client.
That and the issue of no easy way to do gRPC calls from a browser yet, and the resulting architecture then needing to have been built the ground up with considerations for an extra piece - a gateway, or something like Envoy to make it more accessible over the web. I feel like that would be okay if I was working on a project with k8 ingress set up already and already utilizing microservices. But for getting running it seems like added complexity. Not hating on it or anything, just sharing my experiences and thoughts, hoping that some of you might've had experience reasoning and dealing with such designs and can offer some insight.
- brinox 5y agoYou can configure the message size limit on your gRPC server, doesn't that fix your issue?
- sjnair96 5y agoAhh, we don't have access to the server. It's closed - an NVIDIA inference engine. Which under the hood talks to their Triton engine. Unfortunately, while Triton allows configuring the limit, the layer in front of it eats our channel options which have the message size configurations.
- CommonGuy 5y agoAnother option could be to "chunk" the messages with client-/server-streaming or bi-directional calls. But if you call your API from a browser, that may not be possible yet
- sjnair96 5y agoYes, but that would involve us digging into the code of the gRPC server, which is unfortunately closed to us :(