3 ms·

The issues in this blog post seem like non-issues. You probably shouldn't be sending back >8kb error messages to the client. The server should log error detail

The issues in this blog post seem like non-issues. You probably shouldn't be sending back >8kb error messages to the client. The server should log error details locally and the logs should be inspected or scraped separately. The client doesn't need to know the full details like the contents of a variable length DS I develop gRPC clients and servers at work that handle about 40k QPS (typically <50ms response times) whilst using default gRPC configuration. The only changes I had to make was to the max message size.

cryptonym 6y ago
Can be quite useful to enable verbose errors during dev and initial tests.
lmm 6y ago
> You probably shouldn't be sending back >8kb error messages to the client. The server should log error details locally and the logs should be inspected or scraped separately. The client doesn't need to know the full details like the contents of a variable length DS You shouldn't, but it's easy to accidentally do. Having a limit on the server and a limit on the client that's (by default) half the limit on the server seems like an unforced error - given that the server does truncate at a certain limit, surely it would be better for that limit to be smaller than the default client limit. Postel's law and all that.
Aeolun 6y ago
> You probably shouldn't be sending back >8kb error messages to the client While I agree, it seems like a strange limit to have as part of your spec.
- JamesNK 6y ago
  The error message is sent as a HTTP trailing header. Their server or client has a header size limit of 8kb.
- thu2111 6y ago
  There is no spec, as far as I can tell. That's why the article says different implementations disagree on the max size of the error message. As for large errors, there are plenty of cases where they can be useful. Not every RPC is crossing a privacy boundary: if an administration CLI performs an RPC that fails, it's more convenient to have lots of debugging information printed to the admin than to force them to go and dig through the server logs. Honestly, gRPC makes me a bit sad. It's clearly a duplicate of Stubby. I don't know what Google use these days, but Stubby was a truly great RPC system. For inexplicable reasons Google constantly choose to badly reimplement their internal tech as part of open sourcing it, and gRPC is like some cheap knockoff in comparison.
  - jsmith45 6y ago
    Part of the problem with open sourcing Google's software is that the software is often integrated with a bunch of the rest of the google stack. This could just be dependencies on little libraries that may or may not already be open source, all the way up to depending on various large components to do some of the heavy lifting around security/scheduling/storage etc. If some library (that the thing being opened up uses) has an open open source version, it is necessary to worry about versioning concerns, because unlike inside google3, the open source versions can no longer make breaking changes merely by updating every single consumer. So the internal version in some cases will have drifted from the open source version. Even when a library or tool needed is already opened sourced, it is not rare for a newly open source program to use a stub version instead of the full tool/library to avoid such headaches. (e.g. I've seen super stripped down alternative to gtest shipped as part of some opened libraries, because trying to utilize the open source version was more hassle than it was worth.) For libraries or tools not open sourced, some form of stub or adapter to some similar publicly available software is needed. In some cases that does not exist, making directly open sourcing basically impossible (but redeveloping the tool might be possible). In other cases there are alternatives available, but they might be lacking features that made the integration nicer inside Google. Overall I can totally understand why Googlers sometimes want to basically redesign an internal tool as the approach to open sourcing the underlying concept, rather than try to publish the equivalent-ish internal tool.
throwaway4good 6y ago
Maybe. But the protocol should not set a size limit on the error response.
ThePhysicist 6y ago
Yep, it's also quite easy to leak sensitive data in errors so you shouldn't just send every error your code produces to the client. Maybe less a concern for gRPC as it's often used internally, but some systems still might propagate the error logs to untrusted clients. In the worst case your traceback will contain environment variables, which in turn might contain database secrets or API tokens (of course it's also bad practice storing secrets in environment variables but people still do it a lot). In our projects we solve this by implementing internal and external error types. External errors (e.g. form validation errors) will be sent to the client, internal errors will just send a generic error message (e.g. "please contact tech support. ID: #fa3ca....") and will be logged internally so we can trace them back to their source and ideally fix them.
- nitrogen 6y ago
  storing secrets in environment variables This is exactly how a lot of frameworks tell you to provide secrets, and it's also how at least some cluster management tools pass secrets into apps as well. It always seemed strange to me that the environment was considered "secure".
est 6y ago
> should log error details locally and the logs should be inspected or scraped separately Adding extra parts to a already complex system does not solving problems. It only transfers the problem to other components. e.g. what happens when the "scraper" also use gRPC and want to yield a >8KB error?

cryptonym 6y ago

Can be quite useful to enable verbose errors during dev and initial tests.

lmm 6y ago

> You probably shouldn't be sending back >8kb error messages to the client. The server should log error details locally and the logs should be inspected or scraped separately. The client doesn't need to know the full details like the contents of a variable length DS You shouldn't, but it's easy to accidentally do. Having a limit on the server and a limit on the client that's (by default) half the limit on the server seems like an unforced error - given that the server does truncate at a certain limit, surely it would be better for that limit to be smaller than the default client limit. Postel's law and all that.

Aeolun 6y ago

> You probably shouldn't be sending back >8kb error messages to the client While I agree, it seems like a strange limit to have as part of your spec.

JamesNK 6y ago
The error message is sent as a HTTP trailing header. Their server or client has a header size limit of 8kb.
thu2111 6y ago
There is no spec, as far as I can tell. That's why the article says different implementations disagree on the max size of the error message. As for large errors, there are plenty of cases where they can be useful. Not every RPC is crossing a privacy boundary: if an administration CLI performs an RPC that fails, it's more convenient to have lots of debugging information printed to the admin than to force them to go and dig through the server logs. Honestly, gRPC makes me a bit sad. It's clearly a duplicate of Stubby. I don't know what Google use these days, but Stubby was a truly great RPC system. For inexplicable reasons Google constantly choose to badly reimplement their internal tech as part of open sourcing it, and gRPC is like some cheap knockoff in comparison.
- jsmith45 6y ago
  Part of the problem with open sourcing Google's software is that the software is often integrated with a bunch of the rest of the google stack. This could just be dependencies on little libraries that may or may not already be open source, all the way up to depending on various large components to do some of the heavy lifting around security/scheduling/storage etc. If some library (that the thing being opened up uses) has an open open source version, it is necessary to worry about versioning concerns, because unlike inside google3, the open source versions can no longer make breaking changes merely by updating every single consumer. So the internal version in some cases will have drifted from the open source version. Even when a library or tool needed is already opened sourced, it is not rare for a newly open source program to use a stub version instead of the full tool/library to avoid such headaches. (e.g. I've seen super stripped down alternative to gtest shipped as part of some opened libraries, because trying to utilize the open source version was more hassle than it was worth.) For libraries or tools not open sourced, some form of stub or adapter to some similar publicly available software is needed. In some cases that does not exist, making directly open sourcing basically impossible (but redeveloping the tool might be possible). In other cases there are alternatives available, but they might be lacking features that made the integration nicer inside Google. Overall I can totally understand why Googlers sometimes want to basically redesign an internal tool as the approach to open sourcing the underlying concept, rather than try to publish the equivalent-ish internal tool.

throwaway4good 6y ago

Maybe. But the protocol should not set a size limit on the error response.

ThePhysicist 6y ago

Yep, it's also quite easy to leak sensitive data in errors so you shouldn't just send every error your code produces to the client. Maybe less a concern for gRPC as it's often used internally, but some systems still might propagate the error logs to untrusted clients. In the worst case your traceback will contain environment variables, which in turn might contain database secrets or API tokens (of course it's also bad practice storing secrets in environment variables but people still do it a lot). In our projects we solve this by implementing internal and external error types. External errors (e.g. form validation errors) will be sent to the client, internal errors will just send a generic error message (e.g. "please contact tech support. ID: #fa3ca....") and will be logged internally so we can trace them back to their source and ideally fix them.

nitrogen 6y ago
storing secrets in environment variables This is exactly how a lot of frameworks tell you to provide secrets, and it's also how at least some cluster management tools pass secrets into apps as well. It always seemed strange to me that the environment was considered "secure".

est 6y ago

> should log error details locally and the logs should be inspected or scraped separately Adding extra parts to a already complex system does not solving problems. It only transfers the problem to other components. e.g. what happens when the "scraper" also use gRPC and want to yield a >8KB error?