5 ms·

Google does not use gRPC internally it, like k8s, it is an open re-implementation of an internal google technology. Also, like k8s & Bazel, it is less advanced

Google does not use gRPC internally it, like k8s, it is an open re-implementation of an internal google technology. Also, like k8s & Bazel, it is less advanced and lower performance than their internal technology. At a high level Google is open sourcing things that have genuine value, but also make integration with their money making services easier.

lhecker 8y ago
> Also, like k8s & Bazel, [gRPC] is less advanced and lower performance than their internal technology. Are you sure about that? As far as I know gRPC is literally just as good as Bazel, which is why Google is even migrating to it internally. For instance this comment agrees with me: https://news.ycombinator.com/item?id=12348286 https://news.ycombinator.com/item?id=12348286
- clhodapp 8y ago
  Note that Bazel is a build system (https://bazel.build https://bazel.build). I believe you are confusing it with Stubby. That's not to say that the general thrust of what you're saying is necessarily wrong though.
- deleted 8y ago
  [deleted]
- ebikelaw 8y ago
  I think you are misreading that comment. He’s saying that stubby is fast, not grpc. In fact performance is the big unknown with grpc adoption within google. It definitely isn’t on par with stubby today and it has to get there before anyone significant will switch to it.
  - Xorlev 8y ago
    I can't comment on raw numbers (because I simply don't have them) but at least for the service I work on, replacing Stubby with gRPC wouldn't really move the needle even if it was 2-3x slower (it might be faster, this is just for illustration) -- we spend our time waiting on IO from other services or crunching numbers in the CPU. Being a Java service, gRPC/Java might well be just as fast or faster than Stubby Java, but I could understand that Stubby C++ has been hyperoptimized over the years vs. gRPC C core which might have a ways to go. By the latest performance dashboard [1, 2], gRPC/Java is leading the pack but gRPC C++ doesn't seem like it's slouching too much either. I seem to remember the C++ impl crushing Java at performance a while back, so I'm sure that'll change in the future. Honestly though? It'd take a _very_ demanding workload such that your RPC system was the bottleneck (so long as they're within constant factors of each other). There are services like that, but they're the exception and not the norm. Most services don't need to do 100kQPS/task. Even then, at that point you're spending a lot of time on serialization/deserialization, auth, logging, etc.. Your service is more than its communication layer, even if that's important to optimize it's still just a minor constant factor. The real problem is inertia. There's a lot of code/tools/patterns built up around Stubby and the semantics of Stubby (including all its features which likely haven't been ported to gRPC yet) and that's difficult to overcome. Our #1 use of gRPC so far I would imagine is at the edge. gRPC is making its way into Android apps since it's pretty trivial for translating proxies to more or less 1:1 convert gRPC to Stubby calls. [1] https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5636470266134528 https://performance-dot-grpc-testing.appspot.com/explore?das... [2] https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5652536396611584&widget=1735297033&container=1012810333 https://performance-dot-grpc-testing.appspot.com/explore?das...
    - ebikelaw 8y ago
      You and I seem to be using a different denominator to quantify "most" services. I'm thinking of it as "most" in terms of who has all the resources / budget. You seem to be thinking of it in terms of sheer number of services or engineers working on them. The fact is that the highly demanding services have the huge majority of the resources, and are the most sensitive to performance issues. If your service uses 10% of Google's datacenter space, you won't accept a 5% or even 1% regression just so you can port to gRPC, because at that scale your team can just staff someone or even several people to maintain the pre-gRPC system forever and still come out ahead on the budget. Totally agree that world-facing APIs will all be gRPC and that makes perfect sense to me.
      - Xorlev 8y ago
        > You seem to be thinking of it in terms of sheer number of services or engineers working on them. I'm not sure where I said that, but yes, that's part of the switching cost. > The fact is that the highly demanding services have the huge majority of the resources, and are the most sensitive to performance issues. If your service uses 10% of Google's datacenter space, you won't accept a 5% or even 1% regression just so you can port to gRPC, The thrust of my statement was that for many services, RPC overhead is minimal. So even a 2x or 3x increase in RPC overhead is still minimal. I agree, a 5% increase in resource utilization for a large service is something that would be weighed. But lets explore that idea for a moment: > because at that scale your team can just staff someone or even several people to maintain the pre-gRPC system forever and still come out ahead on the budget. Not necessarily. Engineers are expensive and becoming ever more expensive while computing resources are becoming increasingly cheaper. Not only that, but engineers tend to be more specialized and so you can't just task anyone to maintain the previous system, it tends to be people with deep expertise already. And those people also have career aims to do more than long-term support of a deprecated system, so there's retention to be considered. Pretending for a moment that all your services except a small handful moved on to somme system B from some system A, if the maintenance burden of maintaining system B starts to eclipse the resource cost of moving to system A (which decreases all the time due to improvements in system B and the increasing cost of maintaining system A, and the monotonic reduction in computing resource cost), then you might well just swallow the 5%-10% increase in resources either permanently or temporarily and come out ahead in the end. Additionally, as system B moves on, staying on system A becomes increasingly risky: security improvements, features, layers which don't know about system A anymore all threaten the stability of your service. If you've checked out the SRE book, you'll know that our SLOs are more important than any one resource. If nobody trusts your service to operate, then they won't use it and then you won't have to worry about resources anymore since the users will have moved on. > because at that scale your team can just staff someone or even several people to maintain the pre-gRPC system forever and still come out ahead on the budget. To reiterate the point above, these roles tend to be fairly specialized and hard to staff. Arguably these same engineers are better tasked making system B good enough to switch to so you can thank system A for its service and show it the door. Bringing this back to Stubby vs. gRPC, it's a pretty academic argument so far. They're both here to stay. And honestly, when we say "Stubby" there's already different versions of Stubby which interoperate with each other and gRPC will not be any different. Likewise, we still use proto1 in addition to proto2 and proto3 (the public versions) since that just takes time and energy to fix. We do make these kinds of decisions every day, and it's not always in favor of reduced resources. If we cared for nothing other than resource utilization, we'd be completely C++, no Java, no Python. Realistically, the cost of maintaining systems with equivalent roles can often lead to one or the other winning out, usually in favor of maintainability so long as their feature sets are roughly equivalent. We're fortunate to be in a position that we can choose code health and uniformity of vision over absolute minimum resource utilization. And again, even if we choose system B (higher resources) over system A, perhaps due to the differences in architecture or design choices the absolute bar for performance of that system will be greater than system A, despite starting lower. Sometimes it takes a critical mass of adopters to really shake out all those issues. I know that quotes from Knuth are often trotted out during these kinds of discussions, but it's true: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." That 3% is where we choose to spend our effort, and that critical 3% includes the ability of our engineering force to make forward progress and not be hindered by too much debt. It also includes real data, check our Google Wide Profiling [1]. > Totally agree that world-facing APIs will all be gRPC and that makes perfect sense to me. Probably not all. We still fully support HTTP/JSON APIs, but at least in our little corner of the world we've chosen to take full advantage of gRPC. Anyways, thanks for letting me stand on my soapbox for a bit. [1] https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36575.pdf https://storage.googleapis.com/pub-tools-public-publication-...
shaklee3 8y ago
You implied this, but Google does use k8s internally. I saw a recent tweet where two employees (both kubernetes maintainers) said it's used more and more.
saalweachter 8y ago
Eh, you shouldn’t use whether an open sourced (Google-driven) equivalent to an internal Google project is widely used internally as any sort of guide to which is better or worse. Google SWEs, like most software engineers, have a lot of work to do, and “learn a new framework that is sort of the same as the framework you’re using now and convert the entire code base to it” is pretty low on anyone’s list. We all have to do it from time to time, but you usually wait until you’ve had a few new employees learn the new framework and use it on a couple of smaller new projects, and then wait until you’re doing a big enough rewrite that the framework change is noise. My guess is that the the major differences between an open source Google project and an equivalent internal one are (1) mistakes that are too deep in the original design are corrected, (2) some edge-case features are not immediately available, (3) the software is written to be run without a team of 10 SREs managing it, and (4) some novel bugs are created that will take a few iterations to correct. I’d be surprised if there was a significant “performance” difference; the people writing it will probably be able to benefit from the experience on the first implementation to not make too many mistakes (and they have solid performance targets to shoot for, as opposed to not knowing what an efficient implementation should look like). Everyone complains about (2) and (4), which are the inevitable result of developing software in the open, but you cannot imagine how wonderful (1) is. I use k8s for my projects at home and borg for my projects at work, and dear god, the configuration language differences alone. (Don’t spit into the wind; don’t tug on Superman’s cape; don’t become the borg configuration expert for your team.)
jsolson 8y ago
Note: mix of facts and opinions ahead; the facts mostly boil down to "all of these technologies _are_ used within Google"; the opinions are my own thoughts on what our internal future looks like. Other Googlers may see a different future or different reasons for the status quo :) We do use gRPC internally, but the bulk of services were built prior to its existence/maturity and use its predecessors. My expectation is that services will eventually migrate, but that's a long way out. I'd say it's less to do with performance than a clear reason for teams to migrate, and several compelling reasons not to (yet). Bazel is mostly a subset of Blaze, and many packages would "just work" if pulled into a Bazel workspace. Those that wouldn't are mostly broken because they rely on proprietary (but in many cases deprecated) Blaze features. I see a handful of changes go by every week migrating packages to newer Bazel-friendly definitions (a big one here is the historical handling of proto_library rules). Kubernetes is used for some things built on GCP, but that's mostly suitable for green-field work (it's easier to integrate with existing things on Borg if you just run on Borg).

lhecker 8y ago

> Also, like k8s & Bazel, [gRPC] is less advanced and lower performance than their internal technology. Are you sure about that? As far as I know gRPC is literally just as good as Bazel, which is why Google is even migrating to it internally. For instance this comment agrees with me: https://news.ycombinator.com/item?id=12348286 https://news.ycombinator.com/item?id=12348286

clhodapp 8y ago
Note that Bazel is a build system (https://bazel.build https://bazel.build). I believe you are confusing it with Stubby. That's not to say that the general thrust of what you're saying is necessarily wrong though.
deleted 8y ago
[deleted]
ebikelaw 8y ago
I think you are misreading that comment. He’s saying that stubby is fast, not grpc. In fact performance is the big unknown with grpc adoption within google. It definitely isn’t on par with stubby today and it has to get there before anyone significant will switch to it.
- Xorlev 8y ago
  I can't comment on raw numbers (because I simply don't have them) but at least for the service I work on, replacing Stubby with gRPC wouldn't really move the needle even if it was 2-3x slower (it might be faster, this is just for illustration) -- we spend our time waiting on IO from other services or crunching numbers in the CPU. Being a Java service, gRPC/Java might well be just as fast or faster than Stubby Java, but I could understand that Stubby C++ has been hyperoptimized over the years vs. gRPC C core which might have a ways to go. By the latest performance dashboard [1, 2], gRPC/Java is leading the pack but gRPC C++ doesn't seem like it's slouching too much either. I seem to remember the C++ impl crushing Java at performance a while back, so I'm sure that'll change in the future. Honestly though? It'd take a _very_ demanding workload such that your RPC system was the bottleneck (so long as they're within constant factors of each other). There are services like that, but they're the exception and not the norm. Most services don't need to do 100kQPS/task. Even then, at that point you're spending a lot of time on serialization/deserialization, auth, logging, etc.. Your service is more than its communication layer, even if that's important to optimize it's still just a minor constant factor. The real problem is inertia. There's a lot of code/tools/patterns built up around Stubby and the semantics of Stubby (including all its features which likely haven't been ported to gRPC yet) and that's difficult to overcome. Our #1 use of gRPC so far I would imagine is at the edge. gRPC is making its way into Android apps since it's pretty trivial for translating proxies to more or less 1:1 convert gRPC to Stubby calls. [1] https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5636470266134528 https://performance-dot-grpc-testing.appspot.com/explore?das... [2] https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5652536396611584&widget=1735297033&container=1012810333 https://performance-dot-grpc-testing.appspot.com/explore?das...
  - ebikelaw 8y ago
    You and I seem to be using a different denominator to quantify "most" services. I'm thinking of it as "most" in terms of who has all the resources / budget. You seem to be thinking of it in terms of sheer number of services or engineers working on them. The fact is that the highly demanding services have the huge majority of the resources, and are the most sensitive to performance issues. If your service uses 10% of Google's datacenter space, you won't accept a 5% or even 1% regression just so you can port to gRPC, because at that scale your team can just staff someone or even several people to maintain the pre-gRPC system forever and still come out ahead on the budget. Totally agree that world-facing APIs will all be gRPC and that makes perfect sense to me.
    - Xorlev 8y ago
      > You seem to be thinking of it in terms of sheer number of services or engineers working on them. I'm not sure where I said that, but yes, that's part of the switching cost. > The fact is that the highly demanding services have the huge majority of the resources, and are the most sensitive to performance issues. If your service uses 10% of Google's datacenter space, you won't accept a 5% or even 1% regression just so you can port to gRPC, The thrust of my statement was that for many services, RPC overhead is minimal. So even a 2x or 3x increase in RPC overhead is still minimal. I agree, a 5% increase in resource utilization for a large service is something that would be weighed. But lets explore that idea for a moment: > because at that scale your team can just staff someone or even several people to maintain the pre-gRPC system forever and still come out ahead on the budget. Not necessarily. Engineers are expensive and becoming ever more expensive while computing resources are becoming increasingly cheaper. Not only that, but engineers tend to be more specialized and so you can't just task anyone to maintain the previous system, it tends to be people with deep expertise already. And those people also have career aims to do more than long-term support of a deprecated system, so there's retention to be considered. Pretending for a moment that all your services except a small handful moved on to somme system B from some system A, if the maintenance burden of maintaining system B starts to eclipse the resource cost of moving to system A (which decreases all the time due to improvements in system B and the increasing cost of maintaining system A, and the monotonic reduction in computing resource cost), then you might well just swallow the 5%-10% increase in resources either permanently or temporarily and come out ahead in the end. Additionally, as system B moves on, staying on system A becomes increasingly risky: security improvements, features, layers which don't know about system A anymore all threaten the stability of your service. If you've checked out the SRE book, you'll know that our SLOs are more important than any one resource. If nobody trusts your service to operate, then they won't use it and then you won't have to worry about resources anymore since the users will have moved on. > because at that scale your team can just staff someone or even several people to maintain the pre-gRPC system forever and still come out ahead on the budget. To reiterate the point above, these roles tend to be fairly specialized and hard to staff. Arguably these same engineers are better tasked making system B good enough to switch to so you can thank system A for its service and show it the door. Bringing this back to Stubby vs. gRPC, it's a pretty academic argument so far. They're both here to stay. And honestly, when we say "Stubby" there's already different versions of Stubby which interoperate with each other and gRPC will not be any different. Likewise, we still use proto1 in addition to proto2 and proto3 (the public versions) since that just takes time and energy to fix. We do make these kinds of decisions every day, and it's not always in favor of reduced resources. If we cared for nothing other than resource utilization, we'd be completely C++, no Java, no Python. Realistically, the cost of maintaining systems with equivalent roles can often lead to one or the other winning out, usually in favor of maintainability so long as their feature sets are roughly equivalent. We're fortunate to be in a position that we can choose code health and uniformity of vision over absolute minimum resource utilization. And again, even if we choose system B (higher resources) over system A, perhaps due to the differences in architecture or design choices the absolute bar for performance of that system will be greater than system A, despite starting lower. Sometimes it takes a critical mass of adopters to really shake out all those issues. I know that quotes from Knuth are often trotted out during these kinds of discussions, but it's true: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." That 3% is where we choose to spend our effort, and that critical 3% includes the ability of our engineering force to make forward progress and not be hindered by too much debt. It also includes real data, check our Google Wide Profiling [1]. > Totally agree that world-facing APIs will all be gRPC and that makes perfect sense to me. Probably not all. We still fully support HTTP/JSON APIs, but at least in our little corner of the world we've chosen to take full advantage of gRPC. Anyways, thanks for letting me stand on my soapbox for a bit. [1] https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36575.pdf https://storage.googleapis.com/pub-tools-public-publication-...

shaklee3 8y ago

You implied this, but Google does use k8s internally. I saw a recent tweet where two employees (both kubernetes maintainers) said it's used more and more.

saalweachter 8y ago

Eh, you shouldn’t use whether an open sourced (Google-driven) equivalent to an internal Google project is widely used internally as any sort of guide to which is better or worse. Google SWEs, like most software engineers, have a lot of work to do, and “learn a new framework that is sort of the same as the framework you’re using now and convert the entire code base to it” is pretty low on anyone’s list. We all have to do it from time to time, but you usually wait until you’ve had a few new employees learn the new framework and use it on a couple of smaller new projects, and then wait until you’re doing a big enough rewrite that the framework change is noise. My guess is that the the major differences between an open source Google project and an equivalent internal one are (1) mistakes that are too deep in the original design are corrected, (2) some edge-case features are not immediately available, (3) the software is written to be run without a team of 10 SREs managing it, and (4) some novel bugs are created that will take a few iterations to correct. I’d be surprised if there was a significant “performance” difference; the people writing it will probably be able to benefit from the experience on the first implementation to not make too many mistakes (and they have solid performance targets to shoot for, as opposed to not knowing what an efficient implementation should look like). Everyone complains about (2) and (4), which are the inevitable result of developing software in the open, but you cannot imagine how wonderful (1) is. I use k8s for my projects at home and borg for my projects at work, and dear god, the configuration language differences alone. (Don’t spit into the wind; don’t tug on Superman’s cape; don’t become the borg configuration expert for your team.)

jsolson 8y ago

Note: mix of facts and opinions ahead; the facts mostly boil down to "all of these technologies _are_ used within Google"; the opinions are my own thoughts on what our internal future looks like. Other Googlers may see a different future or different reasons for the status quo :) We do use gRPC internally, but the bulk of services were built prior to its existence/maturity and use its predecessors. My expectation is that services will eventually migrate, but that's a long way out. I'd say it's less to do with performance than a clear reason for teams to migrate, and several compelling reasons not to (yet). Bazel is mostly a subset of Blaze, and many packages would "just work" if pulled into a Bazel workspace. Those that wouldn't are mostly broken because they rely on proprietary (but in many cases deprecated) Blaze features. I see a handful of changes go by every week migrating packages to newer Bazel-friendly definitions (a big one here is the historical handling of proto_library rules). Kubernetes is used for some things built on GCP, but that's mostly suitable for green-field work (it's easier to integrate with existing things on Borg if you just run on Borg).