5 ms·
>why not open source Spanner [or] BigTable? Because it's genuinely difficult to open source things like that. They're way too tied to Google and have many clos
by jchw 8y ago
>why not open source Spanner [or] BigTable?
Because it's genuinely difficult to open source things like that. They're way too tied to Google and have many closed-source dependencies on internal libraries and services. I think they regret missing the opportunity on some of these and that may reflect why Google has started open sourcing more 'core' things like gRPC and Bazel - it might make it easier.
- dagenix 8y agoGood points. But, they also faced the same issue with Borg - presumably it was hard to open source, so, they created Kubernetes instead. There is nothing stopping Google from doing the same thing with Spanner or BigTable. Such a move would probably be well received by the community. It seems likely that people at Google would want to work on such projects - AIUI, CockrochDB was created by ex-Googlers. (And I didn't mean to focus on those technologies too specifically - with HBase and CockroachDB already existing, maybe Google wouldn't want to compete, but, I suspect they have other technologies for which creating open source versions would be well received, but they choose not to). My point isn't that Google could directly open source those technologies. Nor is it that they should create open source versions of them. I'm more just musing on how when the selfish reasons to do something line up with the good reasons to do something, you tend to get cool results. At a previous job, we had this process that everyone was supposed to do via a script. Doing the process via the script meant we got the logging that we wanted and such. Everyone was supposed to do it this way. However, there were no technical controls preventing anyone from doing that process by hand. We just told people not to. So, one day, a colleague asks me how to make sure that everyone does the process the right way. And my answer is that everyone already does - the script is way easier to run that to do it by hand. The good thing to do (run it the way you were told to), lined up with the selfish reason (it was way easier to do that way). Making sure those two things were in alignment was effectively as powerful as making it impossible to do the wrong way. (since then, the process has been significantly changed). I feel like its a similar case here. The good thing to do was to create a cool technology and make it free for everyone. The selfish thing to do, was to create a technology that made business sense. The result, is Kubernetes was created as open source (at least, thats my theory). If that wasn't the good thing to do, maybe the person writing the article doesn't talk to the VP on the shuttle about it and it never happens. If it wasn't the selfish thing to do (business case), maybe the VP wouldn't approve it. But, getting those two things aligned, results in positive reinforcement and results.
- jchw 8y agoThey mention some of the benefits of open sourcing in this article: >We always believed that open-sourcing Kubernetes was the right way to go, bringing many benefits to the project. For one, feedback loops were essentially instantaneous — if there was a problem or something didn’t work quite right, we knew about it immediately. But most importantly, we were able to work with lots of great engineers, many of whom really understood the needs of businesses who would benefit from deploying containers (have a look at the Kubernetes blog for perspectives from some of the early contributors). It was a virtuous cycle: the work of talented engineers led to more interest in the project, which further increased the rate of improvement and usage. Of course, open sourcing software was never about charity - but I do think that it was always a community thing, and big companies choosing to be a part of that community are neither selfish or selfless for simply participating.
- dagenix 8y ago> Of course, open sourcing software was never about charity - but I do think that it was always a community thing, and big companies choosing to be a part of that community are neither selfish or selfless for simply participating. Strongly agree. And to be clear, I didn't use the term "selfish" as a pejorative - I think its pretty reasonable to expect any entity to analyze how a decision will directly benefit them and to act accordingly.
- mrep 8y agoTotally! You could steal any big tech companies entire source code and it wouldn't do you much good in replicating it as you would realize most of it is entirely dependent on thousands of internal services maintained from that source code that depend on each other and if each one is not running on the companies specific infrastructure it is designed to run on, it won't work. Also, all those services are dependent on the companies internal company specific server hardware and network designs that take years to plan, buy, and build which by the time you did, would be out of date.
- akqu 8y agoI think that is an underappreciated point. What makes a lot of open source software productive only exists in proprietary code, infrastructure and knowledge guarded as a competitive advantage at large companies. I have never really understood why companies like that gets so much open source credit while someone independently trying to make a living shipping a proprietary product for an open source system, making that system more attractive, gets a lot of hate.
- Game_Ender 8y agoGoogle does not use gRPC internally it, like k8s, it is an open re-implementation of an internal google technology. Also, like k8s & Bazel, it is less advanced and lower performance than their internal technology. At a high level Google is open sourcing things that have genuine value, but also make integration with their money making services easier.
- lhecker 8y ago> Also, like k8s & Bazel, [gRPC] is less advanced and lower performance than their internal technology. Are you sure about that? As far as I know gRPC is literally just as good as Bazel, which is why Google is even migrating to it internally. For instance this comment agrees with me: https://news.ycombinator.com/item?id=12348286 https://news.ycombinator.com/item?id=12348286
- clhodapp 8y agoNote that Bazel is a build system (https://bazel.build https://bazel.build). I believe you are confusing it with Stubby. That's not to say that the general thrust of what you're saying is necessarily wrong though.
- deleted 8y ago[deleted]
- ebikelaw 8y agoI think you are misreading that comment. He’s saying that stubby is fast, not grpc. In fact performance is the big unknown with grpc adoption within google. It definitely isn’t on par with stubby today and it has to get there before anyone significant will switch to it.
- Xorlev 8y agoI can't comment on raw numbers (because I simply don't have them) but at least for the service I work on, replacing Stubby with gRPC wouldn't really move the needle even if it was 2-3x slower (it might be faster, this is just for illustration) -- we spend our time waiting on IO from other services or crunching numbers in the CPU. Being a Java service, gRPC/Java might well be just as fast or faster than Stubby Java, but I could understand that Stubby C++ has been hyperoptimized over the years vs. gRPC C core which might have a ways to go. By the latest performance dashboard [1, 2], gRPC/Java is leading the pack but gRPC C++ doesn't seem like it's slouching too much either. I seem to remember the C++ impl crushing Java at performance a while back, so I'm sure that'll change in the future. Honestly though? It'd take a _very_ demanding workload such that your RPC system was the bottleneck (so long as they're within constant factors of each other). There are services like that, but they're the exception and not the norm. Most services don't need to do 100kQPS/task. Even then, at that point you're spending a lot of time on serialization/deserialization, auth, logging, etc.. Your service is more than its communication layer, even if that's important to optimize it's still just a minor constant factor. The real problem is inertia. There's a lot of code/tools/patterns built up around Stubby and the semantics of Stubby (including all its features which likely haven't been ported to gRPC yet) and that's difficult to overcome. Our #1 use of gRPC so far I would imagine is at the edge. gRPC is making its way into Android apps since it's pretty trivial for translating proxies to more or less 1:1 convert gRPC to Stubby calls. [1] https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5636470266134528 https://performance-dot-grpc-testing.appspot.com/explore?das... [2] https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5652536396611584&widget=1735297033&container=1012810333 https://performance-dot-grpc-testing.appspot.com/explore?das...
- simonebrunozzi 8y agoDisagree. Take a look at CockroachDB, mostly based on Spanner. Google open sourcing spanner would have helped tremendously.