6 ms·

My understanding is that Google has recently deviated from this strategy. The result of the strategy you mention is that the industry standardized on other comp

My understanding is that Google has recently deviated from this strategy. The result of the strategy you mention is that the industry standardized on other companies' implementation of ideas that came from Google: Hadoop (MapReduce), HDFS (GFS), ZooKeeper (Chubby, and more. For examples of newer open source projects that see more active maintenance from Google, see Kubernetes and TensorFlow.

tehbeard 8y ago
Does that deviation lineup with Google's entry into offering cloud computing?
- skybrian 8y ago
  Pretty much. That's when it became clear that Google would have to support whatever is popular outside Google.
yingw787 8y ago
I know some people who develop on top of Tensorflow; from my conversations with them, Tensorflow's moat is Google making a lot of breaking changes by incorporating a lot of new functionality. I've also heard complaints that the online documentation isn't terribly great for triaging not-happy paths, to the point where you kind of have to just dig through the source code to figure out what's going on. Also, if you want to poach the maintainers, you would somehow have to poach them away from Google (which isn't happening, since ML at scale is something Google does best, and is an ongoing field of research). You can't become as good at using Tensorflow as Google is by simply forking the project.
shereadsthenews 8y ago
But K8s is not a development of Google software. It is developed specifically for the public, it throws out all of the interesting parts of Borg, and Google themselves don't use it, or barely do. As for that other stuff it seems to have worked out fine for Google: they describe obsolete technologies and the outside world develops hideous analogs of those and uses them for decades. Hadoop for example is just an unbelievably bad implementation of map reduce as it was ten years ago and is laughable compared to what Google replaced it with. HDFS is a joke of GFS which Google turned off eight years back. It's really remarkable the way the industry is essentially self-disabling in this regard. Meanwhile Google does not burden itself with trying to adopt every idea they read in a paper, and maintain a significant cost and efficiency advantage by doing so.
- dikei 8y ago
  Well, nobody should be using MapReduce now, and HDFS now is a lot better than 8 years ago. Without Google using, validating and releasing those design, we might be stuck with MPI and NFS for a lot longer.
  - wbl 8y ago
    How many GMM can MapReduce do? What about lattice quantum chronodynamics performance? MPI and Lustre exist for a reason: map reduce isn't great for all problems.
    - dikei 8y ago
      MR never claimed to be great for all problems. It main selling points was big-data and easy-of-use. Sure, MPI might blow MR out of the water in term of number-crunching, but it is also way harder to use.
- bb88 8y ago
  Implementation matters to google, more than to say the average company that uses Hadoop. At "Google-scale" small imperfections become huge imperfections. What's good for the bottom 90% of tech companies probably isn't for the top 10%.
  - breakingcups 8y ago
    I'd argue Kubernetes isn't the best choice for the bottom 90%. There's a lot of companies you could describe as tech and many are doing just fine in the old world of manual application provisioning.
    - ksec 8y ago
      >I'd argue Kubernetes isn't the best choice for the bottom 90%. Exactly. Instead of having something so simple that scales for 90% of everyone's needs. We have solution that Most enterprise wants and filter down from top to bottom. And it is true in almost all Tech things related.
rayvy 8y ago
Gonna just chime in and mention Yahoo for Hadoop (yes I know Big Table was Google), and Zookeeper. Great tech started at Yahoo, but they didn’t (unlike today) make too much of a fuss/self-pat-on-the-back about it
- atombender 8y ago
  Those projects were initiated by Yahoo, but the designs are taken directly from Google papers. Hadoop (HDFS and Map/Reduce) was based on the GFS and Map/Reduce papers. ZooKeeper was based on the Chubby paper.

tehbeard 8y ago

Does that deviation lineup with Google's entry into offering cloud computing?

skybrian 8y ago
Pretty much. That's when it became clear that Google would have to support whatever is popular outside Google.

yingw787 8y ago

I know some people who develop on top of Tensorflow; from my conversations with them, Tensorflow's moat is Google making a lot of breaking changes by incorporating a lot of new functionality. I've also heard complaints that the online documentation isn't terribly great for triaging not-happy paths, to the point where you kind of have to just dig through the source code to figure out what's going on. Also, if you want to poach the maintainers, you would somehow have to poach them away from Google (which isn't happening, since ML at scale is something Google does best, and is an ongoing field of research). You can't become as good at using Tensorflow as Google is by simply forking the project.

shereadsthenews 8y ago

But K8s is not a development of Google software. It is developed specifically for the public, it throws out all of the interesting parts of Borg, and Google themselves don't use it, or barely do. As for that other stuff it seems to have worked out fine for Google: they describe obsolete technologies and the outside world develops hideous analogs of those and uses them for decades. Hadoop for example is just an unbelievably bad implementation of map reduce as it was ten years ago and is laughable compared to what Google replaced it with. HDFS is a joke of GFS which Google turned off eight years back. It's really remarkable the way the industry is essentially self-disabling in this regard. Meanwhile Google does not burden itself with trying to adopt every idea they read in a paper, and maintain a significant cost and efficiency advantage by doing so.

dikei 8y ago
Well, nobody should be using MapReduce now, and HDFS now is a lot better than 8 years ago. Without Google using, validating and releasing those design, we might be stuck with MPI and NFS for a lot longer.
- wbl 8y ago
  How many GMM can MapReduce do? What about lattice quantum chronodynamics performance? MPI and Lustre exist for a reason: map reduce isn't great for all problems.
  - dikei 8y ago
    MR never claimed to be great for all problems. It main selling points was big-data and easy-of-use. Sure, MPI might blow MR out of the water in term of number-crunching, but it is also way harder to use.
bb88 8y ago
Implementation matters to google, more than to say the average company that uses Hadoop. At "Google-scale" small imperfections become huge imperfections. What's good for the bottom 90% of tech companies probably isn't for the top 10%.
- breakingcups 8y ago
  I'd argue Kubernetes isn't the best choice for the bottom 90%. There's a lot of companies you could describe as tech and many are doing just fine in the old world of manual application provisioning.
  - ksec 8y ago
    >I'd argue Kubernetes isn't the best choice for the bottom 90%. Exactly. Instead of having something so simple that scales for 90% of everyone's needs. We have solution that Most enterprise wants and filter down from top to bottom. And it is true in almost all Tech things related.

rayvy 8y ago

Gonna just chime in and mention Yahoo for Hadoop (yes I know Big Table was Google), and Zookeeper. Great tech started at Yahoo, but they didn’t (unlike today) make too much of a fuss/self-pat-on-the-back about it

atombender 8y ago
Those projects were initiated by Yahoo, but the designs are taken directly from Google papers. Hadoop (HDFS and Map/Reduce) was based on the GFS and Map/Reduce papers. ZooKeeper was based on the Chubby paper.