6 ms·
But K8s is not a development of Google software. It is developed specifically for the public, it throws out all of the interesting parts of Borg, and Google the
by shereadsthenews 8y ago
But K8s is not a development of Google software. It is developed specifically for the public, it throws out all of the interesting parts of Borg, and Google themselves don't use it, or barely do. As for that other stuff it seems to have worked out fine for Google: they describe obsolete technologies and the outside world develops hideous analogs of those and uses them for decades. Hadoop for example is just an unbelievably bad implementation of map reduce as it was ten years ago and is laughable compared to what Google replaced it with. HDFS is a joke of GFS which Google turned off eight years back. It's really remarkable the way the industry is essentially self-disabling in this regard. Meanwhile Google does not burden itself with trying to adopt every idea they read in a paper, and maintain a significant cost and efficiency advantage by doing so.
- dikei 8y agoWell, nobody should be using MapReduce now, and HDFS now is a lot better than 8 years ago. Without Google using, validating and releasing those design, we might be stuck with MPI and NFS for a lot longer.
- wbl 8y agoHow many GMM can MapReduce do? What about lattice quantum chronodynamics performance? MPI and Lustre exist for a reason: map reduce isn't great for all problems.
- dikei 8y agoMR never claimed to be great for all problems. It main selling points was big-data and easy-of-use. Sure, MPI might blow MR out of the water in term of number-crunching, but it is also way harder to use.
- bb88 8y agoImplementation matters to google, more than to say the average company that uses Hadoop. At "Google-scale" small imperfections become huge imperfections. What's good for the bottom 90% of tech companies probably isn't for the top 10%.
- breakingcups 8y agoI'd argue Kubernetes isn't the best choice for the bottom 90%. There's a lot of companies you could describe as tech and many are doing just fine in the old world of manual application provisioning.
- ksec 8y ago>I'd argue Kubernetes isn't the best choice for the bottom 90%. Exactly. Instead of having something so simple that scales for 90% of everyone's needs. We have solution that Most enterprise wants and filter down from top to bottom. And it is true in almost all Tech things related.
- koffiezet 8y agoI don't agree at all. While the large majority does not need the scalability it offers, it can benefit from all the other stuff applying the 'best practices' offers. The problem is - that many people do not stick to the best practices and do not know how to build containerised applications. Implementing the whole "DevOps" idea just becomes a whole lot easier when developers don't even have the concept of their snowflake server anymore. And yes - k8s has a ton of overhead and is pretty complex to get into at first, but it all makes sense. There are many points that could be criticised about it, it's far from perfect, but having a standardised way of deploying whatever application has been a massive game changer in the development environments I've been thrown in. Source/disclaimer: I'm a consultant that has seen quite a few k8s/openshift fuckups and success stories, both on large and small scale.
- StreamBright 8y agoI am a consultant working on Hadoop installations across the globe. As an average I usually able to save 70% disk usage and 30% overall cost by changing the defaults to a reasonable value as well as migrating companies out of HDFS to something like S3. I have spent majority of my career (10 years) working on Hadoop and I can tell you that it is a terrible piece of software with insane ineffciency all over the place. If you would switch over all the Hadoop installations on Earth at once to something more reasonable it would be visible on the global CO2 production chart quite a bit. What is good for the bottom 90% of companies is a financial question not a technological question. It is a bit unfurtunate that Hadoop is so popular and nobody cares about efficiency, not even the Hadoop vendors (maybe with the exception of MapR, which is not opensource).
- ntoshev 8y agoGoogle offers their most recent infrastructure as a service in GCP, e.g. Cloud Dataflow, and it hasn't exactly taken the world by storm. Industry standards matter, even if they are inferior implementations; the differential is just not that big.
- pm90 8y agoDataflow is based on oss Apache airflow. I don’t know how well it’s doing in the wild but every IT admin I’ve worked with are super excited to use it.
- manojlds 8y agoAirflow and Dataflow are not related. Google open sourced dataflow as Beam - https://beam.apache.org https://beam.apache.org
- omeid2 8y agoCan you mention some of the interesting parts of Borg that are missing in k8s?
- ithkuil 8y agoamong other things, I miss Autopilot and generally the extensive machinery to help with massive capacity planning. Think of Autopilot as an automation that tweaks a pod's request/limits according to what it actually needs in order to reduce waste and thus improve cluster utilization. (I _think_ this no longer qualifies as secret after https://github.com/kubernetes/kubernetes/issues/44095 https://github.com/kubernetes/kubernetes/issues/44095) That said, k8s is quite extensible and it would definitely be possible to add such a component as a controller.
- ec109685 8y agoAutopilot is coming as VPA: https://stupefied-goodall-e282f7.netlify.com/contributors/design-proposals/autoscaling/vertical-pod-autoscaler/ https://stupefied-goodall-e282f7.netlify.com/contributors/de...
- davidopp__ 8y ago> it throws out all of the interesting parts of Borg This is not true. It throws out the Google-specific parts of Borg (like integration with Google's service discovery, load balancing, and monitoring systems) and improves a number of things compared to Borg. For a good reference on the evolution of Borg into Kubernetes, I recommend the recent Kubernetes Podcast interview with Brian Grant: https://kubernetespodcast.com/episode/043-borg-omega-kubernetes-beyond/ https://kubernetespodcast.com/episode/043-borg-omega-kuberne... > Google themselves don't use it This is not true, and the reasons why it hasn't replaced Borg are related to the integrations I mentioned above (which will take time to integrate or replace) and the zillions of lines of borg config that have built up over the years, rather than concerns that people outside of Google would have (production-worthiness, reliability, etc.) (Disclaimer: I worked on Borg at Google, and now work on Kubernetes at Google.)
- shereadsthenews 8y agoUnfortunately we can't discuss the parts of Google's platform that aren't in Kubernetes on this forum. If we could, I think I could defend my statement reasonably well. But perhaps you just don't think that the pieces I would mention qualify as interesting.