5 ms·

For a decade of container control, cAdvisor is really not a good product. Fundamentally flawed and seriously lacking documentation. This is an area where Google

For a decade of container control, cAdvisor is really not a good product. Fundamentally flawed and seriously lacking documentation. This is an area where Google is a miss, they leave to many projects dangling and half working. Luckily there are better alternatives on the market.

thrownaway2424 11y ago
The problem with extrapolating like you just did is that Kubernetes has not been in use at Google for a decade, and it is not related to the container infrastructure at Google. When you are evaluating Kubernetes it's important to know that no major project at Google has used Kubernetes for anything, ever. Kubernetes is a weird project where Google is trying to give the public a large-scale container infrastructure, but not _their_ large-scale container infrastructure.
- jamesblonde 11y ago
  I tend to agree. It's another case of - here is an API to access our products so you can play around with it. The scalable, powerful scheduler, however is not included. There are no other batteries that fit it, apart from GCE. Kind of like Blaze and TensorFlow - nice and shiny, but hollow (missing the good distributed filling).
  - skj 11y ago
    The internal google scheduling system is simply not appropriate for anyone other than Google to use. A huge number of machines, all alike, with a huge number of trusted binaries that can be multiplexed onto these machines without fear that they're going to break the jail and cause havoc (since there is a solid trail from source to running artifact). It's just not the reality that other companies exist in.
    - justicezyx 11y ago
      Scheduler is actually one relatively simple piece of the whole picture. If scheduling is a pain, then Kubernetes would just addressed. The fact that Kubernetes did not choose scheduler, means that it is actually not a big problem, at least not the biggest one.
    - mike_hearn 11y ago
      Borg jobs are not trusted! The system sandboxes them, prevents them spying on other jobs data files, and assumes they might abuse system resources in arbitrary ways. The days when all Google machines trusted all Google employees is long in the past.
    - stonogo 11y ago
      That is not an argument against releasing the code. Why would Google assign itself as gatekeeper? I personally could use this code on supercomputers now. I don't work for a supercomputing company; my use case is academic work and computational science. I absolutely have thousands of huge machines that I multiplex trusted binaries to -- and scheduling is not a trivial problem. So what's the real reason nobody gets to see this code?
      - thockingoog 11y ago
        The code is literally millions of LoC, all of which needs to be audited for stuff we can't release for whatever reasons. All that code is built upon layer after layer of Google internal stuff. Open-sourcing Borg means open-sourcing Chubby, internal form of gRPC (older), hundreds of libs, monitoring infrastructure, etc. Net result is O(50M) LoC. And when someone sends us a patch - then what? The cost of doing it is simply prohibitive. I'd love to do it, it's just not practical and has no RoI.
        stonogo 11y ago
        That's a much more sensible reason! If the code is truly Google specific, then I agre. It sounded to me like the code was not released because nobody else has a lot of computers, which I found odd. Thank you for the details!
  - thockingoog 11y ago
    "Powerful scheduling" is such a tiny piece of what Kubernetes does, it's funny. Yeah, Borg's scheduler is faster and more scalable and has more features. It also has 12 years of optimizations under its belt. I have 100% confidence that, should Kubernetes be around 10 years from now,this will be a non-issue.
    - x0rg 11y ago
      Sure, even though having multiple schedulers support (for different type of workloads) would be great to increase the cluster utilization, which is one selling point of such systems. I understand that Kubernetes is developed in the open and with the community but the heavy marketing as the "solution you can use now, directly from the creators of Omega" makes some people think it's ready, perfect and will fix all of their problems, but that's simply not true.
      - thockingoog 11y ago
        We have multi-scheduler support in v1.2 :)
        x0rg 11y ago
        Great, the docs are a bit bad for that, I'll have a look at the code :-)
  - thockingoog 11y ago
    As for "no other batteries that fit it" - I am confused. We do run on AWS, OpenStack, and other cloud providers, as well as on bare metal. It's not like nobody is using this thing. In fact, just a finger in the wind, I'd guess the number of people using it outside of Google Cloud is several times more than people using it on Google Cloud.
  - SEJeff 11y ago
    FWIW, you can plug mesos in as a Kubernetes scheduler. From the OSS world, that is about as heavy duty as you're going to get and proven on quite large (although not google large) 10k+ node clusters.
- thockingoog 11y ago
  As cited above, we simply CAN'T opensource Borg. It's enormous, and it is deeply, DEEPLY entangled with millions of LoC of Google code. Nobody could untangle that. And even if we DID untangle it, it's alien technology. It does not make any attempt to meet people where they are. It does not focus on open standards or simple solutions to simple problems. On top of that, it's got 10+ years of semi-organic growth in it. There are a lot of mistakes that have been made that we simply have to live with internally. Also, it's C++, for which there is approximately ZERO opensource community. We made a very strategic decision to rebuild it. It embodies many lessons from Borg and Omega (both things we got right and things we botched). It is implemented in an easier-to-approach language (Go) which has an active OSS community. It specifically focuses on "legacy apps" (everything written up to and including today) and open standards (HTTP, JSON, REST). I've never been shy about my opinion that I do hope to supplant Borg one day, but that day is necessarily years away. Of course no major project has ever used it, the whole thing didn't exist just 2 years ago.
  - deleted 11y ago
    [deleted]
  - gjvc 11y ago
    >> Also, it's C++, for which there is approximately ZERO opensource community. Rubbish, and saying so undermines the rest of your points.
    - zeveb 11y ago
      >> Also, it's C++, for which there is approximately ZERO opensource community. > Rubbish, and saying so undermines the rest of your points. Hardly rubbish. Although there are open-source projects which use C++, I and many others avoid them like the plague. I think he meant 'approximately ZERO' in the Spolsky sense, which is 'sure, there are some, but in the grand scheme of things they're indistinguishable from ε.
    - thockingoog 11y ago
      Wonderful rebuttal. Proof? There are some successful projects, but that is not a community. There are some libs, but that is not a community. The Go community is vibrant and growing. Go is an easy language to learn (and I say that as someone who LIKES the power of C++) and it is not a total joke to ask people who report bugs to jump in and try to fix them. C++ is simply NOT approachable by mere mortals, and would have made for a very different community and a much slower pace. And I say that as someone who detests many facets of Go - but it's just better at Getting Things Done than C++.
- TheIronYuppie 11y ago
  Disclaimer: I work at Google on Kubernetes. Tim knows more about this than almost anyone, but I will add one point - we have used Kubernetes for significant internal projects, and plan to continue expanding its usage over time. To Tim's point, though, it'll take time. The thought that you could move literally millions of lines of code and applications over to a new platform in just 10 months (the amount of time that Kubernetes has been GA) is... optimistic.
  - Estragon 11y ago
    When did the internal projects start using Kubernetes? I heard that its uptake within Google has been very anemic.
    - thockingoog 11y ago
      Some internal projects started evaluating Kubernetes well before 1.0. We don;t talk about them much because they are, well, internal. This is NOT in competition with Borg, though. Not yet.
rodionos 11y ago
cAdvisor is pretty good at what it does. At its core, it's a metrics collections daemon that iterates over cgroup controllers and sends them to a backend server for centralized alerting and long-term retention. It does other helpful things with the docker API such as extracting container labels so that you can rightsize resource allocations based on workload metadata. What's not to like?

thrownaway2424 11y ago

The problem with extrapolating like you just did is that Kubernetes has not been in use at Google for a decade, and it is not related to the container infrastructure at Google. When you are evaluating Kubernetes it's important to know that no major project at Google has used Kubernetes for anything, ever. Kubernetes is a weird project where Google is trying to give the public a large-scale container infrastructure, but not _their_ large-scale container infrastructure.

jamesblonde 11y ago
I tend to agree. It's another case of - here is an API to access our products so you can play around with it. The scalable, powerful scheduler, however is not included. There are no other batteries that fit it, apart from GCE. Kind of like Blaze and TensorFlow - nice and shiny, but hollow (missing the good distributed filling).
- skj 11y ago
  The internal google scheduling system is simply not appropriate for anyone other than Google to use. A huge number of machines, all alike, with a huge number of trusted binaries that can be multiplexed onto these machines without fear that they're going to break the jail and cause havoc (since there is a solid trail from source to running artifact). It's just not the reality that other companies exist in.
  - justicezyx 11y ago
    Scheduler is actually one relatively simple piece of the whole picture. If scheduling is a pain, then Kubernetes would just addressed. The fact that Kubernetes did not choose scheduler, means that it is actually not a big problem, at least not the biggest one.
  - mike_hearn 11y ago
    Borg jobs are not trusted! The system sandboxes them, prevents them spying on other jobs data files, and assumes they might abuse system resources in arbitrary ways. The days when all Google machines trusted all Google employees is long in the past.
  - stonogo 11y ago
    That is not an argument against releasing the code. Why would Google assign itself as gatekeeper? I personally could use this code on supercomputers now. I don't work for a supercomputing company; my use case is academic work and computational science. I absolutely have thousands of huge machines that I multiplex trusted binaries to -- and scheduling is not a trivial problem. So what's the real reason nobody gets to see this code?
    - thockingoog 11y ago
      The code is literally millions of LoC, all of which needs to be audited for stuff we can't release for whatever reasons. All that code is built upon layer after layer of Google internal stuff. Open-sourcing Borg means open-sourcing Chubby, internal form of gRPC (older), hundreds of libs, monitoring infrastructure, etc. Net result is O(50M) LoC. And when someone sends us a patch - then what? The cost of doing it is simply prohibitive. I'd love to do it, it's just not practical and has no RoI.
      - stonogo 11y ago
        That's a much more sensible reason! If the code is truly Google specific, then I agre. It sounded to me like the code was not released because nobody else has a lot of computers, which I found odd. Thank you for the details!
- thockingoog 11y ago
  "Powerful scheduling" is such a tiny piece of what Kubernetes does, it's funny. Yeah, Borg's scheduler is faster and more scalable and has more features. It also has 12 years of optimizations under its belt. I have 100% confidence that, should Kubernetes be around 10 years from now,this will be a non-issue.
  - x0rg 11y ago
    Sure, even though having multiple schedulers support (for different type of workloads) would be great to increase the cluster utilization, which is one selling point of such systems. I understand that Kubernetes is developed in the open and with the community but the heavy marketing as the "solution you can use now, directly from the creators of Omega" makes some people think it's ready, perfect and will fix all of their problems, but that's simply not true.
    - thockingoog 11y ago
      We have multi-scheduler support in v1.2 :)
      - x0rg 11y ago
        Great, the docs are a bit bad for that, I'll have a look at the code :-)
- thockingoog 11y ago
  As for "no other batteries that fit it" - I am confused. We do run on AWS, OpenStack, and other cloud providers, as well as on bare metal. It's not like nobody is using this thing. In fact, just a finger in the wind, I'd guess the number of people using it outside of Google Cloud is several times more than people using it on Google Cloud.
- SEJeff 11y ago
  FWIW, you can plug mesos in as a Kubernetes scheduler. From the OSS world, that is about as heavy duty as you're going to get and proven on quite large (although not google large) 10k+ node clusters.
thockingoog 11y ago
As cited above, we simply CAN'T opensource Borg. It's enormous, and it is deeply, DEEPLY entangled with millions of LoC of Google code. Nobody could untangle that. And even if we DID untangle it, it's alien technology. It does not make any attempt to meet people where they are. It does not focus on open standards or simple solutions to simple problems. On top of that, it's got 10+ years of semi-organic growth in it. There are a lot of mistakes that have been made that we simply have to live with internally. Also, it's C++, for which there is approximately ZERO opensource community. We made a very strategic decision to rebuild it. It embodies many lessons from Borg and Omega (both things we got right and things we botched). It is implemented in an easier-to-approach language (Go) which has an active OSS community. It specifically focuses on "legacy apps" (everything written up to and including today) and open standards (HTTP, JSON, REST). I've never been shy about my opinion that I do hope to supplant Borg one day, but that day is necessarily years away. Of course no major project has ever used it, the whole thing didn't exist just 2 years ago.
- deleted 11y ago
  [deleted]
- gjvc 11y ago
  >> Also, it's C++, for which there is approximately ZERO opensource community. Rubbish, and saying so undermines the rest of your points.
  - zeveb 11y ago
    >> Also, it's C++, for which there is approximately ZERO opensource community. > Rubbish, and saying so undermines the rest of your points. Hardly rubbish. Although there are open-source projects which use C++, I and many others avoid them like the plague. I think he meant 'approximately ZERO' in the Spolsky sense, which is 'sure, there are some, but in the grand scheme of things they're indistinguishable from ε.
  - thockingoog 11y ago
    Wonderful rebuttal. Proof? There are some successful projects, but that is not a community. There are some libs, but that is not a community. The Go community is vibrant and growing. Go is an easy language to learn (and I say that as someone who LIKES the power of C++) and it is not a total joke to ask people who report bugs to jump in and try to fix them. C++ is simply NOT approachable by mere mortals, and would have made for a very different community and a much slower pace. And I say that as someone who detests many facets of Go - but it's just better at Getting Things Done than C++.
TheIronYuppie 11y ago
Disclaimer: I work at Google on Kubernetes. Tim knows more about this than almost anyone, but I will add one point - we have used Kubernetes for significant internal projects, and plan to continue expanding its usage over time. To Tim's point, though, it'll take time. The thought that you could move literally millions of lines of code and applications over to a new platform in just 10 months (the amount of time that Kubernetes has been GA) is... optimistic.
- Estragon 11y ago
  When did the internal projects start using Kubernetes? I heard that its uptake within Google has been very anemic.
  - thockingoog 11y ago
    Some internal projects started evaluating Kubernetes well before 1.0. We don;t talk about them much because they are, well, internal. This is NOT in competition with Borg, though. Not yet.

rodionos 11y ago

cAdvisor is pretty good at what it does. At its core, it's a metrics collections daemon that iterates over cgroup controllers and sends them to a backend server for centralized alerting and long-term retention. It does other helpful things with the docker API such as extracting container labels so that you can rightsize resource allocations based on workload metadata. What's not to like?