5 ms·

>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-

>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?

bajsejohannes 9y ago
You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48c9a600e405 https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...
- outside1234 9y ago
  That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.
  - kyrra 9y ago
    Packaging was (is?) a bit different, because it existed before Docker existed. Pdf of the presentation: https://www.usenix.org/sites/default/files/conference/protected-files/lisa_2014_talk.pdf https://www.usenix.org/sites/default/files/conference/protec... And a recording of the talk: https://www.usenix.org/conference/lisa14/conference-program/presentation/mcnutt https://www.usenix.org/conference/lisa14/conference-program/...
    - puzzle 9y ago
      Googlers, including the ones working on Kubernetes, like to pick on MPM, but that's probably because they haven't run major services deployed from Docker images.
  - puzzle 9y ago
    Before it ran on F1/Spanner, Adwords ran on sharded MySQL on Borg. Not for its entire early life, but for quite a few years. Later in life, Checkout and maybe Wallet ran on MySQL on Borg, too. So did YouTube, which used Vitess, a sharding layer (now ported to Kubernetes and open sourced). Bigtable, Spanner and even Colossus/D run in containers on Borg.
    - tayo42 9y ago
      How is colossus running in a container in borg? That's their network file system that makes running databases in containers possible?
      - puzzle 9y ago
        There's a bootstrapping issue, of course, but that's how it works. And it's even crazier than what you're picturing. What if I told you that Bigtable runs on top of Colossus, but Colossus itself stores metadata about files, including Bigtable's files, in... Bigtable? It's really turtles all the way down, the last of which is, luckily, Chubby. Some of the details here: http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...
- daxfohl 9y ago
  Wow, I'd never heard of Fargate until this. It seems so somewhat obvious now for things that Lambda doesn't fit. Heroku for containers, not VMs.
  - manigandham 9y ago
    There's also Azure Container Instances: https://azure.microsoft.com/en-us/services/container-instances/ https://azure.microsoft.com/en-us/services/container-instanc... And Google has AppEngine, now with Flexible Environment that can run containers: https://cloud.google.com/appengine/docs/flexible/dotnet/quickstart https://cloud.google.com/appengine/docs/flexible/dotnet/quic...
- puzzle 9y ago
  Kubernetes might be simpler than Borg in so many ways (let me count the ones I care about...), but it also has better features that Borg did not implement (labels and selectors) or that are offered only by some other internal services, which obviously are configured through very different mechanisms (ingress).
lobster_johnson 9y ago
It's my understanding, based on comments by googlers here on HN, that Google does run a bunch of apps on GKE. We don't know about which apps, but it's not surprising that they want to dogfood their own cloud platform.

>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?

bajsejohannes 9y ago

You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48c9a600e405 https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...

outside1234 9y ago
That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.
- kyrra 9y ago
  Packaging was (is?) a bit different, because it existed before Docker existed. Pdf of the presentation: https://www.usenix.org/sites/default/files/conference/protected-files/lisa_2014_talk.pdf https://www.usenix.org/sites/default/files/conference/protec... And a recording of the talk: https://www.usenix.org/conference/lisa14/conference-program/presentation/mcnutt https://www.usenix.org/conference/lisa14/conference-program/...
  - puzzle 9y ago
    Googlers, including the ones working on Kubernetes, like to pick on MPM, but that's probably because they haven't run major services deployed from Docker images.
- puzzle 9y ago
  Before it ran on F1/Spanner, Adwords ran on sharded MySQL on Borg. Not for its entire early life, but for quite a few years. Later in life, Checkout and maybe Wallet ran on MySQL on Borg, too. So did YouTube, which used Vitess, a sharding layer (now ported to Kubernetes and open sourced). Bigtable, Spanner and even Colossus/D run in containers on Borg.
  - tayo42 9y ago
    How is colossus running in a container in borg? That's their network file system that makes running databases in containers possible?
    - puzzle 9y ago
      There's a bootstrapping issue, of course, but that's how it works. And it's even crazier than what you're picturing. What if I told you that Bigtable runs on top of Colossus, but Colossus itself stores metadata about files, including Bigtable's files, in... Bigtable? It's really turtles all the way down, the last of which is, luckily, Chubby. Some of the details here: http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...
daxfohl 9y ago
Wow, I'd never heard of Fargate until this. It seems so somewhat obvious now for things that Lambda doesn't fit. Heroku for containers, not VMs.
- manigandham 9y ago
  There's also Azure Container Instances: https://azure.microsoft.com/en-us/services/container-instances/ https://azure.microsoft.com/en-us/services/container-instanc... And Google has AppEngine, now with Flexible Environment that can run containers: https://cloud.google.com/appengine/docs/flexible/dotnet/quickstart https://cloud.google.com/appengine/docs/flexible/dotnet/quic...
puzzle 9y ago
Kubernetes might be simpler than Borg in so many ways (let me count the ones I care about...), but it also has better features that Borg did not implement (labels and selectors) or that are offered only by some other internal services, which obviously are configured through very different mechanisms (ingress).

lobster_johnson 9y ago

It's my understanding, based on comments by googlers here on HN, that Google does run a bunch of apps on GKE. We don't know about which apps, but it's not surprising that they want to dogfood their own cloud platform.