8 ms·

As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't nece

by tazjin 5y ago

As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't necessarily think that's for the better. Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" than Kubernetes ever did to me. I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares.

_hzrk 5y ago
> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. I know that Borg was written in Java and Kubernetes in Go. Though the latter had a reputation in the beginning as a systems programming language, its purpose was actually to build large-scale cloud infrastructure projects with it and it proved formidably well suited for the task. It compiles fast, anyone can read it, good tooling and it is efficient for the layer on which it is meant to be deployed. Go, as Java is one of the most productive languages in use today, judging by the ecosystems they have spawned.
- tazjin 5y ago
  Borg is C++.
  - johnnycerberus 5y ago
    There you Go (pun intended). Go even replaced a low-level language like C++ and achieved the same result in the end. I don't know why I thought it's Java, probably the first Kube was initially in Java. It's even better that they managed to pull that off.
    - tazjin 5y ago
      But it did not replace it. Google runs on Borg. Go lets you build things fast, but the lack of strong typing and the vast amount of language pitfalls make maintenance hard in the long run. The community also has the attitude to pretend that these pitfalls don't actually exist, which is very different from C++ where most peculiar behaviours are well-understood and controlled.
      - johnnycerberus 5y ago
        Sorry, I don't think I have been explicit in my comment. The language of K8s, a system which is the so-called direct descendant of Borg, replaced C++ at that layer. Also, Go is strongly typed.
        mijoharas 5y ago
        I'd argue that go is statically typed rather than strongly typed due to `interface{}`. > There is no real agreement on what "strongly typed" means, although the most widely used definition in the professional literature is that in a "strongly typed" language, it is not possible for the programmer to work around the restrictions imposed by the type system. This term is almost always used to describe statically typed languages. (random definition found googling).
        still_grokking 5y ago
        Actually Google knew at the time they "designed" k8s that Borg doesn't scale due to fundamental design flaws in it's basic architecture. Still they reused the exact same architecture for k8s. Who wants to know the details of those scaling issues can google for the Omega paper. The "proper"™ solution to those design flaws was implemented in Mesos (and to my knowledge nowhere else until now).
        ksec 5y ago
        >The "proper"™ solution to those design flaws was implemented in Mesos And yet somehow Mesos failed... sigh.
    - farresito 5y ago
      > Go even replaced a low-level language like C++ and achieved the same result in the end Did it, though? Honest question. I get the feeling that it ended up competing with Java (and Python) more than it ended up replacing C++. The C++ folks seem to be way more into Rust than Go.
      - johnnycerberus 5y ago
        I'm a Java fan and an ex-Scala aficionado. I would have hoped that JVM would eat the pie when it comes to cloud deployments but it didn't happen. Like Scala.js never happened and TypeScript became the type system of the web. JVM languages will remain at the application layer, concerned with microservices, data processing, streaming, databases, etc. It's not what folks seem to be way more into, it's all about tradeoffs. I am talking here about layers and the suitability of a language at a specific layer. I don't know about Rust, but Go proved that it can handle it. If Rust would prove to be better, that would be great, but only time will tell. Until now it did not happen, instead people are trying to fit Rust in scenarios where a higher-level language would go round in circles (GUI programming, microservices, game development etc.). For Java it is too late, if GraalVM with native images and value types would have been released earlier, maybe we could say that Java could compete with Go at that layer, but it is not, the train has left the station long time ago. Only if Oracle comes out of the cave and throw a lot of money at a JVM alternative to Kubernetes, which is likely to happen in the foreseeable future, given the investments and the attention the Java platform has received recently.
        EdwardDiego 5y ago
        Plenty of K8s operators written in Java. Sure, it's not not underpinning K8s itself, but tbh, what language K8s is written in doesn't really matter, so long as it works.
        johnnycerberus 5y ago
        Yes, exactly my thoughts, the problem is that devops generally are used to Python/Go and in my company though they don't make it mandatory, they recommend Go. Also, they have a repulsive reaction to everything .NET and JVM :).
- EdwardDiego 5y ago
  I just wish klog would go die in a fire, I'm so confused at why logging is so terrible in Go.
  - GlennS 5y ago
    Can you expand on this? My experience with Go has just been making small changes to small programs. So, I don't know what the normal experience is. My experience with logging varies from: print (works fine I guess) import logging (this is pretty good - means I don't have to parse my logs before deciding where to send them) import slf4j (6 logging frameworks and a logging framework disintermediation framework)
    - EdwardDiego 5y ago
      I elaborated a bit in my reply above, but basically, multiple logging frameworks with incompatible APIs, few of which offer the fine-grained control the person running the app might need. But I'm used to the JVM world. And when I first met slf4j, I was like, wtf is this crap, but I appreciate it now as the embodiment of the desire to standardise logging across the Java ecosystem. While using slf4j does make it trivial to swap out logging frameworks in an app, in my 13 years at my last job, we only did that once, from log4j to Logback, so that's not so important. But yeah, what I miss from Java land in Go logging is the common approach - loggers and appenders are (usually) configured outside of code, the user can provide their own configuration at runtime to override the config shipped in the jar to troubleshoot issues - especially when you can configure the logging lib to check the conf file every X seconds for changes - allows you to change logger levels on the fly without restarting the app (ditto Logback's JMX configurator). And lastly, no matter the logging library, configuring them is near identical.
  - cpuguy83 5y ago
    klog is the descendant of glog which is a go implementation of Google's logging. Go has nothing to do with it except it's the logging that k8s took on (originally it used glog). -- edit typo
    - 5y ago
proxysna 5y ago
> I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares. Consul is used for service discovery. Fast, reliable and easily extensible. Yet to have a serious issue with it.
- pronoiac 5y ago
  Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and products, based on feedback.
  - mijoharas 5y ago
    Could you expand a little on your problems with Consul? (I have no experience with it myself).
    - kerblang 5y ago
      Not OP, but go look at the consul documentation. In fact, just look at "Configuration" page alone: https://www.consul.io/docs/agent/options https://www.consul.io/docs/agent/options - it goes FOREVER. Maybe you don't need most of that, but whatever you do need, you're going to have to find it somewhere in there. And yes, the error messages can be utterly cryptic. One thing: Don't ever let junior devs try to create a cluster out of their own laptops (yes they will think it sounds like a good idea) as this will be a never-ending nightmare.
    - pronoiac 5y ago
      It's been a while, so I'm fuzzy on some details: * in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged * in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers * second-hand, in self-hosted installations, it caused a number of hard-to-troubleshoot outages * something about circular dependencies and going by "wait _n_ seconds" rather than by healthiness It was reliable enough that it could gather a really significant blast radius, and it had different gnarly failure modes, so documentation could be irrelevant from case to case.
gilbetron 5y ago
> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. How is Go not suited to those? I'm not seeing it - and are you comparing Go to Java or C++?
- papito 5y ago
  Go is a very hard language to work on as a team. It does not push a lot of conventions on you, and when it does, they just seem weird and anachronistic. Even the original k8s codebase was notoriously horrible in Go, and that was "from the source". Ironically, HashiCorp has a better Go codebase, and that's where I picked best practices from, not from Google. The problem with Go is that the Googlers fail to challenge some of the design decisions. The giants of computer science working on it at the company cannot be questioned, and that just leads to a broken feedback process. As someone said, Go is the new C, skipping decades of new language design concepts.
  - tazjin 5y ago
    > The problem with Go is that the Googlers fail to challenge some of the design decisions. The giants of computer science working on it at the company cannot be questioned, and that just leads to a broken feedback process. Well, they can be questioned but they're not very receptive to it, so most people don't bother. During my time at Alphabet I didn't see much use of Go anyways, other than in glue tooling (including a horrific large-scale incident due to non-obvious behaviour of Go code in such a glue tool).
  - cpuguy83 5y ago
    The original k8s codebase was Java, then rewritten in Go by Java developers. K8s is not a good example of Go, and it has nothing to do with the size of the project.
  - still_grokking 5y ago
    > As someone said, Go is the new C, skipping decades of new language design concepts. Well, some are doomed to repeat their errors. In fact C was even crappy for it's own time.
jeffbee 5y ago
Borg is what k8s could be if there were any kind of ground rules and people were willing to call out some use cases as being legitimately stupid. Compared to k8s, Borg is a model of usability and simplicity.
- dilyevsky 5y ago
  Ground rules as in “you have to compile with 100s of these google3 libraries to run a simple app”?
  - jeffbee 5y ago
    I don't think that's really true at all. You can pretty easily run any damned thing in Borg. If you want it to do stuff with other Google services then you need a lot of Google code, but if you just want to run 100000 replicas of `/bin/sleep 100000`, Borg makes it easy.
    - p_l 5y ago
      Because from what I heard, all the other crucial to me benefits of k8s are provided through linking of lots and lots of Google internal libraries?
    - dilyevsky 5y ago
      Sure but none of the stuff like authentication and access to any google systems will work. Kubernetes is complex bc it can take damn near any docker container and it can run seamlessly
omneity 5y ago
Nomad was implemented based on the Borg paper.

by tazjin 5y ago

_hzrk 5y ago

> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. I know that Borg was written in Java and Kubernetes in Go. Though the latter had a reputation in the beginning as a systems programming language, its purpose was actually to build large-scale cloud infrastructure projects with it and it proved formidably well suited for the task. It compiles fast, anyone can read it, good tooling and it is efficient for the layer on which it is meant to be deployed. Go, as Java is one of the most productive languages in use today, judging by the ecosystems they have spawned.

tazjin 5y ago
Borg is C++.
- johnnycerberus 5y ago
  There you Go (pun intended). Go even replaced a low-level language like C++ and achieved the same result in the end. I don't know why I thought it's Java, probably the first Kube was initially in Java. It's even better that they managed to pull that off.
  - tazjin 5y ago
    But it did not replace it. Google runs on Borg. Go lets you build things fast, but the lack of strong typing and the vast amount of language pitfalls make maintenance hard in the long run. The community also has the attitude to pretend that these pitfalls don't actually exist, which is very different from C++ where most peculiar behaviours are well-understood and controlled.
    - johnnycerberus 5y ago
      Sorry, I don't think I have been explicit in my comment. The language of K8s, a system which is the so-called direct descendant of Borg, replaced C++ at that layer. Also, Go is strongly typed.
      - mijoharas 5y ago
        I'd argue that go is statically typed rather than strongly typed due to `interface{}`. > There is no real agreement on what "strongly typed" means, although the most widely used definition in the professional literature is that in a "strongly typed" language, it is not possible for the programmer to work around the restrictions imposed by the type system. This term is almost always used to describe statically typed languages. (random definition found googling).
      - still_grokking 5y ago
        Actually Google knew at the time they "designed" k8s that Borg doesn't scale due to fundamental design flaws in it's basic architecture. Still they reused the exact same architecture for k8s. Who wants to know the details of those scaling issues can google for the Omega paper. The "proper"™ solution to those design flaws was implemented in Mesos (and to my knowledge nowhere else until now).
        ksec 5y ago
        >The "proper"™ solution to those design flaws was implemented in Mesos And yet somehow Mesos failed... sigh.
  - farresito 5y ago
    > Go even replaced a low-level language like C++ and achieved the same result in the end Did it, though? Honest question. I get the feeling that it ended up competing with Java (and Python) more than it ended up replacing C++. The C++ folks seem to be way more into Rust than Go.
    - johnnycerberus 5y ago
      I'm a Java fan and an ex-Scala aficionado. I would have hoped that JVM would eat the pie when it comes to cloud deployments but it didn't happen. Like Scala.js never happened and TypeScript became the type system of the web. JVM languages will remain at the application layer, concerned with microservices, data processing, streaming, databases, etc. It's not what folks seem to be way more into, it's all about tradeoffs. I am talking here about layers and the suitability of a language at a specific layer. I don't know about Rust, but Go proved that it can handle it. If Rust would prove to be better, that would be great, but only time will tell. Until now it did not happen, instead people are trying to fit Rust in scenarios where a higher-level language would go round in circles (GUI programming, microservices, game development etc.). For Java it is too late, if GraalVM with native images and value types would have been released earlier, maybe we could say that Java could compete with Go at that layer, but it is not, the train has left the station long time ago. Only if Oracle comes out of the cave and throw a lot of money at a JVM alternative to Kubernetes, which is likely to happen in the foreseeable future, given the investments and the attention the Java platform has received recently.
      - EdwardDiego 5y ago
        Plenty of K8s operators written in Java. Sure, it's not not underpinning K8s itself, but tbh, what language K8s is written in doesn't really matter, so long as it works.
        johnnycerberus 5y ago
        Yes, exactly my thoughts, the problem is that devops generally are used to Python/Go and in my company though they don't make it mandatory, they recommend Go. Also, they have a repulsive reaction to everything .NET and JVM :).
EdwardDiego 5y ago
I just wish klog would go die in a fire, I'm so confused at why logging is so terrible in Go.
- GlennS 5y ago
  Can you expand on this? My experience with Go has just been making small changes to small programs. So, I don't know what the normal experience is. My experience with logging varies from: print (works fine I guess) import logging (this is pretty good - means I don't have to parse my logs before deciding where to send them) import slf4j (6 logging frameworks and a logging framework disintermediation framework)
  - EdwardDiego 5y ago
    I elaborated a bit in my reply above, but basically, multiple logging frameworks with incompatible APIs, few of which offer the fine-grained control the person running the app might need. But I'm used to the JVM world. And when I first met slf4j, I was like, wtf is this crap, but I appreciate it now as the embodiment of the desire to standardise logging across the Java ecosystem. While using slf4j does make it trivial to swap out logging frameworks in an app, in my 13 years at my last job, we only did that once, from log4j to Logback, so that's not so important. But yeah, what I miss from Java land in Go logging is the common approach - loggers and appenders are (usually) configured outside of code, the user can provide their own configuration at runtime to override the config shipped in the jar to troubleshoot issues - especially when you can configure the logging lib to check the conf file every X seconds for changes - allows you to change logger levels on the fly without restarting the app (ditto Logback's JMX configurator). And lastly, no matter the logging library, configuring them is near identical.
- cpuguy83 5y ago
  klog is the descendant of glog which is a go implementation of Google's logging. Go has nothing to do with it except it's the logging that k8s took on (originally it used glog). -- edit typo
  - 5y ago

proxysna 5y ago

> I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares. Consul is used for service discovery. Fast, reliable and easily extensible. Yet to have a serious issue with it.

pronoiac 5y ago
Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and products, based on feedback.
- mijoharas 5y ago
  Could you expand a little on your problems with Consul? (I have no experience with it myself).
  - kerblang 5y ago
    Not OP, but go look at the consul documentation. In fact, just look at "Configuration" page alone: https://www.consul.io/docs/agent/options https://www.consul.io/docs/agent/options - it goes FOREVER. Maybe you don't need most of that, but whatever you do need, you're going to have to find it somewhere in there. And yes, the error messages can be utterly cryptic. One thing: Don't ever let junior devs try to create a cluster out of their own laptops (yes they will think it sounds like a good idea) as this will be a never-ending nightmare.
  - pronoiac 5y ago
    It's been a while, so I'm fuzzy on some details: * in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged * in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers * second-hand, in self-hosted installations, it caused a number of hard-to-troubleshoot outages * something about circular dependencies and going by "wait _n_ seconds" rather than by healthiness It was reliable enough that it could gather a really significant blast radius, and it had different gnarly failure modes, so documentation could be irrelevant from case to case.

gilbetron 5y ago

> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. How is Go not suited to those? I'm not seeing it - and are you comparing Go to Java or C++?

papito 5y ago
Go is a very hard language to work on as a team. It does not push a lot of conventions on you, and when it does, they just seem weird and anachronistic. Even the original k8s codebase was notoriously horrible in Go, and that was "from the source". Ironically, HashiCorp has a better Go codebase, and that's where I picked best practices from, not from Google. The problem with Go is that the Googlers fail to challenge some of the design decisions. The giants of computer science working on it at the company cannot be questioned, and that just leads to a broken feedback process. As someone said, Go is the new C, skipping decades of new language design concepts.
- tazjin 5y ago
  > The problem with Go is that the Googlers fail to challenge some of the design decisions. The giants of computer science working on it at the company cannot be questioned, and that just leads to a broken feedback process. Well, they can be questioned but they're not very receptive to it, so most people don't bother. During my time at Alphabet I didn't see much use of Go anyways, other than in glue tooling (including a horrific large-scale incident due to non-obvious behaviour of Go code in such a glue tool).
- cpuguy83 5y ago
  The original k8s codebase was Java, then rewritten in Go by Java developers. K8s is not a good example of Go, and it has nothing to do with the size of the project.
- still_grokking 5y ago
  > As someone said, Go is the new C, skipping decades of new language design concepts. Well, some are doomed to repeat their errors. In fact C was even crappy for it's own time.

jeffbee 5y ago

Borg is what k8s could be if there were any kind of ground rules and people were willing to call out some use cases as being legitimately stupid. Compared to k8s, Borg is a model of usability and simplicity.

dilyevsky 5y ago
Ground rules as in “you have to compile with 100s of these google3 libraries to run a simple app”?
- jeffbee 5y ago
  I don't think that's really true at all. You can pretty easily run any damned thing in Borg. If you want it to do stuff with other Google services then you need a lot of Google code, but if you just want to run 100000 replicas of `/bin/sleep 100000`, Borg makes it easy.
  - p_l 5y ago
    Because from what I heard, all the other crucial to me benefits of k8s are provided through linking of lots and lots of Google internal libraries?
  - dilyevsky 5y ago
    Sure but none of the stuff like authentication and access to any google systems will work. Kubernetes is complex bc it can take damn near any docker container and it can run seamlessly

omneity 5y ago

Nomad was implemented based on the Borg paper.