5 ms·

You know, I think, maybe it's not a coincidence that almost all high-scale infrastructure tooling is written in Go nowadays.

kuschku 10y ago
Almost all high-scale infrastructure tooling? Are there many databases, parallelism frameworks, indexing systems, or machine learning frameworks in go? I’d be interested if you can name any fields where non-Google go projects dominate. I only know of containers.
- cube2222 10y ago
  Service Discovery and configuration (Consul, etcd), Monitoring (Prometheus and Influx), non-container cluster management (Nomad), databases (CockroachDB), messaging (NATS, Jocko), Terraform. Other than that, I think everybody noticed the frequent posts "xxx database written in Go". Also, most high scale computing companies now use Go for their infrastructure parts. EDIT: You can also check out all the stuff the CoreOS team puts out.
  - kuschku 10y ago
    I’ve actually never used any of them, and none of the devs or ops people I usually talk to used them either. We never even heard of any of those except for CockroachDB. Are you sure these are "dominating the entire industry"-products, as you mentioned before? Or is it more that they’re only used in SV?
    - cube2222 10y ago
      Totally not, I'm working in Poland. And, I'm sorry to say, if you haven't heard about Consul, then you're really not in a position to judge if infrastructure tools are or aren't written in Go. Consul is the de'facto industry standard for service discovery for a few years already. EDIT: Cockroach is actually the most SV'ish of all those.
      - kuschku 10y ago
        > Consul is the de'facto industry standard for service discovery for a few years already. So standard that it doesn’t even have a Wikipedia page? Nor is there even one for this specific type of datacenter-oriented service discovery? This is all pretty niche tech, used by a handful of startups that engineer technology for billions of users, but not used anywhere else. I’ve never seen any companies on national scale operate with this – nor would they need to.
        cube2222 10y ago
        The thing is, I know a lot of companies using it at nationwide scale (non-startups). I think there's a reason you are getting downvoted. Anyways, Service Discovery like this is less niche than containers are. If containers are still niche for you, then there you go.
        kuschku 10y ago
        > I think there's a reason you are getting downvoted. Well, so are you, your comments are even so grey I can barely read them anymore (and I can’t downvote you). Regarding the rest: Containers are far less niche than Consul is. For service discovery, the market isn’t nearly as monopolized as containers are. And containers are far more used than consul is – most places I’ve seen build their own service discovery systems, or hardcode a main node and then have services in containers register with that, etc. While for containers, at the moment, there’s almost only docker, due to the ecosystem.
      - maloney 10y ago
        etcd is also used quite a bit for service discovery, but it's written in Go too :)
        cube2222 10y ago
        Are you using it personally? I never met anyone using it for that. How's your experience with it been? (in case you did use it)
    - koffiezet 10y ago
      In the ops world, these tools are pretty much the standard. If you look at the state of many of the tools targeted at infrastructure were, with in the best scenario a decently maintained Python code-base - it's no surprise the ops world jumped on Go. Go's main attraction there is that it's a simple, script-like language that compiles to easily deployed static binaries. That enables people who's main job is not necessarily coding, but with a good scripting background to solve problems by writing more complex/performance-critical stuff which was previously not feasible in languages like Python, Ruby, ... These projects requiring more performance would have required a seasoned dedicated C/C++ coder before, but now they can be started and maintainer by people initially not focusing on programming, but on solving problems they are facing in their daily job. They don't have the time to invest learning a language with complex typesystems where initially, you'll be fighting the language itself more than solving actual problems. For programmers, this investment is useful, it's their main skill and is worth something. Ops people have other things to worry about, and just want to solve problems. It does mean wheels are being reinvented, but the major projects now have pretty smart people working on them, hitting the limits of the language and it's garbage collector, pushing them forward.
      - ex_amazon_sde 10y ago
        > In the ops world, these tools are pretty much the standard Thankfully, this is true only in the HN bubble. One example: Amazon.
        cube2222 10y ago
        Amazon is a big enough company that it's obvious that they'll roll their own stuff usually.

kuschku 10y ago

Almost all high-scale infrastructure tooling? Are there many databases, parallelism frameworks, indexing systems, or machine learning frameworks in go? I’d be interested if you can name any fields where non-Google go projects dominate. I only know of containers.

cube2222 10y ago
Service Discovery and configuration (Consul, etcd), Monitoring (Prometheus and Influx), non-container cluster management (Nomad), databases (CockroachDB), messaging (NATS, Jocko), Terraform. Other than that, I think everybody noticed the frequent posts "xxx database written in Go". Also, most high scale computing companies now use Go for their infrastructure parts. EDIT: You can also check out all the stuff the CoreOS team puts out.
- kuschku 10y ago
  I’ve actually never used any of them, and none of the devs or ops people I usually talk to used them either. We never even heard of any of those except for CockroachDB. Are you sure these are "dominating the entire industry"-products, as you mentioned before? Or is it more that they’re only used in SV?
  - cube2222 10y ago
    Totally not, I'm working in Poland. And, I'm sorry to say, if you haven't heard about Consul, then you're really not in a position to judge if infrastructure tools are or aren't written in Go. Consul is the de'facto industry standard for service discovery for a few years already. EDIT: Cockroach is actually the most SV'ish of all those.
    - kuschku 10y ago
      > Consul is the de'facto industry standard for service discovery for a few years already. So standard that it doesn’t even have a Wikipedia page? Nor is there even one for this specific type of datacenter-oriented service discovery? This is all pretty niche tech, used by a handful of startups that engineer technology for billions of users, but not used anywhere else. I’ve never seen any companies on national scale operate with this – nor would they need to.
      - cube2222 10y ago
        The thing is, I know a lot of companies using it at nationwide scale (non-startups). I think there's a reason you are getting downvoted. Anyways, Service Discovery like this is less niche than containers are. If containers are still niche for you, then there you go.
        kuschku 10y ago
        > I think there's a reason you are getting downvoted. Well, so are you, your comments are even so grey I can barely read them anymore (and I can’t downvote you). Regarding the rest: Containers are far less niche than Consul is. For service discovery, the market isn’t nearly as monopolized as containers are. And containers are far more used than consul is – most places I’ve seen build their own service discovery systems, or hardcode a main node and then have services in containers register with that, etc. While for containers, at the moment, there’s almost only docker, due to the ecosystem.
    - maloney 10y ago
      etcd is also used quite a bit for service discovery, but it's written in Go too :)
      - cube2222 10y ago
        Are you using it personally? I never met anyone using it for that. How's your experience with it been? (in case you did use it)
  - koffiezet 10y ago
    In the ops world, these tools are pretty much the standard. If you look at the state of many of the tools targeted at infrastructure were, with in the best scenario a decently maintained Python code-base - it's no surprise the ops world jumped on Go. Go's main attraction there is that it's a simple, script-like language that compiles to easily deployed static binaries. That enables people who's main job is not necessarily coding, but with a good scripting background to solve problems by writing more complex/performance-critical stuff which was previously not feasible in languages like Python, Ruby, ... These projects requiring more performance would have required a seasoned dedicated C/C++ coder before, but now they can be started and maintainer by people initially not focusing on programming, but on solving problems they are facing in their daily job. They don't have the time to invest learning a language with complex typesystems where initially, you'll be fighting the language itself more than solving actual problems. For programmers, this investment is useful, it's their main skill and is worth something. Ops people have other things to worry about, and just want to solve problems. It does mean wheels are being reinvented, but the major projects now have pretty smart people working on them, hitting the limits of the language and it's garbage collector, pushing them forward.
    - ex_amazon_sde 10y ago
      > In the ops world, these tools are pretty much the standard Thankfully, this is true only in the HN bubble. One example: Amazon.
      - cube2222 10y ago
        Amazon is a big enough company that it's obvious that they'll roll their own stuff usually.