11 ms·

All of the server backends at my company are written in Go. This was a result of me writing a couple servers in Python a few years back, ending up with lots of

• devsquid 10y ago
  You would like to write an entire server without using a GC?
  • asolove 10y ago
    Presumably he is complaining about the GC implementation's characteristics, not its existence. You can read a description of the team's design decisions here: https://blog.golang.org/go15gc https://blog.golang.org/go15gc but a simple summary would be: memory is cheap and getting cheaper, so focus on making GC fast rather than small memory footprints.
    • merb 10y ago
      actually thats what the Java G1GC (which will be the default in Java 9) does aswell. However I would still favor java instead of go
    • stincity 10y ago
      Hasn't go been improving the GC with it's rounds of updates to the language?
      • kasey_junk 10y ago
        Yes, with an emphasis on speed not memory.
    • nameless912 10y ago
      And in my opinion, this is a great tradeoff. Ultimately, adding more RAM to a server (or clicking a button on the AWS console) is a very easy fix. However, reducing latency is almost never easy. If they can prevent GC stuttering, I'll take the higher memory overhead every time.
      • tracker1 10y ago
        GC has gotten significantly better the last few releases, I don't think most applications will notice.
  • haberman 10y ago
    People do it all the time. For example, in C++ with modern owning pointers like std::unique_ptr<> and std::shared_ptr<> it's really not that bad.
  • dboreham 10y ago
    I would not like to write a server with GC. Source: I've written many servers in both C and Java and given the choice will never use a GC language again for a server. Holding out some hope for Rust..
    • dikaiosune 10y ago
      There are definitely some pain points building a server in Rust, but I think on the whole the tradeoff is worth it, and will become moreso as the ecosystem matures.
    • netheril96 10y ago
      Care to elaborate?
• tomcam 10y ago
  Invaluable perspective from the trenches--thanks.
• inglor 10y ago
  To be fair, Go's GC got A LOT better over the past few versions and it's getting much better still. What I would really like to be able to do though is be able to write unmanaged blocks when I need them and know better than the compiler rather than having to "run my own heap" on top of a buffer like I typically have to do in managed languages that don't have that opt out.
  • yxhuvud 10y ago
    Crystal might be worth checking out if you have that kind of need.
  • weberc2 10y ago
    Have you considered pausing the GC for those blocks? Or am I misunderstanding your use case?
    • infogulch 10y ago
      How do you do this? I didn't know there was a way to temporarily disable the GC for specific blocks.
      • weberc2 10y ago
        You can toggle the GC with https://golang.org/pkg/runtime/debug/#SetGCPercent https://golang.org/pkg/runtime/debug/#SetGCPercent. Turn it off before you enter your block and enable it again at the end of your block.
        • amelius 10y ago
          But can you guarantee that the GC actually makes progress that way?
          • weberc2 10y ago
            What does it mean for a GC to "make progress"? My understanding was that the OP wanted to stop the GC for a particular code block, so it would seem that any "progress making" would be undesirable.
• spriggan3 10y ago
  > We continue to use Go because of its strengths, but it just really surprises me how little Google seems to care about the language and ecosystem. Go is certainly a language that is used at Google, but AFAIK a lot of "Googlers" don't really like it and don't use it. It certainly not the "official language at Google", given the weight of C++ and Java there. But that's the consequence of being opinionated. Using Go means having Rob Pike over your shoulder telling you how to write code. And he made sure you can't escape that fact since there is no place for "ninja coding" with Go.
  • mseepgood 10y ago
    What is "ninja coding"? Sounds stupid.
    • nameless912 10y ago
      I think it refers to the Perl-era idea that good code should be somehow "clever" rather than maintainable. It's how bad programmers who spend hours agonizing over how to reduce their line count (presumably to save disk space?) justify their behavior.
      • wyager 10y ago
        For optimal code maintainability, there is a happy compromise between excessively terse and excessively verbose code. There is a lot of code in the wild that's too spread out.
        • nameless912 10y ago
          Well sure, but erring on the side of verbosity at least ensures that someone can follow your thinking as long as the code is well structured. I would prefer to read 25 lines of decent Python over 3 lines of regexes in Perl.
          • Drdrdrq 10y ago
            Sure, but at least I would prefer 25 lines of decent Python over 125 lines of overly verbose Java. Nothing is black and white, it is easy to make code too verbose and thus very taxing for the reader.
• erikpukinskis 10y ago
  > They autogenerate all API libraries for golang, which means they're not idiomatic, are convoluted to use, and the documentation is a jungle. Are idiomatic API libraries really a good thing? If an API is used heavily throughout your application, doesn't that mean it's time to wrap that part of the API in some idiomatic wrapper code that is meaningful in your domain? Functions are idiomatic in all languages. Personally, I'd rather APIs mostly stick to simple functions, and let the application developer build idioms around them if they like. And if the API call involves a web request, I am happy to just make the HTTP call myself. Half the time you need to understand the interchange format to debug your code anyway. My experience is that wrappers rarely protect you from that. I guess if your application is 90% calls to an API, and you use a huge range of different functions in that API then you'd want a good, complete library. But how common is that?
  • stcredzero 10y ago
    Functions are idiomatic in all languages. Personally, I'd rather APIs mostly stick to simple functions, and let the application developer build idioms around them if they like. Idiomatic APIs can be a real PITA, especially when you call them from a different language. KISS should reign supreme. (With bad 80's special effects in a made for TV movie filmed at a carnival.) If you leave it up to the client developer to build the idiomatic facade/interface, then the API is simpler, the code is more properly idiomatic in the end, and everyone is happier.
  • akavlie 10y ago
    You're arguing that it's good to (intentionally) break with common code idioms for certain classes of libraries?? Also, have you actually seen the auto-generated APIs he's referring to? Here are the Godocs for the BigQuery API: https://godoc.org/google.golang.org/api/bigquery/v2 https://godoc.org/google.golang.org/api/bigquery/v2 Here's some example code demonstrating the API's bad habits -- such as deeply nested struct pointers, and Do() methods everywhere: https://github.com/google/google-api-go-client/blob/master/examples/bigquery.go#L205 https://github.com/google/google-api-go-client/blob/master/e...
    • 10y ago
• courtf 10y ago
  We run a cluster of P2P, GPU heavy machines that use Go to ingest byte streams of raw radar data, store that info in btrees, and render, cache & serve map tiles that are drawn on the fly in response to http requests. We are not using much outside the stdlib (opengl and gdal bindings). Garbage collection has become very fast in recent versions of the language. It's quite painless now, and we did struggle with it in the past. I'd take the current GC over dealing with other strategies unless you are a very specific, >60fps kind of situation. We literally don't think about the GC anymore. Random stats from one of our busier nodes, over the past 20 min: 40GB currently in use, 391GB allocated (and freed) in the last 20min average of 2.6ms GC pauses every 70.6s 95th percentile on GC time is 7.26ms For dependency management, there's a lot of tools that make things more automated and I think probably offer people what they are expecting/used to from other environments. This has been a sticky point for sure. I've settled on a bare bones solution that seems to work well: just using git submodules in the vendor directory within the project repo. Does everything I need (pin to version of a fork, build immediately after a clone and a submodule update --init). As of Go1.5, the vendor directory is supported by the compiler, and it will look there first to resolve dependencies. While this does work with recursive dependencies, you will want to flatten the dependency graph as much as possible if types are being used across multiple deps because types are distinguished by their full import paths. This encourages breaking things into modules so I'm ok with it. First-tier support is a problem I have no doubt, but not one we've suffered from personally. I'd say that debugging comes up a lot too, and Go doesn't have an official, reliable option for debugging. The Delve project is getting close to that though it looks like. I personally debug with the stdlib pprof package (provides an http server that prints stack traces for all goroutines currently active, allows cpu/memory/blocking profiles to be requested), print statements, general testing/experimentation and newrelic.
  • dalailambda 10y ago
    How do you find the cgo overhead? Specifically with something like OpenGL, it seems like it would be a big pain point for rendering.
    • courtf 10y ago
• mattetti 10y ago
  We've been running splice.com on Go for 3 years now and handle 5TB of audio/binary data per day. Our memory usage is around 10-15MB per server and the GC pause time has been really low. You do need to stream your IOs instead of reading everything in memory. In regards to dependency management, we honestly had no issues and now with vendoring, it's even easier. We do use a main repo with lots of smaller packages and only a few 3rd party dependencies that are vendored or available via private git repos. We don't use Google cloud but I heard they have 2 repos, one that has auto generated code and one that has hand written code (but less complete).
  • cheez 10y ago
    Your tagline should be: splice.com - no more sample packs. Haha. Nice business model.
  • kkirsche 10y ago
    Could you elaborate on what you mean by stream your IO? Is this just using buffered io it is there a concept I could read up on here?
    • mattetti 10y ago
      yes buffered io, you use readers to read/write small chunks at a time instead of loading everything at once. Go offers way to do both since in some cases, such as loading an entire file in memory can be fine/better/faster.
  • p0nce 10y ago
    OT about splice.com: how do you version control DAW projects, you have a custom parser + text diff? Would you support Orion and Reaper? :)
• chrisbroadfoot 10y ago
  > Google itself doesn't actually have any real API libraries for their cloud services. We're (I'm one of the contributors) working on them. Check them out here: https://github.com/GoogleCloudPlatform/gcloud-golang https://github.com/GoogleCloudPlatform/gcloud-golang Datastore docs: http://godoc.org/google.golang.org/cloud/datastore http://godoc.org/google.golang.org/cloud/datastore
• pvelagal 10y ago
  Slide 13 (https://talks.golang.org/2016/applicative.slide#13 https://talks.golang.org/2016/applicative.slide#13) is interesting. I was expecting Go to be very close to C/C++ on the X axis (fast/efficient) as it doesn't use VM, but it is more close to Java ?
  • eudoxus 10y ago
    It's true that Go compiles down to an executable and doesn't run on a VM like Java, but it does have it own runtime compiled into it, which manages the GC and goroutine scheduler to name two, so there is definitely some more overhead to Go then just straight C/C++
    • pcwalton 10y ago
      In addition, the language isn't designed for zero-cost abstractions like C++ is. You see this in the design of things like defer, which requires allocating records on the heap in certain cases, as compared to exceptions in C++ which can be implemented in a zero-cost manner.
      • weberc2 10y ago
        I think interfaces are a better example, personally.
  • _ph_ 10y ago
    Ideally, Go should be pretty close to C, but currently there are reasons for it lagging behind (though there are also Go programs which perform better than their C counterparts): - The Go compiler certainly isn't as sophisticated as the better C compilers. It looks that the code produces by Go 1.7 is quite improves vs. 1.6 with the SSA compiler and I would expect further gains in future releases. - The Go compiler does not offer optimization levels and overall is optimized for compilation speed to, so these tradeoffs could be limiting. - While the Go GC is very good and improving, so that the pauses are very low, it does use CPU, also while running in parallel to the program. How much exactly very much depends on the allocation behavior of the program, but should be accounted for.
• nvarsj 10y ago
  > Dependency management is a nightmare Is it that bad now in 1.6 with vendor support? And a tool like `govendor` makes it easy to stick things inside of vendor. > it just really surprises me how little Google seems to care about the language and ecosystem To be blunt, google's priority is google, not the open source community or other companies using golang. Dependency management wasn't a priority because of mono repo. Having said that, they are pretty good about improving things for everyone, but it will never be like a company such as Typesafe whose product is the language and tooling itself.
  • fideloper 10y ago
    is godep relevant still, or is 1.6 making something like govendor a "better" way to go?
    • rosadoft 10y ago
      I switched to Glide, a vendor-focused package manager, on 1.5. I think the community has already decided in favor of this approach. Post 1.5, Godeps' workflow just feels weird and unnecessary.
    • themartorana 10y ago
      Use godep everywhere. It supports the vendor folder, and I find it's still a great solution (and saves me using submodules).
    • sdboyer 10y ago
      it is, and works well enough for individuals, but doesn't make for the friendliest ecosystem.
• ngrilly 10y ago
  > Google itself doesn't actually have any real API libraries for their cloud services. They autogenerate all API libraries for golang, which means they're not idiomatic, are convoluted to use, and the documentation is a jungle. Are you sure? Google offers 2 Google Cloud APIs for Go: - Google API Client Library for Go, which is auto-generated like you wrote; - Google Cloud Platform Client Library for Go, which is intended to be idiomatically-designed. See https://cloud.google.com/go/apis https://cloud.google.com/go/apis.
• IndianAstronaut 10y ago
  Another major issue I have had with Go is database connectivity. The db drivers are really lackijg for Go. Makes it tough for those of us who use Teradata or Hive.
  • geodel 10y ago
    Well drivers are mostly written by database vendors or respective community in case of open source db. There is not much Go can do there.
• pmarreck 10y ago
  Have a look at Elixir and Phoenix. GC is per-process so no global slowdown. Extremely fast response times and uptimes. Many other features that mesh nicely with webserving (some courtesy of Erlang's VM). And a completely opposite philosophy from Go when it comes to error handling. Erlang/Elixir embraces failure (and immediately logs and restarts the process); Go seems to ignore it (unless you explicitly check, which to me seems... insane. For reasons why failing fast is better than failing silently, read https://blog.codinghorror.com/whats-worse-than-crashing/ https://blog.codinghorror.com/whats-worse-than-crashing/ .)
  • zalmoxes 10y ago
    To any Go programmers ignoring errors is insane. I haven't seen code outside of simple examples ignore errors, and even there it's discouraged.
    • deleted 10y ago
      [deleted]
    • pmarreck 10y ago
      But you still have to check for errors after every line of code where they are possible (if you're covering all the bases), no? Due to lack of a traditional exception model? Elixir/Erlang at least have pattern-matching which makes checks like that (from functions that return a non-OK value on errors) basically inlined: `{:ok, val} = call_some_func(with_args)` If it doesn't match on the :ok (i.e., an error occurred), you get a match error which gets logged, and the process typically gets killed and restarted by a supervisor process in a millisecond.
• etatoby 10y ago
  Just a quick note about dependency management: we use "go get" to download / update dependencies, integrate / test, and then just include them all in the project's git repo. Never had an issue.
• amelius 10y ago
  > I tried a couple different server libraries on Python but they all seemed to struggle with even tiny loads. Not sure what was up with that Could the problem have been the "global interpreter lock"? The GIL makes it impossible for multiple threads to simultaneously perform certain basic operations, effectively making the system single-threaded.
• trustyhank 10y ago
  Im surprised to hear your issues with Python, even Django can handle decent load depending what youre asking of it.
• sdboyer 10y ago
  dep mgmt, at least, is poised to improve a lot in the coming months. we've got growing consensus around some metadata files, and i'm nearly done with my SAT solver (github.com/sdboyer/vsolver), which will be in glide, and maybe others, soon.