5 ms·

I believe you usually want as much control of memory management as possible as it can be a great place for finding performance improvements and also help in del

by maxmcd 4y ago

I believe you usually want as much control of memory management as possible as it can be a great place for finding performance improvements and also help in delivering very consistent performance. Automatic memory management is a great aid in reduction of the complexity you have to deal with, but you presumably want to deal with that completely in the context of a database. Seems like this: https://www.cockroachlabs.com/blog/why-go-was-the-right-choice-for-cockroachdb/ https://www.cockroachlabs.com/blog/why-go-was-the-right-choi... and more specifically this: https://www.cockroachlabs.com/community/tech-talks/challenges-writing-massive-complex-go-application/ https://www.cockroachlabs.com/community/tech-talks/challenge... would be good to explore for how cockroach has made it work w/ Go+GC.

naasking 4y ago
> I believe you usually want as much control of memory management as possible as it can be a great place for finding performance improvements and also help in delivering very consistent performance. Sure, but premature optimization is the root of all evil right? Get something functional, then make it performant by handling the problematic cases. GC doesn't prevent this progression, it's just a significant help to getting something functional. GC can sometimes lead you into designs that can't be optimized without a change in abstraction, but it's not at all obvious that you would have landed on the right abstraction if you didn't have the GC to begin with anyway. I think getting something working as fast as possible lets you gather the data you need to make a performant design, and having to handle all of that complexity up front without a GC just delays the data gathering stage.
- flagsrule 4y ago
  And if the GC ever starts slowing you down, just run a profiler and eliminate the allocations. It's usually as simple as replacing a dependency or using sync.Pool in the hot path.
- fsociety 4y ago
  If you end up being wrong you will either have to fight the language or completely rewrite it in another. It is not premature optimization but an intentional design choice, and a valid one to make.
  - naasking 4y ago
    > If you end up being wrong you will either have to fight the language or completely rewrite it in another. This is mostly a myth.
- mattpallissard 4y ago
  > premature optimization is the root of all evil right? Making sure they're aren't a bunch of alloc's or GC spent in the hot path of a high performance library is hardly a premature optimization. If you're tackling problems in DB land changes are you already have a fairly clear picture of what needs to happen and have a list of shortcomings you're trying to avoid. Memory layout, buffers, wals, etc are all things that should be accounted for upfront.
  - naasking 4y ago
    > Making sure they're aren't a bunch of alloc's or GC spent in the hot path of a high performance library is hardly a premature optimization. You often don't know the what the hot paths are.
czx4f4bd 4y ago
You can allocate memory manually in Go and manage it yourself using Go's `unsafe` package, so there's an escape route once Go's GC became a limiting factor. Here's a blog post about it in the context of Dgraph: https://dgraph.io/blog/post/manual-memory-management-golang-jemalloc/ https://dgraph.io/blog/post/manual-memory-management-golang-...
- notwokeno 4y ago
  The real issue with go is (or was, I haven't messed with it for a few years) its un-optimized goroutine scheduler that makes channels slow if you start using them a lot.
  - flagsrule 4y ago
    Yes, refactoring from channels to atomic package can easily give 10x speedup. Sometimes we have to communicate by sharing memory :)

by maxmcd 4y ago

naasking 4y ago

> I believe you usually want as much control of memory management as possible as it can be a great place for finding performance improvements and also help in delivering very consistent performance. Sure, but premature optimization is the root of all evil right? Get something functional, then make it performant by handling the problematic cases. GC doesn't prevent this progression, it's just a significant help to getting something functional. GC can sometimes lead you into designs that can't be optimized without a change in abstraction, but it's not at all obvious that you would have landed on the right abstraction if you didn't have the GC to begin with anyway. I think getting something working as fast as possible lets you gather the data you need to make a performant design, and having to handle all of that complexity up front without a GC just delays the data gathering stage.

flagsrule 4y ago
And if the GC ever starts slowing you down, just run a profiler and eliminate the allocations. It's usually as simple as replacing a dependency or using sync.Pool in the hot path.
fsociety 4y ago
If you end up being wrong you will either have to fight the language or completely rewrite it in another. It is not premature optimization but an intentional design choice, and a valid one to make.
- naasking 4y ago
  > If you end up being wrong you will either have to fight the language or completely rewrite it in another. This is mostly a myth.
mattpallissard 4y ago
> premature optimization is the root of all evil right? Making sure they're aren't a bunch of alloc's or GC spent in the hot path of a high performance library is hardly a premature optimization. If you're tackling problems in DB land changes are you already have a fairly clear picture of what needs to happen and have a list of shortcomings you're trying to avoid. Memory layout, buffers, wals, etc are all things that should be accounted for upfront.
- naasking 4y ago
  > Making sure they're aren't a bunch of alloc's or GC spent in the hot path of a high performance library is hardly a premature optimization. You often don't know the what the hot paths are.

czx4f4bd 4y ago

You can allocate memory manually in Go and manage it yourself using Go's `unsafe` package, so there's an escape route once Go's GC became a limiting factor. Here's a blog post about it in the context of Dgraph: https://dgraph.io/blog/post/manual-memory-management-golang-jemalloc/ https://dgraph.io/blog/post/manual-memory-management-golang-...

notwokeno 4y ago
The real issue with go is (or was, I haven't messed with it for a few years) its un-optimized goroutine scheduler that makes channels slow if you start using them a lot.
- flagsrule 4y ago
  Yes, refactoring from channels to atomic package can easily give 10x speedup. Sometimes we have to communicate by sharing memory :)