3 ms·

Honest question: isn't Go a bit of a weird choice for a database given the GC? Why not C/C++/Rust?

by dj_mc_merlin 4y ago

kangda123 4y ago
Ease of use
naasking 4y ago
Why can't a database be written in a garbage collected language?
- maxmcd 4y ago
  I believe you usually want as much control of memory management as possible as it can be a great place for finding performance improvements and also help in delivering very consistent performance. Automatic memory management is a great aid in reduction of the complexity you have to deal with, but you presumably want to deal with that completely in the context of a database. Seems like this: https://www.cockroachlabs.com/blog/why-go-was-the-right-choice-for-cockroachdb/ https://www.cockroachlabs.com/blog/why-go-was-the-right-choi... and more specifically this: https://www.cockroachlabs.com/community/tech-talks/challenges-writing-massive-complex-go-application/ https://www.cockroachlabs.com/community/tech-talks/challenge... would be good to explore for how cockroach has made it work w/ Go+GC.
  - naasking 4y ago
    > I believe you usually want as much control of memory management as possible as it can be a great place for finding performance improvements and also help in delivering very consistent performance. Sure, but premature optimization is the root of all evil right? Get something functional, then make it performant by handling the problematic cases. GC doesn't prevent this progression, it's just a significant help to getting something functional. GC can sometimes lead you into designs that can't be optimized without a change in abstraction, but it's not at all obvious that you would have landed on the right abstraction if you didn't have the GC to begin with anyway. I think getting something working as fast as possible lets you gather the data you need to make a performant design, and having to handle all of that complexity up front without a GC just delays the data gathering stage.
    - flagsrule 4y ago
      And if the GC ever starts slowing you down, just run a profiler and eliminate the allocations. It's usually as simple as replacing a dependency or using sync.Pool in the hot path.
    - fsociety 4y ago
      If you end up being wrong you will either have to fight the language or completely rewrite it in another. It is not premature optimization but an intentional design choice, and a valid one to make.
      - naasking 4y ago
        > If you end up being wrong you will either have to fight the language or completely rewrite it in another. This is mostly a myth.
    - mattpallissard 4y ago
      > premature optimization is the root of all evil right? Making sure they're aren't a bunch of alloc's or GC spent in the hot path of a high performance library is hardly a premature optimization. If you're tackling problems in DB land changes are you already have a fairly clear picture of what needs to happen and have a list of shortcomings you're trying to avoid. Memory layout, buffers, wals, etc are all things that should be accounted for upfront.
      - naasking 4y ago
        > Making sure they're aren't a bunch of alloc's or GC spent in the hot path of a high performance library is hardly a premature optimization. You often don't know the what the hot paths are.
  - czx4f4bd 4y ago
    You can allocate memory manually in Go and manage it yourself using Go's `unsafe` package, so there's an escape route once Go's GC became a limiting factor. Here's a blog post about it in the context of Dgraph: https://dgraph.io/blog/post/manual-memory-management-golang-jemalloc/ https://dgraph.io/blog/post/manual-memory-management-golang-...
    - notwokeno 4y ago
      The real issue with go is (or was, I haven't messed with it for a few years) its un-optimized goroutine scheduler that makes channels slow if you start using them a lot.
      - flagsrule 4y ago
        Yes, refactoring from channels to atomic package can easily give 10x speedup. Sometimes we have to communicate by sharing memory :)
eatonphil 4y ago
If you like/trust dbdb.io, most databases are written in C++ or Java. C# and Go are also high on the list. https://dbdb.io/stats https://dbdb.io/stats
Ingaz 4y ago
C/C++/Rust - will not solve anything magically. Anyway you will need to implement some kind of memory management for DB. Go GC creates problems and they are non-trivial (as all memory management) but it's not so twisted as in JVM. Examples: - DGraph: migrated from RocksDB (C++) to Badger (pure go, developed themselves, based on jemalloc) - CocroachDB - pure go for storage engine - go 1.20 proposed feature: package `arena` for region-based memory management

Honest question: isn't Go a bit of a weird choice for a database given the GC? Why not C/C++/Rust?

kangda123 4y ago

Ease of use

naasking 4y ago

Why can't a database be written in a garbage collected language?

maxmcd 4y ago
I believe you usually want as much control of memory management as possible as it can be a great place for finding performance improvements and also help in delivering very consistent performance. Automatic memory management is a great aid in reduction of the complexity you have to deal with, but you presumably want to deal with that completely in the context of a database. Seems like this: https://www.cockroachlabs.com/blog/why-go-was-the-right-choice-for-cockroachdb/ https://www.cockroachlabs.com/blog/why-go-was-the-right-choi... and more specifically this: https://www.cockroachlabs.com/community/tech-talks/challenges-writing-massive-complex-go-application/ https://www.cockroachlabs.com/community/tech-talks/challenge... would be good to explore for how cockroach has made it work w/ Go+GC.
- naasking 4y ago
  > I believe you usually want as much control of memory management as possible as it can be a great place for finding performance improvements and also help in delivering very consistent performance. Sure, but premature optimization is the root of all evil right? Get something functional, then make it performant by handling the problematic cases. GC doesn't prevent this progression, it's just a significant help to getting something functional. GC can sometimes lead you into designs that can't be optimized without a change in abstraction, but it's not at all obvious that you would have landed on the right abstraction if you didn't have the GC to begin with anyway. I think getting something working as fast as possible lets you gather the data you need to make a performant design, and having to handle all of that complexity up front without a GC just delays the data gathering stage.
  - flagsrule 4y ago
    And if the GC ever starts slowing you down, just run a profiler and eliminate the allocations. It's usually as simple as replacing a dependency or using sync.Pool in the hot path.
  - fsociety 4y ago
    If you end up being wrong you will either have to fight the language or completely rewrite it in another. It is not premature optimization but an intentional design choice, and a valid one to make.
    - naasking 4y ago
      > If you end up being wrong you will either have to fight the language or completely rewrite it in another. This is mostly a myth.
  - mattpallissard 4y ago
    > premature optimization is the root of all evil right? Making sure they're aren't a bunch of alloc's or GC spent in the hot path of a high performance library is hardly a premature optimization. If you're tackling problems in DB land changes are you already have a fairly clear picture of what needs to happen and have a list of shortcomings you're trying to avoid. Memory layout, buffers, wals, etc are all things that should be accounted for upfront.
    - naasking 4y ago
      > Making sure they're aren't a bunch of alloc's or GC spent in the hot path of a high performance library is hardly a premature optimization. You often don't know the what the hot paths are.
- czx4f4bd 4y ago
  You can allocate memory manually in Go and manage it yourself using Go's `unsafe` package, so there's an escape route once Go's GC became a limiting factor. Here's a blog post about it in the context of Dgraph: https://dgraph.io/blog/post/manual-memory-management-golang-jemalloc/ https://dgraph.io/blog/post/manual-memory-management-golang-...
  - notwokeno 4y ago
    The real issue with go is (or was, I haven't messed with it for a few years) its un-optimized goroutine scheduler that makes channels slow if you start using them a lot.
    - flagsrule 4y ago
      Yes, refactoring from channels to atomic package can easily give 10x speedup. Sometimes we have to communicate by sharing memory :)

eatonphil 4y ago

If you like/trust dbdb.io, most databases are written in C++ or Java. C# and Go are also high on the list. https://dbdb.io/stats https://dbdb.io/stats

Ingaz 4y ago

C/C++/Rust - will not solve anything magically. Anyway you will need to implement some kind of memory management for DB. Go GC creates problems and they are non-trivial (as all memory management) but it's not so twisted as in JVM. Examples: - DGraph: migrated from RocksDB (C++) to Badger (pure go, developed themselves, based on jemalloc) - CocroachDB - pure go for storage engine - go 1.20 proposed feature: package `arena` for region-based memory management