5 ms·

CockroachDB is a terrible technology that causes almost guaranteed data corruption due to its lack of ACID guarantees and is written in a language with a GC whi

CockroachDB is a terrible technology that causes almost guaranteed data corruption due to its lack of ACID guarantees and is written in a language with a GC which contirbutes to GC pauses. The dev team refuses to listen to feedback to port their code to C ;(.

Guvante 5y ago
Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument when it comes to any production system that it isn't even worth discussing. The reality is there are limited resources to work on any given project and "rewrite" is generally not the correct way to fix a given problem. Especially since the first thing you are going to need to do is show that a network system isn't resilient to GC pauses or that GC pauses are frequent enough to impact throughput (very very few applications are actually materially impacted by sub second spikes in latency).
- pkolaczk 5y ago
  GC pauses are not the only GC issue. Thrashing the caches and blowing up memory usage by an order of magnitude can be also very bad for performance. In a database system memory is very precious - the more of it you can use for caching / buffering users data, the better the performance. As for the subsecond spikes in latency, these tend to multiply in a distributed system. If serving a client request takes N internal requests, the likelihood of hitting a GC pause somewhere is much larger than if you did only one local request. Not sure about Go, but none of the "free" Java's GC guarantees low pauses. There is STW fallback even in the most advanced ones like ZGC. So you never know when it stops for more than 1s.
  - ngrilly 5y ago
    Go GC pauses are bounded at 0.5 ms: https://blog.golang.org/ismmkeynote https://blog.golang.org/ismmkeynote
    - pkolaczk 5y ago
      Is it a hard guarantee or just a soft goal like in G1, which tries to stay within the target pause milliseconds, but there are many ways it can fail to? Can it handle 100+ GB large heaps? Even if it is a hard guarantee, then, from the link you posted, it is not even generational, so it will scan the whole heap quite frequently, and that is going to influence average performance quite visibly - you definitely dont want a database system to access all its cached memory once in a while. Database systems are really all about memory and I/O management. You shouldn't outsource those core features to a universal algorithm, unless you wish to forgo any competitive advantage (at least in performance department). So this pushes the devs into the off-heap manual memory management territory, where dragons live (at least in Java, again - maybe Go is better in that regard). I've been there, and I don't recommend.
- redis_mlc 5y ago
  > Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument False. All popular databases are written in C, and continue because of the GC issue. I would not use a general-purpose database written in Java because of GC, for example. We'll see how well Go works in practise. > sub second spikes in latency Go is supposed to be sub-second GC pause latency, but understand that most SQL queries are sub-millisecond, so GC latency is still a significant issue compared to query times. Go might be acceptable now for niche databases like column-store for certain use cases, though. Also, see the excellent comment above about distributed systems and server cache issues. You can't do application performance analysis with GC literally everywhere. The puerile knee-jerk hatred for C on HN has to stop - almost every software you use is written in C, from scripting languages to operating systems to web servers to databases. Source: DBA who's worked with current databases, as well as a custom database written in Java with significant (ie. brutal) GC problems that required a total refactor (rewrite) to "work" at all.
  - mappu 5y ago
    I'd like to reply specifically to the Go is supposed to be sub-second GC pause latency part - Go's GC is crazy fast, it pauses for only microseconds, comparable to the socket transport overhead. It can't be thought of as Java-like at all.
    - ngrilly 5y ago
      Yes. https://blog.golang.org/ismmkeynote https://blog.golang.org/ismmkeynote
  - ngrilly 5y ago
    The fact that database engines written in Java exhibit high latency is not a reason to dismiss all other programming languages except C and C++. It still makes sense to consider modern languages like Rust, Go, Zig, etc.
  - lmm 5y ago
    I've run production systems on H2 that ran rings around the same company's dedicated-DBA systems.
andreimatei1 5y ago
> its lack of ACID guarantees Our transactions implementation is our crown jewel. You might want to check your sources.
hugofirth 5y ago
This is daft. CockroachDB exhibits extremely strong isolation and consistency levels. I believe it is strictly serializable under most circumstances? Also - anyone who says that x database must be rewritten “because GC” is just making an incredibly un nuanced argument about a nuanced problem. People have built production ready databases in both Java and Go. If you care about low/predictable tail latencies then you have a bunch of other more important problems to solve before you worry about the behaviour of a modern garbage collector. For example: how good is your cache hit ratio? How are your synchronous replication protocols affected by grey failures? That kind of thing.
- _benedict 5y ago
  I believe it only guarantees serializable isolation. You may get strict serializable, but it doesn’t appear to be the case that you will know for sure if any transactions were not linearizable.

Guvante 5y ago

Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument when it comes to any production system that it isn't even worth discussing. The reality is there are limited resources to work on any given project and "rewrite" is generally not the correct way to fix a given problem. Especially since the first thing you are going to need to do is show that a network system isn't resilient to GC pauses or that GC pauses are frequent enough to impact throughput (very very few applications are actually materially impacted by sub second spikes in latency).

pkolaczk 5y ago
GC pauses are not the only GC issue. Thrashing the caches and blowing up memory usage by an order of magnitude can be also very bad for performance. In a database system memory is very precious - the more of it you can use for caching / buffering users data, the better the performance. As for the subsecond spikes in latency, these tend to multiply in a distributed system. If serving a client request takes N internal requests, the likelihood of hitting a GC pause somewhere is much larger than if you did only one local request. Not sure about Go, but none of the "free" Java's GC guarantees low pauses. There is STW fallback even in the most advanced ones like ZGC. So you never know when it stops for more than 1s.
- ngrilly 5y ago
  Go GC pauses are bounded at 0.5 ms: https://blog.golang.org/ismmkeynote https://blog.golang.org/ismmkeynote
  - pkolaczk 5y ago
    Is it a hard guarantee or just a soft goal like in G1, which tries to stay within the target pause milliseconds, but there are many ways it can fail to? Can it handle 100+ GB large heaps? Even if it is a hard guarantee, then, from the link you posted, it is not even generational, so it will scan the whole heap quite frequently, and that is going to influence average performance quite visibly - you definitely dont want a database system to access all its cached memory once in a while. Database systems are really all about memory and I/O management. You shouldn't outsource those core features to a universal algorithm, unless you wish to forgo any competitive advantage (at least in performance department). So this pushes the devs into the off-heap manual memory management territory, where dragons live (at least in Java, again - maybe Go is better in that regard). I've been there, and I don't recommend.
redis_mlc 5y ago
> Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument False. All popular databases are written in C, and continue because of the GC issue. I would not use a general-purpose database written in Java because of GC, for example. We'll see how well Go works in practise. > sub second spikes in latency Go is supposed to be sub-second GC pause latency, but understand that most SQL queries are sub-millisecond, so GC latency is still a significant issue compared to query times. Go might be acceptable now for niche databases like column-store for certain use cases, though. Also, see the excellent comment above about distributed systems and server cache issues. You can't do application performance analysis with GC literally everywhere. The puerile knee-jerk hatred for C on HN has to stop - almost every software you use is written in C, from scripting languages to operating systems to web servers to databases. Source: DBA who's worked with current databases, as well as a custom database written in Java with significant (ie. brutal) GC problems that required a total refactor (rewrite) to "work" at all.
- mappu 5y ago
  I'd like to reply specifically to the Go is supposed to be sub-second GC pause latency part - Go's GC is crazy fast, it pauses for only microseconds, comparable to the socket transport overhead. It can't be thought of as Java-like at all.
  - ngrilly 5y ago
    Yes. https://blog.golang.org/ismmkeynote https://blog.golang.org/ismmkeynote
- ngrilly 5y ago
  The fact that database engines written in Java exhibit high latency is not a reason to dismiss all other programming languages except C and C++. It still makes sense to consider modern languages like Rust, Go, Zig, etc.
- lmm 5y ago
  I've run production systems on H2 that ran rings around the same company's dedicated-DBA systems.

andreimatei1 5y ago

> its lack of ACID guarantees Our transactions implementation is our crown jewel. You might want to check your sources.

hugofirth 5y ago

This is daft. CockroachDB exhibits extremely strong isolation and consistency levels. I believe it is strictly serializable under most circumstances? Also - anyone who says that x database must be rewritten “because GC” is just making an incredibly un nuanced argument about a nuanced problem. People have built production ready databases in both Java and Go. If you care about low/predictable tail latencies then you have a bunch of other more important problems to solve before you worry about the behaviour of a modern garbage collector. For example: how good is your cache hit ratio? How are your synchronous replication protocols affected by grey failures? That kind of thing.

_benedict 5y ago
I believe it only guarantees serializable isolation. You may get strict serializable, but it doesn’t appear to be the case that you will know for sure if any transactions were not linearizable.