4 ms·

A database is a server. Your malloc isn't deterministic, either. According to the Go Wikipedia article, as of 2017: > Garbage collection pauses should be sig

by ken 7y ago

A database is a server. Your malloc isn't deterministic, either. According to the Go Wikipedia article, as of 2017: > Garbage collection pauses should be significantly shorter than they were in Go 1.7, usually under 100 microseconds and often as low as 10 microseconds. According to [1], "A trivial SELECT can take in the order of 0.1ms to execute server-side", i.e., 100 microseconds. Any I/O will of course significantly increase that. This sounds like an entirely reasonable price to pay. Writing high-level functionality in C in 2019 would not give me much faith in their product. [1]: https://www.2ndquadrant.com/en/blog/postgresql-latency-pipelining-batching/ https://www.2ndquadrant.com/en/blog/postgresql-latency-pipel...

aloknnikhil 7y ago
For me, the issue with GC never has been about how long it runs for. It's been about the unpredictability of when it can kick in. Granted a malloc can be unpredictable too (albeit very very rarely), but at least you know a malloc is the only time your process could stall waiting for memory. Databases make extensive use of scratch space for undo logs and redo logs and especially while processing JOINs. It's tremendous pressure on the GC that results in frequent performance dips. With C/C++/Rust, you'd pipeline the query stages to malloc/free that space outside the hot path to minimize query response times.
- irfansharif 7y ago
  I'll repeat what was mentioned elsewhere. The GC pauses in Go are in the order of microseconds. We've done a ton of work keeping our memory usage under tight bounds and keeping a close eye on exactly how the go GC processes interplay with query execution threads (what processors are consumed, GC pressure buildup, where/when the stalls are introduced, etc.) Consistently we've found that given you're invariably dealing with network latencies (it's a distributed db after all), the microsecond GC pauses are almost entirely dwarfed to matter (I'm not excluding p99s here).
- tom_mellior 7y ago
  > With C/C++/Rust, you'd pipeline the query stages to malloc/free that space outside the hot path to minimize query response times. I don't understand why you seem to assert that GC'd languages would force you to put allocations on the hot path. Surely in Go you can also allocate that space elsewhere?
  - aloknnikhil 7y ago
    I was primarily referring to the free on the hot path. GCd languages don't force any allocation paradigms. But due to the unpredictability of the GC, you could be in the middle of a query execution and the GC can kick in. With C/C++/Rust, you're free to maintain your own a allocator that could have a buffer pool of pages that can be recycled and freed when the database is idle. Under a large load, the micro spikes in latency can have an avalanche effect that leads to a build up of queries. I've dealt with customer cases where the MySQL/InnoDB lock manager caused connection spikes due to a brief (< 10 us) global write lock taken while the system was under load.
    - tom_mellior 7y ago
      > But due to the unpredictability of the GC, you could be in the middle of a query execution and the GC can kick in. Yes, that is a possibility. Typically there are two ways to deal with this in a GC'd language: (a) You know that GC is only triggered by certain operations that might allocate, and you avoid those operations in your critical hot section, or (b) you know how to disable the GC temporarily, while you are in that critical hot section. For Go the latter is apparently done by calling this function: https://golang.org/pkg/runtime/debug/#SetGCPercent https://golang.org/pkg/runtime/debug/#SetGCPercent I'm not a Go developer myself, but I imagine people developing such low-latency software in Go know about this.
jandrewrogers 7y ago
Throughput optimization in database engines often relies on operation latency being visible and predictable to the execution scheduler. Disk I/O has this property, a GC does not, so they are not comparable in terms of their impact on throughput. An important class of schedule-based architectural optimizations are rendered ineffective if you are running a GC in the background. This only matters to databases that actually implement these optimizations. If a database does not then a GC has a much smaller impact. But it also means that, all other things being equal, the database will never be competitive with a non-GC design since those optimizations can provide an integer factor improvement in performance. Also, many database engines aren't doing malloc() at runtime. 100 microseconds of CPU time is an integer factor larger than many atomic database operations in a fast database engine. An operation taking an order of magnitude longer than the scheduler would expect based on runtime state has real adverse consequences.
- mumblemumble 7y ago
  So, before we theorize that Cockroach can't possibly be fast enough to meet anyone's needs, perhaps we should look at some performance figures? I haven't ever touched it myself, but, if the benchmarks that Cockroach Labs posts on their blog are to be believed, it's pretty respectable for what it is.
  - jandrewrogers 7y ago
    You are misrepresenting what was written. Most databases don't need to be optimal to be useful, merely adequate, and few attempt to be optimal in any kind of absolute sense. I still use PostgreSQL regularly and it unambiguously falls into this "far from optimal" category. However, as data velocity and volume grow, optimization of absolute performance and hardware efficiency has an increasingly large impact on the cost of operating a database and the kinds of applications you can economically run. Databases that sacrifice major optimizations at an architectural level will have no chance to be competitive with databases that do not over the long term as average data volume and velocity grows. At the scale of database operations many companies are using today, these optimizations literally save them tens of millions of dollars per year on infrastructure. CockroachDB produces what looks like a fine product and has every right to make whatever technical decisions they wish. That the architecture has made substantial performance sacrifices is not theoretical though, and they don't pretend to operate in markets that require any kind of absolute operational efficiency.
paulddraper 7y ago
C may not be an attractive choice in 2019, but C++ and Rust certainly are. Especially if you're working at the level where you want/need vectorization.

by ken 7y ago

aloknnikhil 7y ago

For me, the issue with GC never has been about how long it runs for. It's been about the unpredictability of when it can kick in. Granted a malloc can be unpredictable too (albeit very very rarely), but at least you know a malloc is the only time your process could stall waiting for memory. Databases make extensive use of scratch space for undo logs and redo logs and especially while processing JOINs. It's tremendous pressure on the GC that results in frequent performance dips. With C/C++/Rust, you'd pipeline the query stages to malloc/free that space outside the hot path to minimize query response times.

irfansharif 7y ago
I'll repeat what was mentioned elsewhere. The GC pauses in Go are in the order of microseconds. We've done a ton of work keeping our memory usage under tight bounds and keeping a close eye on exactly how the go GC processes interplay with query execution threads (what processors are consumed, GC pressure buildup, where/when the stalls are introduced, etc.) Consistently we've found that given you're invariably dealing with network latencies (it's a distributed db after all), the microsecond GC pauses are almost entirely dwarfed to matter (I'm not excluding p99s here).
tom_mellior 7y ago
> With C/C++/Rust, you'd pipeline the query stages to malloc/free that space outside the hot path to minimize query response times. I don't understand why you seem to assert that GC'd languages would force you to put allocations on the hot path. Surely in Go you can also allocate that space elsewhere?
- aloknnikhil 7y ago
  I was primarily referring to the free on the hot path. GCd languages don't force any allocation paradigms. But due to the unpredictability of the GC, you could be in the middle of a query execution and the GC can kick in. With C/C++/Rust, you're free to maintain your own a allocator that could have a buffer pool of pages that can be recycled and freed when the database is idle. Under a large load, the micro spikes in latency can have an avalanche effect that leads to a build up of queries. I've dealt with customer cases where the MySQL/InnoDB lock manager caused connection spikes due to a brief (< 10 us) global write lock taken while the system was under load.
  - tom_mellior 7y ago
    > But due to the unpredictability of the GC, you could be in the middle of a query execution and the GC can kick in. Yes, that is a possibility. Typically there are two ways to deal with this in a GC'd language: (a) You know that GC is only triggered by certain operations that might allocate, and you avoid those operations in your critical hot section, or (b) you know how to disable the GC temporarily, while you are in that critical hot section. For Go the latter is apparently done by calling this function: https://golang.org/pkg/runtime/debug/#SetGCPercent https://golang.org/pkg/runtime/debug/#SetGCPercent I'm not a Go developer myself, but I imagine people developing such low-latency software in Go know about this.

jandrewrogers 7y ago

Throughput optimization in database engines often relies on operation latency being visible and predictable to the execution scheduler. Disk I/O has this property, a GC does not, so they are not comparable in terms of their impact on throughput. An important class of schedule-based architectural optimizations are rendered ineffective if you are running a GC in the background. This only matters to databases that actually implement these optimizations. If a database does not then a GC has a much smaller impact. But it also means that, all other things being equal, the database will never be competitive with a non-GC design since those optimizations can provide an integer factor improvement in performance. Also, many database engines aren't doing malloc() at runtime. 100 microseconds of CPU time is an integer factor larger than many atomic database operations in a fast database engine. An operation taking an order of magnitude longer than the scheduler would expect based on runtime state has real adverse consequences.

mumblemumble 7y ago
So, before we theorize that Cockroach can't possibly be fast enough to meet anyone's needs, perhaps we should look at some performance figures? I haven't ever touched it myself, but, if the benchmarks that Cockroach Labs posts on their blog are to be believed, it's pretty respectable for what it is.
- jandrewrogers 7y ago
  You are misrepresenting what was written. Most databases don't need to be optimal to be useful, merely adequate, and few attempt to be optimal in any kind of absolute sense. I still use PostgreSQL regularly and it unambiguously falls into this "far from optimal" category. However, as data velocity and volume grow, optimization of absolute performance and hardware efficiency has an increasingly large impact on the cost of operating a database and the kinds of applications you can economically run. Databases that sacrifice major optimizations at an architectural level will have no chance to be competitive with databases that do not over the long term as average data volume and velocity grows. At the scale of database operations many companies are using today, these optimizations literally save them tens of millions of dollars per year on infrastructure. CockroachDB produces what looks like a fine product and has every right to make whatever technical decisions they wish. That the architecture has made substantial performance sacrifices is not theoretical though, and they don't pretend to operate in markets that require any kind of absolute operational efficiency.

paulddraper 7y ago

C may not be an attractive choice in 2019, but C++ and Rust certainly are. Especially if you're working at the level where you want/need vectorization.