9 ms·
New ScyllaDB Go Driver: Faster Than GoCQL and Its Rust Counterpart
- metadat 4y agoClever ideas to optimize this baby. Nice work. Hadn't heard of the pre-coalesce millisecond pile up technique. Favorited, thank you, sincerely!
- loosescrews 4y agoThis is similar to Nagle's algorithm (controllable on Linux with TCPNODELAY).
- gpderetta 4y ago> Hadn't heard of the pre-coalesce millisecond pile up technique. This is basically Nagling and/or TCP_CORK right?
- habibur 4y ago> We also paid close attention to proper memory management, producing as little garbage as possible. The key. And I was wondering how can a tracing GC outperform a non-tracing-GC memory manager.
- forrestthewoods 4y agoThe key to garbage collection is… to go super far out of your way to avoid allocating memory. This is not ideal.
- girvo 4y agoI'm probably biased because I live in firmware development these days, but that's true even for non-garbage collected languages when it comes to making sure things are fast
- ekidd 4y agoSpeaking as someone who has spent time optimizing C++ and Rust, memory allocation in hot loops is often where performance goes to die. GC or not, if you want to fast, reducing allocation is one of the first things to benchmark. (One fast way to manage allocations is to use an arena allocator that allocates memory by incrementing a pointer, and frees memory all at once. This is pretty effective for simple, short-lived requests.)
- forrestthewoods 4y agoYes, minimize malloc in all cases. The difference is that GC languages are fundamentally designed around the concept that it’s cheap and easy to malloc/free. Avoiding allocations can be excruciatingly difficult. In C++ you also need to minimize allocations, but it’s radically easier to do in C++ than in C#.
- throwaway81523 4y ago> And I was wondering how can a tracing GC outperform a non-tracing GC memory manager. The cliché is that malloc/free style memory management has to touch all the garbage in order to free it, while a semispace GC only has to copy the live data once in a while. The garbage is ignored.
- masklinn 4y agoAfaik Go uses a non-moving GC, so it can’t be a semi space.
- pkolaczk 4y agoHowever, when a tracing GC runs, it has to touch massive amounts of cold data and pushes hot data out of cache. Traditional malloc/free touches data in small chunks and freeing happens close to the last use, so when most of the data is still hot in cache. Stable, predictable performance is often more important than the peak performance.
- chrisseaton 4y agoManual memory management means your memory management is tightly interleaved with application code, with both competing for the same limited cache space. A tracing GC batches memory management, using cache better and not as frequently evicting your application’s data out of cache. A GC can also let you use more efficient concurrent data structures - many sophisticated concurrent objects require a tracing GC for implementing correctly - which can improve the performance of your application code.
- tialaramex 4y ago> many sophisticated concurrent objects require a tracing GC for implementing correctly What sort of "sophisticated concurrent objects" are you thinking of?
- chrisseaton 4y agoImplementing many lock-free data structures without a GC is still a research topic, for example requiring conversion to use epoch allocation.
- tialaramex 4y agoThis still seems very vague, I was looking for concrete examples where quite different arrangements like hazard pointers are for some reason impossible.
- chrisseaton 4y agoHazard pointers, epoch reclamation yeah are both examples of tools to work around the lack of GC, but they require new work to figure out how to apply them and they harm your throughput which is probably why you wanted these data structures in the first place. https://medium.com/@tylerneely/fear-and-loathing-in-lock-free-programming-7158b1cdd50c https://medium.com/@tylerneely/fear-and-loathing-in-lock-fre...
- yencabulator 4y ago
- psaux 4y agoOne of my fav companies! Dor is an amazing person, highly recommend working with them. We started to move to ScyllaDB at my last engineering job. Doing 100TB’s a day in IoT Data.
- ZephyrBlu 4y agoWhat is that amount of data used for? Is it along the lines of “we want to collect all the data we can in case we want to use or analyze it at some point”, or are there real use cases?
- PeterCorless 4y agoFor IIoT, there's digital twins, machine health, sensor data aggregation, and a lot more. While a lot of it can be classified as "write once, read hardly ever," it is vital for triggers, alarms, real-time failure and security alerts, and, eventually being forwarded over to analytics systems for longer-term trends.
- snapetom 4y agoMy last job was far less data than yours (more like 100's of GB/day), and it wasn't surprising it handled it with ease. I love that DB so much. So performant, and far easier to admin than Cassandra.
- henrydark 4y agoWith all this work to remove allocations I get the feeling that what really is needed is C++ with Go's concurrency syntax and runtime
- deleted 4y ago[deleted]
- pjmlp 4y agoIt is a myth that C and C++ are free of such issues, it is always a matter of how much one cares about performance. https://groups.google.com/a/chromium.org/g/chromium-dev/c/EUqoIz2iFU4/m/kPZ5ZK0K3gEJ https://groups.google.com/a/chromium.org/g/chromium-dev/c/EU...
- avgcorrection 4y agoWhy C++?
- insanitybit 4y agoAre there simple benchmarks that I can run for the Rust counterpart? I've worked a bit on the scylla rust code and I see plenty of room for improving efficiency (there's a lot of unnecessary allocation imo, and the hashing algorithm is 10x slower than it needs to be), but I don't want to make a PR for improvements without evidence. > The big difference between our Rust and Go drivers comes from coalescing; however, even with this optimization disabled in the Go driver, it’s still a bit faster. For anyone who's wondering, the Rust driver has coalescing support as of 9 days ago.
- mvelbaum 4y agoI wonder if Tokio is also a reason for worse performance compared to Go's concurrency runtime.
- raverbashing 4y agoI wonder how many people are using 'async' just for the sake of it without a real need for it and shooting themselves in the foot while at it
- pkolaczk 4y agoActually in this case async is the only way to get sane performance and both drivers deliver excellent performance thanks to async. I've been using Scylla Rust driver in my C* benchmarking project and it is an order of magnitude faster than the tools which use threads. https://github.com/pkolaczk/latte https://github.com/pkolaczk/latte
- raverbashing 4y agoCool, good to know. I know threads are a limiting factor, but sometimes people jump into async while the problem is somewhere else
- pkolaczk 4y ago
- pjmlp 4y agoVery good write up how performance can be improved without the typical rewrite in X.
- erichocean 4y agoAnd yet…ScyllaDB is famous for being a 10x faster rewrite of Cassandra (written in Java) in C++. Your general comment is correct. I see it often with GPU algorithms which, no surprise, are also much faster on CPUs (using something like ISPC to compile them).
- pjmlp 4y agoA performance improvement that could have been obtained by only rewriting in C++ the critical paths and integrate them via JNI, instead of rewriting the world. An approach that tends to be ignored by those rewrite X in Y.
- pornel 4y agoThis is a dare to get people rewrite their Rust driver for performance.
- avgcorrection 4y agoGo/Rust rivalry is a meme.
- _wldu 4y agoEvery language dreams of being faster and safer than C and C++. Can't have it both ways.
- yencabulator 4y agoA different language can enable easy expression of designs that would be nightmare to maintain in C/C++: https://go.dev/talks/2013/oscon-dl.slide https://go.dev/talks/2013/oscon-dl.slide
- usrusr 4y agoAlmost reads like a case where an initial implementation in rust forced a clear mental image of ownership that could then be transferred into another language much easier than it would have been to reach the same clarity outside of pedantic reign of the rust compiler.
- philosopher1234 4y agoI think this is possible but I see no evidence in the article that would support this interpretation
- lukeqsee 4y agoScyllaDB's obsession with performance by working closely with deep understanding of hardware and software and not simply adding more machines is really impressive. They consistently demonstrate that we are under using our CPUs compared to potential.
- nvarsj 4y agoI think it’s a common principle of modern computing. We trade productivity for performance all the time. The idea being machines are fast enough it doesn’t matter. There are order of magnitude gains to be made at most levels in the modern stack - it’s just the effort required is immense.
- rob74 4y agoThe effort doesn't have to be immense though. I bet there are plenty of "low-hanging performance fruit" in most codebases, it's just that there's no real reward to pick them...
- jerf 4y agoMy own experience backs this. I don't sit here obsessing about performance. In a database context that is appropriate but I'm not implementing databases. Nor do I prematurely optimize. I just give some thought to the matter occasionally, especially at the architecture level, and as a result I tend to produce systems that often surprise my fellow programmers at its performance. And I strongly assert that is not because I'm some sort of design genius... ask me about my design mistakes! I've got 'em. It's just that I try a little. My peer's intuitions are visibly tuned for systems where nobody even tried. I'm not sure how much "low hanging" fruit there is, though. A lot of modern slowdown is architectural. Nobody sat down and thought about the flow through the system holistically at the design phase, and the system rolled out with a design that intrinsically depends on a synchronous network transaction every time the user types a key, or the code passes back and forth between three layers of architecture internally getting wrapped and unwrapped in intermediate objects a billion times per second (... loops are terrible magnifiers of architecture failures, a minor "oops" becomes a performance disaster when done a billion times...) when a better design could have just done the thing in one shot, etc. I think a lot of the things we have fundamental performance issues with are actually so hard to fix they all but require new programs to be written in a lot of cases. Then again, there is also visibly a lot of code in the world that has simply never been run through a profiler, not even for fun (and it is so much fun to profile a code base that has never been profiled before, I highly recommend it, no sarcasm), and it's hard to get a statistically-significant sense of how much of the performance issues we face are low-hanging fruit and how much are bad architecture.
- bborud 4y agoSomewhat unrelated observation: I have never looked at SchyllaDB so I went to the web page. In the most prominent space they take a dump on the competition. Normally that would be a red flag for me, but in this case it made me curious. Now I want to know more. :-)
- wejick 4y agoWill this be compatible with other DB using CQL? Like Cassandra itself or Yugabyte for example.
- PeterCorless 4y agoYes. ScyllaDB writes all of its drivers to be backward/generically compatible with other CQL-based databases like Cassandra, etc. There are some specific features like shard-aware queries and shard-aware ports that naturally won't apply. But they will work.
- midislack 4y agoGo d-driver? What the heck? What's this even mean?
- neoyagami 4y agoIs anyone using scyllaDB in production nowdays? I tried some years ago with dataloss and kinda spooked the bejesus out of me(and went to cassandra)
- PeterCorless 4y ago400+ companies. https://www.scylladb.com/users/ https://www.scylladb.com/users/
- Sin2x 4y agoHow does it compare to Clickhouse regarding speed?
- PeterCorless 4y agoClickhouse is a column store designed for analytics [OLAP] workloads. It would compete with, say, Apache Druid or Apache Pinot. ScyllaDB is a wide column store which is, in fact, a row store; you can call it a "key-key-value," since it had a partitioning key and a clustering [or "sort" key]. Which is more for transactional workloads [OLTP]. So it is more comparable with Cassandra or DynamoDB. So they are really designed for different sorts of things. That being said, ScyllaDB has some features, like Workload Prioritization, so you can run analytics, like range or full table scans against it without hammering your incoming transactions. But it wasn't designed specifically for that.
- Sin2x 4y agoThanks, for some reason I thought they aim at comparable usecases.
- pengaru 4y agoSounds like GoCQL and its Rust counterpart are poorly implemented.
- hintymad 4y agoIs DB driver a bottleneck in applications? Somehow I usually see bottlenecks in other places in a service, and the db bottlenecks are usually on the database-side instead on the driver side.