5 ms·

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of a

by sreque 9y ago

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect p99 latencies. It's pretty much accepted everywhere that GCs perform terribly for databases. Modern GCs are great at handling small, very short-lived memory allocations, and that's about it. Just about any other workload and manual memory management ends up being a much better use of your time than GC tuning.

nvarsj 9y ago
So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.
- the8472 9y ago
  OpenJDK/Hotspot is not the only game in town. Those who really need low latencies can opt to use other JVMs (some commercial) with GCs that provide very low pause times, usually at the expense of some percentage points of throughput. In large corporate environments that might not be a problem.
- thefounder 9y ago
  Apparently these people enjoy GC/JVM languages more than C++.
  - chii 9y ago
    GC languages like Java is much easier to write, and can be made performant when required.
    - nvarsj 9y ago
      Yes, this is the reason why Java is chosen. But I feel pretty strongly that databases are system engineering problems, and should be written using a proper systems language. Something like Java makes implementation easier, but operation more difficult and costly.
- jjirsa 9y ago
  To be fair, a lot of big companies know how to tune the JVM. A TON of HUGE companies write a LOT of java. What you consider a constant struggle, a lot of very large companies consider trivial.
  - nvarsj 9y ago
    I'm not sure it's trivial. Tuning the JVM is an entire cottage industry. JVM performance experts can make 1000+/day tuning the JVM and are in high demand. Companies spend huge amounts of engineering effort to keep the JVM running smoothly. I used to be involved in this side of things pretty heavily at a HFT firm, which almost exclusively used Java. In my opinion it's a colossal waste of resources. Classic example of using the wrong tool for the job.
    - koffiezet 9y ago
      And it's still cheaper to hire a guy with a skill like that for a few weeks, or even keep him permanently - and keep a larger development team of cheaper C# or Java devs, than it is to replace them all with higher-payed C/C++ devs which would probably take longer to get the same functionality up & running.
- manigandham 9y ago
  Many of these open-source databases started as internal projects inside big companies, where Java/JVM allowed for more productivity and cross-platform deployment with more skill reuse of the team. Then they grew from there and now it's too late to rewrite the whole thing. If you were starting a database-focused company from the beginning than choosing C++ is a better decision, which is exactly what ScyllaDB has done with their cassandra clone. Along with general algorithm and decision improvements, it'll provide 10-100x the same performance at lower latency on the same server. We're also starting to see more projects written in Go now, which is still a managed runtime but usually better at handling these kinds of low-level systems.
  - nvarsj 9y ago
    Go is a much better choice for systems work. Largely because the GC has a low pause (sub ms) target. I'd still be hesitant to use it for very latency sensitive things, or memory intensive applications. Prometheus, for example, has struggled with golang's memory management (bad memory fragmentation, wasteful memory usage). But I think it's a great compromise if you don't want to deal with memory management.
    - manigandham 9y ago
      Sure, I would also .NET/C# to the list now. .NET Core on Linux is very fast and there are some great developments around fast low-level (yet managed) managed memory manipulation that can lead to some very fast software.
- mlla 9y ago
  Good Q. You might like to check ScyllaDB written in C++, which is supposed to have considerably better performance than Cassandra (also low tail-latency) and a level of compatibility with it: https://www.scylladb.com/ https://www.scylladb.com/
b4lancesh33t 9y ago
> Given that stop-the-world pauses are never evenly distributed throughout time That is not a given. And, even distribution is only part of the equation. If they are sufficiently short, then even being somewhat unevenly distributed should not have much of an impact on latency. For example, if the max length of a pause were 1ms, and 99p latency were 15ms, you'd have to be fairly unlucky to see a 33% increase in latency99 due to GC. That would entail 5 of 25 pauses happening during a 20ms period in a 1s window. (This idea is not purely hypothetical. For example, Go's GC has very low STW periods.) > It's pretty much accepted everywhere Eh. Apparently everyone thinks C is the best language for cryptography and other secure but not particularly perf sensitive code. Go figure. Sometimes the wisdom of the masses is not wisdom. Best not to appeal to it during argumentation.
- katastic 9y ago
  Let me guess. I can feel it coming. You're about to segway into talking about Rust?
- eptcyka 9y ago
  I'd love to see how Go's GC performs when running an application similar to Cassandra, on multiple cores, with gigabytes of memory allocated.
  - the8472 9y ago
    Go's GC is non-moving, incremental and single-generation. Hotspot GCs are all moving collectors so they make different tradeoffs. IBM's metronome collector would be the closest one to Go's properties.
  - b4lancesh33t 9y ago
    This is a thing others have already done, although I don't have citations to hand.
- sreque 9y ago
  You're right on your first point. I responded a little harshly to OP because I believe they responded too harshly to the blog post. The P99 GC latency data they give is us not sufficient to explain their perf gains. More likely, the metric they were tracking was well-correlated with their perf gains, but not the only or even primary cause. The other perf gains could have been caused by reduction in older generation collection times, for instance.

by sreque 9y ago

nvarsj 9y ago

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

the8472 9y ago
OpenJDK/Hotspot is not the only game in town. Those who really need low latencies can opt to use other JVMs (some commercial) with GCs that provide very low pause times, usually at the expense of some percentage points of throughput. In large corporate environments that might not be a problem.
thefounder 9y ago
Apparently these people enjoy GC/JVM languages more than C++.
- chii 9y ago
  GC languages like Java is much easier to write, and can be made performant when required.
  - nvarsj 9y ago
    Yes, this is the reason why Java is chosen. But I feel pretty strongly that databases are system engineering problems, and should be written using a proper systems language. Something like Java makes implementation easier, but operation more difficult and costly.
jjirsa 9y ago
To be fair, a lot of big companies know how to tune the JVM. A TON of HUGE companies write a LOT of java. What you consider a constant struggle, a lot of very large companies consider trivial.
- nvarsj 9y ago
  I'm not sure it's trivial. Tuning the JVM is an entire cottage industry. JVM performance experts can make 1000+/day tuning the JVM and are in high demand. Companies spend huge amounts of engineering effort to keep the JVM running smoothly. I used to be involved in this side of things pretty heavily at a HFT firm, which almost exclusively used Java. In my opinion it's a colossal waste of resources. Classic example of using the wrong tool for the job.
  - koffiezet 9y ago
    And it's still cheaper to hire a guy with a skill like that for a few weeks, or even keep him permanently - and keep a larger development team of cheaper C# or Java devs, than it is to replace them all with higher-payed C/C++ devs which would probably take longer to get the same functionality up & running.
manigandham 9y ago
Many of these open-source databases started as internal projects inside big companies, where Java/JVM allowed for more productivity and cross-platform deployment with more skill reuse of the team. Then they grew from there and now it's too late to rewrite the whole thing. If you were starting a database-focused company from the beginning than choosing C++ is a better decision, which is exactly what ScyllaDB has done with their cassandra clone. Along with general algorithm and decision improvements, it'll provide 10-100x the same performance at lower latency on the same server. We're also starting to see more projects written in Go now, which is still a managed runtime but usually better at handling these kinds of low-level systems.
- nvarsj 9y ago
  Go is a much better choice for systems work. Largely because the GC has a low pause (sub ms) target. I'd still be hesitant to use it for very latency sensitive things, or memory intensive applications. Prometheus, for example, has struggled with golang's memory management (bad memory fragmentation, wasteful memory usage). But I think it's a great compromise if you don't want to deal with memory management.
  - manigandham 9y ago
    Sure, I would also .NET/C# to the list now. .NET Core on Linux is very fast and there are some great developments around fast low-level (yet managed) managed memory manipulation that can lead to some very fast software.
mlla 9y ago
Good Q. You might like to check ScyllaDB written in C++, which is supposed to have considerably better performance than Cassandra (also low tail-latency) and a level of compatibility with it: https://www.scylladb.com/ https://www.scylladb.com/

b4lancesh33t 9y ago

> Given that stop-the-world pauses are never evenly distributed throughout time That is not a given. And, even distribution is only part of the equation. If they are sufficiently short, then even being somewhat unevenly distributed should not have much of an impact on latency. For example, if the max length of a pause were 1ms, and 99p latency were 15ms, you'd have to be fairly unlucky to see a 33% increase in latency99 due to GC. That would entail 5 of 25 pauses happening during a 20ms period in a 1s window. (This idea is not purely hypothetical. For example, Go's GC has very low STW periods.) > It's pretty much accepted everywhere Eh. Apparently everyone thinks C is the best language for cryptography and other secure but not particularly perf sensitive code. Go figure. Sometimes the wisdom of the masses is not wisdom. Best not to appeal to it during argumentation.

katastic 9y ago
Let me guess. I can feel it coming. You're about to segway into talking about Rust?
eptcyka 9y ago
I'd love to see how Go's GC performs when running an application similar to Cassandra, on multiple cores, with gigabytes of memory allocated.
- the8472 9y ago
  Go's GC is non-moving, incremental and single-generation. Hotspot GCs are all moving collectors so they make different tradeoffs. IBM's metronome collector would be the closest one to Go's properties.
- b4lancesh33t 9y ago
  This is a thing others have already done, although I don't have citations to hand.
sreque 9y ago
You're right on your first point. I responded a little harshly to OP because I believe they responded too harshly to the blog post. The P99 GC latency data they give is us not sufficient to explain their perf gains. More likely, the metric they were tracking was well-correlated with their perf gains, but not the only or even primary cause. The other perf gains could have been caused by reduction in older generation collection times, for instance.