4 ms·

Which one is easier? Adding a few lines of GC configuration or rewriting a service in C++ using Seastar and binding processes to cores? What sort of performanc

Which one is easier? Adding a few lines of GC configuration or rewriting a service in C++ using Seastar and binding processes to cores? What sort of performance measurement uses default configurations? What is even the point of not tuning the GC to your application workload? I have spent the last 15 years on running Java apps in production and to optimize for the p99 latency is really not that hard. Optimize p99.99999 is a whole different subject though. I don't understand what is the point of comparing a default GC setting that is for hello world applications to a software that is optimized every way possible. Apples to oranges. It is a grat marketing gimmick though. Look ma no performance! Look here, so much faster. We live in a single dimension word, yay!

manigandham 6y ago
Easier for who? At some point, the architecture needs to change to overcome a fundamental limit. Developers take on that work to make the performance and operations easier for their users. Seastar is the foundation of Scylla, which shows that rewriting in C++ can deliver magnitudes more performance which is not possible by just tuning Cassandra on the JVM. In fact, Datastax has now copied the Scylla approach in Cassandra but still lags behind drastically with performance.
- StreamBright 6y ago
  >> Easier for who? For any decent SRE out there. >> magnitudes more performance which is not possible by just tuning Cassandra Magnitudes?? Are you talking about the order of magnitude? You should read the ScyllaDB performance report first. https://www.scylladb.com/product/benchmarks/aws-i3-metal-benchmark/ https://www.scylladb.com/product/benchmarks/aws-i3-metal-ben... Avg. 99.9% Latency (ms): 9.9 vs. Avg. 99.9% Latency (ms): 474.4 While there is no significant latency difference in lover percentile tiers. Where Scylla really shines is TCO. Some companies trade SRE time for license cost, some other companies tune GC.
  - agallego 6y ago
    who cares about lower percentage tiers in the world of big data? what does the p30 or 50 even matter at large scale? See Gil Tene's talk on how not to measure latency. https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU What matters is what most of your customers will get - p99, p999, p9999, p100.
    - StreamBright 6y ago
      You are assuming that all workloads are customer facing.
    - throwdbaaway 6y ago
      Ironically, Gil Tene uses the talk to sell you the Azul C4 pauseless GC, which shall easily invalidate your 10x lower p99 latency claim. Of course we also have zgc and shenandoah these days.
  - manigandham 6y ago
    Yes, you get 10x the throughput while maintaining far better tail latency with Scylla over Cassandra. How is 10ms vs 475 not a major improvement? How is 4 nodes vs 40 not a major improvement? If you're an SRE than how is managing 4 servers with far less tuning and maintenance not a major improvement? Also 99.9% percentile still matters. They're testing with 300k ops/sec which means 300/sec are facing extreme latency spikes that can be enough to fail and/or cause cascading issues through out the application. There's no metric where Cassandra is better here and you can't tune your way to the same performance in the first place which is the whole point of Scylla. What even is your claim here? Spend more to get less?
    - throwdbaaway 6y ago
      2-3x maybe, 10x not likely, unless you compare it to untuned cassandra. What makes it possible to run cassandra/scylla on nodes with TBs of data density is the TWCS compaction strategy from Jeff Jirsa. He was just a cassandra power user at the time, and I like to think that the invention was possible because of Java. So, next time you read an ad piece from scylla about replacing 40 mid size boxes running CMS with 4 big boxes, don't forget about TWCS.
agallego 6y ago
I understand tail latencies - https://www.youtube.com/watch?v=WdFYY3vEcxo https://www.youtube.com/watch?v=WdFYY3vEcxo - my prev open source project smf (https://github.com/smfrpc/smf https://github.com/smfrpc/smf) uses gill tene's HDR histogram. I think you have a very superficial understanding of seastar. It is not simply marketing, it is a suite of tools and techniques to build low latency software - Specifically for IO intensive apps. Glauber wrote a good into here https://www.scylladb.com/2018/06/12/scylla-leverages-control-theory/ https://www.scylladb.com/2018/06/12/scylla-leverages-control.... Seastar is a fundamentally different way of programming from what you mentioned above. Let me give you an example. Seastar takes all the memory up front - never gives it back to the operating system (you can control how much via -m2G, etc). This gives you deterministic allocation latency, is just incrementing a couple of pointers. Memory is split evenly across the number of cores and the way you communicate between cores is message passing - which means you explicitly tell which thread is allowed to read which inbox (similar to actors) - i wrote about it here in 2017 https://www.alexgallego.org/concurrency/smf/2017/12/16/future.html https://www.alexgallego.org/concurrency/smf/2017/12/16/futur... The point of seastar is to not tune the GC for each application workload So to bring that up means that you missed the whole point of seastar. Instead the programmer explicitly reserves memory units for each subsystem - say 30% for the RPC, 20% for the app specific page-cache (since it's all DMA no kernel page cache), 20% for write-behinds, etc. (obviously in practice most of this is dynamic). It is not one dimension as suggested and not apples to oranges. it is apples to apples. You have a service, you connect your clients - unchanged - and one has better latency. that simple. It may be your experience that when you download a bin kafka say 2.4.1 you change the GC settings but in a multi-tenant environment that's a moving target. Most enterprises I have talked to, just use the default script to startup kafka w.r.t gc memory settings. (they may change some writers settings, caching, etc) At the end of the day there is no substitute for testing in your own app with your own firewall settings w/ your own hardware. The result should still give you 10x lower latency.
- StreamBright 6y ago
  I am familiar with Seastar too. It is one component that is pretty useless by itself. What is relevant in this topic is what is around it, the functionality that you provide. This is why Scylla is copying Cassandra. You can come up with a nice way of programming whatever you want but the end of the way the business functionality is what matters and there are different tradeoffs involved, still.

manigandham 6y ago

Easier for who? At some point, the architecture needs to change to overcome a fundamental limit. Developers take on that work to make the performance and operations easier for their users. Seastar is the foundation of Scylla, which shows that rewriting in C++ can deliver magnitudes more performance which is not possible by just tuning Cassandra on the JVM. In fact, Datastax has now copied the Scylla approach in Cassandra but still lags behind drastically with performance.

StreamBright 6y ago
>> Easier for who? For any decent SRE out there. >> magnitudes more performance which is not possible by just tuning Cassandra Magnitudes?? Are you talking about the order of magnitude? You should read the ScyllaDB performance report first. https://www.scylladb.com/product/benchmarks/aws-i3-metal-benchmark/ https://www.scylladb.com/product/benchmarks/aws-i3-metal-ben... Avg. 99.9% Latency (ms): 9.9 vs. Avg. 99.9% Latency (ms): 474.4 While there is no significant latency difference in lover percentile tiers. Where Scylla really shines is TCO. Some companies trade SRE time for license cost, some other companies tune GC.
- agallego 6y ago
  who cares about lower percentage tiers in the world of big data? what does the p30 or 50 even matter at large scale? See Gil Tene's talk on how not to measure latency. https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU What matters is what most of your customers will get - p99, p999, p9999, p100.
  - StreamBright 6y ago
    You are assuming that all workloads are customer facing.
  - throwdbaaway 6y ago
    Ironically, Gil Tene uses the talk to sell you the Azul C4 pauseless GC, which shall easily invalidate your 10x lower p99 latency claim. Of course we also have zgc and shenandoah these days.
- manigandham 6y ago
  Yes, you get 10x the throughput while maintaining far better tail latency with Scylla over Cassandra. How is 10ms vs 475 not a major improvement? How is 4 nodes vs 40 not a major improvement? If you're an SRE than how is managing 4 servers with far less tuning and maintenance not a major improvement? Also 99.9% percentile still matters. They're testing with 300k ops/sec which means 300/sec are facing extreme latency spikes that can be enough to fail and/or cause cascading issues through out the application. There's no metric where Cassandra is better here and you can't tune your way to the same performance in the first place which is the whole point of Scylla. What even is your claim here? Spend more to get less?
  - throwdbaaway 6y ago
    2-3x maybe, 10x not likely, unless you compare it to untuned cassandra. What makes it possible to run cassandra/scylla on nodes with TBs of data density is the TWCS compaction strategy from Jeff Jirsa. He was just a cassandra power user at the time, and I like to think that the invention was possible because of Java. So, next time you read an ad piece from scylla about replacing 40 mid size boxes running CMS with 4 big boxes, don't forget about TWCS.

agallego 6y ago

I understand tail latencies - https://www.youtube.com/watch?v=WdFYY3vEcxo https://www.youtube.com/watch?v=WdFYY3vEcxo - my prev open source project smf (https://github.com/smfrpc/smf https://github.com/smfrpc/smf) uses gill tene's HDR histogram. I think you have a very superficial understanding of seastar. It is not simply marketing, it is a suite of tools and techniques to build low latency software - Specifically for IO intensive apps. Glauber wrote a good into here https://www.scylladb.com/2018/06/12/scylla-leverages-control-theory/ https://www.scylladb.com/2018/06/12/scylla-leverages-control.... Seastar is a fundamentally different way of programming from what you mentioned above. Let me give you an example. Seastar takes all the memory up front - never gives it back to the operating system (you can control how much via -m2G, etc). This gives you deterministic allocation latency, is just incrementing a couple of pointers. Memory is split evenly across the number of cores and the way you communicate between cores is message passing - which means you explicitly tell which thread is allowed to read which inbox (similar to actors) - i wrote about it here in 2017 https://www.alexgallego.org/concurrency/smf/2017/12/16/future.html https://www.alexgallego.org/concurrency/smf/2017/12/16/futur... The point of seastar is to not tune the GC for each application workload So to bring that up means that you missed the whole point of seastar. Instead the programmer explicitly reserves memory units for each subsystem - say 30% for the RPC, 20% for the app specific page-cache (since it's all DMA no kernel page cache), 20% for write-behinds, etc. (obviously in practice most of this is dynamic). It is not one dimension as suggested and not apples to oranges. it is apples to apples. You have a service, you connect your clients - unchanged - and one has better latency. that simple. It may be your experience that when you download a bin kafka say 2.4.1 you change the GC settings but in a multi-tenant environment that's a moving target. Most enterprises I have talked to, just use the default script to startup kafka w.r.t gc memory settings. (they may change some writers settings, caching, etc) At the end of the day there is no substitute for testing in your own app with your own firewall settings w/ your own hardware. The result should still give you 10x lower latency.

StreamBright 6y ago
I am familiar with Seastar too. It is one component that is pretty useless by itself. What is relevant in this topic is what is around it, the functionality that you provide. This is why Scylla is copying Cassandra. You can come up with a nice way of programming whatever you want but the end of the way the business functionality is what matters and there are different tradeoffs involved, still.