16 ms·
How Palo Alto Networks Replaced Kafka with ScyllaDB for Stream Processing
- smohare 4y ago
- switchbak 4y agoIt's a 27 min video from an actual customer doing real things with this product. The product in particular has a reputation for being obsessive about performance. Personally, I think this could be interesting and useful for those with similar needs.
- redwood 4y agoInteresting to see space for multiple commercial backers of Cassandra Anyone seeing Cassandra adoption for new use cases in the public cloud?
- deleted 4y ago[deleted]
- throwaway81523 4y agoScyllaDB and its related parts like Seastar always struck me as real performance-oriented programming, though it was based on leveraging language tech (C++14 early on) that was painful. I wonder if a nicer approach is possible nowadays.
- scottcodie 4y agoI get not wanting to add yet-another-system to reduce operational complexity but it seems more economical to use a system like Flink to do a time windowed join and emit single records to be written to a persistence store. The Flink time window can be sufficiently large to encompass the disparity between ingest and event time without much RAM consumption by using a RocksDB state backend on the operator. Let me know if I miss something, every use case is different :)
- majormajor 4y agoThey don't go into much of the detail of their "event consumers" but it certainly sounds like something Flink can handle, although Flink itself can be yet another operational set of headaches on top of Kafka. It also seems like something simple enough (simple processing-time windows) that they might not need Flink for the consumer. I have a hard time recommending Flink over dumber consumers or a more micro-batch vs true streaming approach unless you're doing something that really needs the long-lived in-worker keyed state and the ability to do things like streaming joins and all. Otherwise the Flink gotchas and nasty surprises can outweigh the ease of which it lets you do what you want to do.
- dikei 4y agoI have written Flink pipelines that solved similar problems to this post: sessionization of time-skew data points from multiple sources with variable and possibly large delays; simple and economical is not how I would describe it :) . If one of your datasource get lag behind, Flink would buffer a huge amount of data waiting for it to catch up due to how watermark work when joining 2 streams, and you would still encounter out-of-memory error even with RocksDB from time to time if your session window get too large. In addition, with our state size frequently reached hundred of GBs, recovering from failure was not exactly fast either.
- DeathArrow 4y agoAs I understood, they wanted to eliminate the MQ layer. If they used Flink, they would still need to keep Kafka and they would be introducing another layer of complexity with Flink. So, instead of simplifying they would make the stack more complex.
- cvccvroomvroom 4y agoThis reads like sales copy. It's freemium FOSS-washed crippleware with a radioactive license (AGPL). Hard pass. I'll stick to FOSS solutions that don't require licenses to unlock closed-source components and can be patched by a community and/or yourself. Edit: Previously, there are other commercialish OSS NoSQL solutions for large-scale apps that are less proprietary with better licenses like Couchbase (not Cassandra CQL).
- newaccount2021 4y agowhy the hostility? they're trying to build a sustainable business so their product can have a future, which in turn benefits those who can avail themselves of their open source is paying for support really such a raw deal if you are using their produce to generate your own revenue?
- cvccvroomvroom 4y agoYou don't understand AGPL and have a consumer, pay-to-play mindset... sorry, but no sale. Come back when all of the features are available to everyone and it's a BSD-2 license with optional support.
- PeterCorless 4y agoScyllaDB Open Source is a fully functional NoSQL database. It is far from "crippleware" as many of our FOSS users can and have publicly attested. Contributors can submit their own patches to ScyllaDB. Contributing Guidelines: https://github.com/scylladb/scylla/blob/4bd4aa2e887278c4912f65b80548faaad4e05951/CONTRIBUTING.md https://github.com/scylladb/scylla/blob/4bd4aa2e887278c4912f...
- bluelightning2k 4y agoIf a general system becomes good enough, you see it displace specialized systems. In this case the Kafka paradigm can be replaced because there is such a performant NoSql DB. It's kind of like how standalone cameras became less and less desirable as phone cameras got better. Standalone could do better quality - but this matters less once both options are really good. There is some 'good enough' point where you hit vastly diminishing returns & simplifying into just phones became worthwhile. Databases (certainly Scylla) may be hitting a point where specializing, actively optimizing, etc. are less desirable than just reusing one good system.
- fizwhiz 4y agoFrom earlier in the article: > Clock skew across different sensors: Sensors might be located across different datacenters, computers, and networks, so their clocks might not be synchronized to the millisecond. And later on in their final solution > Implementation4 cons: Producers and consumers must have synchronized clocks (up to a certain resolution) How do they reconcile this skew in their final solution?
- dikei 4y agoIt's simple, and the same in all implementations: `Wait some amount of time to allow related events to arrive`. For implementations 4, I suppose it's done by reading only data point with `insert_time < NOW() - some_delays`. I think the requirement `Producers and consumers must have synchronized clocks (up to a certain resolution)` is simply so that there's no unnecessary lag between producers and consumers.
- fizwhiz 4y agoCan you walk me through how waiting some amount of time actually improves their guarantee? It's not like they're using Google's TrueTime api that provides bounds on time. How is "unnecessary lag" avoided between Producers & Consumers?
- dikei 4y ago> Clock skew across different sensors: Sensors might be located across different datacenters, computers, and networks, so their clocks might not be synchronized to the millisecond. We have to assume that although the sensors time are not synchronized to the milliseconds, but their time are still accurate to a certain degree (say ~1 seconds). Then factoring in the time for data to travel from sensors to the collector, if you wait for 5 secs, most events would have arrived and get stored in database. Events that arrive after that watermark would probably be safe to discard or become irrelevant. Waiting also solve the out-of-order problem, since you can sort the events again. > Implementation4 cons: Producers and consumers must have synchronized clocks (up to a certain resolution) So let's say the consumer's clock is slower than the producer's by 5 seconds, i.e when the producer thinks the time is 10, the consumer thinks the time is only 5. In this case, even if the producer has already written `insert_time = 10`, consumer will only read up to `insert_time = 5` and there's an extra 5 seconds worth of lag on top the of wait introduce by the algorithm.
- throwawaythekey 4y agoDoes any one on here have some real world experience with scylla? We currently make heavy use of dynamo and are interested in something cheaper/faster. The marketing material is pretty compelling but I'm unsure of how hard scylla is to operate at scale.
- zeckalpha 4y agoThey have some high a availability functionality in the open source version but not commercial. If the experts won’t run it in that mode, why do you think you’ll do a better job?
- manigandham 4y agoWhat are you talking about? High-availability is a core part of the Cassandra data model and node architecture. Scylla is designed to be highly available, and the enterprise version only adds features to make it easier to operate.
- PeterCorless 4y agoJust to note that ScyllaDB Open Source sometimes is some months to a year ahead of the Enterprise release in terms of features. Though I am not sure what specific "high availability" feature poster above is referring to. Otherwise, as you note, ScyllaDB Enterprise will be a superset of our Open Source offering. Also to minimize lag between Open Source an feature availability, we just announced new Feature releases for Enterprise which will be coming later this year and into the future. https://www.scylladb.com/2022/06/08/new-scylladb-enterprise-support-and-versioning/ https://www.scylladb.com/2022/06/08/new-scylladb-enterprise-...
- manigandham 4y agoUsed it on my past adtech startup handling billions of http requests per day (backend having multiple more calls). Solid product, it's since reached parity with Cassandra and added even more features. Great team, very helpful. It's far easier to run than Cassandra because of its performance so you need less nodes overall (which is the single biggest issue in C* ops). The enterprise edition also has better compaction strategies and an automated system to schedule it all. What scale are you looking at?
- DeathArrow 4y agoIs there any more recent technical review of ScyllaDB than this? https://jepsen.io/analyses/scylla-4.2-rc3 https://jepsen.io/analyses/scylla-4.2-rc3
- DeathArrow 4y ago>Like the first solution, normalized data is stored in a database – but in this implementation, it’s a NoSQL database instead of a relational database. What means data normalization in a NoSQL context? I think most normal forms make sense in a context where we have tables, rows and Relational algebra.
- EdwardDiego 4y agoWell, that was very uninformative. > To meet our performance expectations, Kafka must work from memory, and we don’t have much memory to give it... ...Even the smallest customer required two or three Kafka instances A) What performance expectations? Timeliness? B) What's "not much" memory? C) And why don't you have that much? D) When you say instance, do you mean "broker", or actual clusters? Why did the smallest customer need 2 to 3 of them?
- skinnyarms 4y agoD) Sounds like clusters to me: "Palo Alto Networks provides each customer a unique deployment."
- infogulch 4y agoWhy does scylla not need clustering when kafka does?
- EdwardDiego 4y agoErr, Scylla does... here's some link text from their docs. > Provision and manage a Scylla cluster in your environment.
- supermatt 4y agoIm not seeing the "stream processing" piece here. Looks like they went from polling an RDBMS to some triggered querying of scylla, and then on to polling scylla. i.e. they went from polling an RDBMS to polling Scylla. They didnt replace kafka with anything so now their implementation isnt reactive. This is effectively no different that implementing a message queue in a database, with all the negatives that brings. They are sharding for each consumer to prevent multiple consumption due to lack of locks. What if a consumer goes down? How does it manage its own state? All things managed by kafka (or pretty much any MQ) out of the box, and now they have to implement ALL of that themselves - none of which is mentioned in the article.
- Thev00d00 4y agoThis is clearly just an advertisement, and not even an informative one!
- infogulch 4y agoSo they're using Scylla to manually do what kafka does, basically processors polling for new records in a shard and updating their watermark once its done processing. I'm surprised that this is faster than just using kafka alone, though one of the reasons why they wanted to avoid kafka is dealing with deployment complexity and memory usage of kafka clusters.