7 ms·

What issues did they run into with Cassandra? It's not easy to build a scalable database like Cassandra. There are good reasons to write your own distributed st

What issues did they run into with Cassandra? It's not easy to build a scalable database like Cassandra. There are good reasons to write your own distributed storage system but the authors need to add more details about Keevo and specific issues they ran into with Cassandra.

dis-sys 9y ago
not cool enough to be used as clickbait any more.
StreamBright 9y ago
Well there are several issues with Cassandra: - it is not ACID the worst possible way "Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended." "This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight transactions (LWT) are for, and that's what the authors of this piece used. However! Fundamentally, mixing serializable (LWT) and non-serializable (plain UPDATE) ops will produce unpredictable results and that's what bit them here.Basically the same as if you marked half the accesses to a concurrently-updated Java variable with "synchronized" and left it off of the other half as an "optimization."Don't take shortcuts and you won't get burned." - there are several things that needs to be tuning (like GC) that is not trivial to do - modeling is challenging say the least, easy to create hotspots that the developers are not aware of
- anonymous7777 9y ago
  " it is not ACID the worst possible way" Vow what a great revelation. "Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended." This is not true. Cassandra is row level atomic but I guess it also depends on the version you use maybe. Can you talk about which version and provide a test that satisfies your assertion ? "there are several things that needs to be tuning (like GC) that is not trivial to do". Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead. I guess these days people can make up whatever they want without providing any valid tests that prove their assertions. It all comes down is either love & hate of a programming language or someone wants to put some fancy sounding tools in their resume!!
- anonymous7077 9y ago
  "it is not ACID the worst possible way" Vow what a great revelation. "Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended." This is not true. Cassandra is row level atomic but I guess it also depends on the version you use maybe. Can you talk about which version and provide a test that satisfies your assertion ? "there are several things that needs to be tuning (like GC) that is not trivial to do". Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead. I guess these days people can make up whatever they want without providing any valid tests that prove their assertions. It all comes down to is either love & hate of a programming language or someone wants to put some fancy sounding tools in their resume!!
  - nemothekid 9y ago
    > Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead. I won't comment on the other points - but I managed a medium sized Cassandra cluster for a couple of years, and the GC point is valid. It has nothing to do with Java's GC being more advanced. It's easier to bypass the Go GC with stack allocations (not possible in Java), and many of Cassandra's processes (compaction, repair) end up being very GC heavy. GC tuning ends up being a function of your workload and if you ignore it, background processes like repair can adversely affect projection nodes, or throw nodes in a loop. - and adversely giving the node more memory can make things worse. Cassandra has left a bad taste in my mouth for Java-based databases.
    - StreamBright 9y ago
      Yeah I can show you one of the systems: GC of death: https://imgur.com/a/6RAX7 https://imgur.com/a/6RAX7 Write latency: https://imgur.com/a/Us0MO https://imgur.com/a/Us0MO Typical Cassandra user has these sort of problems. I have one client where they identified the issue (data modeling) and fixed it by redesigning their tables before I got there. Some of the Cassandra users are not even aware. The company where the pictures are from engaged me because the system did not meet with business requirements anymore. Could not insert data into the cluster, their ETL jobs were running for 20 hours and if one failed they did not have data for their business. We could speed it up to run it for 4 hours without remodeling the data. With remodeling it Cassandra was not the bottleneck anymore.
JelteF 9y ago
Main developer of Keevo at Stream here and you ask a very good question. I've asked myself the same question quite a lot. And I think it actually warrants it own dedicated blog post at some point. To give some idea though, the main reasons are, cost, simplicity, control and wanting to have a very good understanding of the database internals. Cassandra is very scalable, but it's not very efficient. The hosting costs for our cassandra cluster were so big that it was infeasible to run it in another region as well. Appart from that we've had (partial) downtime a couple of times because one node just started going crazy because of unclear reasons. Keevo solves this by not trying to be cassandra, but be much simpler. It doesn't do schema's or indexes. All it is is a very fast ordered key value store that is stored to disk and replicated automatically to multiple servers (using raft). Any other features we need, we build on top of this usually outside of Keevo itself. This simplicity saves us a lot of hosting costs and makes its performance much more predictable and easier to debug. Last but not least a very important advantage is control and understanding of the database internals. Because we build Keevo ourselves, we know the performance and consistency tradeoffs it has and can change/improve them when needed. I hope this helps in understanding our choice. It's definitely not something I would recommend for most companies, but since our product is storage at its core it makes sense for us.
- pbowyer 9y ago
  Thanks for the comment and look forward to the dedicated blog post :) Was Riak KV ever considered?
  - JelteF 9y ago
    Not as far as I know. We did benchmark a lot of embedded storages before deciding on Rocksdb though.
  - ddorian43 9y ago
    Wasn't riak, like, less efficient than cassandra ?
    - StreamBright 9y ago

dis-sys 9y ago

not cool enough to be used as clickbait any more.

StreamBright 9y ago

Well there are several issues with Cassandra: - it is not ACID the worst possible way "Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended." "This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight transactions (LWT) are for, and that's what the authors of this piece used. However! Fundamentally, mixing serializable (LWT) and non-serializable (plain UPDATE) ops will produce unpredictable results and that's what bit them here.Basically the same as if you marked half the accesses to a concurrently-updated Java variable with "synchronized" and left it off of the other half as an "optimization."Don't take shortcuts and you won't get burned." - there are several things that needs to be tuning (like GC) that is not trivial to do - modeling is challenging say the least, easy to create hotspots that the developers are not aware of

anonymous7777 9y ago
" it is not ACID the worst possible way" Vow what a great revelation. "Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended." This is not true. Cassandra is row level atomic but I guess it also depends on the version you use maybe. Can you talk about which version and provide a test that satisfies your assertion ? "there are several things that needs to be tuning (like GC) that is not trivial to do". Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead. I guess these days people can make up whatever they want without providing any valid tests that prove their assertions. It all comes down is either love & hate of a programming language or someone wants to put some fancy sounding tools in their resume!!
anonymous7077 9y ago
"it is not ACID the worst possible way" Vow what a great revelation. "Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended." This is not true. Cassandra is row level atomic but I guess it also depends on the version you use maybe. Can you talk about which version and provide a test that satisfies your assertion ? "there are several things that needs to be tuning (like GC) that is not trivial to do". Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead. I guess these days people can make up whatever they want without providing any valid tests that prove their assertions. It all comes down to is either love & hate of a programming language or someone wants to put some fancy sounding tools in their resume!!
- nemothekid 9y ago
  > Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead. I won't comment on the other points - but I managed a medium sized Cassandra cluster for a couple of years, and the GC point is valid. It has nothing to do with Java's GC being more advanced. It's easier to bypass the Go GC with stack allocations (not possible in Java), and many of Cassandra's processes (compaction, repair) end up being very GC heavy. GC tuning ends up being a function of your workload and if you ignore it, background processes like repair can adversely affect projection nodes, or throw nodes in a loop. - and adversely giving the node more memory can make things worse. Cassandra has left a bad taste in my mouth for Java-based databases.
  - StreamBright 9y ago
    Yeah I can show you one of the systems: GC of death: https://imgur.com/a/6RAX7 https://imgur.com/a/6RAX7 Write latency: https://imgur.com/a/Us0MO https://imgur.com/a/Us0MO Typical Cassandra user has these sort of problems. I have one client where they identified the issue (data modeling) and fixed it by redesigning their tables before I got there. Some of the Cassandra users are not even aware. The company where the pictures are from engaged me because the system did not meet with business requirements anymore. Could not insert data into the cluster, their ETL jobs were running for 20 hours and if one failed they did not have data for their business. We could speed it up to run it for 4 hours without remodeling the data. With remodeling it Cassandra was not the bottleneck anymore.

JelteF 9y ago

Main developer of Keevo at Stream here and you ask a very good question. I've asked myself the same question quite a lot. And I think it actually warrants it own dedicated blog post at some point. To give some idea though, the main reasons are, cost, simplicity, control and wanting to have a very good understanding of the database internals. Cassandra is very scalable, but it's not very efficient. The hosting costs for our cassandra cluster were so big that it was infeasible to run it in another region as well. Appart from that we've had (partial) downtime a couple of times because one node just started going crazy because of unclear reasons. Keevo solves this by not trying to be cassandra, but be much simpler. It doesn't do schema's or indexes. All it is is a very fast ordered key value store that is stored to disk and replicated automatically to multiple servers (using raft). Any other features we need, we build on top of this usually outside of Keevo itself. This simplicity saves us a lot of hosting costs and makes its performance much more predictable and easier to debug. Last but not least a very important advantage is control and understanding of the database internals. Because we build Keevo ourselves, we know the performance and consistency tradeoffs it has and can change/improve them when needed. I hope this helps in understanding our choice. It's definitely not something I would recommend for most companies, but since our product is storage at its core it makes sense for us.

pbowyer 9y ago
Thanks for the comment and look forward to the dedicated blog post :) Was Riak KV ever considered?
- JelteF 9y ago
  Not as far as I know. We did benchmark a lot of embedded storages before deciding on Rocksdb though.
- ddorian43 9y ago
  Wasn't riak, like, less efficient than cassandra ?
  - StreamBright 9y ago