5 ms·
The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, a
by kerblang 6y ago
The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional.
Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition key column is "day_of_week", then you have 7 partitions. New problem: Your partitions need to be < 300 MB or cassandra falls over dead.
In fact you'll soon realize that over the lifetime of your table, keeping those partitions under control may end up forcing you to use various date tricks like putting year/month/day in as "artificial" partition keys.
Of course if you put everything in the partition keys instead of clustering keys, let me note again that you have to put each partition key column in every where clause, in which case you may have trouble querying for batches of data.
Furthermore, when you do put clustering columns in your where clause, they have to be in order declared; so if your CC's are a, b, and c, then your where clause can use (a), (a,b), or (a,b,c); but if it has b, it has to have a, and if it has c, it has to have a&b. This is because storage is hierarchical. (Same rules for "order by", btw) (and no there's no "group by")
This is when you start realizing: Oh, you mean it's really not like "SQL without joins". No, not even close.
- jimbokun 6y agoBest of summary of the Cassandra data model I've seen: HashMap<PartitionKey, OrderedSet<ClusterKey>> So you can lookup data by the ParititionKey, then perform range queries on the ClusterKey to filter data belonging to that partition. If your access patterns look like that, Cassandra could be a great fit. For anything else, you probably want to consider a different data store.
- koolba 6y agoHa! That's pretty succinct. Though HashMap should be just "Map" and the value of the top level map is not a set, it's another map: Map<PartitionKey, SortedMap<ClusterKey, Record>>
- deleted 6y ago[deleted]
- garyrichardson 6y agoMaking CQL SQL-Like was the worst possible thing the designers could have done. Newbies want to do SQL joins and filtering with it until they've been a few times. C* is essentially scaled/automated MySQL Sharding and blob storage.. except you pay a huge cost for any server side filtering. The Partition Key would be the equivalent of your MySQL shard id. ClusterKey is your primary key. if you treat everything like PUT $paritionKey, $primaryKey, $data and GET $partitionKey, $primaryKey you get a slow and massively scaling redis. If you do anything else with it you'll likely start regretting your choices and looking for a replacement database.
- fennecfoxen 6y agoWhat you are expected to do with Cassandra, to achieve your horizontal scalability, is (a) denormalize by each supported query up front, and (b) fan out all of those (often by time) so they never get too big. So for your "primary" data store for a type data, you might use the date for the partition key. Then if you do a lot of day-of-week queries, you'd store a second copy grouped by year-month-weekday, and query as many months as you need, and combine them. If you are collecting data for several sites, have a copy that's stored by year-month-weekday-site. And set up your software so it's easy to say "write in these 5 places, keyed off this combination of values." And because it's in all these denormalized places, you can forget updating anything atomically. So you go to append-only mode and if you have an "update" you record that as a separate entry after the first and your application's understanding of the data store includes coalescing the events and applying the update. (Maybe if you need it you have a separate process coalescing updates after the fact, but in the background, non-critical-like.) And you're right. This is very much not SQL. It's also super obnoxious and support for it in extant frameworks is pretty minimal (they tend to be rather myopically SQL-oriented, ill suited to take advantage of this model). It also uses lots of storage, on the premise that disk is cheap. But what all this accomplishes is crazy horizontal scalability, and when you make any one query, then all the data is right there on disk lined up neatly, so it all gets streamed out in a predictable amount of time. For a few select applications, this consistent scalability is worth the pain.
- MichaelMoser123 6y ago> New problem: Your partitions need to be < 300 MB or cassandra falls over dead. people work around that by storing the data in something like S3 and then keeping the handle in cassandra. Yet another hack on top of another hack.
- aasasd 6y agoPretty sure that's what the article is about.
- AtlasBarfed 6y agoDo you fill postgres with 100MB+ blobs? You'd do the same "hack" there.
- MichaelMoser123 6y agoyes, but postgress doesn't have the problem that it fails if you have more than 100MB in all records, in total.
- PeterCorless 6y agoWhile part of this is Cassandra's architecture, it may also be a limitation of the designer's way of thinking. Recently we had a hackathon project where we decided to implement the s3 API on top of Scylla (a Cassandra-compatible database) and were able to chunkify binary large objects (BLOBs) of up to 5 TB, and even turn Scylla into a FUSE filesystem. https://www.scylladb.com/2020/12/15/scylladb-blog-scylladb-developer-hackathon-scylla-s3/ https://www.scylladb.com/2020/12/15/scylladb-blog-scylladb-d...
- manigandham 6y agoThe issue seems to be a misunderstanding of what Cassandra is. It's an (advanced/nested) key-value database, so of course you need to specify the key to do anything.
- jeffbee 6y agoThe misunderstanding is understandable, since the Apache Cassandra site fails to state what Cassandra is, even directly below the heading "What is Cassandra?"
- mgkimsal 6y agoWell... it is "the right choice when you need scalability and high availability without compromising performance". I'm not sure how much more description you need! :)
- aasasd 6y agoThat's normal by now. Pretty much all software/service sites would do better by just using Wikipedia's description of themselves as the landing page. At least, better for the user. Probably because Wikipedia actually needs to explain the thing's meaning, and a standard marketing page would be slapped with plaques about non-encyclopedic content, in no time.
- aasasd 6y agoNoob question here. > Your partitions need to be < 300 MB or cassandra falls over dead Why is that? Is that some kind of engineered-in limitation?
- manigandham 6y agoThere's a default limit of 2 billion cells (rows * cols) but larger partitions also mean larger indexes and more serialization, repair, compaction and memory overhead which doesn't work well with Cassandra's Java/JVM GC requirements. It's getting better but likely will never reach the performance potential of Scylla which is Cassandra reimplemented in C++ with a much better sharded architecture.
- _benedict 6y agoNo, it's just that the storage layer is low priority to improve for this use case, but it'll happen eventually.
- cnlwsu 6y agoIt’s not a real requirement. Just a performance thing. Each partition has an index and as it grows the deserialization of it to find the queried start column takes longer. It is lazy loads but there a perf hit. Easier to have a rule of thumb like “100mb partitions” and have more fixed tps per core etc expectations.
- kerblang 6y agoReplying to myself because I can't seem to find an edit button: If you're interested in Cassandra, the free online DataStax training is highly recommended. You'll learn all the gotchas ahead of time instead of after you went to production, and you really, really don't want the latter.
- noncoml 6y ago> Your partitions need to be < 300 MB or cassandra falls over dead. Is that a typo? 300MB sounds ridiculously low..