13 ms·

Couple of things as always: Cassandra works really bad with fat nodes (lots of data on one node), and much much better with a lot of small nodes, and 100PB wit

Couple of things as always: Cassandra works really bad with fat nodes (lots of data on one node), and much much better with a lot of small nodes, and 100PB with 300K nodes confirms this. Scylla scales better vertically, but don't know how much. Some comments are already comparing this to pgsql/mysql/whatever. Please don't. You can't make the same queries even though the language seems to support it. Cassandra is good at ingesting data, bad at deleting, really really bad at anything remotely relational. Errors are almost pointless. I'm going to point at an older comment of mine on cassandra: https://news.ycombinator.com/item?id=20430925#20432564 https://news.ycombinator.com/item?id=20430925#20432564 The takeaway should be: Yes, cassandra/scylla can be really fast and scale a lot. But it is also very probably unusable for your use case. Don't trust what the CQL language says you can do. Don't get me started on how bad the CQL language is, either.

raverbashing 4y ago
> older comment of mine on cassandra: https://news.ycombinator.com/item?id=20430925#20432564 https://news.ycombinator.com/item?id=20430925#20432564 Wow, that's surprisingly bad For real, I'm not against java, but I find time and again that those services that depend on the JVM have lots of "weird stuff". Not because Java is bad but Java fans (which you usually are when you start to develop something this big) think a lot of this crap is acceptable.
- pmcf 4y ago
  In my (extensive) experience in infrastructure. When people say the JVM is the problem, the JVM is never the problem. It's usually just a symptom of something else and lazy ops people just want to blame something and throw up their hands. I've never had to "tune a JVM" to make things work. I'll give you an example in Spark. We had a huge job that was failing and after checking the logs, it was when results were being spooled to disk. More log diving showed a lot of GC on every node. At that point we could have gone down the route of tuning something in the JVM, but more digging found the real culprit. IOstats when the jobs ran showed while reading data, writes were completely blocked and write latency was in the 100s of ms. The spark executor trying to dump data was blocked and the first symptom that things were falling apart was... GC. We changed the scheduler on the nodes and magically everything worked great. The VM in JVM is virtual machine. Same rules for resources apply and if you run out of resources, don't expect the mythical ops faeries to save you.
  - ibgeek 4y ago
    Can you expand on which scheduler you swapped out? OS? Spark? Something else?
    - pmcf 4y ago
      Sure I can answer both. The OS scheduler CFQ is generally bad for high volume disk applications. Used deadline in this case. Amy's Guide is still a treasure trove of info and a solid recommended read: https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html https://tobert.github.io/pages/als-cassandra-21-tuning-guide... If you are using Spark in a bare metal cluster with spark-submit, the above advice applies. In Kubernetes, never use the default pod scheduler with analytic workloads. Great choices are Volcano(https://volcano.sh/en/ https://volcano.sh/en/) and Yunikorn(https://yunikorn.apache.org/ https://yunikorn.apache.org/). Also great and evolving projects to support and contribute if you can.
deleted 4y ago
[deleted]
_benedict 4y ago
Your statement about data propagation in that linked comment is at least misleading. A write at quorum will always be visible instantly to a read at quorum.
- Luker88 4y ago
  I only wrote that quorum is not a transaction (no transactions in cassandra), and that is not a consensus. While quorum looks a lot like consensus, it is not since what is returned to the client is the latest timestamp. Different nodes could have and return different data. So Quorum write+local_quorum read might fail even if you are in the datacenter that accepted the write. Quorum is also on the total copies of the data, not a quorum of DC, so in certain (weird) multi-DC setups you could have a quorum in a single DC. In general though I think that the data consistency options of cassandra (quorum/local/N) are a good idea but underdeveloped. Anyway my point was: cassandra has too many pitfalls and eliminating those restricts the use case by a lot more than people realize. Plus the naming of all features look designed to trick you into thinking soewthing else
  - _benedict 4y ago
    No, you wrote that you have to write and wait several minutes for data to replicate. This is straight-up false. Yes, if you mix consistency levels incorrectly you will do it wrong and maybe get stale data, but that is a different criticism. I agree that it is easy for unsophisticated users to incorrectly use consistency levels in complex topologies, and I hope we will introduce mechanisms to prevent users making such mistakes in future. But that was not your claim, and in my experience users do understand consistency levels just fine. There are lots of valid criticisms to point at various use cases with Cassandra, but this was just incorrect.
    - skyde 4y ago
      it’s worse than waiting actually. Because client A might write value « 1 » and after client B write value « 2 ». Then client C can read Value 2 for 5 minute and then cassandra internal read-repair eventually put value 1 on all replica and value 2 is lost forever.
curiousDog 4y ago
And not to mention, for 90% of the use cases out there Postgres or MySql on aurora or whatever will be more than enough
- JensRantil 4y ago
  More like 99,9% of use cases. But yeah.
- scarface74 4y ago
  If I know what my access patterns are, DynamoDB is often the better choice.
jhgg 4y ago
What is wrong with CQL? The only gotcha I am aware of is UPDATE/INSERT really just meaning UPSERT.
- mnd999 4y ago
  It encourages you to think you can use secondary indexes to make different types of query over the same data like SQL, which you mostly can’t.
- Luker88 4y ago
  In no particular order: * LWT (Lightweight Transactions) are nothing like what you expect when you head "transactions". LWT are single row only. * error handling is asinine. E.g: A quorum insert fails. is the data there or not? Maybe! * material views. Added, removed, readded, deprecated. * big limits on secondary indexes which only work inside each single partition * different format of timestamp for insert and select * BATCH is atomic, but not isolated. ...In general CQL feels like a hack. In classic SQL you understand early that you can compose queries, subqueries and such. CQL leads to you to believe that this is possible, only for you to find out that it isn't. The docs are full of stuff like "You can do this query, except in this case, unless this other case in which you can again, but not in this other". It's just exception on top of exception. Even the documentation does not list all exceptions (I managed to find a couple in my last work).
  - jlokier 4y ago
    > A quorum insert fails. is the data there or not? Maybe! How is this point fundamentally different from SQL? In SQL, after you send COMMIT to a remote database and get back a network error, you don't know if your INSERTed data is committed or not. You'll have to reconnect and check for it.
    - Luker88 4y ago
      In CQL queries can fail even if your connection is still up and running. In this case it means that it could not write to 3 nodes, but it is not telling you if it did not write anything or at least once. Writing just once means that the data will be replicated. So if only 1 of the 3 nodes is up, it will still write there and then return error, even if the coordinator knows that the other nodes are not up. Queries have a maximum running time (only configurable at the database level if I remember correctly), and if your write exceeds this, it returns error. The replication still goes on and the data is still there. Network errors are different category, since you might not even know if your command was sent or not. Cassandra kind of sidesteps the network errors though since the client connects to multiple coordinators in the cluster.
YZF 4y ago
What you're saying applies more or less no any NoSQL database. Unusable for your use case is a bit of a big statement. There's plenty of cases where you don't need the power of relational databases and relational databases can't/don't typically scale horizontally the way NoSQL can. And really NoSQL can because of its attributes. If all you need is key/value lookups or time series, you're not looking for any joins or random reads that don't agree with the native ordering/partitioning then it's hard to beat NoSQL scalability/ingest-wise.
- eusto 4y ago
  I agree, but I think the part that's not stated is that when picking such a technology, you basically commit to not needing joins or random reads. If you do, as the application evolves, you're really out of luck. Generally, that commitment cannot be made upfront. Typically, solutions like time-series or document databases are deployed alongside other components in a much larger system. If you are looking for a single database solution, then maybe NoSQL is not for you
- Luker88 4y ago
  True, which is why I said that cassandra has its uses. The problem is that the documentation and the language tries really hard to sell you something else, and I have seen multiple people wasting way too much time on this.
- skyde 4y ago
  why settle for noSQL when we can use newSQL? Cockroachdb, CitusDB, TiDB,Yugabyte …
  - YZF 4y ago
    I don't think those have the same performance characteristics. At the end of the day you can't have your cake and eat it. You can't keep large scale indices that facilitate fast joins without writing them out to disk and you can't keep them in sync with your data without transactionality/atomicity. This boils down to tradeoffs in terms of locking/consensus/scale/indices/data locality. That said if you're going to be emulating this over noSQL anyways then there's an argument for using something that does it for you (assuming it meets your requirements).

raverbashing 4y ago

> older comment of mine on cassandra: https://news.ycombinator.com/item?id=20430925#20432564 https://news.ycombinator.com/item?id=20430925#20432564 Wow, that's surprisingly bad For real, I'm not against java, but I find time and again that those services that depend on the JVM have lots of "weird stuff". Not because Java is bad but Java fans (which you usually are when you start to develop something this big) think a lot of this crap is acceptable.

pmcf 4y ago
In my (extensive) experience in infrastructure. When people say the JVM is the problem, the JVM is never the problem. It's usually just a symptom of something else and lazy ops people just want to blame something and throw up their hands. I've never had to "tune a JVM" to make things work. I'll give you an example in Spark. We had a huge job that was failing and after checking the logs, it was when results were being spooled to disk. More log diving showed a lot of GC on every node. At that point we could have gone down the route of tuning something in the JVM, but more digging found the real culprit. IOstats when the jobs ran showed while reading data, writes were completely blocked and write latency was in the 100s of ms. The spark executor trying to dump data was blocked and the first symptom that things were falling apart was... GC. We changed the scheduler on the nodes and magically everything worked great. The VM in JVM is virtual machine. Same rules for resources apply and if you run out of resources, don't expect the mythical ops faeries to save you.
- ibgeek 4y ago
  Can you expand on which scheduler you swapped out? OS? Spark? Something else?
  - pmcf 4y ago
    Sure I can answer both. The OS scheduler CFQ is generally bad for high volume disk applications. Used deadline in this case. Amy's Guide is still a treasure trove of info and a solid recommended read: https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html https://tobert.github.io/pages/als-cassandra-21-tuning-guide... If you are using Spark in a bare metal cluster with spark-submit, the above advice applies. In Kubernetes, never use the default pod scheduler with analytic workloads. Great choices are Volcano(https://volcano.sh/en/ https://volcano.sh/en/) and Yunikorn(https://yunikorn.apache.org/ https://yunikorn.apache.org/). Also great and evolving projects to support and contribute if you can.

deleted 4y ago

[deleted]

_benedict 4y ago

Your statement about data propagation in that linked comment is at least misleading. A write at quorum will always be visible instantly to a read at quorum.

Luker88 4y ago
I only wrote that quorum is not a transaction (no transactions in cassandra), and that is not a consensus. While quorum looks a lot like consensus, it is not since what is returned to the client is the latest timestamp. Different nodes could have and return different data. So Quorum write+local_quorum read might fail even if you are in the datacenter that accepted the write. Quorum is also on the total copies of the data, not a quorum of DC, so in certain (weird) multi-DC setups you could have a quorum in a single DC. In general though I think that the data consistency options of cassandra (quorum/local/N) are a good idea but underdeveloped. Anyway my point was: cassandra has too many pitfalls and eliminating those restricts the use case by a lot more than people realize. Plus the naming of all features look designed to trick you into thinking soewthing else
- _benedict 4y ago
  No, you wrote that you have to write and wait several minutes for data to replicate. This is straight-up false. Yes, if you mix consistency levels incorrectly you will do it wrong and maybe get stale data, but that is a different criticism. I agree that it is easy for unsophisticated users to incorrectly use consistency levels in complex topologies, and I hope we will introduce mechanisms to prevent users making such mistakes in future. But that was not your claim, and in my experience users do understand consistency levels just fine. There are lots of valid criticisms to point at various use cases with Cassandra, but this was just incorrect.
  - skyde 4y ago
    it’s worse than waiting actually. Because client A might write value « 1 » and after client B write value « 2 ». Then client C can read Value 2 for 5 minute and then cassandra internal read-repair eventually put value 1 on all replica and value 2 is lost forever.

curiousDog 4y ago

And not to mention, for 90% of the use cases out there Postgres or MySql on aurora or whatever will be more than enough

JensRantil 4y ago
More like 99,9% of use cases. But yeah.
scarface74 4y ago
If I know what my access patterns are, DynamoDB is often the better choice.

jhgg 4y ago

What is wrong with CQL? The only gotcha I am aware of is UPDATE/INSERT really just meaning UPSERT.

mnd999 4y ago
It encourages you to think you can use secondary indexes to make different types of query over the same data like SQL, which you mostly can’t.
Luker88 4y ago
In no particular order: * LWT (Lightweight Transactions) are nothing like what you expect when you head "transactions". LWT are single row only. * error handling is asinine. E.g: A quorum insert fails. is the data there or not? Maybe! * material views. Added, removed, readded, deprecated. * big limits on secondary indexes which only work inside each single partition * different format of timestamp for insert and select * BATCH is atomic, but not isolated. ...In general CQL feels like a hack. In classic SQL you understand early that you can compose queries, subqueries and such. CQL leads to you to believe that this is possible, only for you to find out that it isn't. The docs are full of stuff like "You can do this query, except in this case, unless this other case in which you can again, but not in this other". It's just exception on top of exception. Even the documentation does not list all exceptions (I managed to find a couple in my last work).
- jlokier 4y ago
  > A quorum insert fails. is the data there or not? Maybe! How is this point fundamentally different from SQL? In SQL, after you send COMMIT to a remote database and get back a network error, you don't know if your INSERTed data is committed or not. You'll have to reconnect and check for it.
  - Luker88 4y ago
    In CQL queries can fail even if your connection is still up and running. In this case it means that it could not write to 3 nodes, but it is not telling you if it did not write anything or at least once. Writing just once means that the data will be replicated. So if only 1 of the 3 nodes is up, it will still write there and then return error, even if the coordinator knows that the other nodes are not up. Queries have a maximum running time (only configurable at the database level if I remember correctly), and if your write exceeds this, it returns error. The replication still goes on and the data is still there. Network errors are different category, since you might not even know if your command was sent or not. Cassandra kind of sidesteps the network errors though since the client connects to multiple coordinators in the cluster.

YZF 4y ago

What you're saying applies more or less no any NoSQL database. Unusable for your use case is a bit of a big statement. There's plenty of cases where you don't need the power of relational databases and relational databases can't/don't typically scale horizontally the way NoSQL can. And really NoSQL can because of its attributes. If all you need is key/value lookups or time series, you're not looking for any joins or random reads that don't agree with the native ordering/partitioning then it's hard to beat NoSQL scalability/ingest-wise.

eusto 4y ago
I agree, but I think the part that's not stated is that when picking such a technology, you basically commit to not needing joins or random reads. If you do, as the application evolves, you're really out of luck. Generally, that commitment cannot be made upfront. Typically, solutions like time-series or document databases are deployed alongside other components in a much larger system. If you are looking for a single database solution, then maybe NoSQL is not for you
Luker88 4y ago
True, which is why I said that cassandra has its uses. The problem is that the documentation and the language tries really hard to sell you something else, and I have seen multiple people wasting way too much time on this.
skyde 4y ago
why settle for noSQL when we can use newSQL? Cockroachdb, CitusDB, TiDB,Yugabyte …
- YZF 4y ago
  I don't think those have the same performance characteristics. At the end of the day you can't have your cake and eat it. You can't keep large scale indices that facilitate fast joins without writing them out to disk and you can't keep them in sync with your data without transactionality/atomicity. This boils down to tradeoffs in terms of locking/consensus/scale/indices/data locality. That said if you're going to be emulating this over noSQL anyways then there's an argument for using something that does it for you (assuming it meets your requirements).