10 ms·

ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

114 points by haint 11y ago

prohor 11y ago
The license is Affero GPL, which means you need to open-source your code even if you use it for a service. "Traditional" GPL was effective only while redistributing. That means you would need to go for commercial license whenever you build a service on it. Which in fact is a fair approach for a business model when there is a company behind an open source project. Especially that this time there is no lock-in. You could always come back to Cassandra.
- mappu 11y ago
  The virality doesn't cross the database interface layer. Modifications to the database software must be shared, yes, but your client application is outside the reach of the AGPL and can remain proprietary.
philipov 11y ago
Wow, that autoadvancing website is a deal breaker.
- eip 11y ago
  NoScript FTW
- dmarti 11y ago
  ScyllaDB web person here. If I made it so that you could block one script and get a home page without the horizontally scrolling thingy (but have all the other JS stuff work including syntax highlighting and graphs), would you come back? ( dmarti@scylladb.com )
nattaylor 11y ago
>The Scylla design, right, is based on a modern shared-nothing approach. Scylla runs multiple engines, one per core, each with its own memory, CPU and multi-queue NIC. We can easily reach 1 million CQL operations on a single commodity server. In addition, Scylla targets consistent low latency, under 1 ms, for inserts, deletes, and reads. Interesting. From: http://www.scylladb.com/technology/architecture/ http://www.scylladb.com/technology/architecture/
- whalesalad 11y ago
  So on virtualized hardware, namely AWS, I'm sure the benchmarks won't be so magnificent. Needing a dedicated nic per core is a big deal unless you're at a pretty large scale.
  - vitalyd 11y ago
    Not dedicated NIC per core, but multi-queue NIC having its queues serviced by dedicated cores.
  - acconsta 11y ago
    How big do you have to be to lease hardware?
    - bwohlergo 11y ago
      leasing is often cheaper than the alternatives
      - acconsta 11y ago
        That's what I thought. I think deploying something like this would be easier than the parent comment suggests.
  - jandrewrogers 11y ago
    A modern Ethernet chipset has a large number of independent hardware queues. These can be assigned to VMs for direct access to the NIC, bypassing the hypervisor. AWS, since you used that example, offers instances with this type of direct bypass. Just to pull an example from memory, the ubiquitous Intel 82599 10GbE NIC silicon has up to 128 TX and RX queues in hardware. IIRC, these are bundled in pairs for direct access in virtualized environments, so in principle you could have 64 virtual cores each with their own dedicated physical hardware queue. This is almost certainly what they were talking about. That is the whole point of this feature in Ethernet silicon; it gives cores (virtual or physical) dedicate network hardware off a single NIC.
domlebo70 11y ago
Literally zero mention that in the event of a network partition they will just drop messages on the floor. This is fine as a cache, perhaps replacing Redis... but as a Cassandra replacement this is pretty scary
- nnx 11y ago
  Source?
- vitalyd 11y ago
  Where did you get this from? I hope that's not a conclusion from the benchmark doing single server load testing.
dschiptsov 11y ago
Finally, back to sanity of great old-school products, like Informix, by dropping Java (the whole scam) for C++14 and by paying attention to details of an underlying OS (again). Same trend, by the way, is in Android development.
- _Codemonkeyism 11y ago
  10x speedup (same algorithms, same architecture) replacing Java with C++ is not possible (~2x at max). One of the latest benchmarks I've seen is "Comparison of Programming Languages in Economics" [1] for code without any IO just number crunching, has a 1.91 to 2.69 speedup of using C++ compared to Java. So any code involving IO is going to be slower. Replacing bad Java code with excellent machine aligned C++ a 10x speedup is possible. [1] https://github.com/jesusfv/Comparison-Programming-Languages-Economics https://github.com/jesusfv/Comparison-Programming-Languages-...
  - cbsmith 11y ago
    It's particularly flawed given: a) IO is such a large portion of the problem b) Hypertable isn't just way, way faster.
    - acconsta 11y ago
      Are there any recent comparative benchmarks of hypertable? I looked around but couldn't find any.
      - cbsmith 11y ago
        Not that I know of. Anecdotally though, it has some advantages, but doesn't exactly crush the competition.
    - _Codemonkeyism 11y ago
      "a) IO is such a large portion of the problem" I'm very interested in a cluster benchmark therefor, say 10 servers, as Cassandra claims to scale very well. With a cluster IO has a higher performance impact than one server with local RAID IO.
jandrewrogers 11y ago
Very nice. Broadly speaking, this is the correct style of architecture for a database engine on modern hardware. It is vastly more efficient in terms of throughput than the more traditional architectures common in open source. It also lends itself to elegant, compact implementations. I've been using similar architectures for several years now. While I have not benchmarked their particular implementation, my first-hand experience is that these types of implementations are always at least 10x faster on the same hardware than a nominally equivalent open source database engine, so the performance claim is completely believable. One of my longstanding criticisms of open source data infrastructure has always been the very poor operational efficiency at a basic architectural level; many closed source companies have made a good business arbitraging the gross efficiency differences.
- acconsta 11y ago
  Agreed, but which architectural features are you referring to?
  - z92 11y ago
    Basically ditched Java in favor of C++, and used a C++ framework called Seastar. "The Scylla design, right, is based on a modern shared-nothing approach. Scylla runs multiple engines, one per core, each with its own memory, CPU and multi-queue NIC." http://www.scylladb.com/technology/architecture/ http://www.scylladb.com/technology/architecture/
  - jandrewrogers 11y ago
    Over the last decade, the distributed system nature of modern server hardware internals has become painfully evident in how software architectures scale on a single machine. The traditional approaches -- multithreading, locking, lock-free structures, etc -- are all forms of coordination and agreement in a distributed system, with the attendant scalability problems if not used very carefully. At some point several years ago, a few people noticed that if you attack the problem of scalable distribution within a single server the same way you would in large distributed systems (e.g. shared nothing architectures) that you could realize huge performance increases on a single machine. The caveat is that the software architectures look unorthodox. The general model looks like this: - one process per core, each locked to a single core - use locked local RAM only (effectively limiting NUMA) - direct dedicated network queue (bypass kernel) - direct storage I/O (bypass kernel) If you do it right, you minimize the amount of silicon that is shared between processes which has surprisingly large performance benefits. Linux has facilities that make this relatively straightforward too. As a consequence, adjacent cores on the same CPU have only marginally more interaction with each other than cores on different machines entirely. Treating a single server as a distributed cluster of 1-core machines, and writing the software in such a way that the operating system behavior reflects that model to the extent possible, is a great architecture for extreme performance but you rarely see it outside of closed source software. As a corollary, garbage-collected languages do not work for this at all.
finalight 11y ago
will this be the next docker in the nosql database?
- ketralnis 11y ago
  What does this mean?
  - mappu 11y ago
    It's nonsensical buzzwords. I guess the poster's underlying question is "will this database become hyped as the Next Big Thing"
- superpaul 11y ago
  I don't even see any connection how could this be the next docker in the nosql world... in other words i didn't get at all what you mean...
mappu 11y ago
Numbers look great, but so do /dev/null's. What guarantees does it make? Has it been through Jepsen yet?
- acconsta 11y ago
  Honestly, Cassandra's Jepsen didn't set a high bar: https://aphyr.com/posts/294-call-me-maybe-cassandra/ https://aphyr.com/posts/294-call-me-maybe-cassandra/
  - cbsmith 11y ago
    Except that problem has been largely addressed now.
    - saurik 11y ago
      I really really really want to see Aphyr attack the patched version to see if he thinks the fix actually worked.
      - gegtik 11y ago
        Datastax is presenting on the topic at their Summit on thursday http://cassandrasummit-datastax.com/agenda/testing-cassandra-guarantees-under-diverse-failure-modes-with-jepsen/ http://cassandrasummit-datastax.com/agenda/testing-cassandra...
    - acconsta 11y ago
      Right, I should add that it was two years ago. My point is that the age of a project has nothing to do with the correctness of its Paxos implementation.
      - cbsmith 11y ago
        Jepsen's finding wasn't that there was a bug in Paxos. It was in how it handled conflicts.
acconsta 11y ago
It's exciting to finally see this. Cassandra's strengths were in its distributed architecture (no master, tunable consistency, etc.). The database engine itself has always been a bit of a mess (https://issues.apache.org/jira/browse/CASSANDRA-8099 https://issues.apache.org/jira/browse/CASSANDRA-8099).
mbfg 11y ago
Wait, did I read that right? the test was with (1) one server? What's the point of that? Smells like a cooked up test.
- glommer 11y ago
  The point of that is to show how efficient a node can be, because that is what is replaced. All the external facing things for scylla is the same as Cassandra. That includes all the ring stuff and all network protocols. So you should expect similar cluster behavior.
  - kcw39217 11y ago
    Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure....... There is nothing commodity about a server with 128GB RAM. When you introduce other nodes, you get chatter and network traffic....
    - bpicolo 11y ago
      > There is nothing commodity about a server with 128GB RAM. Except I can launch higher than that on EC2, so that's not fact.
    - sciurus 11y ago
      Nothing commodity about a server with 128GB RAM? At list price, you can configure one of dell's entry-level servers with 128GB of RAM for less than $3,500. http://www.dell.com/us/business/p/poweredge-rack-servers http://www.dell.com/us/business/p/poweredge-rack-servers
      - mbfg 11y ago
        Dell's servers you point to do not have 48 logical cores, either. That cpu runs $2.2K by itself.
      - deleted 11y ago
jhugg 11y ago
"A Cassandra compatible NoSQL column store, at 1MM transactions/sec per server." Personal pet-peeve of mine. Using "TPS" or "Transactions/sec" to measure something that is in no way transactional. Maybe ops/sec, reads/sec, updates/sec, or something...
- JoeAltmaier 11y ago
  Add my pet peeve: not listing latency stats. Big Tables does millions of ops/sec but it can take 5(!) seconds to complete one. That's the stat that matters to customers.
  - lucindo 11y ago
    http://www.scylladb.com/technology/cassandra-vs-scylla-latency-benchmark/ http://www.scylladb.com/technology/cassandra-vs-scylla-laten...
    - JoeAltmaier 11y ago
      Thanks!
    - rakoo 11y ago
      > The test hardware configuration includes: > 1 DB server (Cassandra / Scylla) The whole point of Cassandra is to run a cluster of servers to handle load at scale with minimal friction instead of having to buy a big single machine or spend all your time/money trying to run a clustered RDBMS. This test doesn't measure the correct thing.
      - acconsta 11y ago
        Cassandra's performance scales linearly with the number of nodes though, so per-node performance definitely matters. Probably not 10x, but probably not 1x either.
deleted 11y ago
[deleted]
kcw39217 11y ago
which JVM did they use? What was the flags passed to the JVM?

114 points by haint 11y ago

The license is Affero GPL, which means you need to open-source your code even if you use it for a service. "Traditional" GPL was effective only while redistributing. That means you would need to go for commercial license whenever you build a service on it. Which in fact is a fair approach for a business model when there is a company behind an open source project. Especially that this time there is no lock-in. You could always come back to Cassandra.

mappu 11y ago
The virality doesn't cross the database interface layer. Modifications to the database software must be shared, yes, but your client application is outside the reach of the AGPL and can remain proprietary.

philipov 11y ago

Wow, that autoadvancing website is a deal breaker.

eip 11y ago
NoScript FTW
dmarti 11y ago
ScyllaDB web person here. If I made it so that you could block one script and get a home page without the horizontally scrolling thingy (but have all the other JS stuff work including syntax highlighting and graphs), would you come back? ( dmarti@scylladb.com )

nattaylor 11y ago

>The Scylla design, right, is based on a modern shared-nothing approach. Scylla runs multiple engines, one per core, each with its own memory, CPU and multi-queue NIC. We can easily reach 1 million CQL operations on a single commodity server. In addition, Scylla targets consistent low latency, under 1 ms, for inserts, deletes, and reads. Interesting. From: http://www.scylladb.com/technology/architecture/ http://www.scylladb.com/technology/architecture/

whalesalad 11y ago
So on virtualized hardware, namely AWS, I'm sure the benchmarks won't be so magnificent. Needing a dedicated nic per core is a big deal unless you're at a pretty large scale.
- vitalyd 11y ago
  Not dedicated NIC per core, but multi-queue NIC having its queues serviced by dedicated cores.
- acconsta 11y ago
  How big do you have to be to lease hardware?
  - bwohlergo 11y ago
    leasing is often cheaper than the alternatives
    - acconsta 11y ago
      That's what I thought. I think deploying something like this would be easier than the parent comment suggests.
- jandrewrogers 11y ago
  A modern Ethernet chipset has a large number of independent hardware queues. These can be assigned to VMs for direct access to the NIC, bypassing the hypervisor. AWS, since you used that example, offers instances with this type of direct bypass. Just to pull an example from memory, the ubiquitous Intel 82599 10GbE NIC silicon has up to 128 TX and RX queues in hardware. IIRC, these are bundled in pairs for direct access in virtualized environments, so in principle you could have 64 virtual cores each with their own dedicated physical hardware queue. This is almost certainly what they were talking about. That is the whole point of this feature in Ethernet silicon; it gives cores (virtual or physical) dedicate network hardware off a single NIC.

domlebo70 11y ago

Literally zero mention that in the event of a network partition they will just drop messages on the floor. This is fine as a cache, perhaps replacing Redis... but as a Cassandra replacement this is pretty scary

nnx 11y ago
Source?
vitalyd 11y ago
Where did you get this from? I hope that's not a conclusion from the benchmark doing single server load testing.

dschiptsov 11y ago

Finally, back to sanity of great old-school products, like Informix, by dropping Java (the whole scam) for C++14 and by paying attention to details of an underlying OS (again). Same trend, by the way, is in Android development.

_Codemonkeyism 11y ago
10x speedup (same algorithms, same architecture) replacing Java with C++ is not possible (~2x at max). One of the latest benchmarks I've seen is "Comparison of Programming Languages in Economics" [1] for code without any IO just number crunching, has a 1.91 to 2.69 speedup of using C++ compared to Java. So any code involving IO is going to be slower. Replacing bad Java code with excellent machine aligned C++ a 10x speedup is possible. [1] https://github.com/jesusfv/Comparison-Programming-Languages-Economics https://github.com/jesusfv/Comparison-Programming-Languages-...
- cbsmith 11y ago
  It's particularly flawed given: a) IO is such a large portion of the problem b) Hypertable isn't just way, way faster.
  - acconsta 11y ago
    Are there any recent comparative benchmarks of hypertable? I looked around but couldn't find any.
    - cbsmith 11y ago
      Not that I know of. Anecdotally though, it has some advantages, but doesn't exactly crush the competition.
  - _Codemonkeyism 11y ago
    "a) IO is such a large portion of the problem" I'm very interested in a cluster benchmark therefor, say 10 servers, as Cassandra claims to scale very well. With a cluster IO has a higher performance impact than one server with local RAID IO.

jandrewrogers 11y ago

Very nice. Broadly speaking, this is the correct style of architecture for a database engine on modern hardware. It is vastly more efficient in terms of throughput than the more traditional architectures common in open source. It also lends itself to elegant, compact implementations. I've been using similar architectures for several years now. While I have not benchmarked their particular implementation, my first-hand experience is that these types of implementations are always at least 10x faster on the same hardware than a nominally equivalent open source database engine, so the performance claim is completely believable. One of my longstanding criticisms of open source data infrastructure has always been the very poor operational efficiency at a basic architectural level; many closed source companies have made a good business arbitraging the gross efficiency differences.

acconsta 11y ago
Agreed, but which architectural features are you referring to?
- z92 11y ago
  Basically ditched Java in favor of C++, and used a C++ framework called Seastar. "The Scylla design, right, is based on a modern shared-nothing approach. Scylla runs multiple engines, one per core, each with its own memory, CPU and multi-queue NIC." http://www.scylladb.com/technology/architecture/ http://www.scylladb.com/technology/architecture/
- jandrewrogers 11y ago
  Over the last decade, the distributed system nature of modern server hardware internals has become painfully evident in how software architectures scale on a single machine. The traditional approaches -- multithreading, locking, lock-free structures, etc -- are all forms of coordination and agreement in a distributed system, with the attendant scalability problems if not used very carefully. At some point several years ago, a few people noticed that if you attack the problem of scalable distribution within a single server the same way you would in large distributed systems (e.g. shared nothing architectures) that you could realize huge performance increases on a single machine. The caveat is that the software architectures look unorthodox. The general model looks like this: - one process per core, each locked to a single core - use locked local RAM only (effectively limiting NUMA) - direct dedicated network queue (bypass kernel) - direct storage I/O (bypass kernel) If you do it right, you minimize the amount of silicon that is shared between processes which has surprisingly large performance benefits. Linux has facilities that make this relatively straightforward too. As a consequence, adjacent cores on the same CPU have only marginally more interaction with each other than cores on different machines entirely. Treating a single server as a distributed cluster of 1-core machines, and writing the software in such a way that the operating system behavior reflects that model to the extent possible, is a great architecture for extreme performance but you rarely see it outside of closed source software. As a corollary, garbage-collected languages do not work for this at all.

finalight 11y ago

will this be the next docker in the nosql database?

ketralnis 11y ago
What does this mean?
- mappu 11y ago
  It's nonsensical buzzwords. I guess the poster's underlying question is "will this database become hyped as the Next Big Thing"
superpaul 11y ago
I don't even see any connection how could this be the next docker in the nosql world... in other words i didn't get at all what you mean...

mappu 11y ago

Numbers look great, but so do /dev/null's. What guarantees does it make? Has it been through Jepsen yet?

acconsta 11y ago
Honestly, Cassandra's Jepsen didn't set a high bar: https://aphyr.com/posts/294-call-me-maybe-cassandra/ https://aphyr.com/posts/294-call-me-maybe-cassandra/
- cbsmith 11y ago
  Except that problem has been largely addressed now.
  - saurik 11y ago
    I really really really want to see Aphyr attack the patched version to see if he thinks the fix actually worked.
    - gegtik 11y ago
      Datastax is presenting on the topic at their Summit on thursday http://cassandrasummit-datastax.com/agenda/testing-cassandra-guarantees-under-diverse-failure-modes-with-jepsen/ http://cassandrasummit-datastax.com/agenda/testing-cassandra...
  - acconsta 11y ago
    Right, I should add that it was two years ago. My point is that the age of a project has nothing to do with the correctness of its Paxos implementation.
    - cbsmith 11y ago
      Jepsen's finding wasn't that there was a bug in Paxos. It was in how it handled conflicts.

acconsta 11y ago

It's exciting to finally see this. Cassandra's strengths were in its distributed architecture (no master, tunable consistency, etc.). The database engine itself has always been a bit of a mess (https://issues.apache.org/jira/browse/CASSANDRA-8099 https://issues.apache.org/jira/browse/CASSANDRA-8099).

mbfg 11y ago

Wait, did I read that right? the test was with (1) one server? What's the point of that? Smells like a cooked up test.

glommer 11y ago
The point of that is to show how efficient a node can be, because that is what is replaced. All the external facing things for scylla is the same as Cassandra. That includes all the ring stuff and all network protocols. So you should expect similar cluster behavior.
- kcw39217 11y ago
  Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure....... There is nothing commodity about a server with 128GB RAM. When you introduce other nodes, you get chatter and network traffic....
  - bpicolo 11y ago
    > There is nothing commodity about a server with 128GB RAM. Except I can launch higher than that on EC2, so that's not fact.
  - sciurus 11y ago
    Nothing commodity about a server with 128GB RAM? At list price, you can configure one of dell's entry-level servers with 128GB of RAM for less than $3,500. http://www.dell.com/us/business/p/poweredge-rack-servers http://www.dell.com/us/business/p/poweredge-rack-servers
    - mbfg 11y ago
      Dell's servers you point to do not have 48 logical cores, either. That cpu runs $2.2K by itself.
    - deleted 11y ago

jhugg 11y ago

"A Cassandra compatible NoSQL column store, at 1MM transactions/sec per server." Personal pet-peeve of mine. Using "TPS" or "Transactions/sec" to measure something that is in no way transactional. Maybe ops/sec, reads/sec, updates/sec, or something...

JoeAltmaier 11y ago
Add my pet peeve: not listing latency stats. Big Tables does millions of ops/sec but it can take 5(!) seconds to complete one. That's the stat that matters to customers.
- lucindo 11y ago
  http://www.scylladb.com/technology/cassandra-vs-scylla-latency-benchmark/ http://www.scylladb.com/technology/cassandra-vs-scylla-laten...
  - JoeAltmaier 11y ago
    Thanks!
  - rakoo 11y ago
    > The test hardware configuration includes: > 1 DB server (Cassandra / Scylla) The whole point of Cassandra is to run a cluster of servers to handle load at scale with minimal friction instead of having to buy a big single machine or spend all your time/money trying to run a clustered RDBMS. This test doesn't measure the correct thing.
    - acconsta 11y ago
      Cassandra's performance scales linearly with the number of nodes though, so per-node performance definitely matters. Probably not 10x, but probably not 1x either.

deleted 11y ago

[deleted]

kcw39217 11y ago

which JVM did they use? What was the flags passed to the JVM?