5 ms·

I haven't dealt with the ops side of running our clusters, and the details are not fresh in my mind, so I don't remember much about the storage engines. But we

by dvirsky 11y ago

I haven't dealt with the ops side of running our clusters, and the details are not fresh in my mind, so I don't remember much about the storage engines. But we encountered surprises on almost any front when taking C* to production.

frik 11y ago
Given your experience: Have you considered a NewSQL db like MariaDB with one of its modern storage engines? Also Facebook moved from their Cassandra to Hadoop/Hive a long time ago (and the MySQL side was never touched).
- dvirsky 11y ago
  We've moved most of our data to redis (which we were using way before cassandra), with a bit of classic MySQL for off-line data, and Redshift for analytics. Right now redis is not using cluster mode, but we'll switch to it eventually. It's a big move, but I find redis more predictable, and for our data sizes (10s of Gs), cheaper to maintain.
  - manishsharan 11y ago
    If you moved your data off Caasandra to Redis, I am going to assume you must be using a lot of Lua to maintain some relationships in data in Redis. How is the performance? I have been experimenting with Redis and my lua scripts are long and I have been wondering how they will fare under production level loads.
    - fweespeech 11y ago
      A Redis Hash is similar enough to Casandra's data structure I doubt they'd need that. row key = key column = field I know Lua is "an option" in Redis but I avoid it like the plague because under production loads, Redis hashes perform and are easy to maintain w/o Lua.
      - dvirsky 11y ago
        More or less this. Row = HASH key, sorted sets for secondary key indexing, and also for iterating primary key entries. I've built an abstraction layer on top of redis to automate all that and it works rather well.
  - chisleu 11y ago
    > We've moved most of our data to redis Redis is neet if your information fits in RAM, but that's not what Cassandra is for.
    - fweespeech 11y ago
      Eh...as long as you are ready and able to shard the dataset, fitting the data in RAM is quite possible for TB sized datasets. You can easily get commodity servers with 256GB of RAM per node x 10 shards. Cluster or not, as per your use case.
    - dvirsky 11y ago
      the dropping prices of RAM lead us to re-evaluate what we consider "data that fits in RAM". at 10Gs it's just more economic, and even if we go up one order of magnitude, the serving speed of redis still makes it economic for a lot of workloads.
  - lobster_johnson 11y ago
    Do you use Redis as the master data store? Are you using Redis Cluster, or unclustered? My concern with Redis is reliability. Redis Cluster has problems with consistency [1], whereas unclustered Redis -- well, it syncs to disk every 10 seconds or so, but even then I'm concerned that the reliability of its on-disk structures haven't been as battle-tested as, say, PostgreSQL. Or has it? [1] https://aphyr.com/posts/307-call-me-maybe-redis-redux https://aphyr.com/posts/307-call-me-maybe-redis-redux
    - fweespeech 11y ago
      > well, it syncs to disk every 10 seconds or so Yeah, about that: http://redis.io/topics/persistence http://redis.io/topics/persistence > Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes when no fsync is in progress.) but you can only lose one second worth of writes.
      - lobster_johnson 11y ago
        RDB is what I was thinking of. I didn't know they had added a write-ahead log. It seems you can combine RDB and AOF, which is nice.
        fweespeech 11y ago
        Yeah, you can. Its quite nice, the only real problem with it is the reliability of the clustering is subpar. If you are happy with a Master/Slave setup and occasionally having to deal with data loss due to Master failures [e.g. losing a second or two of data that wasn't replicated], it works nicely. Just realize I wouldn't use Redis as a long term persisting of data because of issues like this: https://muut.com/blog/news/april-2014-service-failure.html https://muut.com/blog/news/april-2014-service-failure.html

by dvirsky 11y ago

frik 11y ago

Given your experience: Have you considered a NewSQL db like MariaDB with one of its modern storage engines? Also Facebook moved from their Cassandra to Hadoop/Hive a long time ago (and the MySQL side was never touched).

dvirsky 11y ago
We've moved most of our data to redis (which we were using way before cassandra), with a bit of classic MySQL for off-line data, and Redshift for analytics. Right now redis is not using cluster mode, but we'll switch to it eventually. It's a big move, but I find redis more predictable, and for our data sizes (10s of Gs), cheaper to maintain.
- manishsharan 11y ago
  If you moved your data off Caasandra to Redis, I am going to assume you must be using a lot of Lua to maintain some relationships in data in Redis. How is the performance? I have been experimenting with Redis and my lua scripts are long and I have been wondering how they will fare under production level loads.
  - fweespeech 11y ago
    A Redis Hash is similar enough to Casandra's data structure I doubt they'd need that. row key = key column = field I know Lua is "an option" in Redis but I avoid it like the plague because under production loads, Redis hashes perform and are easy to maintain w/o Lua.
    - dvirsky 11y ago
      More or less this. Row = HASH key, sorted sets for secondary key indexing, and also for iterating primary key entries. I've built an abstraction layer on top of redis to automate all that and it works rather well.
- chisleu 11y ago
  > We've moved most of our data to redis Redis is neet if your information fits in RAM, but that's not what Cassandra is for.
  - fweespeech 11y ago
    Eh...as long as you are ready and able to shard the dataset, fitting the data in RAM is quite possible for TB sized datasets. You can easily get commodity servers with 256GB of RAM per node x 10 shards. Cluster or not, as per your use case.
  - dvirsky 11y ago
    the dropping prices of RAM lead us to re-evaluate what we consider "data that fits in RAM". at 10Gs it's just more economic, and even if we go up one order of magnitude, the serving speed of redis still makes it economic for a lot of workloads.
- lobster_johnson 11y ago
  Do you use Redis as the master data store? Are you using Redis Cluster, or unclustered? My concern with Redis is reliability. Redis Cluster has problems with consistency [1], whereas unclustered Redis -- well, it syncs to disk every 10 seconds or so, but even then I'm concerned that the reliability of its on-disk structures haven't been as battle-tested as, say, PostgreSQL. Or has it? [1] https://aphyr.com/posts/307-call-me-maybe-redis-redux https://aphyr.com/posts/307-call-me-maybe-redis-redux
  - fweespeech 11y ago
    > well, it syncs to disk every 10 seconds or so Yeah, about that: http://redis.io/topics/persistence http://redis.io/topics/persistence > Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes when no fsync is in progress.) but you can only lose one second worth of writes.
    - lobster_johnson 11y ago
      RDB is what I was thinking of. I didn't know they had added a write-ahead log. It seems you can combine RDB and AOF, which is nice.
      - fweespeech 11y ago
        Yeah, you can. Its quite nice, the only real problem with it is the reliability of the clustering is subpar. If you are happy with a Master/Slave setup and occasionally having to deal with data loss due to Master failures [e.g. losing a second or two of data that wasn't replicated], it works nicely. Just realize I wouldn't use Redis as a long term persisting of data because of issues like this: https://muut.com/blog/news/april-2014-service-failure.html https://muut.com/blog/news/april-2014-service-failure.html