9 ms·

The trouble with Cassandra as an object storage metadata database

84 points by jasim 6y ago

SMFloris 6y ago
A while back we explored the use of Cassandra. We wanted to keep some event related data there and for it to be relatively fast read-wise in order for us to do all sorts of reporting based on it. So we wrote allot and wanted to read fast. Seemed like a perfect store for our timestamped events, especially since we wanted to not even use deletes and has in-build record deduplication via its primary key. Turns out, it is not that perfect. Other than what the article described, I can also add: 1. It has a steep learning curve, but you do get to see the advantages while you learn it. But then, everything comes crumbling down. 2. The setup is a pain locally. Then it is a pain to set it up in prod and manage it. The tooling itself feels very unfinished and basic. 3. No querying outside primary index on AWS Keyspace if you want it managed. Also, any managed variants are EXPENSIVE. I mean, every database is fast if you only query by the primary index so why pay extra? It is just not worth it. For example, we winded up using MongoDb and it turned out to be fast, scalable, had mature tooling and we can keep tons of event related metadata in it and it is easy to manage and doesn't cost a fortune.
- killtimeatwork 6y ago
  I think Cassandra is a better fit for interactive use cases, not for reporting. Also, basically it's super heavy duty and it should start to shine when you're really serving entire Internet (on the scale of Reddit, Expedia etc.) and your Cassandra cluster is distributed across DCs across the world. I haven't really worked in this space for a couple of years so I don't know if the cloud offerings have already completely matched Cassandra's features and robustness.
  - AtlasBarfed 6y ago
    Dynamo allegedly has global replication now. Of course, I have been totally unable to determine how they merge rows/partitions without cell timestamps. It's a black box. I was just on a "Keyspaces" meeting where the sales dude basically described dynamo billing, dynamo provisioning, feature shortfalls obviously due to dynamo, but would not admit it was dynamo. It was bizarre.
- deleted 6y ago
  [deleted]
- snapetom 6y ago
  We initially looked at Cassandra. We liked its use case, we liked its scalability. However we also ran into maintenance and setup pains. We ended up going with ScyllaDB, which is a drop in replacement for Cassandra. It’s written in C. Much easier in resource demands and we didn’t have to deal with Zookeeper directly.
  - bbromhead 6y ago
    Cassandra doesn't require Zookeeper
    - AtlasBarfed 6y ago
      And Scylladb may have a better thread-per-core model and no GC pauses, it basically has the exact same management challenges as Cassandra. The parent comment almost seems generated by AI.
      - snapetom 6y ago
        Dumb statements all around, but clearly you've never been in a Cassandra environment vs Scylla. Scylla is far, far more reliable, easier to get up and running, and required a bit less supervision than Cassandra.
    - snapetom 6y ago
      Ah, good to know. Our admins set up Zookeeper with Cassandra, so I had always assumed it was part of the deal.
snapetom 6y ago
Meh. About 80% of that article discusses known limitations of Cassandra, which isn't specific to a use case of object store metadata storage. In the little that it actually does specifically talk about that use case, if your object store reflects the Cassandra limitations (flat, infrequent mutability), I don't see why Cassandra would be a bad choice.
toolslive 6y ago
Yes. the problem with "eventual consistency" is that "eventually" never happens about 1 in a million times, and with millions of storage objects to manage you can't have that. So what's the alternative as a metadata store for object storage ? a consensus algorithm (paxos, raft, mencius, ...) on top of local key value stores.
- deleted 6y ago
  [deleted]
francoisdevlin 6y ago
A tangent - I love using minio in dev & test for their s3 simulators. Being able to throw away a bucket and start from scratch, and having everything self contained in my docker-compose command is a real blessing. Has anyone ever used min.io for production stuff? What are the pros & cons over vanilla s3?
- mildbyte 6y ago
  We use it in anger at Splitgraph [0] to let people upload/download datasets and Postgres table snapshots (instead of storing them directly in S3). Pros: * Less platform-dependent. By self-managing it, we can also run deploys to GCP / Azure / Scaleway / other providers without writing a separate adapter for e.g. Azure Blob Storage. * Python API [1] much more pleasant to use than boto3 (and can speak to normal S3). It doesn't do everything that boto3 does, but it supports everything we need (e.g. pre-signed URLs). * minio server itself supports a large chunk of S3's functionality (e.g. SELECT API / AssumeRole / bucket versioning) * Don't pay per request and for egress: this was a big deal since people might want to download large amounts of data from us (or make a bunch of small requests to download/upload a subset of data). Cons: * Have to manage own infrastructure. We run it on managed VMs so it's semi-managed, but we still have to provision block storage, set up backup policies etc. * In a similar vein, scaling and availability all have to be DIY [2]. We haven't run into situations yet where Minio would be the bottleneck, but it might be something to keep in mind. * Obviously not as seamless: you don't get things like Glacier or integration with other IAM. [0] https://www.splitgraph.com/ https://www.splitgraph.com/ [1] https://github.com/minio/minio-py https://github.com/minio/minio-py [2] https://docs.min.io/docs/distributed-minio-quickstart-guide.html https://docs.min.io/docs/distributed-minio-quickstart-guide....
- ddorian43 6y ago
  It doesn't work for a lot of small files scenario, there is too much overhead. You can't +1 new machines, you need to add whole clusters. If you ask too many questions you get invited for a premium subscription. You can't add +1 server, you need to add in clusters. There is no public big production scenario of 10PB+. There is no community of big production users (like there is with ceph).
- 6y ago
KaiserPro 6y ago
I would have hoped its obvious. Your metadata database needs to be the fastest and most reliable store out of everything. It can't be eventually consistent without partitioning your datastore. Even then you'll end up partitioning your data neatly into the same failure zone. Cassandra has basically one usecase: high volume writes, with a few batch reads. Cassandra is not really optimised for high reads. Most of the time postgres will do fine.
- adev_ 6y ago
  > Most of the time postgres will do fine. For something that scale horizontally, I would probably recommend to use something like FoundationDB for this use case [^1]. A transactional Key-Value store is exactly what you want for this kind of use case. [^1]: https://apple.github.io/foundationdb/ https://apple.github.io/foundationdb/
  - ddorian43 6y ago
    Yes now you are maintaining 5+ types of services for foundationdb. While you can vertically-scale a single PostgreSQL (with failover/ha) server to 50+TB of NVME and grow until you can throw other engineers at the problem. Example: wasabi.com uses mysql for metadata.
    - adev_ 6y ago
      I'm curious on you describing the 5+ types of services fdb requires. In my world it requires 1.
      - ddorian43 6y ago
        Heading to the docs again, looks like they were consolidated. I remember you had to run coordinator nodes, data nodes, log storage nodes, cluster controller etc.
- athenot 6y ago
  > Cassandra has basically one usecase: high volume writes, with a few batch reads. Yes. It's also a good choice for horizontal scaling, but only if the other DBs won't do.
segmondy 6y ago
If you really want to see and get hints on what you can do with Cassandra, head over to Netflix tech blogs. They use Casandra extensively.
- mrwd021 6y ago
  really? what do they use Cassandra for?
  - CraigJPerry 6y ago
    Back in 2014 when i last used Cassandra, Netflix were using it for some kind of metrics store i believe. They also published the nicer-to-use client back then (astyanx or something it was called - nicer API than Hector). My memory is letting me down here on specifics - in my defence it's been a long time since i answered questions on stack overflow about Cassandra. I absolutely loved working with the product even if the sharp edges cut me a few times (ultimately my own fault not Cassandra's).
  - redis_mlc 6y ago
    Netflix uses it for everything that you'd use Postgres for but: - they have Cassandra committers (most end up going to Apple iCloud) - they have mgmt. support to make it work - they are/were on tokens for years, not vtokens, since their internal tools were token-based (They started with Oracle Enterprise (and a little MySQL), tried Mongo briefly, then decided to make Cassandra work for them.) Source: original Netflix Cassandra team member.
icegreentea2 6y ago
What does "Bottom line. Write your metadata atomically with your object. Never separate them." mean from the perspective of picking a system?
- eternalban 6y ago
  If multiple distinct writes to DB must be atomic (as in a transaction), an AP system is likely a poor choice. AP systems are designed for 'eventual consistency'. If EC is a poor fit for 'consistent object data and metadata' in your application, do not pick an AP system. An AP system may be a reasonable choice for an ODB if your added layer of transactionality is exploiting a domain specific characteristics that allows for fast-path, efficient, distributed transactions on top of a bare object level AP system (such as Cassandra). Since this is all very niche and technically demanding, it is almost always a better choice for the general team to choose a mature CP database as backing for metadata-rich objects and/or domains that demand transactional support for systems of record.
KingOfCoders 6y ago
The one thing about Cassandra: Most probably you don't need it because you do not have the write performance needs and will not have them in the next five years. Scaling early is the death of many startups [1] Postgres with a time database will be sufficient for most needs. [1] https://www.duetpartners.com/why-is-premature-scaling-still-the-biggest-startup-killer/ https://www.duetpartners.com/why-is-premature-scaling-still-...
u678u 6y ago
Its frustrating when it doesn't recommend something better. Every DB has tradeoffs if you choose something over Cassandra you have to give up something else.
ddorian43 6y ago
Storing metadata together with data will just make it harder,slower to query metadata (since it will reside in hdd in most cases). You may think "it will be cached in ram, because it's small", yes, but then you'll end up querying many nodes just for metadata queries. Yes it's nicer to manage only 1 system, but in big scenarios it's probably better to separate. You can have 50+TB of nvme in 1 server, so your metadata layer probably doesn't need to horizontally scale. Imagine if you lose some objects (because you lost some replicas). You won't even know WHICH objects you lost, because the metadata is gone together with the data. Having separate, you can add a 5-replicas to metadata to be even safer compared to the usual 3 replicas.
- anonymousDan 6y ago
  What kind of 'metadata' are they referring to here? Can you give some concrete examples?
  - ddorian43 6y ago
    Metadata of the object storage. In the simplest is a table of <bucket_id, filename, location_of_object_in_storage, file_size, object_metadata>
kerblang 6y ago
The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition key column is "day_of_week", then you have 7 partitions. New problem: Your partitions need to be < 300 MB or cassandra falls over dead. In fact you'll soon realize that over the lifetime of your table, keeping those partitions under control may end up forcing you to use various date tricks like putting year/month/day in as "artificial" partition keys. Of course if you put everything in the partition keys instead of clustering keys, let me note again that you have to put each partition key column in every where clause, in which case you may have trouble querying for batches of data. Furthermore, when you do put clustering columns in your where clause, they have to be in order declared; so if your CC's are a, b, and c, then your where clause can use (a), (a,b), or (a,b,c); but if it has b, it has to have a, and if it has c, it has to have a&b. This is because storage is hierarchical. (Same rules for "order by", btw) (and no there's no "group by") This is when you start realizing: Oh, you mean it's really not like "SQL without joins". No, not even close.
- jimbokun 6y ago
  Best of summary of the Cassandra data model I've seen: HashMap<PartitionKey, OrderedSet<ClusterKey>> So you can lookup data by the ParititionKey, then perform range queries on the ClusterKey to filter data belonging to that partition. If your access patterns look like that, Cassandra could be a great fit. For anything else, you probably want to consider a different data store.
  - koolba 6y ago
    Ha! That's pretty succinct. Though HashMap should be just "Map" and the value of the top level map is not a set, it's another map: Map<PartitionKey, SortedMap<ClusterKey, Record>>
  - deleted 6y ago
    [deleted]
eternalban 6y ago
So these guys have a storage system on top of Amazon S3 with a distributed (timestamp based) RW mutex locking, that uses NTP (github.com/beevik/ntp). That seems to be about it. https://github.com/minio/minio/blob/master/pkg/bucket/object/lock/lock.go https://github.com/minio/minio/blob/master/pkg/bucket/object... https://github.com/minio/minio/blob/master/pkg/dsync/drwmutex.go https://github.com/minio/minio/blob/master/pkg/dsync/drwmute... And this is their test: https://github.com/minio/minio/blob/master/pkg/dsync/dsync_test.go https://github.com/minio/minio/blob/master/pkg/dsync/dsync_t... I just browsed quickly but it is littered with Amazon Simple Storage Service hardcoded bits like this: https://github.com/minio/minio/blob/master/pkg/bucket/object/lock/lock.go#L96 https://github.com/minio/minio/blob/master/pkg/bucket/object... There is not a single document that I can find that discusses the MinIO architecture. I guess "MinIO is a simple wrapper around S3 with a homegrown distributed state tech using NTP and it is 'fast!'" does not make for a sexy doc. The column "trouble with OSS distributed DBs" without Jepsen tests has probably already been written. There should also be one about "competitor's mature product bashing blogs are HN clickbait".
pid_0 6y ago
From an ops perspective, managing Cassandra is a bitch. Just use a managed service unless you have the money to hire a dedicated Cassandra expert. I’m so glad I’m done with ops
- XorNot 6y ago
  Also be wary with any company which "also does Cassandra" amongst a suite of products and isn't FAANG scale. Chances are they absolutely don't have the headcount to properly manage it and were just filling out the portfolio they wanted to offer.
- mthomassen 6y ago
  Just to provide a counter statement; at $work we definatively do not have the appropriate headcount to even start considering running our own distrubuted datastores. However, we _do_ use Cassandra. And while many persons shared the sentiment of 'it being to hard to maintain' when the first few projects started to incorporate Cassandra, the truth is; there never has been any failure or incident, or even technical hurdle related to the usage/maintenance of the -admittedly smallish- Cassandra cluster (for 2 years now). The costs of operating the cluster are ~5K/month (that's what our service provider charges us for 24/7 ops). I consider this a scam since averaged maintenance costs are perhaps ~1 hour per month. (normally peaks at 40.000 writes/sec, 500 reads/sec, 200GB)
  - ddorian43 6y ago
    Yes, but 200GB and those read/write rates of data can reside in a single server even entirely in-memory, it's not a big amount.
omginternets 6y ago
This is a restatement of the Cassandra docs paired with the usual misunderstanding of CAP. An AP system is not a choice of "I'll have availability and partition tolerance, please". It's the choice of availability given a partition. The whole point of CAP is that when partitions occur, there is a forced choice -- this is why it's incorrect to ask for a CA system. The Minio team are manifestly excellent engineers, but insightless posts that contain subtle misunderstandings of CAP do nothing to showcase that competence.
- acscott 6y ago
  Do you mean AP is the choice of losing consistency given a partition?
  - deleted 6y ago
    [deleted]
  - llarsson 6y ago
    That's how it is usually expressed, yes.
  - omginternets 6y ago
    Oh duh... You were pointing out a typo in my comment. I'm leaving my original reply below because what the hell. ~~~~ Yes, exactly. Think of it this way: you don't actually have any control over partitioning, therefore partitions are a given. So CAP is expressed as such: given a partition, choose between consistency and availability. EDIT: I personally find PACELC [0] less confusing, and more nuanced than CAP, but they basically say the same thing. [0] https://en.wikipedia.org/wiki/PACELC_theorem https://en.wikipedia.org/wiki/PACELC_theorem
anonymousDan 6y ago
I don't quite get what they are referring to when they talk about 'metadata' here. Are they talking mostly about something internal to the database, something to do with the schema, or some kind of additional data used to enrich a particular object?
- nemothekid 6y ago
  Minio is an s3 clone, so I assume you metadata they mean details about the object you uploaded. So if the actual data is a video file, the metadata would be the headers, filesize, what host/drive the file is stored on. Essentially when you ask for minio.io/my/video/file some service has to transform that to 192.168.0.1/customer/drive/ahe123.blob
  - anonymousDan 6y ago
    Ahhh,I see thanks.
gryn 6y ago
can anyone explain to me why someone would want to use cassandra when not handling internet scale stuff? the team am in uses it, but after many times asking why it was chosen since it seems a poor fit for our uses cases compared to a relational DB, the only justification I was given is a that the cluster is easier to maintain for our ops guys.
- ddorian43 6y ago
  Contrary to popular beliefs, programmers aren't very rational people, but still prone to emotional beliefs when choosing databases. Another reason is for "resume driven development".
jeff_vader 6y ago
One of my favourite tech overkills I've seen in my career is Cassandra database to store couple million records. Yup. Needless to say, it was later converted to PostgreSQL. The guy who did it is still advertising the fact on his online CV.
chrislusf 6y ago
I am working on SeaweedFS, which supports S3 API for object store, and can also use Cassandra as the metadata db. Cassandra has been performing well for most SeaweedFS users. The article listed many known Cassandra characteristics and cited them as limitations. However, it all depends on use cases. There are no file system that works for all cases, and not all of them needs ACID, CA vs CP, etc. The rest points are not convincing either. They are related to how to design the data structure better. Actually, SeaweedFS can use many other database/KV stores as the metadata DB. The list includes Redis, Cassandra, HBase, MySql, Postgres, Etcd, ElasticSearch, etc. https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores I did find one drawback for Cassandra as the metadata store though. One use case is that the customer uploaded a lot of zip files to one folder /tmp, unzip them, and then moved to a final folder. The rate is about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow. The solution was to use Redis for that /tmp folder, and still use Cassandra for the rest of folders. With Redis B-tree structure, the creation and deletion are cheap. So it is all depends on use cases.
- ddorian43 6y ago
  > With Redis B-tree structure, the creation and deletion are cheap. Pretty sure redis has no b-tree but hashmap. But redis has to do no compaction at all since it can directly delete the stuff on disk. > The rate is about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow. The problem here also is the gc-perdiod. Since the database has async-replication, it needs to keep deleted rows for some days so downed replicas can replicate. So they stay in disk/memory for some days actually and don't get compacted. While in a CP system they would be.
AtlasBarfed 6y ago
"I wanted to store something that could scale, but wanted to store and access my data in a way that didn't scale. Also, I don't understand CAP, distributed transactions, or distributed systems. I had a bad time." Somewhere else in the comments: "Yeah, we went with MongoDB."

84 points by jasim 6y ago

A while back we explored the use of Cassandra. We wanted to keep some event related data there and for it to be relatively fast read-wise in order for us to do all sorts of reporting based on it. So we wrote allot and wanted to read fast. Seemed like a perfect store for our timestamped events, especially since we wanted to not even use deletes and has in-build record deduplication via its primary key. Turns out, it is not that perfect. Other than what the article described, I can also add: 1. It has a steep learning curve, but you do get to see the advantages while you learn it. But then, everything comes crumbling down. 2. The setup is a pain locally. Then it is a pain to set it up in prod and manage it. The tooling itself feels very unfinished and basic. 3. No querying outside primary index on AWS Keyspace if you want it managed. Also, any managed variants are EXPENSIVE. I mean, every database is fast if you only query by the primary index so why pay extra? It is just not worth it. For example, we winded up using MongoDb and it turned out to be fast, scalable, had mature tooling and we can keep tons of event related metadata in it and it is easy to manage and doesn't cost a fortune.

killtimeatwork 6y ago
I think Cassandra is a better fit for interactive use cases, not for reporting. Also, basically it's super heavy duty and it should start to shine when you're really serving entire Internet (on the scale of Reddit, Expedia etc.) and your Cassandra cluster is distributed across DCs across the world. I haven't really worked in this space for a couple of years so I don't know if the cloud offerings have already completely matched Cassandra's features and robustness.
- AtlasBarfed 6y ago
  Dynamo allegedly has global replication now. Of course, I have been totally unable to determine how they merge rows/partitions without cell timestamps. It's a black box. I was just on a "Keyspaces" meeting where the sales dude basically described dynamo billing, dynamo provisioning, feature shortfalls obviously due to dynamo, but would not admit it was dynamo. It was bizarre.
deleted 6y ago
[deleted]
snapetom 6y ago
We initially looked at Cassandra. We liked its use case, we liked its scalability. However we also ran into maintenance and setup pains. We ended up going with ScyllaDB, which is a drop in replacement for Cassandra. It’s written in C. Much easier in resource demands and we didn’t have to deal with Zookeeper directly.
- bbromhead 6y ago
  Cassandra doesn't require Zookeeper
  - AtlasBarfed 6y ago
    And Scylladb may have a better thread-per-core model and no GC pauses, it basically has the exact same management challenges as Cassandra. The parent comment almost seems generated by AI.
    - snapetom 6y ago
      Dumb statements all around, but clearly you've never been in a Cassandra environment vs Scylla. Scylla is far, far more reliable, easier to get up and running, and required a bit less supervision than Cassandra.
  - snapetom 6y ago
    Ah, good to know. Our admins set up Zookeeper with Cassandra, so I had always assumed it was part of the deal.

snapetom 6y ago

Meh. About 80% of that article discusses known limitations of Cassandra, which isn't specific to a use case of object store metadata storage. In the little that it actually does specifically talk about that use case, if your object store reflects the Cassandra limitations (flat, infrequent mutability), I don't see why Cassandra would be a bad choice.

toolslive 6y ago

Yes. the problem with "eventual consistency" is that "eventually" never happens about 1 in a million times, and with millions of storage objects to manage you can't have that. So what's the alternative as a metadata store for object storage ? a consensus algorithm (paxos, raft, mencius, ...) on top of local key value stores.

deleted 6y ago
[deleted]

francoisdevlin 6y ago

A tangent - I love using minio in dev & test for their s3 simulators. Being able to throw away a bucket and start from scratch, and having everything self contained in my docker-compose command is a real blessing. Has anyone ever used min.io for production stuff? What are the pros & cons over vanilla s3?

mildbyte 6y ago
We use it in anger at Splitgraph [0] to let people upload/download datasets and Postgres table snapshots (instead of storing them directly in S3). Pros: * Less platform-dependent. By self-managing it, we can also run deploys to GCP / Azure / Scaleway / other providers without writing a separate adapter for e.g. Azure Blob Storage. * Python API [1] much more pleasant to use than boto3 (and can speak to normal S3). It doesn't do everything that boto3 does, but it supports everything we need (e.g. pre-signed URLs). * minio server itself supports a large chunk of S3's functionality (e.g. SELECT API / AssumeRole / bucket versioning) * Don't pay per request and for egress: this was a big deal since people might want to download large amounts of data from us (or make a bunch of small requests to download/upload a subset of data). Cons: * Have to manage own infrastructure. We run it on managed VMs so it's semi-managed, but we still have to provision block storage, set up backup policies etc. * In a similar vein, scaling and availability all have to be DIY [2]. We haven't run into situations yet where Minio would be the bottleneck, but it might be something to keep in mind. * Obviously not as seamless: you don't get things like Glacier or integration with other IAM. [0] https://www.splitgraph.com/ https://www.splitgraph.com/ [1] https://github.com/minio/minio-py https://github.com/minio/minio-py [2] https://docs.min.io/docs/distributed-minio-quickstart-guide.html https://docs.min.io/docs/distributed-minio-quickstart-guide....
ddorian43 6y ago
It doesn't work for a lot of small files scenario, there is too much overhead. You can't +1 new machines, you need to add whole clusters. If you ask too many questions you get invited for a premium subscription. You can't add +1 server, you need to add in clusters. There is no public big production scenario of 10PB+. There is no community of big production users (like there is with ceph).
6y ago

KaiserPro 6y ago

I would have hoped its obvious. Your metadata database needs to be the fastest and most reliable store out of everything. It can't be eventually consistent without partitioning your datastore. Even then you'll end up partitioning your data neatly into the same failure zone. Cassandra has basically one usecase: high volume writes, with a few batch reads. Cassandra is not really optimised for high reads. Most of the time postgres will do fine.

adev_ 6y ago
> Most of the time postgres will do fine. For something that scale horizontally, I would probably recommend to use something like FoundationDB for this use case [^1]. A transactional Key-Value store is exactly what you want for this kind of use case. [^1]: https://apple.github.io/foundationdb/ https://apple.github.io/foundationdb/
- ddorian43 6y ago
  Yes now you are maintaining 5+ types of services for foundationdb. While you can vertically-scale a single PostgreSQL (with failover/ha) server to 50+TB of NVME and grow until you can throw other engineers at the problem. Example: wasabi.com uses mysql for metadata.
  - adev_ 6y ago
    I'm curious on you describing the 5+ types of services fdb requires. In my world it requires 1.
    - ddorian43 6y ago
      Heading to the docs again, looks like they were consolidated. I remember you had to run coordinator nodes, data nodes, log storage nodes, cluster controller etc.
athenot 6y ago
> Cassandra has basically one usecase: high volume writes, with a few batch reads. Yes. It's also a good choice for horizontal scaling, but only if the other DBs won't do.

segmondy 6y ago

If you really want to see and get hints on what you can do with Cassandra, head over to Netflix tech blogs. They use Casandra extensively.

mrwd021 6y ago
really? what do they use Cassandra for?
- CraigJPerry 6y ago
  Back in 2014 when i last used Cassandra, Netflix were using it for some kind of metrics store i believe. They also published the nicer-to-use client back then (astyanx or something it was called - nicer API than Hector). My memory is letting me down here on specifics - in my defence it's been a long time since i answered questions on stack overflow about Cassandra. I absolutely loved working with the product even if the sharp edges cut me a few times (ultimately my own fault not Cassandra's).
- redis_mlc 6y ago
  Netflix uses it for everything that you'd use Postgres for but: - they have Cassandra committers (most end up going to Apple iCloud) - they have mgmt. support to make it work - they are/were on tokens for years, not vtokens, since their internal tools were token-based (They started with Oracle Enterprise (and a little MySQL), tried Mongo briefly, then decided to make Cassandra work for them.) Source: original Netflix Cassandra team member.

icegreentea2 6y ago

What does "Bottom line. Write your metadata atomically with your object. Never separate them." mean from the perspective of picking a system?

eternalban 6y ago
If multiple distinct writes to DB must be atomic (as in a transaction), an AP system is likely a poor choice. AP systems are designed for 'eventual consistency'. If EC is a poor fit for 'consistent object data and metadata' in your application, do not pick an AP system. An AP system may be a reasonable choice for an ODB if your added layer of transactionality is exploiting a domain specific characteristics that allows for fast-path, efficient, distributed transactions on top of a bare object level AP system (such as Cassandra). Since this is all very niche and technically demanding, it is almost always a better choice for the general team to choose a mature CP database as backing for metadata-rich objects and/or domains that demand transactional support for systems of record.

KingOfCoders 6y ago

The one thing about Cassandra: Most probably you don't need it because you do not have the write performance needs and will not have them in the next five years. Scaling early is the death of many startups [1] Postgres with a time database will be sufficient for most needs. [1] https://www.duetpartners.com/why-is-premature-scaling-still-the-biggest-startup-killer/ https://www.duetpartners.com/why-is-premature-scaling-still-...

u678u 6y ago

Its frustrating when it doesn't recommend something better. Every DB has tradeoffs if you choose something over Cassandra you have to give up something else.

ddorian43 6y ago

Storing metadata together with data will just make it harder,slower to query metadata (since it will reside in hdd in most cases). You may think "it will be cached in ram, because it's small", yes, but then you'll end up querying many nodes just for metadata queries. Yes it's nicer to manage only 1 system, but in big scenarios it's probably better to separate. You can have 50+TB of nvme in 1 server, so your metadata layer probably doesn't need to horizontally scale. Imagine if you lose some objects (because you lost some replicas). You won't even know WHICH objects you lost, because the metadata is gone together with the data. Having separate, you can add a 5-replicas to metadata to be even safer compared to the usual 3 replicas.

anonymousDan 6y ago
What kind of 'metadata' are they referring to here? Can you give some concrete examples?
- ddorian43 6y ago
  Metadata of the object storage. In the simplest is a table of <bucket_id, filename, location_of_object_in_storage, file_size, object_metadata>

kerblang 6y ago

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition key column is "day_of_week", then you have 7 partitions. New problem: Your partitions need to be < 300 MB or cassandra falls over dead. In fact you'll soon realize that over the lifetime of your table, keeping those partitions under control may end up forcing you to use various date tricks like putting year/month/day in as "artificial" partition keys. Of course if you put everything in the partition keys instead of clustering keys, let me note again that you have to put each partition key column in every where clause, in which case you may have trouble querying for batches of data. Furthermore, when you do put clustering columns in your where clause, they have to be in order declared; so if your CC's are a, b, and c, then your where clause can use (a), (a,b), or (a,b,c); but if it has b, it has to have a, and if it has c, it has to have a&b. This is because storage is hierarchical. (Same rules for "order by", btw) (and no there's no "group by") This is when you start realizing: Oh, you mean it's really not like "SQL without joins". No, not even close.

jimbokun 6y ago
Best of summary of the Cassandra data model I've seen: HashMap<PartitionKey, OrderedSet<ClusterKey>> So you can lookup data by the ParititionKey, then perform range queries on the ClusterKey to filter data belonging to that partition. If your access patterns look like that, Cassandra could be a great fit. For anything else, you probably want to consider a different data store.
- koolba 6y ago
  Ha! That's pretty succinct. Though HashMap should be just "Map" and the value of the top level map is not a set, it's another map: Map<PartitionKey, SortedMap<ClusterKey, Record>>
- deleted 6y ago
  [deleted]

eternalban 6y ago

So these guys have a storage system on top of Amazon S3 with a distributed (timestamp based) RW mutex locking, that uses NTP (github.com/beevik/ntp). That seems to be about it. https://github.com/minio/minio/blob/master/pkg/bucket/object/lock/lock.go https://github.com/minio/minio/blob/master/pkg/bucket/object... https://github.com/minio/minio/blob/master/pkg/dsync/drwmutex.go https://github.com/minio/minio/blob/master/pkg/dsync/drwmute... And this is their test: https://github.com/minio/minio/blob/master/pkg/dsync/dsync_test.go https://github.com/minio/minio/blob/master/pkg/dsync/dsync_t... I just browsed quickly but it is littered with Amazon Simple Storage Service hardcoded bits like this: https://github.com/minio/minio/blob/master/pkg/bucket/object/lock/lock.go#L96 https://github.com/minio/minio/blob/master/pkg/bucket/object... There is not a single document that I can find that discusses the MinIO architecture. I guess "MinIO is a simple wrapper around S3 with a homegrown distributed state tech using NTP and it is 'fast!'" does not make for a sexy doc. The column "trouble with OSS distributed DBs" without Jepsen tests has probably already been written. There should also be one about "competitor's mature product bashing blogs are HN clickbait".

pid_0 6y ago

From an ops perspective, managing Cassandra is a bitch. Just use a managed service unless you have the money to hire a dedicated Cassandra expert. I’m so glad I’m done with ops

XorNot 6y ago
Also be wary with any company which "also does Cassandra" amongst a suite of products and isn't FAANG scale. Chances are they absolutely don't have the headcount to properly manage it and were just filling out the portfolio they wanted to offer.
mthomassen 6y ago
Just to provide a counter statement; at $work we definatively do not have the appropriate headcount to even start considering running our own distrubuted datastores. However, we _do_ use Cassandra. And while many persons shared the sentiment of 'it being to hard to maintain' when the first few projects started to incorporate Cassandra, the truth is; there never has been any failure or incident, or even technical hurdle related to the usage/maintenance of the -admittedly smallish- Cassandra cluster (for 2 years now). The costs of operating the cluster are ~5K/month (that's what our service provider charges us for 24/7 ops). I consider this a scam since averaged maintenance costs are perhaps ~1 hour per month. (normally peaks at 40.000 writes/sec, 500 reads/sec, 200GB)
- ddorian43 6y ago
  Yes, but 200GB and those read/write rates of data can reside in a single server even entirely in-memory, it's not a big amount.

omginternets 6y ago

This is a restatement of the Cassandra docs paired with the usual misunderstanding of CAP. An AP system is not a choice of "I'll have availability and partition tolerance, please". It's the choice of availability given a partition. The whole point of CAP is that when partitions occur, there is a forced choice -- this is why it's incorrect to ask for a CA system. The Minio team are manifestly excellent engineers, but insightless posts that contain subtle misunderstandings of CAP do nothing to showcase that competence.

acscott 6y ago
Do you mean AP is the choice of losing consistency given a partition?
- deleted 6y ago
  [deleted]
- llarsson 6y ago
  That's how it is usually expressed, yes.
- omginternets 6y ago
  Oh duh... You were pointing out a typo in my comment. I'm leaving my original reply below because what the hell. ~~~~ Yes, exactly. Think of it this way: you don't actually have any control over partitioning, therefore partitions are a given. So CAP is expressed as such: given a partition, choose between consistency and availability. EDIT: I personally find PACELC [0] less confusing, and more nuanced than CAP, but they basically say the same thing. [0] https://en.wikipedia.org/wiki/PACELC_theorem https://en.wikipedia.org/wiki/PACELC_theorem

anonymousDan 6y ago

I don't quite get what they are referring to when they talk about 'metadata' here. Are they talking mostly about something internal to the database, something to do with the schema, or some kind of additional data used to enrich a particular object?

nemothekid 6y ago
Minio is an s3 clone, so I assume you metadata they mean details about the object you uploaded. So if the actual data is a video file, the metadata would be the headers, filesize, what host/drive the file is stored on. Essentially when you ask for minio.io/my/video/file some service has to transform that to 192.168.0.1/customer/drive/ahe123.blob
- anonymousDan 6y ago
  Ahhh,I see thanks.

gryn 6y ago

can anyone explain to me why someone would want to use cassandra when not handling internet scale stuff? the team am in uses it, but after many times asking why it was chosen since it seems a poor fit for our uses cases compared to a relational DB, the only justification I was given is a that the cluster is easier to maintain for our ops guys.

ddorian43 6y ago
Contrary to popular beliefs, programmers aren't very rational people, but still prone to emotional beliefs when choosing databases. Another reason is for "resume driven development".

jeff_vader 6y ago

One of my favourite tech overkills I've seen in my career is Cassandra database to store couple million records. Yup. Needless to say, it was later converted to PostgreSQL. The guy who did it is still advertising the fact on his online CV.

chrislusf 6y ago

I am working on SeaweedFS, which supports S3 API for object store, and can also use Cassandra as the metadata db. Cassandra has been performing well for most SeaweedFS users. The article listed many known Cassandra characteristics and cited them as limitations. However, it all depends on use cases. There are no file system that works for all cases, and not all of them needs ACID, CA vs CP, etc. The rest points are not convincing either. They are related to how to design the data structure better. Actually, SeaweedFS can use many other database/KV stores as the metadata DB. The list includes Redis, Cassandra, HBase, MySql, Postgres, Etcd, ElasticSearch, etc. https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores I did find one drawback for Cassandra as the metadata store though. One use case is that the customer uploaded a lot of zip files to one folder /tmp, unzip them, and then moved to a final folder. The rate is about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow. The solution was to use Redis for that /tmp folder, and still use Cassandra for the rest of folders. With Redis B-tree structure, the creation and deletion are cheap. So it is all depends on use cases.

ddorian43 6y ago
> With Redis B-tree structure, the creation and deletion are cheap. Pretty sure redis has no b-tree but hashmap. But redis has to do no compaction at all since it can directly delete the stuff on disk. > The rate is about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow. The problem here also is the gc-perdiod. Since the database has async-replication, it needs to keep deleted rows for some days so downed replicas can replicate. So they stay in disk/memory for some days actually and don't get compacted. While in a CP system they would be.

AtlasBarfed 6y ago

"I wanted to store something that could scale, but wanted to store and access my data in a way that didn't scale. Also, I don't understand CAP, distributed transactions, or distributed systems. I had a bad time." Somewhere else in the comments: "Yeah, we went with MongoDB."