4 ms·

I'm working on a project where the backend devs decided to use Cassandra to manage a few hundred entities. Due to different access modes they had to denormalize

I'm working on a project where the backend devs decided to use Cassandra to manage a few hundred entities. Due to different access modes they had to denormalize the data into multiple tables and we're constantly running into consistency issues. A fucking sqlite db with a few tables and indexes would do just fine. Also, the tool has maybe a dozen DAUs. But it's web scale, so, yeah…

twblalock 4y ago
That's not a scale problem. That's a problem of database choice. Cassandra is not good at that sort of query pattern at any scale.
bee_rider 4y ago
This completely tangential, but any time I see someone talking about normalizing a database, I briefly think they are talking about doing something like: x/|x| Ya know, like normalizing a vector. Of course this brief confusion doesn't matter, I sort it out quickly, and it is totally clear language to other database people -- it is just a funny quirk of overloaded lingo. But I wonder if as big data and machine learning folks (since there's some overlap with Linear Algebra there) ever get their lingo mixed up. "To predict where the ridesharing customers will want to go, we will apply the taxicab norm to our database."
- adepressedthrow 4y ago
  While the overloading is annoying, database normalization actually has a strict definition that is not unlike the standard mathematical use; that is, to imply consistency (to make normal): https://en.wikipedia.org/wiki/Database_normalization https://en.wikipedia.org/wiki/Database_normalization I would also point out that normalization within mathematics does not have a single concrete definition either, and it depends on your domain and desired property to stabilize: https://en.wikipedia.org/wiki/Normalization_(statistics) https://en.wikipedia.org/wiki/Normalization_(statistics)
  - bee_rider 4y ago
    Just to be clear -- I'm not annoyed really, I actually find these little momentary confusions to be a charming little quirk of our language. And I think it is not a big deal anyway, 99% of the time we're talking inside our fields where it is obvious what we mean.
jsiaajdsdaa 4y ago
Why would a team decide to use nosql to manage "a few hundred" entities? Nosql databases excel at simple lookups and gets by ID, rather than trying to sort and join pretty much ad hoc.
- winrid 4y ago
  NoSQL might not be the right term here. Maybe you mean "columnar databases"? Ex Mongo sorts just fine, and can optimize sorting with indexes.
- lmm 4y ago
  Why would you ever use SQL? The SQL transaction model doesn't make a lot of sense for most use cases (including essentially anything that uses the internet), and trivial usability things like being able to have collection columns without having to define a separate table all add up.
  - akhmatova 4y ago
    That must have been what they were thinking when they made the decision to go with Cassandra.
    - lmm 4y ago
      I'm working on a system that's using Cassandra on a small scale (like, probably less than a million entities) now, it's great. Haven't had any consistency issues because we've designed our data flow properly (SQL will help you paper over a bad data flow for longer but that's a double-edged sword), and getting true just-works HA (master-master where a failover doesn't interrupt your writes) out of the box is a huge advantage over something like Postgres.
      - akhmatova 4y ago
        Oh, I don't dispute that there are valid use cases. But the above instance sounds like a variant of the "because, bwah, SQL is hard" line of thought that seems to have powered much of the (now historical) NoSQL boom.
mbrodersen 4y ago
You are absolutely right. 95% of problems in software development are created by the software developers themselves. The fastest/smartest developers I have worked with simply make less dumb mistakes. They keep it simple. That’s all they have to do to be 10x more productive.

twblalock 4y ago

That's not a scale problem. That's a problem of database choice. Cassandra is not good at that sort of query pattern at any scale.

bee_rider 4y ago

This completely tangential, but any time I see someone talking about normalizing a database, I briefly think they are talking about doing something like: x/|x| Ya know, like normalizing a vector. Of course this brief confusion doesn't matter, I sort it out quickly, and it is totally clear language to other database people -- it is just a funny quirk of overloaded lingo. But I wonder if as big data and machine learning folks (since there's some overlap with Linear Algebra there) ever get their lingo mixed up. "To predict where the ridesharing customers will want to go, we will apply the taxicab norm to our database."

adepressedthrow 4y ago
While the overloading is annoying, database normalization actually has a strict definition that is not unlike the standard mathematical use; that is, to imply consistency (to make normal): https://en.wikipedia.org/wiki/Database_normalization https://en.wikipedia.org/wiki/Database_normalization I would also point out that normalization within mathematics does not have a single concrete definition either, and it depends on your domain and desired property to stabilize: https://en.wikipedia.org/wiki/Normalization_(statistics) https://en.wikipedia.org/wiki/Normalization_(statistics)
- bee_rider 4y ago
  Just to be clear -- I'm not annoyed really, I actually find these little momentary confusions to be a charming little quirk of our language. And I think it is not a big deal anyway, 99% of the time we're talking inside our fields where it is obvious what we mean.

jsiaajdsdaa 4y ago

Why would a team decide to use nosql to manage "a few hundred" entities? Nosql databases excel at simple lookups and gets by ID, rather than trying to sort and join pretty much ad hoc.

winrid 4y ago
NoSQL might not be the right term here. Maybe you mean "columnar databases"? Ex Mongo sorts just fine, and can optimize sorting with indexes.
lmm 4y ago
Why would you ever use SQL? The SQL transaction model doesn't make a lot of sense for most use cases (including essentially anything that uses the internet), and trivial usability things like being able to have collection columns without having to define a separate table all add up.
- akhmatova 4y ago
  That must have been what they were thinking when they made the decision to go with Cassandra.
  - lmm 4y ago
    I'm working on a system that's using Cassandra on a small scale (like, probably less than a million entities) now, it's great. Haven't had any consistency issues because we've designed our data flow properly (SQL will help you paper over a bad data flow for longer but that's a double-edged sword), and getting true just-works HA (master-master where a failover doesn't interrupt your writes) out of the box is a huge advantage over something like Postgres.
    - akhmatova 4y ago
      Oh, I don't dispute that there are valid use cases. But the above instance sounds like a variant of the "because, bwah, SQL is hard" line of thought that seems to have powered much of the (now historical) NoSQL boom.

mbrodersen 4y ago

You are absolutely right. 95% of problems in software development are created by the software developers themselves. The fastest/smartest developers I have worked with simply make less dumb mistakes. They keep it simple. That’s all they have to do to be 10x more productive.