3 ms·

>You will more or less have to write a full-fledged DBMS in your application code. Yes, but only if your data is highly relational, and needs to be operated on

>You will more or less have to write a full-fledged DBMS in your application code. Yes, but only if your data is highly relational, and needs to be operated on in that way (e.g. transactionally across records, needing joins to do anything useful, querying by secondary indices, etc.). Depending on your domain, this could be a large amount of your data, or not. Just like anything, Kafka and Kafka Streams are tools that can be used and misused. There is perhaps an exciting future where something like ksqlDB makes even highly relational data sit well in Kafka, but we're definitely not there yet, and I'm not terribly confident about getting there.

yawaramin 6y ago
> Yes, but only if your data is highly relational, and needs to be operated on in that way (e.g. transactionally across records, needing joins to do anything useful, querying by secondary indices, etc.). Ah, you mean that if the data is in a normalized form. Relational data means, basically, data in tabular form. Tabular data can be normalized or not. > Depending on your domain, this could be a large amount of your data, or not. Assuming you are talking about data in normalized form, that's orthogonal to the domain. That's in fact the major breakthrough of relational databases–that data across domains can be modelled as relational and normalized. Now, in the relatively rarer cases that you don't care about normalizing data, a stream can be useful. But in the vast majority of cases IMHO, we should be looking at a proper relational database like Postgres before thinking about streams.
- pgwhalen 6y ago
  I agree that _normalized_ is a more precise term for what I'm describing, I only used relational because contemporary discourse seems to favor that term in these kinds of discussions and seems to imply a high degree of normalization. >Assuming you are talking about data in normalized form, that's orthogonal to the domain. I don't agree with this though. Nothing about a software architecture is _ever_ orthogonal to the domain.

yawaramin 6y ago

> Yes, but only if your data is highly relational, and needs to be operated on in that way (e.g. transactionally across records, needing joins to do anything useful, querying by secondary indices, etc.). Ah, you mean that if the data is in a normalized form. Relational data means, basically, data in tabular form. Tabular data can be normalized or not. > Depending on your domain, this could be a large amount of your data, or not. Assuming you are talking about data in normalized form, that's orthogonal to the domain. That's in fact the major breakthrough of relational databases–that data across domains can be modelled as relational and normalized. Now, in the relatively rarer cases that you don't care about normalizing data, a stream can be useful. But in the vast majority of cases IMHO, we should be looking at a proper relational database like Postgres before thinking about streams.

pgwhalen 6y ago
I agree that _normalized_ is a more precise term for what I'm describing, I only used relational because contemporary discourse seems to favor that term in these kinds of discussions and seems to imply a high degree of normalization. >Assuming you are talking about data in normalized form, that's orthogonal to the domain. I don't agree with this though. Nothing about a software architecture is _ever_ orthogonal to the domain.