5 ms·
Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly
by j-pb 6y ago
Tbh, It's a weird blog post coming from the materialize folks, considering they know better.
The "event sourced" arch they sketched is missing pieces.
Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them.
Throwing shade for maketings sake is something that they should be above.
I mean c'mon, I'd argue that Postgres enhanced with Materialize isn't a database anymore either, but in a good sense!
It's building material. A hybrid between MQ, DB, backend logic & frontend logic.
The reduction in application logic and the increase in reliability you can get from reactive systems is insane.
SQL is declarative, reactive Materialize streams are declarative on a whole new level.
Once that tech makes it into other parts of computing like the frontend, development will be so much better, less code, less bug, a lot more fun.
Imagine that your react component could simply declare all the data it needs from a db, and the system will figure out all the caching and rerendering.
So yeah, they have awesome tech with many advantages, so I don't get why they bad-mouth other architectures.
- dkhenry 6y agoI am not terribly surprised. The materialize team was previously at CockroachDB which also had a habit of putting out marketing material like this.
- j-pb 6y agoMaybe I'm biased because I'm such a huge Frank McSherry fanboy. The differential dataflow work he does in rust is simply awesome, and he writes great papers too! Little known fun fact, the rust type- and borrow-checker uses a datalog engine internally to express typing rules, and that engine was written and improved by Frank McSherry. So whenever you hit compile on a rust program, you're using a tiny bit of Materialize tech.
- _jal 6y agoI ignored them for a long time because of it. I just assumed they were lightweights being annoying because they didn't have anything else. It serves as anti-marketing, at least to me.
- dwelch2344 6y agoTotally agree with this. "Kafka is terrible if you use it with poor practices" should be the title of this. It's totally clickbait targeting the "Database Inside Out" articles / concept, which if you read the most common 3-page high level it explicitly states you should be materializing up into other data sources / structures / systems / etc To date, I've never written code that reads from a Kafka topic that wasn't taking data and transforming + materializing it into domain intelligence
- arjunnarayan 6y agoHi! I'm one of the two authors here. At Materialize, we're definitely of the 'we are a bunch of voices, we are people rather than corp-speak, and you get our largely unfiltered takes' flavor. This is my (and George's from Fivetran) take. In particular this is not Frank's take, as you attribute below :) > SQL is declarative, reactive Materialize streams are declarative on a whole new level. Thank you for the kind words about our tech, I'm flattered! That said, this dream is downstream of Kafka. Most of our quibbles with the Kafka-as-database architecture are to do with the fact that that architecture neglects the work that needs to be done _upstream_ of Kafka. That work is best done with an OLTP database. Funnily enough, neither of us are building OLTP databases, but this piece largely is a defense of OLTP databases (if you're curious, yes, I'd recommend CockroachDB), and their virtues at that head of the data pipeline. Kafka has its place - and when its used downstream of CDC from said OLTP database (using, e.g. Debezium), we could not be happier with it (and we say so). The best example is in foreign key checks. It is not good if you ever need to enforce foreign key checks (which translates to checking a denormalization of your source data _transactionally_ with deciding whether to admit or deny an event). This is something that you may not need in your data pipeline on day 1, but adding that in later is a trivial schema change with an OLTP database, and exceedingly difficult with a Kafka-based event sourced architecture. > Normally you'd have single writer instances that are locked to the corresponding Kafka partition, which ensure strong transactional guarantees, IF you need them. This still does not deal with the use-case of needing to add a foreign key check. You'd have to: 1. Log "intents to write" rather than writes themselves in Topic A 2. Have a separate denormalization computed and kept in a separate Topic B, which can be read from. This denormalization needs to be read until the intent propagates from Topic A. 3. Convert those intents into commits. 4. Deal with all the failure cases in a distributed system, e.g. cleaning up abandoned intents, etc. If you use an OLTP database, and generate events into Kafka via CDC, you get the best of both worlds. And hopefully, yes, have a reactive declarative stack downstream of that as well!
- skybrian 6y agoIt seems like Kafka could be used to record requests and responses (upstream of the database) and approved changes (downstream of the database), but isn't good for the approval process itself?
- georgewfraser 6y agoWe're trying to address a real problem that is happening in our industry: VPs of eng and principal engineers at startups are adopting the "Kappa Architecture" / "Turning the Database Inside Out", without realizing how much functionality from traditional database systems they are leaving behind. This has led to a barrage of consistency bugs in everything from food-delivery apps to the "unread message count" in LinkedIn. We're at the peak of the hype cycle for Kafka, and it's being used in all kinds of places it doesn't belong. For 99% of companies, a traditional DBMS is the right foundation.