2 ms·
I agree with you about being willing and unafraid of getting the depth in the concepts that matter. But that's also why I agree with you that the question of th
by sanderjd 2mo ago
I agree with you about being willing and unafraid of getting the depth in the concepts that matter. But that's also why I agree with you that the question of the server technology is the less important thing. I just think that leads to the conclusion "maybe stick with whatever you're already using for oltp for longer?" more so than to "you may as well introduce kafka".
I think both your questions, about correctness and about the comparison point, are answered by thinking through the empirical metrics you're trying to improve.
I often feel a bit out of step with other engineers on this point, but to me, "correctness" is not a binary yes/no, it is also a continuum. It's one of the metrics you probably want to be pushing to a large number of 9s, but trying to get to actually 100% has low ROI for what most people work on. (Not for everything! But for most things.) If you can already get to five 9s with postgres, but more 9s with kafka, is that worth it? Maybe, but maybe not. If you're missing way more messages than that and corrupting data because of it or some other bad thing, and this is the culprit, then that's certainly a good time to compare the trade offs of fixing your implementation or switching to a "proper" system for this.
And to "compared to what?", the answer is, compared to the current values of the metrics you've decided you care about. One of those might be the complexity of the implementation and the necessity to maintain specific expertise in a bespoke implementation of non-trivial complexity. This, to me, is usually the compelling reason to "buy" rather than build. But I think it is too often left too implicit and vibes-y instead of making it explicit and seeking to quantify the impact.