4 ms·

When you have a monolith database that everybody wants to access Kafka with CDC is a very good pattern for teams to adopt. The alternative is modifying your ap

When you have a monolith database that everybody wants to access Kafka with CDC is a very good pattern for teams to adopt. The alternative is modifying your app for arbitrary data access and dealing with auth or giving direct data access. Giving direct access to your database is fast but leads to very strong data coupling. Exposing the data through the app layer is slower than cdc kafka integration but comes with a long term commitment to support that access. I have seen these type of data access queries easily hit performance edge cases. There are other pros and cons but those are the ones that end up being the most troubling. Often a single apps data can be incredibly valuable and many teams want to access it. So coming up with patterns that 1. Protect the app 2. Protect the data store 3. Decrease the maintenance cost of the app 4. Decrease maintenance cost of the datatype 5. Add sight complexities from kafka. Can be very valuable.

derefr 6y ago
> Giving direct access to your database is fast but leads to very strong data coupling. Or you could just give "direct" access to (a DB role that only has been granted SELECT on) a single schema within your database, entirely consisting of views. Then you've got a stable data-access API that you can iterate your data model behind, with nothing more than your DB itself. (And you don't need to put much thought into setting it up, if you don't want to; your stable schema-API-layer can just start off as a bunch of views with 1:1 relationships to your real tables, which each just "SELECT * FROM their_underlying_table".)
- barrkel 6y ago
  Change data capture, though. That is so so useful for tearing bits of the monolith off.
  - tacticus 6y ago
    And so painful in the short, medium and long run for anyone who has to troubleshoot or operate it.
    - longcommonname 6y ago
      Maintenance costs of any software should be taken into consideration. I can say that we've had our troubles with kafka sure. But app teams 100% love working with it instead of having to have familiarity with all the various data sources we have. Working with all sorts of different data but all of its kafka is much easier than all sorts of data from Many data sources.
      - tacticus 6y ago
        The app teams i've seen loving it were app teams that siloed off ops but hey maybe your use cases were not mine. I'm seeing too many places using it in some form of RPC style system as part of some "workflow" orchestration magic system with unicorns and ponies for everyone
- longcommonname 6y ago
  That's a useful stategy, yes. But scaling your database is one of the more costly things you can scale.
rebyn 6y ago
I once looked into authen/authorization with Kafka topics and whether one can only be allowed to subscribe to specific ones and came up short with any solution to how to do it with Kafka. From my POV, points: 1 to 4 listed above now have been moved to one single point (Kafka) and its capabilities are a lot limited than those other tools/entrypoints (of data access) mentioned. Have there been new development to Kafka's IAM?
- andyroid 6y ago
  You could inject a custom authorizer and use something like OPA for fine grained access control. See https://github.com/Bisnode/opa-kafka-plugin https://github.com/Bisnode/opa-kafka-plugin for an example.
thethought 6y ago
To save some time for others. CDC -> Change Data Capture
pm90 6y ago
Wouldn’t apps relying on CDC events be strongly coupled as well? They’re just not synchronously coupled.
- longcommonname 6y ago
  I mostly disagree, but in a native implementation you probably do have strong data coupling. You can have cdc feed to another system that performs enrichment or transformation. Cdc also enables all sorts of non obvious changes that are infra focused. It's much easier to update your version of sql server, db2, mongo, etc. If you don't have to worry about every other app and how it connects to your shared service. You may think that you're being smooth by updating a A record or Cname when doing a migration but very often there's an app out there accessing using an ip.

derefr 6y ago

> Giving direct access to your database is fast but leads to very strong data coupling. Or you could just give "direct" access to (a DB role that only has been granted SELECT on) a single schema within your database, entirely consisting of views. Then you've got a stable data-access API that you can iterate your data model behind, with nothing more than your DB itself. (And you don't need to put much thought into setting it up, if you don't want to; your stable schema-API-layer can just start off as a bunch of views with 1:1 relationships to your real tables, which each just "SELECT * FROM their_underlying_table".)

barrkel 6y ago
Change data capture, though. That is so so useful for tearing bits of the monolith off.
- tacticus 6y ago
  And so painful in the short, medium and long run for anyone who has to troubleshoot or operate it.
  - longcommonname 6y ago
    Maintenance costs of any software should be taken into consideration. I can say that we've had our troubles with kafka sure. But app teams 100% love working with it instead of having to have familiarity with all the various data sources we have. Working with all sorts of different data but all of its kafka is much easier than all sorts of data from Many data sources.
    - tacticus 6y ago
      The app teams i've seen loving it were app teams that siloed off ops but hey maybe your use cases were not mine. I'm seeing too many places using it in some form of RPC style system as part of some "workflow" orchestration magic system with unicorns and ponies for everyone
longcommonname 6y ago
That's a useful stategy, yes. But scaling your database is one of the more costly things you can scale.

rebyn 6y ago

I once looked into authen/authorization with Kafka topics and whether one can only be allowed to subscribe to specific ones and came up short with any solution to how to do it with Kafka. From my POV, points: 1 to 4 listed above now have been moved to one single point (Kafka) and its capabilities are a lot limited than those other tools/entrypoints (of data access) mentioned. Have there been new development to Kafka's IAM?

andyroid 6y ago
You could inject a custom authorizer and use something like OPA for fine grained access control. See https://github.com/Bisnode/opa-kafka-plugin https://github.com/Bisnode/opa-kafka-plugin for an example.

thethought 6y ago

To save some time for others. CDC -> Change Data Capture

pm90 6y ago

Wouldn’t apps relying on CDC events be strongly coupled as well? They’re just not synchronously coupled.

longcommonname 6y ago
I mostly disagree, but in a native implementation you probably do have strong data coupling. You can have cdc feed to another system that performs enrichment or transformation. Cdc also enables all sorts of non obvious changes that are infra focused. It's much easier to update your version of sql server, db2, mongo, etc. If you don't have to worry about every other app and how it connects to your shared service. You may think that you're being smooth by updating a A record or Cname when doing a migration but very often there's an app out there accessing using an ip.