4 ms·

> When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the p

> When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the producer to know when it will be able to "see" that downstream update. For certain use cases it would be helpful to be able to guarantee that derived data views have been updated when a produce request gets acknowledged, allowing Kafka to act as a log for a true database with strong read-your-own-writes semantics. Just don't use Kafka. Write to the downstream datastore directly. Then you know your data is committed and you have a database to query.

menzoic 2y ago
Writing directly to the datastore ignores the need for queuing the writes. How do you solve for that need?
- immibis 2y ago
  Why do you need to queue the writes?
  - smitty1e 2y ago
    A proficient coder can write a program to accomplish a task in the singular. In the plural, accomplishing that task in a performant way at enterprise scale seems to involve turning every function call into an asynchronous, queued service of some sort. Which then begets additional deployment and monitoring services. A queued problem requires a cute solution, bringing acute pain.
  - vergessenmir 2y ago
    some writes might fail, you may need to retry, the data store may be temporarily available etc. There may be many things that go wrong and how you handle this depends on your data guarantees and consistency requirements. If you're not queuing what are you doing when a write fails, throwing away the data?
    - bushbaba 1y ago
      Some Kafka writes might fail. Hence the Kafka client having a queue with retries.
    - immibis 1y ago
      As bushbaba points out, the same things may happen with Kafka. The standard regex joke really works for all values of X. Some people, when confronted with a problem, think "I know - I'll use a queue!" Now they have two problems. Adding a queue to the system does not make it faster or more reliable. It makes it more asynchronous (because of the queue), slower (because the computer has to do more stuff) and less reliable (because there are more "moving parts"). It's possible that a queue is a required component of a design which is more reliable and faster, but this can only be known about the specific design, not the general case. I'd start by asking why your data store is randomly failing to write data. I've never encountered Postgres randomly failing to write data. There are certainly conditions that can cause Postgres to fail to write data, but they aren't random and most of them are bad enough to require an on-call engineer to come and fix the system anyway - if the problem could be resolved automatically, it would have been. If you want to be resilient to events like the database disk being full (maybe it's a separate analytics database that's less important from the main transactional database) then adding a queue (on a separate disk) can make sense, so you can continue having accurate analytics after upgrading the analytics database storage. In this case you're using the queue to create a fault isolation boundary. It just kicks the can down the road though, since if the analytics queue storage fills up, you still have to either drop the analytics or fail the client's request. You have the same problem but now for the queue. Again, it could be a reasonable design, but not by default and you'd have to evaluate the whole design to see whether it's reasonable.
wvh 2y ago
The problem is that you don't know who's listening. You don't want all possible interested parties to hammer the database. Hence the events in between. Arguably, I'd not use Kafka to store actual data, just to notify in-flight.
- Spivak 2y ago
  For read only queries, hammer away I can scale reads nigh infinitely horizontally. There's no secret sauce that makes it so that only kafka can do this.
  - throwaway7783 1y ago
    It's not about scaling reads, but coordinating consumers so no more than one consumer processes same messages. That means some kind of locking, that means scaling issues.
- immibis 1y ago
  But you do know who's listening, because you were the one who installed all the listeners. ("you" can be plural.) This reminds me of the OOP vs DOD debate again. OOP adherents say they don't know all the types of data their code operates on; DOD adherents say they actually do, since their program contains a finite number of classes, and a finite subset of those classes can be the ones called in this particular virtual function call. What you mean is that your system is structured in such a way that it's as if you don't know who's listening. Which is okay, but you should be explicit that it's a design choice, not a law of physics, so when that design choice no longer serves you well, you have the right to change it. (Sometimes you really don't know, because your code is a library or has a plugin system. In such cases, this doesn't apply.) > Arguably, I'd not use Kafka to store actual data, just to notify in-flight. I believe people did this initially and then discovered the non-Kafka copy of the data is redundant, so got rid of it, or relegated it to the status of a cache. This type of design is called Event Sourcing.
  - wvh 1y ago
    I have worked for financial institutions where random departments have random interests in certain data transactions. You (as in a dev team in one such department) have no say in who touches the data, from where, and how it's used. Kafka is used as a corporate message bus to let e.g. the accountant department know something happened in another department. Those "listening" departments don't have devs and are not involved in development, they operate more on the MS BI level of PowerPoint. So yes, in large companies, your development team is just a small cog, you don't set policy for what happens to the data you gather. And in some sectors, like finances, you are an especially small cog with little power, which might sound strange if you only ever worked for a software startup.
thom 2y ago
Of course, if you don't have separate downstream and upstream datastores, you don't have anything to do in the first place.
rakoo 2y ago
Alternatively, your write doesn't have to be fire-and-forget: downstream datastores can also write to kafka (this time fire-and-forget) and the initial client can wait for that event to acknowledge the initial write
salomonk_mur 1y ago
Yeah... Not happening when you have scores of clients running down your database. The reason message queue systems exist is scale. Good luck sending a notification at 9am to your 3 million users and keeping your database alive in the sudden influx of activity. You need to queue that load.
- nitwit005 1y ago
  Kafka is itself a database. Sending a message requires what is essentially a database insert. You're still doing a DB commit either way.
  - lamp_book 1y ago
    It's more of a commit log/write-ahead log/replication stream than a DBMS - consider that DBMSs typically include these in addition to their primary storage.
    - nitwit005 1y ago
      It's clearly not a full blown database product, but it's still got the core elements of a database. Your data is getting replicated to multiple instances, written to disk, and an index is created for quick lookup. It's just the table that's getting is essentially append only (excepting the cleanup processes it supports).
      - lamp_book 1y ago
        My point was more about the two having separate use cases - but Kafka handles write throughput much easier compared to an RDBMS due to lack of transactions and not needing to manipulate the primary table data structures.

menzoic 2y ago

Writing directly to the datastore ignores the need for queuing the writes. How do you solve for that need?

immibis 2y ago
Why do you need to queue the writes?
- smitty1e 2y ago
  A proficient coder can write a program to accomplish a task in the singular. In the plural, accomplishing that task in a performant way at enterprise scale seems to involve turning every function call into an asynchronous, queued service of some sort. Which then begets additional deployment and monitoring services. A queued problem requires a cute solution, bringing acute pain.
- vergessenmir 2y ago
  some writes might fail, you may need to retry, the data store may be temporarily available etc. There may be many things that go wrong and how you handle this depends on your data guarantees and consistency requirements. If you're not queuing what are you doing when a write fails, throwing away the data?
  - bushbaba 1y ago
    Some Kafka writes might fail. Hence the Kafka client having a queue with retries.
  - immibis 1y ago
    As bushbaba points out, the same things may happen with Kafka. The standard regex joke really works for all values of X. Some people, when confronted with a problem, think "I know - I'll use a queue!" Now they have two problems. Adding a queue to the system does not make it faster or more reliable. It makes it more asynchronous (because of the queue), slower (because the computer has to do more stuff) and less reliable (because there are more "moving parts"). It's possible that a queue is a required component of a design which is more reliable and faster, but this can only be known about the specific design, not the general case. I'd start by asking why your data store is randomly failing to write data. I've never encountered Postgres randomly failing to write data. There are certainly conditions that can cause Postgres to fail to write data, but they aren't random and most of them are bad enough to require an on-call engineer to come and fix the system anyway - if the problem could be resolved automatically, it would have been. If you want to be resilient to events like the database disk being full (maybe it's a separate analytics database that's less important from the main transactional database) then adding a queue (on a separate disk) can make sense, so you can continue having accurate analytics after upgrading the analytics database storage. In this case you're using the queue to create a fault isolation boundary. It just kicks the can down the road though, since if the analytics queue storage fills up, you still have to either drop the analytics or fail the client's request. You have the same problem but now for the queue. Again, it could be a reasonable design, but not by default and you'd have to evaluate the whole design to see whether it's reasonable.

wvh 2y ago

The problem is that you don't know who's listening. You don't want all possible interested parties to hammer the database. Hence the events in between. Arguably, I'd not use Kafka to store actual data, just to notify in-flight.

Spivak 2y ago
For read only queries, hammer away I can scale reads nigh infinitely horizontally. There's no secret sauce that makes it so that only kafka can do this.
- throwaway7783 1y ago
  It's not about scaling reads, but coordinating consumers so no more than one consumer processes same messages. That means some kind of locking, that means scaling issues.
immibis 1y ago
But you do know who's listening, because you were the one who installed all the listeners. ("you" can be plural.) This reminds me of the OOP vs DOD debate again. OOP adherents say they don't know all the types of data their code operates on; DOD adherents say they actually do, since their program contains a finite number of classes, and a finite subset of those classes can be the ones called in this particular virtual function call. What you mean is that your system is structured in such a way that it's as if you don't know who's listening. Which is okay, but you should be explicit that it's a design choice, not a law of physics, so when that design choice no longer serves you well, you have the right to change it. (Sometimes you really don't know, because your code is a library or has a plugin system. In such cases, this doesn't apply.) > Arguably, I'd not use Kafka to store actual data, just to notify in-flight. I believe people did this initially and then discovered the non-Kafka copy of the data is redundant, so got rid of it, or relegated it to the status of a cache. This type of design is called Event Sourcing.
- wvh 1y ago
  I have worked for financial institutions where random departments have random interests in certain data transactions. You (as in a dev team in one such department) have no say in who touches the data, from where, and how it's used. Kafka is used as a corporate message bus to let e.g. the accountant department know something happened in another department. Those "listening" departments don't have devs and are not involved in development, they operate more on the MS BI level of PowerPoint. So yes, in large companies, your development team is just a small cog, you don't set policy for what happens to the data you gather. And in some sectors, like finances, you are an especially small cog with little power, which might sound strange if you only ever worked for a software startup.

thom 2y ago

Of course, if you don't have separate downstream and upstream datastores, you don't have anything to do in the first place.

rakoo 2y ago

Alternatively, your write doesn't have to be fire-and-forget: downstream datastores can also write to kafka (this time fire-and-forget) and the initial client can wait for that event to acknowledge the initial write

salomonk_mur 1y ago

Yeah... Not happening when you have scores of clients running down your database. The reason message queue systems exist is scale. Good luck sending a notification at 9am to your 3 million users and keeping your database alive in the sudden influx of activity. You need to queue that load.

nitwit005 1y ago
Kafka is itself a database. Sending a message requires what is essentially a database insert. You're still doing a DB commit either way.
- lamp_book 1y ago
  It's more of a commit log/write-ahead log/replication stream than a DBMS - consider that DBMSs typically include these in addition to their primary storage.
  - nitwit005 1y ago
    It's clearly not a full blown database product, but it's still got the core elements of a database. Your data is getting replicated to multiple instances, written to disk, and an index is created for quick lookup. It's just the table that's getting is essentially append only (excepting the cleanup processes it supports).
    - lamp_book 1y ago
      My point was more about the two having separate use cases - but Kafka handles write throughput much easier compared to an RDBMS due to lack of transactions and not needing to manipulate the primary table data structures.