5 ms·

I've seen successful, but flawed usage. Every use I've seen sent events after database transactions, with the event not part of the transaction. This means you

I've seen successful, but flawed usage. Every use I've seen sent events after database transactions, with the event not part of the transaction. This means you can get both dropped events, and out of order events. My current company has analytics driven by a system like that. I'm sure there's some corrupted data as a result. The main issue being people just don't know how to build and test distributed systems.

mrkeen 2y ago
I had an interview where I was asked how I would guarantee that an event happened in addition to a database update (transactionally). It sounded kind of impossible, I said as much, and then proposed a different approach. The interviewer persisted and claimed that it could be done with 'the outbox pattern'. I disagreed and ended the interview there. Later when I was chatting about it with a former colleague, he said "Oh, they solved the two generals problem?" > Every use I've seen sent events after database transactions, with the event not part of the transaction. Maybe this is what they were doing.
- lastofus 2y ago
  I don't quite see what the outbox pattern has to do with the two generals problem. The point of the outbox pattern is that a durable record of the need to send an event is stored in the DB as part of the DB txn, taking advantage of ACID guarantees. Once you have that durable record in your DB, you can essentially treat your DB as a queue (there's lot of great articles on how to do this with Postgres for instance) for some worker processes to later process the queue records, and send the events. The worker processes in turn can decide if they want to attempt at least once, or at most once delivery of the message. Of course if you choose the later, then maybe your event is never sent, and perhaps that was the point you were trying to make to the interviewer. They key takeaway though is that you are no longer reliant on the original process that stores the DB txn to also send the event, which can fail for any number of reasons, and may have no path to recovery. In other words, at least once delivery is now an option on the table.
  - mrkeen 2y ago
    > I don't quite see what the outbox pattern has to do with the two generals problem. Well then, hopefully you would have found it an unsatisfactory 'solution' and walked away from that interview too ;) > Once you have that durable record in your DB, you can essentially treat your DB as a queue (there's lot of great articles on how to do this with Postgres for instance) for some worker processes to later process the queue records, and send the events. Yeah but I already have a queue I can treat as a queue. It's called Kafka.
- ramchip 2y ago
  I'm a bit confused by the story. Why did you disagree?
  - mrkeen 2y ago
    They asked how I would guarantee that a Postgres update and a Kafka event could both happen transactionally. (P, K) Which sounds like one of those classical impossibility proofs. Their solution was to introduce another part into the system, "the outbox": (P, O) K P and O can form a transaction, but that still leaves the original problem unanswered, how to include K in the transaction.
    - ramchip 2y ago
      The goal of the outbox pattern is at-least-once publishing though, not only-once. You either get P + (eventually) at least one copy of K, or you get no P and no K. Without the outbox you can get P without K or K without P, which lead to consumers out of sync with the producer. This requires the consumer to be able to deal with repeated events to some extent, but you usually want that anyway, since an event can be processed twice even if it appears only once in the topic. For instance if the consumer crashes after processing but before updating the offset.
      - mrkeen 2y ago
        > The goal of the outbox pattern is at-least-once publishing though, not only-once. Right, which is why it's an unacceptable solution to 'transacting over postgres and Kafka', and why I wouldn't want to work for a company that wants me to believe differently. And there's a better solution anyway: just K.
        ramchip 2y ago
        I think you're just using "transaction" in a different sense than what the interviewer meant; "guarantee that an event happened in addition to a database update" sounds like at-least-once to me, and it's normally what you would want in this kind of system.
- nick__m 2y ago
  You could two phase commit with XA compliant broker and database.

mrkeen 2y ago

I had an interview where I was asked how I would guarantee that an event happened in addition to a database update (transactionally). It sounded kind of impossible, I said as much, and then proposed a different approach. The interviewer persisted and claimed that it could be done with 'the outbox pattern'. I disagreed and ended the interview there. Later when I was chatting about it with a former colleague, he said "Oh, they solved the two generals problem?" > Every use I've seen sent events after database transactions, with the event not part of the transaction. Maybe this is what they were doing.

lastofus 2y ago
I don't quite see what the outbox pattern has to do with the two generals problem. The point of the outbox pattern is that a durable record of the need to send an event is stored in the DB as part of the DB txn, taking advantage of ACID guarantees. Once you have that durable record in your DB, you can essentially treat your DB as a queue (there's lot of great articles on how to do this with Postgres for instance) for some worker processes to later process the queue records, and send the events. The worker processes in turn can decide if they want to attempt at least once, or at most once delivery of the message. Of course if you choose the later, then maybe your event is never sent, and perhaps that was the point you were trying to make to the interviewer. They key takeaway though is that you are no longer reliant on the original process that stores the DB txn to also send the event, which can fail for any number of reasons, and may have no path to recovery. In other words, at least once delivery is now an option on the table.
- mrkeen 2y ago
  > I don't quite see what the outbox pattern has to do with the two generals problem. Well then, hopefully you would have found it an unsatisfactory 'solution' and walked away from that interview too ;) > Once you have that durable record in your DB, you can essentially treat your DB as a queue (there's lot of great articles on how to do this with Postgres for instance) for some worker processes to later process the queue records, and send the events. Yeah but I already have a queue I can treat as a queue. It's called Kafka.
ramchip 2y ago
I'm a bit confused by the story. Why did you disagree?
- mrkeen 2y ago
  They asked how I would guarantee that a Postgres update and a Kafka event could both happen transactionally. (P, K) Which sounds like one of those classical impossibility proofs. Their solution was to introduce another part into the system, "the outbox": (P, O) K P and O can form a transaction, but that still leaves the original problem unanswered, how to include K in the transaction.
  - ramchip 2y ago
    The goal of the outbox pattern is at-least-once publishing though, not only-once. You either get P + (eventually) at least one copy of K, or you get no P and no K. Without the outbox you can get P without K or K without P, which lead to consumers out of sync with the producer. This requires the consumer to be able to deal with repeated events to some extent, but you usually want that anyway, since an event can be processed twice even if it appears only once in the topic. For instance if the consumer crashes after processing but before updating the offset.
    - mrkeen 2y ago
      > The goal of the outbox pattern is at-least-once publishing though, not only-once. Right, which is why it's an unacceptable solution to 'transacting over postgres and Kafka', and why I wouldn't want to work for a company that wants me to believe differently. And there's a better solution anyway: just K.
      - ramchip 2y ago
        I think you're just using "transaction" in a different sense than what the interviewer meant; "guarantee that an event happened in addition to a database update" sounds like at-least-once to me, and it's normally what you would want in this kind of system.
nick__m 2y ago
You could two phase commit with XA compliant broker and database.