5 ms·

> It does not provide read-after-write consistency So, this is probably the last "database" you should ever opt to use, unless you're in a very serious bind.

by arnon 7y ago

missosoup 7y ago
What does this actually mean in practical terms for the db?
- Erwin 7y ago
  AIUI you can only write to the database indirectly through Kafka messages which the KSQL then periodically consumes and materializes into tables with syntax almost like SQL. So if you emit an event (like "New User Created") and have a ksql table that summarizes the user count, then Kafka having accepted the event does not mean the ksql tables have also been updated. Contrasted with a traditional database where a COMMIT returning one one session means the second can immediately read it.
  - missosoup 7y ago
    That's standard behaviour in every analytics database that talks to kafka that I can think of. Kafka doesn't really encourage ACK/NACK type processing patterns. Accepting an event usually means the consumer has successfully read it and staged it for whatever is meant to happen next, not that the operation is completed. Now if it were possible for it to accept an event from Kafka but not guarantee that the event will eventually make it into the materialized view but may be lost, that'd be a problem.
    - bladecatcher 7y ago
      > Now if it were possible for it to accept an event from Kafka but not guarantee that the event will eventually make it into the materialized view but may be lost, that'd be a problem. This is usually not a problem these days as it’s possible to guarantee exactly once ingestion using Kafka offsets
      - missosoup 7y ago
        IIRC clickhouse still doesn't have this guarantee. It guarantees exactly one ingestion but not that the ingested event gets processed all the way through to view. And if the processing fails, that event won't be retried and is now gone.
egwor 7y ago
I don't think that KDB does, either? MY understanding of this in KDB, and I could be wrong (I'm not an expert) is that in KDB you can write to the tickerplant database. Those clients subscribing to the realtime will see this in realtime, and then eventually it will get asynchronously written to another DB which allows querying. I guess you'd think of it as eventual consistency. (Please correct me if I'm incorrect/misunderstood the above)
- bladecatcher 7y ago
  You’re right. You don’t get read-your-own-writes guarantees in a typical Kdb tickerplant architecture, as the event streams are propagated to the real-time nodes asynchronously.
bladecatcher 7y ago
I think the usecase for this is streaming queries and aggregations where the query output is materialised asynchronously

> It does not provide read-after-write consistency So, this is probably the last "database" you should ever opt to use, unless you're in a very serious bind.

by arnon 7y ago

> It does not provide read-after-write consistency So, this is probably the last "database" you should ever opt to use, unless you're in a very serious bind.

missosoup 7y ago

What does this actually mean in practical terms for the db?

Erwin 7y ago
AIUI you can only write to the database indirectly through Kafka messages which the KSQL then periodically consumes and materializes into tables with syntax almost like SQL. So if you emit an event (like "New User Created") and have a ksql table that summarizes the user count, then Kafka having accepted the event does not mean the ksql tables have also been updated. Contrasted with a traditional database where a COMMIT returning one one session means the second can immediately read it.
- missosoup 7y ago
  That's standard behaviour in every analytics database that talks to kafka that I can think of. Kafka doesn't really encourage ACK/NACK type processing patterns. Accepting an event usually means the consumer has successfully read it and staged it for whatever is meant to happen next, not that the operation is completed. Now if it were possible for it to accept an event from Kafka but not guarantee that the event will eventually make it into the materialized view but may be lost, that'd be a problem.
  - bladecatcher 7y ago
    > Now if it were possible for it to accept an event from Kafka but not guarantee that the event will eventually make it into the materialized view but may be lost, that'd be a problem. This is usually not a problem these days as it’s possible to guarantee exactly once ingestion using Kafka offsets
    - missosoup 7y ago
      IIRC clickhouse still doesn't have this guarantee. It guarantees exactly one ingestion but not that the ingested event gets processed all the way through to view. And if the processing fails, that event won't be retried and is now gone.

egwor 7y ago

I don't think that KDB does, either? MY understanding of this in KDB, and I could be wrong (I'm not an expert) is that in KDB you can write to the tickerplant database. Those clients subscribing to the realtime will see this in realtime, and then eventually it will get asynchronously written to another DB which allows querying. I guess you'd think of it as eventual consistency. (Please correct me if I'm incorrect/misunderstood the above)

bladecatcher 7y ago
You’re right. You don’t get read-your-own-writes guarantees in a typical Kdb tickerplant architecture, as the event streams are propagated to the real-time nodes asynchronously.

bladecatcher 7y ago

I think the usecase for this is streaming queries and aggregations where the query output is materialised asynchronously