5 ms·

There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQ

There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQuery loads in batches. So the author is splitting up change-data segments into individual rows, feeding those rows into Kafka, and then re-batching them on the BQ side for ingestion. The use of a streaming system adds a tremendous amount of complexity for no benefit, because the original data source is batch. A simpler solution would be to process the data in 5-minute batches and use a BigQuery MERGE statement to apply that "patch" to the existing table. You really shouldn't use Kafka unless you have a true streaming data source and a hard latency requirement of < 5 minutes. It's much easier to work with data in small batches, using local disk and blob stores like S3 as your queue.

dillonmckay 7y ago
I recently had a junior dev propose using Kafka as a business ‘event’ system, to trigger or be triggered by webhooks, in order to integrate our API with a third party CRM.
- meddlepal 7y ago
  What's your point? Your comment comes off as dismissive and incomplete. Why was the juniors suggestion bad? It may have totally been a garbage suggestion, but if it was explain the reasoning.
  - _cairn 7y ago
    Not the author of the relevant post but assuming he's tailing off of the first response I can only surmise they meant that in their case Kafka was unnecessary/a bad choice due to the restraints posed (<5 minute delay, streaming input, etc). Agree that without context it's an incomplete comment that does come off as dismissive, ("junior dev") seems to be meant as belittling/insulting.
    - dillonmckay 7y ago
      Seemed like the wrong solution to the problem. I apologize to anybody I have offended. I was hoping for some discourse here.
matchagreentea 7y ago
OP here. Thanks for the feedback! I totally agree with you that using a streaming approach seems like an overkill. Actually making the pipeline truly real-time is part of our future work for Cassandra CDC (This is mentioned in Part 2 of this blogpost, which is expected to be published next week). In Cassandra 4.0, the CDC feature is improved to allow real-time parsing of commit logs. We are running 3.X in production, however we plan to add 4.0 support in the future.
- georgewfraser 7y ago
  What does the 4.0 api look like? I would home that they would expose a high-level interface for change data, similar to SQL Server or Snowflake streams. Most databases leave this as an afterthought, which is so unfortunate.
  - matchagreentea 7y ago
    http://cassandra.apache.org/doc/latest/operating/cdc.html http://cassandra.apache.org/doc/latest/operating/cdc.html The high-level idea is hard-linking the commit log segment file in cdc directory as it is being written + have a separate index file to keep track of offset.
    - pfarnsworth 7y ago
      Relying on an eventual consistency database sounds like a huge challenge in terms of getting absolute consistency between Cassandra and Kafka. There seems to be many edge cases that could cause the data in Kafka to get out of sync, it doesn't sound very reliable. Especially if you want to rely on Kafka for event sourcing later on, which is sounds like you're planning on in the future. For a payments company that sounds very risky. What happens if you write a row to Cassandra but that row is invalidated because of failed writes to the other nodes in the cluster (ex. the write was QUORUM and it succeeded on the node that you're reading from but failed on all other nodes). In Cassandra, when an error occurs, there are no rollbacks, and the row gets read-repaired. It sounds like that row will get outputted to Kafka. How do you "disappear" data written into Kafka if it gets read-repaired?
nightshadetrie 7y ago
Alot of engineers at WePay come from LinkedIn, so there's a special love for Kafka.
- aerioux 7y ago
  While that may be the case, this is not how Kafka is used within LI.

dillonmckay 7y ago

I recently had a junior dev propose using Kafka as a business ‘event’ system, to trigger or be triggered by webhooks, in order to integrate our API with a third party CRM.

meddlepal 7y ago
What's your point? Your comment comes off as dismissive and incomplete. Why was the juniors suggestion bad? It may have totally been a garbage suggestion, but if it was explain the reasoning.
- _cairn 7y ago
  Not the author of the relevant post but assuming he's tailing off of the first response I can only surmise they meant that in their case Kafka was unnecessary/a bad choice due to the restraints posed (<5 minute delay, streaming input, etc). Agree that without context it's an incomplete comment that does come off as dismissive, ("junior dev") seems to be meant as belittling/insulting.
  - dillonmckay 7y ago
    Seemed like the wrong solution to the problem. I apologize to anybody I have offended. I was hoping for some discourse here.

matchagreentea 7y ago

OP here. Thanks for the feedback! I totally agree with you that using a streaming approach seems like an overkill. Actually making the pipeline truly real-time is part of our future work for Cassandra CDC (This is mentioned in Part 2 of this blogpost, which is expected to be published next week). In Cassandra 4.0, the CDC feature is improved to allow real-time parsing of commit logs. We are running 3.X in production, however we plan to add 4.0 support in the future.

georgewfraser 7y ago
What does the 4.0 api look like? I would home that they would expose a high-level interface for change data, similar to SQL Server or Snowflake streams. Most databases leave this as an afterthought, which is so unfortunate.
- matchagreentea 7y ago
  http://cassandra.apache.org/doc/latest/operating/cdc.html http://cassandra.apache.org/doc/latest/operating/cdc.html The high-level idea is hard-linking the commit log segment file in cdc directory as it is being written + have a separate index file to keep track of offset.
  - pfarnsworth 7y ago
    Relying on an eventual consistency database sounds like a huge challenge in terms of getting absolute consistency between Cassandra and Kafka. There seems to be many edge cases that could cause the data in Kafka to get out of sync, it doesn't sound very reliable. Especially if you want to rely on Kafka for event sourcing later on, which is sounds like you're planning on in the future. For a payments company that sounds very risky. What happens if you write a row to Cassandra but that row is invalidated because of failed writes to the other nodes in the cluster (ex. the write was QUORUM and it succeeded on the node that you're reading from but failed on all other nodes). In Cassandra, when an error occurs, there are no rollbacks, and the row gets read-repaired. It sounds like that row will get outputted to Kafka. How do you "disappear" data written into Kafka if it gets read-repaired?

nightshadetrie 7y ago

Alot of engineers at WePay come from LinkedIn, so there's a special love for Kafka.

aerioux 7y ago
While that may be the case, this is not how Kafka is used within LI.