4 ms·
Wrote this after seeing the same failure mode too many times: a MySQL sync job that looks fine in the dashboard but has been silently dropping deletes and inter
by gpaulbagetti 1mo ago
Wrote this after seeing the same failure mode too many times: a MySQL sync job that looks fine in the dashboard but has been silently dropping deletes and intermediate updates for months, because it's comparing snapshots instead of reading the binlog. Tried to lay out exactly what has to be true on the MySQL side (row image, binlog_row_value_options, server-id, retention) for CDC to actually be complete, not just "eventually consistent." Happy to go deeper on any of it in the comments.
- domodomo 1mo agoIt is interesting to me! Do you know if the above applies to Datastream for CloudSQL as well, is it the same thing, or are you talking about a more DIY approach to CDC here? I assume not everyone using MySQL and BigQuery are using CloudSQL and so don’t have Datastream available to them.
- antonvs 1mo agoFYI, Datastream works with pretty much any MySQL instance, including self-hosted and even AWS RDS and MariaDB. See: https://docs.cloud.google.com/datastream/docs/sources-mysql https://docs.cloud.google.com/datastream/docs/sources-mysql Datastream solves the issues mentioned in the article, because it relies on CDC, which is the same solution the article is proposing. Note that the article seems to have a major error in its first sentence: “MySQL CDC syncs miss deletes and intermediate updates.” CDC is a solution to that issue, it doesn’t suffer from it. The second paragraph correctly describes the approach that has limitations: “a scheduled job selects rows…”
- gpaulbagetti 28d agohey thank you for your feedback! The description was giving the wrong idea indeed. I've just updated the article :)
- dcmerritt 1mo ago[flagged]