4 ms·
How big is a row? If you're saying the change speed is "100s of records per second or less" you could easily connect to each database as a replica and stream th
by slap_shot 9y ago
How big is a row? If you're saying the change speed is "100s of records per second or less" you could easily connect to each database as a replica and stream the change data capture to Kafka/Google Pub/Sub or Kinesis and stream the inserts into your destination table.
This should be possible for much less than $100,000. Assuming each database gets its own process for reading the change feed, you're looking at 20 (add more based on individual table/db needs) instances for the change data capture, 3-5 instances for Kafka/ZK brokers, and a handful of machines for reading the changes and writing them into the destination database.
The important thing here is not the number of databases or the size of the tables - its the speed at which inserts happen. 100 inserts/updates per second * 30k tables = 3M per second. That can be handled by the architecture I described above.
If this is a real problem you are trying to solve, I'd love to talk to you. I'm a co-founder of stealth company that is building better tools for engineers solving problems like this.
Our product can solve this exact problem very easily. I'd love to hear what you are working on and show you what we've built. Let me know how to contact you.
Edit: corrected my math.
- dba_leveling_up 9y agoThe size of the rows vary, but very few tables have large blobs, and there should be much fewer than 3M rows changing per second. Does the change capture use triggers or a different method?