3 ms·
Think about the use-case where you want all your Relational data to be denormalised and written to a S3 bucket and then loaded into something like Redshift for
by hashhar 8y ago
Think about the use-case where you want all your Relational data to be denormalised and written to a S3 bucket and then loaded into something like Redshift for analytical loads.
If you start pushing the data from your application itself then you introduce the problem of dual-writes (what if S3 write failed? what if RDBMS failed?).
If instead you treat your RDBMS as the source of truth and use something like PostgresSQL's logical decoding feature to listen to the replication stream and then push the events from that stream to S3 you can achieve guaranteed-eventual-consistency.
The best way would've obviously been to make your application write to a Message Bus (Kafka etc.) and then some other application would persist that message to your RDBMS. This approach requires more development effort and introduces all the issues that distributed systems face: Read your writes, read follow writes, write follow reads etc.
- andy_ppp 8y agoYes, I’ve worked on systems that use Kafka to do exactly this...
- hashhar 8y agoI'd love to discuss more on this topic with you since I am currently working on implementing a proof of concept for the same. Since I've just had almost 1 year of software development experience (2017 grad here) the task feels too daunting to me. Also it's just a one-man team since its a proof of concept.
- hashhar 8y agoOh man, rereading the comments above made me realise i replied to the wrong comment. Doesn't matter though.