I would take this a step further and say NOBODY should write their own ETL. In a world where: 1. SaaS services have APIs 2. Your database is hosted in the clo

I would take this a step further and say NOBODY should write their own ETL. In a world where: 1. SaaS services have APIs 2. Your database is hosted in the cloud 3. You use a standard SQL data warehouse that is also hosted in the cloud. ETL from (1, 2) to (3) is a completely standard problem, and you should be able to buy a fully-automated solution. My company (Fivetran) does this as a service. We've replaced lots of homebrew data pipelines built by our customers, and we always see the same issues: * Homebrew ETL pipelines use fancy big-data tech like Hadoop and Kafka in places where it has no relevance, like syncing your 20 GB Salesforce instance. * Homebrew ETL pipelines don't deal with all the dark corners of the data sources, such as: what happens when someone adds a new custom column? What happens when your MySQL read replica fails over and a new binlog starts? Etc. The lesson being, don't do this yourself.