4 ms·
I'm not the person you were asking, but I, too, use mostly Python for ETL tasks as a good chunk of my job. Python's speed, or lack thereof, is rarely an issue.
by WhyCause 5y ago
I'm not the person you were asking, but I, too, use mostly Python for ETL tasks as a good chunk of my job.
Python's speed, or lack thereof, is rarely an issue. Most tasks that I write/run finish in less than a minute, including the time for transformations. Some of those can get fairly hairy, as well. Most of the time is spent querying the DB and uploading to the remote.
For the tasks where the transformations are too complex, you can greatly decrease the time necessary by tweaking the queries (i.e., limiting rows to just changed rows, etc.). And frankly, once you realize most data doesn't need to update in real time, it doesn't matter how long the transformation step takes (as long as it's fewer than about 23 1/2 hours).