3 ms·
Do you use some kind of library to help with transformations? If you have some volume using plain python can become a bottleneck no? Any tips in this area would
by Cwizard 5y ago
Do you use some kind of library to help with transformations? If you have some volume using plain python can become a bottleneck no? Any tips in this area would be appreciated.
- WhyCause 5y agoI'm not the person you were asking, but I, too, use mostly Python for ETL tasks as a good chunk of my job. Python's speed, or lack thereof, is rarely an issue. Most tasks that I write/run finish in less than a minute, including the time for transformations. Some of those can get fairly hairy, as well. Most of the time is spent querying the DB and uploading to the remote. For the tasks where the transformations are too complex, you can greatly decrease the time necessary by tweaking the queries (i.e., limiting rows to just changed rows, etc.). And frankly, once you realize most data doesn't need to update in real time, it doesn't matter how long the transformation step takes (as long as it's fewer than about 23 1/2 hours).
- elchief 5y agopyspark if big-ol' data sets pandas otherwise
- davinchia 5y agoAirbyte engineer here. We use Dbt to normalise the delivered data at the final destination warehouse. If it's not a destination warehouse, how transformations are handled depends on the connector implementation.