3 ms·
Isn’t pandas centric ETL much more memory intensive and less compute efficient than using SQL?
by gregw2 2y ago
Isn’t pandas centric ETL much more memory intensive and less compute efficient than using SQL?
- thibautdr 2y agoI wrote an article questioning the use of Pandas for ETL. I invite you to read it: https://medium.com/@thibaut_gourdel/should-you-use-pandas-for-etl-0ecf56c84d57 https://medium.com/@thibaut_gourdel/should-you-use-pandas-fo...
- rkozik1989 2y agoThat's kind of the tradeoff you make with any low-code/no-code technology. You leverage prebuilt components and string them together to achieve some kind of task. Which isn't most efficient thing in the world to do but it does work assuming you have enough compute resources to throw at it, and return what you generally achieve is an end product that's completed faster than the traditional development route. You could just use SQL but then you'd have to develop and test the entire infrastructure to support your component-oriented architecture from scratch, and at that point you're kind of just reinventing the wheel because that's basically just pandas with less features. Low-code is kind of just Authorware for a new generation... assuming you're old enough to remember that technology.