4 ms·
The issue is a lot of the tools in the ecosystem are positioning themselves as “do everything” and the supporting marketing/services is all about increasing num
by FridgeSeal 5y ago
The issue is a lot of the tools in the ecosystem are positioning themselves as “do everything” and the supporting marketing/services is all about increasing numbers of workarounds to enable this.
Using pandas but your data doesn’t fit in memory anymore? Here just use this fragile series of packages to distribute it across processors. Don’t reevaluate whether Pandas/Python is still the best choice, just layer more things in.
Run into issues with that approach? Here lift and shift your whole thing into Spark! Don’t ever optimise or actually engineer your process, just layer more things in!!!
- gunshai 5y agoSo guilty of this, haha
- teekert 5y agoYeah me too, But there is a point where you have to stop. I also at some point switched from Excel/Origin to Pandas because of scaling, and now I'm making my first steps in to learning SQL. A point comes where you have to grab and learn a different tool. But it's a difficult to thing to do when you have to get stuff done!
- fifilura 5y agoI think one approach that can help you is to start writing your pandas logic using only apply/merge/groupby etc. I.e. no iterators like df.iterrows() and similar. Once you have done that, the logic will be easily translatable to SQL, and your brain is prepared for the SQL/relational-algebra way of thinking. And good luck! You are doing it right!