4 ms·
Yikes, please tell me they were hired as a junior? Obviously in your case some trivial SQL would have sufficed, which I feel like should have been caught in an
by Grimm1 5y ago
Yikes, please tell me they were hired as a junior?
Obviously in your case some trivial SQL would have sufficed, which I feel like should have been caught in an interview but could be excused for a junior eng.
Also Pandas is not an amazing choice for data engineering tasks in general IMO, type conversion wonkiness is one reason I have questions about people who decide to use Pandas for legitimate data engineering tasks (as opposed to exploratory work or analysis).
- FridgeSeal 5y agoThe issue is a lot of the tools in the ecosystem are positioning themselves as “do everything” and the supporting marketing/services is all about increasing numbers of workarounds to enable this. Using pandas but your data doesn’t fit in memory anymore? Here just use this fragile series of packages to distribute it across processors. Don’t reevaluate whether Pandas/Python is still the best choice, just layer more things in. Run into issues with that approach? Here lift and shift your whole thing into Spark! Don’t ever optimise or actually engineer your process, just layer more things in!!!
- gunshai 5y agoSo guilty of this, haha
- teekert 5y agoYeah me too, But there is a point where you have to stop. I also at some point switched from Excel/Origin to Pandas because of scaling, and now I'm making my first steps in to learning SQL. A point comes where you have to grab and learn a different tool. But it's a difficult to thing to do when you have to get stuff done!
- fifilura 5y agoI think one approach that can help you is to start writing your pandas logic using only apply/merge/groupby etc. I.e. no iterators like df.iterrows() and similar. Once you have done that, the logic will be easily translatable to SQL, and your brain is prepared for the SQL/relational-algebra way of thinking. And good luck! You are doing it right!