5 ms·
1) Learn to do as much in plain Python as possible, focus on lazy evaluation (itertools, yielding, ...), you'll be able to process gigabytes with a tiny memory
by drej 8y ago
1) Learn to do as much in plain Python as possible, focus on lazy evaluation (itertools, yielding, ...), you'll be able to process gigabytes with a tiny memory footprint, deployment will be a breeze etc.
2) Get to know some of the basic Python data processing/science packages like pandas, numpy, scipy etc.
3) Get used to writing short shell scripts - they probably won't be a part of your pipeline, but data engineering, especially development, involves a lot of data prep that coreutils will help you with. Throw in a bit of jq and you'll handle a lot of prep work.
4) Only once you've gotten used to the above, look at Dask, PySpark, Airflow etc. It really depends on your use cases, but chances are you won't have to touch these technologies at all.
Bottom line - wait with the heavyweight tools, they might be needlessly powerful. Also, work closely with DevOps, because the deployment side of things will help you understand consequences of your actions.
- bdhess 8y agoI like this answer. I'd also add: 0) Spend time finding data sets that actually interest you. If you're working with data that you're actually excited about, the questions will flow naturally. And when the questions flow naturally, there's an obvious path to picking up new techniques as you need them.
- drej 8y agoA million times yes. Whenever I teach people anything, I force them to use the tools on problems they actually have. You can solve a lot for Alice and Bob, but I'd rather you solved something for yourself.
- lmiller1990 8y agoThis was the thing that helped me most. I just can’t make myself excited about the price of a house or the size of a tumor. Stats and predictions for my favorite sport, on the other hand, gave me context that naturally made me ask questions regarding the accuracy of my results and how I can improve them.
- zabana 8y agoThanks for your suggestions, I want to deal with the least amount of abstraction possible so I can truly understand the benefits of using industry renowned frameworks (like the ones you've listed in 4.)
- glup 8y agoWhat do yo mean by "least amount of abstraction"? Unless I am misunderstanding what you mean by that, suggestions by user drej entail less abstraction than Spark / Hadoop etc. Also there's a difference between familiarity with "industry-renowned" frameworks and strong knowledge of the core small-scale libraries. FWIW, I'd certainly prefer to hire a developer/data scientist with the latter experience (if you feel like you have already mastered this, then congrats!).
- JacobDotVI 8y agoCalling out the deployment side is huge - You may have tested your ETL software itself, but once real data starts flowing through the pipes all bets are off. You might be interested in this read about the concept of pipeline debt: https://medium.com/@expectgreatdata/down-with-pipeline-debt-introducing-great-expectations-862ddc46782a https://medium.com/@expectgreatdata/down-with-pipeline-debt-... In full disclosure: I'm co-founder of Superconductive Health, our team co-created great-expectations and co-authored the above blog post.
- glup 8y agoDo you have recommendations on resources for learning 1)? I'd add a possible 5) of Jupyter notebooks (or some dashboard framework) + some visualization library for learning to juice actionable analyses from whatever data