4 ms·
Everyone wants to fire up Tensorflow, Keras and PyTorch these days. Fewer people want to work in Airflow and SSIS, spend days tuning ETL, etc. This is the domai
by pbourke 6y ago
Everyone wants to fire up Tensorflow, Keras and PyTorch these days. Fewer people want to work in Airflow and SSIS, spend days tuning ETL, etc. This is the domain of data engineering, which bridges software engineering and data science with a dash of devops. I’ve been working in this field for a couple of years and it’s clear to me that data engineering is a necessary foundation and impact multiplier for data science.
- berzerk 6y agoI'm finding myself really enjoying this type of work and I think I would like to specialize in it. Any good learning resources you used on your path to where you are now?
- tajd 6y agoHow might you recommend moving into this field more?
- stainforth 6y agoAny formalized paths I could take to enter this field?
- castlecrasher2 6y agoI got into data engineering by starting in ETL (DataStage) and learning about cloud services (AWS) on my own and getting my next job in a cloud-based SaaS startup.
- deleted 6y ago[deleted]
- jnwatson 6y agoDon't forget data cleaning. A huge issue I've seen is just getting sufficient data of a high enough quality. Also, (for supervised classification problems) labelling is a big problem. It is almost as if we need a "data janitor" title.
- ajb 6y agoPhht you don't want to call it data janitor; no-one good will want that title. At least call it Data Integrity Engineer or something reasonably high-status.
- WrtCdEvrydy 6y agoMachine Learning Data Integrity Engineer
- anthuswilliams 6y agoAt my company we have just created a position called Data Steward.
- ska 6y agoData Sanitation Engineer :)
- google234123 6y agoData cleaning always sounded suspicious to me.
- jboggan 6y agoAs a data engineer I've often billed myself as the Charlie Kelly of data
- Breza 6y agoThis is a hugely important point. Having the right data is most of the battle in the typical project. I'm fortunate to run the data science department at a company with an amazing and supportive team of devs. We work together to make sure that all data is recorded in the most useful way and we talk regularly. Without their support, my models wouldn't be very useful.