6 ms·
Off-topic, but what is a "data engineer"?
by thearn4 10y ago
Off-topic, but what is a "data engineer"?
- nerdponx 10y agoI'm sure someone can come along with a better description, but it's kinda like devops/sysadmin but specifically for data storage and access. You could also look it up...
- mehaveaccount 10y agoThat's a good description for a data engineer from my end.
- xapata 10y agoI'd include database engineer in that bin. We used to call people DBAs but that seems a bit limiting now.
- sdenton4 10y agoAt some point, people realized that 90% of data science work was building pipes and keeping them clean. And then data engineering was invented so that the data scientists wouldn't have to get their hands as dirty. :)
- fnord123 10y agoA snappier version which gives the flavour: Data scientists use {tensor flow, scikit-learn, parquet, hadoop, spark, scrapers, etc}. Data engineers write {tensor flow, scikit-learn, parquet, hadoop, spark, scrapers, etc}.
- nl 10y agoIn my team, "data engineer" refers to someone who deals with the ETL process, but also some feature engineering (ie, designing and extracting features. There's a lot of cross over here with the work Data Scientists do, but often the tools are different: for example, in the Python world the data engineer will use Pycharms while the data scientist will use Jupyter.