4 ms·
Data is a pretty major component of the programmer's craft, whether it's DBs, I/O, or blobs. Most any experienced programmer is a "Data Engineer".
by angel_j 7y ago
Data is a pretty major component of the programmer's craft, whether it's DBs, I/O, or blobs. Most any experienced programmer is a "Data Engineer".
- threeseed 7y agoYou couldn't be more wrong. Data Engineer as a term came out of the Data Science space. Which means that you will be expected to have skills around Spark, Data Lakes, ETL at scale, validation, schema management and syncing, data catalogs etc. It's not some general skill just like you wouldn't say every programmer is a Network Engineer because they use a HTTP client.
- zeta0x10 7y agoDon't think he is that wrong. Putting aside Spark, the remaining terms are pretty broad. Data Lakes: Is this our new fancy term describing "data". So what distinguishes "data lakes" from "data"? ETL: I would guess 95% of application programs take input, parse it (extract), do some data wrangling (transform) and save the result somewhere else (load). Validation: Again a broad term. Do you mean validation of statistical models? Without validation your predictions are worthless so I guess it is a standard thing to do if you want to do any kind of machine learning. Schema Management and data catalogs: Standard DB stuff I would say. We just like to define new job descriptions. It's the same with DevOps, which seems to be the new term for System Administrator.
- ianamartin 7y agoA shorter version of how I described it above is that Data Engineers intentionally decouple logic from data so that ETL processes can be managed at scale (not necessarily talking about Big Data when I say scale. Sometimes lots of small ETL processes are just as difficult to manage as Big Data).