6 ms·

Data “engineering” is where all the cool kids go with no clue and create insane architectures to justify their incompetence

hobs 2y ago
As a data engineer I have seen absolutely bullshit pass for production, but it doesn't seem that different from all the other bullshit I have seen people deploy in my life. It is one of the few types of jobs I have worked were someone credulously offering adding five more layers to fix an issue with latency is a normal operating procedure though.
parpfish 2y ago
data engineer gives data science a run for their money when it comes to ambiguous job expectations. I’ve seen it mean anything from “distributed computing expert” to “knows SQL”
- marcosdumay 2y ago
  > to “knows SQL” There are still ways to go. I've seen it means "spends the days filling excel spreadsheets".
- sgarland 2y ago
  > knows SQL If by that you mean, “knows the commands to create, fill, and select from a table,” then yes. If you mean, “knows how to create a performant schema and queries that will serve them well into the future,” then no, absolutely not. OTOH, IME data folk are much more cheerful and willing to change things than devs when I point out the innumerable ways their DB choices are choking them. Devs more often fall on the side of “we don’t have time for that on our roadmap; can’t you just fix it?”
  - zelphirkalt 2y ago
    It is often not the devs telling you about the roadmap, but the management. Devs are more willing to fix things, if they are broken, but are not given the time.
- sevensor 2y ago
  > “knows SQL” Ah, clearly you’ve worked with a better class of data scientists than I.
- WorldMaker 2y ago
  I've worked with people where "knows SQL" meant "knows the Access query builder UI, sort of, and demands the ability to query Production databases with the awful SQL auto-generated from the Access UI".
the_af 2y ago
You can replace "data engineering" with "software engineering" in that sentence and it will still hold true...
- fifilura 2y ago
  Or chef or carpenter or automobile builder or priest or...
icedchai 2y ago
I've seen some "data engineering" scripts that were complete messes and beyond crazy. Some examples: Massively over engineered "pipelines" that process a few hundred rows a day, but somehow manage to take forever to run. Developers that didn't know SQL beyond "select * from table", so they do all their summarization in Python. Or, worse, I've seen a Python script calling a shell script calling R calling something else, several more layers deep, when the same result could've all been done in SQL with a few temporary tables. Oh, then I'm asked to "give this a code review before so-and-so does a deployment tomorrow." Uh, it's a little late to address any of the fundamentals, but there are hard coded paths everywhere...
- bob1029 2y ago
  I recently got a bit of a shocked reaction when I proposed to directly load daily files into temporary SQL tables and then use merge commands within the database to load the final tables. My use of code is essentially a shim between an SFTP client and SQL Server in this scenario. Maybe ~200 lines to connect, locate the files, run the bulk load operation, and then invoke the merge commands. Most of the fun bits are in the actual merge scripts. Once your data is safely inside the database (temporary load tables or otherwise), there really isn't a good excuse for pulling it out and playing a bunch of circus tricks on it. Moving and transforming data within the RDBMS is infinitely more reliable than doing it with external tooling. Your ETL code should be entirely about getting the data safely into the RDBMS. It shouldn't even be responsible for testing new/deleted/modified records. You really want to use SQL for that. You'll also be able to recruit more help if everything is neatly contained within the SQL tooling. In my scenario, business analysts can look at the merge commands and quickly iterate on the data pipeline if certain customers have weird quirks. They cannot do the same with some elaborate set of codebases, microservices, etc. One specific thing that really sold me on this path was seeing how CTEs and views can make the T part of ETL 10000000x easier than even the fanciest code helpers like LINQ.
  - snidane 2y ago
    The architecture is sound - typically called ELT these days. Dump contents of upstream straight into a database and apply stateless and deterministic operations to achieve the final result tables. SQL server is where this breaks though. You'll get yelled by DBAs for bad db practices like storing wide text fields without casting them to varchar(32) or varchar(12), primary keys on strings or no indexes at all, and most importantly taking majority of storage on the db host for tbese raw dumps. SQL Server and any traditional database scales by adding machines, so you end up paying compute costs for your storage. If you use a shared disk system with decoupled compute scaling from storage, then your system is the way to go. Ideally these days dump your files into a file storage like s3 and slap a table abstraction over it with some catalog and now you have 100x less storage costs and about 5-10x increased compute power with things like duckdb. Happy data engineering!

hobs 2y ago

As a data engineer I have seen absolutely bullshit pass for production, but it doesn't seem that different from all the other bullshit I have seen people deploy in my life. It is one of the few types of jobs I have worked were someone credulously offering adding five more layers to fix an issue with latency is a normal operating procedure though.

parpfish 2y ago

data engineer gives data science a run for their money when it comes to ambiguous job expectations. I’ve seen it mean anything from “distributed computing expert” to “knows SQL”

marcosdumay 2y ago
> to “knows SQL” There are still ways to go. I've seen it means "spends the days filling excel spreadsheets".
sgarland 2y ago
> knows SQL If by that you mean, “knows the commands to create, fill, and select from a table,” then yes. If you mean, “knows how to create a performant schema and queries that will serve them well into the future,” then no, absolutely not. OTOH, IME data folk are much more cheerful and willing to change things than devs when I point out the innumerable ways their DB choices are choking them. Devs more often fall on the side of “we don’t have time for that on our roadmap; can’t you just fix it?”
- zelphirkalt 2y ago
  It is often not the devs telling you about the roadmap, but the management. Devs are more willing to fix things, if they are broken, but are not given the time.
sevensor 2y ago
> “knows SQL” Ah, clearly you’ve worked with a better class of data scientists than I.
WorldMaker 2y ago
I've worked with people where "knows SQL" meant "knows the Access query builder UI, sort of, and demands the ability to query Production databases with the awful SQL auto-generated from the Access UI".

the_af 2y ago

You can replace "data engineering" with "software engineering" in that sentence and it will still hold true...

fifilura 2y ago
Or chef or carpenter or automobile builder or priest or...

icedchai 2y ago

I've seen some "data engineering" scripts that were complete messes and beyond crazy. Some examples: Massively over engineered "pipelines" that process a few hundred rows a day, but somehow manage to take forever to run. Developers that didn't know SQL beyond "select * from table", so they do all their summarization in Python. Or, worse, I've seen a Python script calling a shell script calling R calling something else, several more layers deep, when the same result could've all been done in SQL with a few temporary tables. Oh, then I'm asked to "give this a code review before so-and-so does a deployment tomorrow." Uh, it's a little late to address any of the fundamentals, but there are hard coded paths everywhere...

bob1029 2y ago
I recently got a bit of a shocked reaction when I proposed to directly load daily files into temporary SQL tables and then use merge commands within the database to load the final tables. My use of code is essentially a shim between an SFTP client and SQL Server in this scenario. Maybe ~200 lines to connect, locate the files, run the bulk load operation, and then invoke the merge commands. Most of the fun bits are in the actual merge scripts. Once your data is safely inside the database (temporary load tables or otherwise), there really isn't a good excuse for pulling it out and playing a bunch of circus tricks on it. Moving and transforming data within the RDBMS is infinitely more reliable than doing it with external tooling. Your ETL code should be entirely about getting the data safely into the RDBMS. It shouldn't even be responsible for testing new/deleted/modified records. You really want to use SQL for that. You'll also be able to recruit more help if everything is neatly contained within the SQL tooling. In my scenario, business analysts can look at the merge commands and quickly iterate on the data pipeline if certain customers have weird quirks. They cannot do the same with some elaborate set of codebases, microservices, etc. One specific thing that really sold me on this path was seeing how CTEs and views can make the T part of ETL 10000000x easier than even the fanciest code helpers like LINQ.
- snidane 2y ago
  The architecture is sound - typically called ELT these days. Dump contents of upstream straight into a database and apply stateless and deterministic operations to achieve the final result tables. SQL server is where this breaks though. You'll get yelled by DBAs for bad db practices like storing wide text fields without casting them to varchar(32) or varchar(12), primary keys on strings or no indexes at all, and most importantly taking majority of storage on the db host for tbese raw dumps. SQL Server and any traditional database scales by adding machines, so you end up paying compute costs for your storage. If you use a shared disk system with decoupled compute scaling from storage, then your system is the way to go. Ideally these days dump your files into a file storage like s3 and slap a table abstraction over it with some catalog and now you have 100x less storage costs and about 5-10x increased compute power with things like duckdb. Happy data engineering!