4 ms·

This doesn't ring true with how I've seen the terms used. Highly scalable distributed systems are the bread and butter of backend engineering in modern tech co

This doesn't ring true with how I've seen the terms used. Highly scalable distributed systems are the bread and butter of backend engineering in modern tech companies. Even specialists working on storage layer tech in particular are not called "data engineers," just backend engineers, although with more systems/infrastructure focus than others. People with the title "data engineer" at my company pretty much do ETL pipelines on Hadoop. I've done some ETL pipelines on Hadoop; seems like a tool that should be in the portfolio of any SWE. But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop; I have even seen them build RPC interfaces out of INSERT statements. Whereas a SWE is looking at a broader suite of options, and probably defaulting to backend services / OLTP databases. Hence my confusion. Maybe our use of the term is anachronistic? Or maybe they just know a bunch of HiveQL or pipeline design patterns that are too complex to fit into the heads of people who can also write services?

squeaky-clean 7y ago
I would say these terms are not exclusive of each other. Like Parallelogram, square, rectangle. All 3 of your descriptions would be different types of Backend Engineers to me, with focuses on systems or data (or both) or something else. Sometimes the storage team is focused on generically keeping storage working while another team actually works on the "big data" applications, but often these teams design and plan very hand-in-hand. > But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop Maybe someone who sees Hadoop as their only hammer but IMO a "good" data engineer would go "How much data? 300MB? And you need to run this how often? Once? You're sure just once? Like really sure? Okay here is a bash script."

squeaky-clean 7y ago

I would say these terms are not exclusive of each other. Like Parallelogram, square, rectangle. All 3 of your descriptions would be different types of Backend Engineers to me, with focuses on systems or data (or both) or something else. Sometimes the storage team is focused on generically keeping storage working while another team actually works on the "big data" applications, but often these teams design and plan very hand-in-hand. > But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop Maybe someone who sees Hadoop as their only hammer but IMO a "good" data engineer would go "How much data? 300MB? And you need to run this how often? Once? You're sure just once? Like really sure? Okay here is a bash script."