3 ms·

The difference is where the difficulty of your problem lies. I work with serving and reporting ads, so most of our actual logic is simple. Glorified ETL work an

The difference is where the difficulty of your problem lies. I work with serving and reporting ads, so most of our actual logic is simple. Glorified ETL work and fancy caching layer really. The problem comes about that we can have peaks of 20k reads and 20k writes per second with strict guarantees on response time and data consistency. All at the same time needing to be kept consistent across multiple datacenters in several regions. Your typical application won't hold up in that environment or can become extremely difficult to maintain. And I've met people at conferences who would say my use case still isn't "big data" and is pennies compared to their data streams. It really does become a class of algorithms and solutions on its own, just dealing with your real-world ability to manage that much data.

closeparen 7y ago
This doesn't ring true with how I've seen the terms used. Highly scalable distributed systems are the bread and butter of backend engineering in modern tech companies. Even specialists working on storage layer tech in particular are not called "data engineers," just backend engineers, although with more systems/infrastructure focus than others. People with the title "data engineer" at my company pretty much do ETL pipelines on Hadoop. I've done some ETL pipelines on Hadoop; seems like a tool that should be in the portfolio of any SWE. But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop; I have even seen them build RPC interfaces out of INSERT statements. Whereas a SWE is looking at a broader suite of options, and probably defaulting to backend services / OLTP databases. Hence my confusion. Maybe our use of the term is anachronistic? Or maybe they just know a bunch of HiveQL or pipeline design patterns that are too complex to fit into the heads of people who can also write services?
- squeaky-clean 7y ago
  I would say these terms are not exclusive of each other. Like Parallelogram, square, rectangle. All 3 of your descriptions would be different types of Backend Engineers to me, with focuses on systems or data (or both) or something else. Sometimes the storage team is focused on generically keeping storage working while another team actually works on the "big data" applications, but often these teams design and plan very hand-in-hand. > But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop Maybe someone who sees Hadoop as their only hammer but IMO a "good" data engineer would go "How much data? 300MB? And you need to run this how often? Once? You're sure just once? Like really sure? Okay here is a bash script."

closeparen 7y ago

This doesn't ring true with how I've seen the terms used. Highly scalable distributed systems are the bread and butter of backend engineering in modern tech companies. Even specialists working on storage layer tech in particular are not called "data engineers," just backend engineers, although with more systems/infrastructure focus than others. People with the title "data engineer" at my company pretty much do ETL pipelines on Hadoop. I've done some ETL pipelines on Hadoop; seems like a tool that should be in the portfolio of any SWE. But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop; I have even seen them build RPC interfaces out of INSERT statements. Whereas a SWE is looking at a broader suite of options, and probably defaulting to backend services / OLTP databases. Hence my confusion. Maybe our use of the term is anachronistic? Or maybe they just know a bunch of HiveQL or pipeline design patterns that are too complex to fit into the heads of people who can also write services?

squeaky-clean 7y ago
I would say these terms are not exclusive of each other. Like Parallelogram, square, rectangle. All 3 of your descriptions would be different types of Backend Engineers to me, with focuses on systems or data (or both) or something else. Sometimes the storage team is focused on generically keeping storage working while another team actually works on the "big data" applications, but often these teams design and plan very hand-in-hand. > But when you give a data engineer a problem, they are sure to suggest an ETL pipeline in Hadoop Maybe someone who sees Hadoop as their only hammer but IMO a "good" data engineer would go "How much data? 300MB? And you need to run this how often? Once? You're sure just once? Like really sure? Okay here is a bash script."