5 ms·

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheapened.

sdairs 1y ago
I think even before dbt turned DE into "just write sql & yaml", there was an appreciable difference in DE vs SE. There was defo some DEs writing a lot of java/scala if they were in Spark heavy co's, but my experience is that DEs were doing a lot more platform engineering (similar to what you suggest), SQL and point-and-click (just because that was the nature of the tooling). I wasn't really seeing many DEs spending a lot of time in an IDE. But I think whats interesting from the post is looking at SEs adopting data infra into their workflow, as opposed to DEs writing more software.
isaacremuant 1y ago
Agreed. Weird distinction to pay less to people who did certain things and you could a high variance between "data engineers". Some who had only done a course and others that had extensive knowledge of software engineering practices were considered the same. Ridiculous.
craneca0 1y ago
yeah, i've seen large fortune 100 data and analytics orgs where the majority of folks with data engineering titles are uncomfortable with even the basics of git.
- vjvjvjvjghv 1y ago
  We have these at my company. They refuse to do any infrastructure work so you have to spoon feed the databases to them ready to go. It’s pretty annoying.
- Foobar8568 1y ago
  Or the basics of SQL...
kentm 1y ago
Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”. Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.
- itsoktocry 1y ago
  Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".
  - ozim 1y ago
    Only in places silly enough to believe software devs/enga cannot write SQL.
  - majormajor 1y ago
    The "write SQL for ETLs all day" job is a risky one right now since LLMs really lower the barrier for dealing with gnarly SQL. So it's still not a bad time to have your resume be as clear as possible that you're the "deals with complex distributed systems" SWE type instead.
    - mrugge 1y ago
      That's more of an analytics engineer role. LLMs lower the barrier to entry, but popular SQL queries are about correctness and flexibility and this often requires deep understanding and ownership of each filter and window function. This lower barrier can quickly can turn into enough rope to hang yourself.
  - CyberDildonics 1y ago
    Why would a software engineer not be able to do both roles?
    - omgwtfbyobbq 1y ago
mandeepj 1y ago
> Data engineering was software engineering from the very beginning. There it is! I found the post title was strange. Thanks for setting the record straight so succinctly.
majormajor 1y ago
"Data Engineering" being considered a different role from "regular" SWE predates DBT by... at least one decade? If not two? Probably folks working with Hadoop vs RDMS DBA jobs.
- snthpy 1y ago
  In work yes, but as a title? I only started seeing it called that around dbt origin.
mrugge 1y ago
Titles in software engineering have never mattered less than they do today. Energy worrying about titles or jealosy over specific tech ownership is best channeled into focus on customer, on problem to solve and on finding the best way to solve it as a team.
omgwtfbyobbq 1y ago
Part of the problem is that a BA/BSA who writes Python, SQL, etc... as part of their day to day work will get lumped with those who don't and their salary doesn't reflect their skills and work product. Obviously the same can apply in any given title, and does with data engineers like you pointed out, but it's not as simple as just title inflation.

sdairs 1y ago

I think even before dbt turned DE into "just write sql & yaml", there was an appreciable difference in DE vs SE. There was defo some DEs writing a lot of java/scala if they were in Spark heavy co's, but my experience is that DEs were doing a lot more platform engineering (similar to what you suggest), SQL and point-and-click (just because that was the nature of the tooling). I wasn't really seeing many DEs spending a lot of time in an IDE. But I think whats interesting from the post is looking at SEs adopting data infra into their workflow, as opposed to DEs writing more software.

isaacremuant 1y ago

Agreed. Weird distinction to pay less to people who did certain things and you could a high variance between "data engineers". Some who had only done a course and others that had extensive knowledge of software engineering practices were considered the same. Ridiculous.

craneca0 1y ago

yeah, i've seen large fortune 100 data and analytics orgs where the majority of folks with data engineering titles are uncomfortable with even the basics of git.

vjvjvjvjghv 1y ago
We have these at my company. They refuse to do any infrastructure work so you have to spoon feed the databases to them ready to go. It’s pretty annoying.
Foobar8568 1y ago
Or the basics of SQL...

kentm 1y ago

Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”. Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.

itsoktocry 1y ago
Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".
- ozim 1y ago
  Only in places silly enough to believe software devs/enga cannot write SQL.
- majormajor 1y ago
  The "write SQL for ETLs all day" job is a risky one right now since LLMs really lower the barrier for dealing with gnarly SQL. So it's still not a bad time to have your resume be as clear as possible that you're the "deals with complex distributed systems" SWE type instead.
  - mrugge 1y ago
    That's more of an analytics engineer role. LLMs lower the barrier to entry, but popular SQL queries are about correctness and flexibility and this often requires deep understanding and ownership of each filter and window function. This lower barrier can quickly can turn into enough rope to hang yourself.
- CyberDildonics 1y ago
  Why would a software engineer not be able to do both roles?
  - omgwtfbyobbq 1y ago

mandeepj 1y ago

> Data engineering was software engineering from the very beginning. There it is! I found the post title was strange. Thanks for setting the record straight so succinctly.

majormajor 1y ago

"Data Engineering" being considered a different role from "regular" SWE predates DBT by... at least one decade? If not two? Probably folks working with Hadoop vs RDMS DBA jobs.

snthpy 1y ago
In work yes, but as a title? I only started seeing it called that around dbt origin.

mrugge 1y ago

Titles in software engineering have never mattered less than they do today. Energy worrying about titles or jealosy over specific tech ownership is best channeled into focus on customer, on problem to solve and on finding the best way to solve it as a team.

omgwtfbyobbq 1y ago

Part of the problem is that a BA/BSA who writes Python, SQL, etc... as part of their day to day work will get lumped with those who don't and their salary doesn't reflect their skills and work product. Obviously the same can apply in any given title, and does with data engineers like you pointed out, but it's not as simple as just title inflation.