3 ms·

Author of the post here, I had no idea it got posted here and just noticed a huge traffic spike. So, hello! Something I got asked a lot in response to this pos

by sdairs 3y ago

Author of the post here, I had no idea it got posted here and just noticed a huge traffic spike. So, hello! Something I got asked a lot in response to this post, or maybe berated for, is that there is a lot more to Data Engineering than SQL. FWIW, I agree. There is so, so much more to DE than SQL. The point of this post was purely to cover entry-level DE, and not 'This is the only thing you'll ever need in your career'. In my view, developing your SQL skills teaches you a lot of the fundamentals of working with data, enough that you can start to use it in a professional setting. From there, you will develop a lot more skills than just SQL, but your journey could take you in so many different paths that I find it hard to agree with folks who say you should always learn X,Y,Z specific tools. Some folks will end up in Python shops, or Scala, or Bash, or no-code, Airflow, Dagster, Prefect, Control-M, Spark, Flank, Pandas, Polrs, BigQuery, Snowflake, Redshift, ClickHouse, MSSQL, Hadoop, dbt....Some might even go down the K8S and Terraform road, becoming more of a platform-focused DE... There is just so much potential variance depending on where you work and who you are, that it doesn't sit well with me to shoe-horn everyone into a pattern of "First learn Airflow, then learn BigQuery, then learn Spark" when you could have a successful career and never touch a single one of those tools. So, the reason I focused on SQL here, is that in my career consulting with hundreds of DE teams - the only consistent skill across every single team has been SQL. I do not believe SQL is all you'll ever need, but I do think its all you need to get started. It's the only skill I see being relavent regardless of which team you join, and that will still be relevant in 15+ years time.

nevi-me 3y ago
Out of curiousity, what then makes a difference between a data engineer and an ETL developer? > There is just so much potential variance depending on where you work and who you are, that it doesn't sit well with me to shoe-horn everyone into a pattern of "First learn Airflow, then learn BigQuery, then learn Spark" when you could have a successful career and never touch a single one of those tools. I agree with this. My observation has been that people who start/stay in 1 place for too long end up knowing some tool(s) very well but not others. So surely then we should abstract away the tools and identify what's common about them (and SQL)? When I produced classroom content for the graduate program that we ran at work, I used to focus more what's being done (ETL, data management, perf optimisation) and primitives like HTTP for REST, bits about DevOps, security, analysis. The idea being that if one is fluent in understanding the underlying problem, tooling becomes easier to pick up. So I still think that there's a roadmap, and that roadmap is the underlying concepts that data engineering tries to solve. I've found that you can teach someone enough about: - ETL & data management (incl modelling) - DBA - DevOps - Software (architecture and development) to the point where they're not as good as the best people in those fields (if they're confining themselves so), but to the point where they become good at the parts of those fields that are relevant to data engineering. That's often my roadmap.
- sdairs 3y ago
  I don't disagree with you, and I believe you're probably teaching people many of the right things that will set them up to do well in the future. But...the tricky part about learning just concepts and not the skills is that they are quite abstract and its very hard for a junior to immediately see how they can applied in practise when getting their first job. They're also quite hard to demonstrate to an employer. Learning SQL gives you some tangible to do that can help you to learn something about many of those areas, maybe not too deep, but while also getting a hands-on skill that easier to demonstrate to an employer. The reason I think SQL is the right skill there is that its specific enough that its actually useful, but its generic enough that its widely applicable to many employers (rather than specialising on one specific tool, like say Spark). For example, you might learn the theory behind internal combustion engines, but does that make you more attractive as a Mechanic's apprentice than a kid who's rebuilt his lawn mower's engine? Neither are 100% perfect, but the hands on experience might be easier to demonstrate that you can be effective on day 1, and you can learn some of that theory on the job. Idk if that's the best analogy, and I'm not saying it's 100% the case here. I did a degree in Compture Science and learnt a lot of theory, its all super interesting and personally I still love understanding all of the theory and low level details behind software...but many of the DEs working in Enterprise don't have that background (or interest). And that's not a bad thing, they're just as competent data engineers as I am. I guess it's a matter of what your end goal (and timeline) is. Some folks aren't looking to be a word-class DE expert, for some its just a job and DE is a booming space with above-average compensation. In that case, you're probably not inclined to sit in a class room and learn all the theory, you just want to land the job and get on with it. Nothing wrong with that at all, some of the best get-stuff-done engineers are like this. For others, even if they find the space fascinating and they want to be at the top, they might be more interested in one very specific hand-on area and they only need as much theory as they pick up day-to-day. Anyway, I don't think your approach is wrong, and everything you're teaching is great for someone who is genuinely interested in the field of DE - but I also don't think its all absolutely necessary if you just want to get your first DE job. RE. DE vs ETL developer; I think DE has become more of an umbrella term like "Software Engineer" which encompasses many different roles. An ETL developer probably falls under Data Engineering these days, its just a specific niche.

by sdairs 3y ago

nevi-me 3y ago

Out of curiousity, what then makes a difference between a data engineer and an ETL developer? > There is just so much potential variance depending on where you work and who you are, that it doesn't sit well with me to shoe-horn everyone into a pattern of "First learn Airflow, then learn BigQuery, then learn Spark" when you could have a successful career and never touch a single one of those tools. I agree with this. My observation has been that people who start/stay in 1 place for too long end up knowing some tool(s) very well but not others. So surely then we should abstract away the tools and identify what's common about them (and SQL)? When I produced classroom content for the graduate program that we ran at work, I used to focus more what's being done (ETL, data management, perf optimisation) and primitives like HTTP for REST, bits about DevOps, security, analysis. The idea being that if one is fluent in understanding the underlying problem, tooling becomes easier to pick up. So I still think that there's a roadmap, and that roadmap is the underlying concepts that data engineering tries to solve. I've found that you can teach someone enough about: - ETL & data management (incl modelling) - DBA - DevOps - Software (architecture and development) to the point where they're not as good as the best people in those fields (if they're confining themselves so), but to the point where they become good at the parts of those fields that are relevant to data engineering. That's often my roadmap.

sdairs 3y ago
I don't disagree with you, and I believe you're probably teaching people many of the right things that will set them up to do well in the future. But...the tricky part about learning just concepts and not the skills is that they are quite abstract and its very hard for a junior to immediately see how they can applied in practise when getting their first job. They're also quite hard to demonstrate to an employer. Learning SQL gives you some tangible to do that can help you to learn something about many of those areas, maybe not too deep, but while also getting a hands-on skill that easier to demonstrate to an employer. The reason I think SQL is the right skill there is that its specific enough that its actually useful, but its generic enough that its widely applicable to many employers (rather than specialising on one specific tool, like say Spark). For example, you might learn the theory behind internal combustion engines, but does that make you more attractive as a Mechanic's apprentice than a kid who's rebuilt his lawn mower's engine? Neither are 100% perfect, but the hands on experience might be easier to demonstrate that you can be effective on day 1, and you can learn some of that theory on the job. Idk if that's the best analogy, and I'm not saying it's 100% the case here. I did a degree in Compture Science and learnt a lot of theory, its all super interesting and personally I still love understanding all of the theory and low level details behind software...but many of the DEs working in Enterprise don't have that background (or interest). And that's not a bad thing, they're just as competent data engineers as I am. I guess it's a matter of what your end goal (and timeline) is. Some folks aren't looking to be a word-class DE expert, for some its just a job and DE is a booming space with above-average compensation. In that case, you're probably not inclined to sit in a class room and learn all the theory, you just want to land the job and get on with it. Nothing wrong with that at all, some of the best get-stuff-done engineers are like this. For others, even if they find the space fascinating and they want to be at the top, they might be more interested in one very specific hand-on area and they only need as much theory as they pick up day-to-day. Anyway, I don't think your approach is wrong, and everything you're teaching is great for someone who is genuinely interested in the field of DE - but I also don't think its all absolutely necessary if you just want to get your first DE job. RE. DE vs ETL developer; I think DE has become more of an umbrella term like "Software Engineer" which encompasses many different roles. An ETL developer probably falls under Data Engineering these days, its just a specific niche.