9 ms·

There is no data engineering roadmap

212 points by sebg 3y ago

catsarebetter 3y ago
Now that I think about it you're right. All the maps for dev careers so far: 1. Staff eng map - https://staffeng.com/stories/ https://staffeng.com/stories/ 2. Bootstrappers - Indiehackers 3. Ecommerce - StarterStory 4. Software Engineering - /u/softwareengineering 5. Higher salary - Blind, reddit 6. Technical interviews - Leetcode forums Looks like the shape of the solution can be forums, books, articles, what do you think you'll do?
charles_f 3y ago
I'm not a data engineer but what about dimensional modelling and stuff like the data warehouse toolkit? I would imagine that it is somewhat important?
- di456 3y ago
  It is important, but when the shift to cheap compute and storage hit an inflection point it became possible to build out wide tables that combine the characteristics of facts and dimensions, and real data modeling took a back seat. That possibility led to bias toward moving quickly with less emphasis on making the data model sustainable. I'm seeing the pendulum start to swing the other way, where the complexity of these scrappy and loosely structured data models is hampering the ability to innovate, and even slowing down the business. The models are often inflexible and hard to maintain with hidden bugs and gotchas.
  - bjornasm 3y ago
    Interesting, this makes me think of a process I am going through, where I have a couple of very wide tables where I feel a need to build fact tables simply to get a better understanding of the data and the domain they are from. I have already stumbled a couple of times as the groupings was not as expected at all. Think this was the last push for me to build those fact tables once and for all, to get an overview and also to be able to add my own inferred groups to the data.
    - di456 3y ago
      Sounds like you are on the right track. A non-technical benefit is there won't be so much context to keep in your head when triaging data issues in the future. A database table is very similar to the separation of concerns problem of core software engineering. A table really just is an abstraction for storing and fetching various data points. Finding the right balance is part art and part science.
- tfehring 3y ago
  A lot of it is dated but the first 3 chapters are still well worth a read for an aspiring DE.
alexleeds 3y ago
I was going to say, ‘Bullsh*t Learn SQL.’ Then I read the post.
- quickthrower2 3y ago
  Then learn what sequence of actions your data store(s) takes based on that SQL. Relational engines are great but can need tuning (add indexes, de/normailze, materialized views or not) and then key/value ones are going to be doing scans from the entire cluster generally unless you get the cluster key in there and ideally the primary key too.
- jldugger 3y ago
  At least you're eventually consistent ^_^
phillipcarter 3y ago
Really enjoyed this article. Made me feel a tad inspired to write a similar one, "There is no Product Management roadmap", tailored for the many people looking to break into that discipline. Loooots of parallels.
- ghiculescu 3y ago
  i think you're probably onto something, what do you think the equivalent to SQL would be in product management?
  - kozikow 3y ago
    Jira? Powerpoint?
    - ghiculescu 3y ago
      Nah. Being good at Jira makes you worse at product management.
  - phillipcarter 3y ago
    It's a little more loosie goosie, but there's really no single tool. The intuition is that a good PM on a team helps guide things along at all stages such that when anyone looks back, they think "this never would have happened without them driving this". But the details of that aren't easily moldable into some framework or tool you can use for other projects, teams, or organizations.
claytonjy 3y ago
I agree with so much in this article: there's no narrow path, SQL is king, and enough with the damn influencers. But how many entry-level DE jobs will hire folks without programming experience? I've never heard of such a thing in the US, perhaps the UK market is different? If you're new to DE you're unlikely to have any experience managing a database or using common ETL tooling, so your transferable skills are basically SQL and python, and that's what every interview process is going to screen you for.
- benrutter 3y ago
  I've worked in a big company that had explicit "junior data engineer" roles, but even those would expect some programming experience - either through a bootcamp, degree or related role like data analysis. But basically yeah, as a UK data engineer, I'd say what you've said is pretty much right on- programming language + SQL will get you hired much more than "I've used this ETL tool"
rectang 3y ago
> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which when it gets changed breaks in unpredictable ways. And the cultural traditions of testing and validation are underdeveloped because SQL has no core testing story — people count on joins being logically correct, but don’t go to the trouble to prove that true. Vector operations are just hard to get right in any ecosystem and if you don’t stay humble and apply skepticism to your work, it’s going to be wrong a lot. I try to do stuff in pandas or similar — which has its own problems, but at least allows for testable library code. But in the field of data engineering, that’s swimming against the tide.
- jerrygenser 3y ago
  I've done written heavily tested code in python. But also heavily tested code in SQL. When I was doing data engineering 5 years ago, dbt was a new tool and I used that to write tests, ci/cd, etc. in data engineering. It was a game changer.
- jci 3y ago
  Can’t speak for SQL _in general_ but in Postgres land pgTAP is wonderful. Invest in writing some factory functions, and you have the ability to write a very expressive test suite. We use Postgres functions to provide a data interface that is thoroughly tested. This allows us to ensure that any migration does not break any contracts.
- xmprt 3y ago
  > Data engineering can be unsatisfying if you thrive on writing reliable systems This is what I needed for me to stop considering data engineering. Not that I was thinking about it a lot. I feel very happy with my current job but the thought of the code I write being so brittle as to paralyze me from making any changes terrifies me.
  - 3y ago
chrismarlow9 3y ago
Good article but a bit short. Anyway two books I can recommend for data engineering: Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB. High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bunch of headaches on the data server side by doing things client side and caching/prefetching more than what the user asked for.
- EstesPark 3y ago
  To add some additional notes - Designing Data Intensive Applications was a heavy read to dive into for me as a junior engineer. I took a step back after the first couple of chapters and read Understanding Distributed Systems[0] which was a fantastic primer. [0] https://understandingdistributed.systems https://understandingdistributed.systems
- SOLAR_FIELDS 3y ago
  Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform data and present it to end users for analysis (BI) in such a way that it’s not painful or arduous to consume it, and ensuring that it is clean and accurate. In other words, how well you move the data has always been a secondary point to how people can access and use said data. Note that I’m not in a dedicated data engineering role, but have worked adjacent to it for the better part of a decade and have done a fair amount of data engineering in that time.
  - Jarwain 3y ago
    This kinda sounds like higher level concerns where the books cover lower level problems?
markus_zhang 3y ago
Does DevOps ever hire juniors? I applied many times as a DE but never got any reply, not even one interview. I'm preparing to get a RH cert and see if it opens some doors.
- wyclif 3y ago
  Sounds like a good idea despite how many people diss certs on this forum. I've been trying to get a entry-level DevOps role for over a year and I can't even get a nibble—nothing! No replies, no interviews. I feel like I'm a competent backend dev, even though a lot of times I feel imposter syndrome when I look at the things other developers are doing.
analyte123 3y ago
Can't forget about SQL. It's also worth noting that designing any central data system without a SQL interface is probably a mistake. The people making it may be more comfortable with a REST interface or Python API, but data users will show up and demand that data be SQL-accessible. So now you get to hack it in, or help them develop their own shadow database that is poorly replicated from the main system.
- idinnoaname 3y ago
  The author stated that the only logical starting point is to learn to solve problems in a dialect of SQL then approach problems of the same type in other dialects of SQL. You’re entirely right, and the author seems to agree to an extent.
iblaine 3y ago
The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst. SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.
- woleium 3y ago
  for a moment there I thought you said "what makes data engineers is excel"
  - jeremyjh 3y ago
    do not speaks its name
- sanderjd 3y ago
  The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.
  - _bk90 3y ago
    A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.
    - sanderjd 3y ago
      I would certainly prefer it if that were the case, but it largely is not, in my experience.
      - _bk90 3y ago
        It seems like your experience doesn't overlap with data engineering, in that case.
dikei 3y ago
One the most important thing about SQL is learn to read the output of EXPLAIN queries: it's the key to understand why your queries are not running as fast as you'd like.
wokwokwok 3y ago
I think there's a fair caveat here: In a context free situation, sure, just learn SQL. It's the workhorse, and you'll definitely use it. If you have to start somewhere, start there. ...buuuut, obviously look at job ads. If all the junior jobs say "experience with SnowPanda" or "Working knowledge of FancyCloudPlatform" or whatever the name of the latest hotness is, you'd be daft not to invest some time and effort into whatever everyone is actually asking for to. You don't need someone to tell you what those things are; literally look at the bullet points on job ads... but don't ignore them.
- victor9000 3y ago
  This also works when reviewing resumes. If you see enough candidates list xyz as a skill, then it's probably worth looking into.
moandcompany 3y ago
But how else will the influencers get their riches if not by selling bootcamps and courses? They've learned like the others to sell shovels during a gold rush.
wirthjason 3y ago
There’s long been framework fatigue in frontend JS work. I feel data engineering has a lot of that now. There’s different schedulers and orchestrators. Databases, ORMs, and wrappers. Cloud providers and hosting platforms. The list goes on. It’s overwhelming.
Sterm 3y ago
Telling data engineers to learn sql because it's what's used in BoringCorp is a bit like telling front end developers in 2003 to learn VB because it's what most user interfaces are written in. Data engineering which pays well happens exactly where databases fail and you need bespoke solutions.
- jmye 3y ago
  They’re speaking to entry-level people, who will not be designing bespoke solutions. SQL is exactly the right recommendation.
  - Sterm 3y ago
    In the same way that VB was to front end developers 20 years ago.
    - jmye 3y ago
      Sure, the language that has underpinned data work for the last few decades and has relevance regardless of what shiny framework you put on top of it is just like VB twenty years ago. Good comparison.
      - Sterm 3y ago
        [dead]
Solvency 3y ago
Am I the only one who finds this to be nearly unreadable on mobile? The full width spaced out text, font, contrast...
- sdairs 3y ago
  Hey, it's my blog - apologies! I am really bad at frontend and this was the first iteration of me trying to build my blog from scratch doing everything myself. It's definitely not perfect! Appreciate the feedback and I'll try to make it more readable on mobile, sorry!
xwdv 3y ago
Data engineer sounds like a great second remote job to pick up. If you already have a remote software engineering job, picking up an entry level data engineer role on the side will allow you to build up that resume in parallel, without having to worry about the lower salary. It’s an investment and gives you a second career branch to eventually fall back on if you get tired of software engineer, or to eventually just have two different six figure jobs with plenty of skill crossover.
slotrans 3y ago
This person is essentially correct.
benrutter 3y ago
I think the idea of a data engineering roadmap is flawed because there isn't a single cookie cutter "data engineer" role to aim for, just like if wouldn't make sense to talk about a single route into becoming a "software engineer". I'm a data engineer but spend most of my time writing code in dask, and use fairly little of SQL. I've been in roles before though where long and complex SQL routines was the main focus. My advice that nobody asked for: focus on 1) general programming skills which are transferable anywhere and 2) do what you personally find interesting. Like SQL? Sweet plenty if people will pay to do that. Obsessed with using scala and spark in combo? Someone'll probably pay you to do that too. Don't focus on being a "data engineer" or any other label- just enjoy programming and find roles that have more of what makes you happy.
Simon_O_Rourke 3y ago
Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. The issues I encountered working as a data engineer many moons ago, is that in a business that's either agnostic to data quality or doesn't invest in reliability, the data engineers tend to become the face of failure. If a Kafka queue begins acting up, or a data ingestion jobs fails because of some upstream data source changes, then it's usually up to the data engineer to... 1. Notice that something's gone wrong. 2. Figure out who in the organization is going to be affected and let them know. 3. Run any triage and comms with the business keeping them in the loop until it's fixed. In my experience, like the guy who only gives good news to the dictator, certain execs/leadership would quite happily let data engineers take the lead for outages, only swooping back in once everything was up and running again to demand a report on "how this happened" and some glib advice to "take time to stop this happening again". While it can be technically rewarding as a career move, it's got to be highlighted that it's only fun and games until you hit your first major outage.
- makeitdouble 3y ago
  TBF, in most companies that have a traditional core business (let's say, selling pharma products), IT is a cost center and your run of the mill engineer is seen as the plumber who's suppose to make everything seamless, as you describe. Short of solving a pain point in a spectacular way, the whole IT dept will be the face of stuff crashing, security issues and budget burning.
- scary-size 3y ago
  This hits home. I communicate a lot more with up and downstream stakeholders as a data engineer. Be it for designing future changes and how those affect others. Or, as you said, triaging outages/delays, figuring out where the missing dependency is. Less problem solving by coding. The SQL thing the article mentions is also true. Even if you work with higher level tech like Spark. SQL gives you a good feel for joins. I also like the fast feedback cycles you get from SQL compared to something Scala/Python based.
xg15 3y ago
Additional useful tech skills besides SQL: grep/sort/uniq, basic bash, jq, python, numpy/scipy.
baazaa 3y ago
The lack of a roadmap is disastrous in terms of professional quality IMO. A lot of orgs have data scientists who are good at coding and maths who can't do their work because the data engineers, who got their jobs because they kinda knew SQL a decade ago, are incompetent. I'm sure this is less of a problem in tech companies which are skimming the cream of the crop, but in other organisations it's apparent that a third-rate data scientist or software engineer is probably much better at their job than a second-rate data engineer. And indeed if I were running an IT dept I'd probably just fill the data engineering roles with decent coders and tell them to reskill a bit.
- samplatt 3y ago
  >the data engineers, who got their jobs because they kinda knew SQL a decade ago, are incompetent. I don't think I've ever been called out more in my life.
- xmcqdpt2 3y ago
  I've been in such an org. The problem is that few decent coders want to do data engineering work, because they'd rather build software and because data engineering pays less and is kind of a dead end in a lot of org. You end up with a lot of DBAs that self-style as Data Engineers.
  - baazaa 3y ago
    I'm in Australia and data engineers here are overpaid, afaict slightly better paid on average than software engineers who are better paid than data people. So you end up with bizarro stuff where a $150k AUD data engineer who basically can't code or even set-up FTP telling some maths PhD data scientist on $100k what he can or can't do. Managers, unable to gauge technical ability, assume the data engineer must be highly skilled because they're getting paid so much. The wages and lack of skill I think are symptomatic of there being no real entry-point in data eng. There's a very slow osmosis of people from genuinely skilled areas into data eng, the main drawback being that the entire culture of data eng is defective with so many unskilled hacks.
    - 3y ago
TrackerFF 3y ago
I work as an (data) analyst, and I'll echo that many data engineers just end up in the engineering role after having worked as a traditional analyst. You start doing the traditional analysis work, and then slowly get more involved on the tooling and pipeline side. Then you become to "go-to" guy on some specific tech, and slowly find yourself doing more and more engineering work, than writing traditional reports. You get offered more money to do the engineering work, due to engineering positions having higher salary caps than the analyst positions. But it is a natural evolution. When you work with analysis, you obviously work very close with the data. You work close with the engineers (if your org / company has them), or become the engineer if there aren't any.
a_c 3y ago
Software always involve data. There is now a tendency for large companies to announce a slice of engineer's day to day to be a standalone position, and others will follow suit thinking they also need a dedicated position/team to handle such operation. And management training indoctrinates you that a team needs objective, measurable result, and sometimes a roadmap. Somewhere down the road, you realised that whatever your goal is, is 2 degrees detached from you business. Data consistency, mean time to recovery, sprint velocity, DORA, SPACE, what else? It is as if one opens a ramen shop to find keeping the light bulbs on all the time the most important job. What about the ramen? In micro service, there is this grains of sand anti-pattern where a service is too granular[1]. Zoom in a bit, you got left-pad as a library. Zoom out a bit, you got identity crisis of granular team. Don't forget to preach communication is important. How else are you going to coordinate all these services, I mean, teams, otherwise [1] https://www.oreilly.com/content/microservices-antipatterns-and-pitfalls/#grains_of_sand_pitfall https://www.oreilly.com/content/microservices-antipatterns-a...
nevi-me 3y ago
I view data engineering as a hybrid of SQL and programming. I'd go as far as saying that it's a mixture of various disciplines. In the end it depends on what you'd be doing as a data engineer. My experience transitioning into the role/ecosystem in the last decade (I'm only 33) + hiring data analysts + scientists + engineers has left me with this impression: Data engineering isn't so much an entry-level career, similar to data science if you don't have some core stat+math background. You could find a job that requires you to write some data pipelines, and it requires just SQL knowledge, and that's fine. However consider what similarity you have with an ETL developer (think Informatica, Oracle, SSIS). I'm probably one of those gatekeepers because I do think there's some roadmap. I agree that it almost always starts with SQL. It's however important to acquire a broad toolkit though, because some people who've gone into data eng roles straight from uni often find themselves only knowing SQL or some bespoke tool that they're using at work, and end up struggling when they encounter the Sparks and glitters of the fiels. Of the hiring processes that I've been involved with (say 20 candidates so far), I've found software engineers to transition better into the role, because most of the time candidates who come from that discipline have more experience debugging performance issues, which translate well into SQL optimisation and generally figuring one's way around tools. Lastly, I'm self-taught. I'm professionally an accountant who started programming while in uni. I was fortunate to move into an analytics team at work and build my foundations from there. The analysts often had Math backgrounds, and I realised that I wouldn't cut it, so I focused on the tooling and giving them data fast and reliably. It was a lot of night-time and weekend tinkering to get to where I am.
- parpfish 3y ago
  > It was a lot of night-time and weekend tinkering to get to where I am. I think that this gets at the difficult part of hiring junior candidates for any role -- you want the people that are capable of learning a new thing if they're given some time to learn, but it's hard to find a signal for 'capable of and willing to learn'.
andelink 3y ago
One of the problems I’ve seen with folks in data engineering is an underdeveloped sense of data modeling and table design. And this article is an example of that. It heavily emphasized SQL, but didn’t mention anything about data modeling. If you aren’t designing your tables thoughtfully, no amount of SQL proficiency will save you.
- deleted 3y ago
  [deleted]
- nevi-me 3y ago
  This reminds me of an interesting project when I was still a consultant. Our partner (audit + consultancy firm) sold the idea of a magical "analytical base table" which would consolidate the client's data in a single place for quicker reporting and analysis. Of course all of this was new to me (the ABT thing really obscured the problem). I was going for quality review with another partner at the firm whose bread & butter are data management. Let's say it was a very harsh review. At the end of the review, he recommended that I learn data modelling properly. He gave me a copy of his Kimball book for the December holidays. I went through and completed it, and on the other side of that we were able to build a coherent dimensional data model to address the client's problem efficiently. That knowledge makes me agree with you, merely knowing SQL or your way around a database is often not enough, and knowing some theory about data modelling (regardless of which type you prefer) is a key element in a data engineer's toolkit.
daemond 3y ago
The author believes that SQL is the core competency of Data Engineering. Now that ChatGPT can translate natural language into good SQLs. So Data Engineering is going to be retired?
- NortySpock 3y ago
  I think it will change or alleviate the tedium of the boring bits of exploring a dataset, figuring out if the dataset can join to another dataset in the way you thought of first, and writing a few of the dashboards. Hopefully it means I can dig into the real problem (correctness, idempotency, and performance) rather than the tedious implementation.
nologic01 3y ago
You cant have a roadmap if you dont have a destination, or at least a direction for travel for the immediate next steps. Remarkably the article doesnt event try to define data engineering, so how you can possibly have a roadmap to it. "SQL" is too vague and jargonistic. Could be just "data analyst". The Engineering aspect is supposed to add a more fundamental data flow approach, less business context driven and more close to the metal. This is the skillset of big data framework (streaming data) nosql etc. We could say it puts more focus on performance, but even that is imprecise because a lot of SQL expertise is also about performant designs and for many people would claim that e.g graph DB's are not essential
blowski 3y ago
(Summary of this article): > Data engineering career roadmaps are a waste of time because they're full of things you don't need to learn. Here instead is an over-simplified data engineering career roadmap. Roadmaps are useful when there's a lot you could learn, you're not sure you have a complete list, a lot of it isn't essential, you don't know what order to do it in, and you don't know what resources to learn from. Many juniors are in that place. To get into a company where they can learn on-the-job or from a mentor is extremely difficult. So a good roadmap is a pretty good alternative to that. Saying "there cannot be a roadmap" or "I didn't need a roadmap, neither do you" is unehlpful at best, damaging at worst. To be an effective junior data engineer needs more than just knowing SQL.
parpfish 3y ago
> People used to say “Software Engineering is not an entry level role”. Something that I ran into myself and have seen other data-people struggle with is that it is hard to get good experience working with data when you don't have access to data. Online SQL tutorials and classes typically have pretty clean/small toy datasets that are only really useful for teaching you the correct syntax. Going from these simple queries that are all ~10 lines to a production system with queries that are 100s of lines is jarring.
- pixl97 3y ago
  And dealing with database sizes that have millions/hundreds of millions of rows. Moving to a system where your indexes are larger than main memory and some really expensive queries may need to spill to temp tables if you don't keep your data sizes under control is also jarring. Big transactions getting deadlocks, and how to read said deadlock graphs and learn how to deal with said locking when you write your SQL starts to become a full time job.
thenerdhead 3y ago
Data engineering reminds me of the George Carlin bit on how names change, but the things don't. Database management -> Data management -> Data architecture -> Database administration -> Big Data engineering -> Data engineering.
- maxlamb 3y ago
  Database administration is more like a "sysadmin of database". IMO not equivalent to the other ones. Data Architecture is like software architecture, so more like high level system design. I do agree though that data engineering is equivalent to what data management referred to in the past.
Ensorceled 3y ago
A data engineer's "hammer" may be SQL but the data engineer's JOB is gluing all that SQL together into a process that works efficiently and reliably. You need to know how those other tools work together, whether it's DBT or Airflow or ... or ... I do agree with ignoring "influencers" they are almost always trying to tie you into their stack or product and don't necessarily have your best interests at heart.
yesiamyourdad 3y ago
Maybe it's just the positions I've ended up in, but my DE roles have required a TON of ops as well. You can build all these pipelines, but you will be absolutely buried in operational work if you can't get the automation squared away. With the role I've been in for the last 2 years, the most significant things we've done were done in Terraform, Ansible, and Prometheus. Sure, there's lots of SQL and Python, some Java, some Scala, but the biggest issue has been getting the system to where we get useful and actionable alerts and downstream data consumers get notified about late or missing data without requiring manual intervention from the data engineers.
- aradox66 3y ago
  Agree with this. I think a good 75% of the value I've delivered in data engineering roles has been on proficiency building out cloud infrastructure, CICD, etc. Teams hire data engineers when the analysts are spending too much time and effort (and failing to succeed at) building data pipelines. Often there is a significant need for engineering and cloud infrastructure improvements beyond fixing up a few wonky python ETLs and broken SQL queries.
- di456 3y ago
  I'll second this too. As a former manager told me, figure out how to scale yourself. Making the end to end development lifestyle easy for peers and partner teams is one way to do that. It could just as easily be called platform or systems engineering with a touch of DevOps. It's like UX for data people. How to make a cohesive experience among the various tools, scripts, services that people use day-to-day so they can use and maintain datasets efficiently.
waffletower 3y ago
I am a Senior Data Engineer professionally and I simply do not use SQL everyday. Services that I test and invoke everyday do use various dialects of SQL, yes. As an almost joking aside, "(f)or Data Engineering, there is only one skill that is absolutely, non-negotiably the first thing you should learn to get started is" Terraform.
sdairs 3y ago
Author of the post here, I had no idea it got posted here and just noticed a huge traffic spike. So, hello! Something I got asked a lot in response to this post, or maybe berated for, is that there is a lot more to Data Engineering than SQL. FWIW, I agree. There is so, so much more to DE than SQL. The point of this post was purely to cover entry-level DE, and not 'This is the only thing you'll ever need in your career'. In my view, developing your SQL skills teaches you a lot of the fundamentals of working with data, enough that you can start to use it in a professional setting. From there, you will develop a lot more skills than just SQL, but your journey could take you in so many different paths that I find it hard to agree with folks who say you should always learn X,Y,Z specific tools. Some folks will end up in Python shops, or Scala, or Bash, or no-code, Airflow, Dagster, Prefect, Control-M, Spark, Flank, Pandas, Polrs, BigQuery, Snowflake, Redshift, ClickHouse, MSSQL, Hadoop, dbt....Some might even go down the K8S and Terraform road, becoming more of a platform-focused DE... There is just so much potential variance depending on where you work and who you are, that it doesn't sit well with me to shoe-horn everyone into a pattern of "First learn Airflow, then learn BigQuery, then learn Spark" when you could have a successful career and never touch a single one of those tools. So, the reason I focused on SQL here, is that in my career consulting with hundreds of DE teams - the only consistent skill across every single team has been SQL. I do not believe SQL is all you'll ever need, but I do think its all you need to get started. It's the only skill I see being relavent regardless of which team you join, and that will still be relevant in 15+ years time.
- nevi-me 3y ago
  Out of curiousity, what then makes a difference between a data engineer and an ETL developer? > There is just so much potential variance depending on where you work and who you are, that it doesn't sit well with me to shoe-horn everyone into a pattern of "First learn Airflow, then learn BigQuery, then learn Spark" when you could have a successful career and never touch a single one of those tools. I agree with this. My observation has been that people who start/stay in 1 place for too long end up knowing some tool(s) very well but not others. So surely then we should abstract away the tools and identify what's common about them (and SQL)? When I produced classroom content for the graduate program that we ran at work, I used to focus more what's being done (ETL, data management, perf optimisation) and primitives like HTTP for REST, bits about DevOps, security, analysis. The idea being that if one is fluent in understanding the underlying problem, tooling becomes easier to pick up. So I still think that there's a roadmap, and that roadmap is the underlying concepts that data engineering tries to solve. I've found that you can teach someone enough about: - ETL & data management (incl modelling) - DBA - DevOps - Software (architecture and development) to the point where they're not as good as the best people in those fields (if they're confining themselves so), but to the point where they become good at the parts of those fields that are relevant to data engineering. That's often my roadmap.

212 points by sebg 3y ago

Now that I think about it you're right. All the maps for dev careers so far: 1. Staff eng map - https://staffeng.com/stories/ https://staffeng.com/stories/ 2. Bootstrappers - Indiehackers 3. Ecommerce - StarterStory 4. Software Engineering - /u/softwareengineering 5. Higher salary - Blind, reddit 6. Technical interviews - Leetcode forums Looks like the shape of the solution can be forums, books, articles, what do you think you'll do?

charles_f 3y ago

I'm not a data engineer but what about dimensional modelling and stuff like the data warehouse toolkit? I would imagine that it is somewhat important?

di456 3y ago
It is important, but when the shift to cheap compute and storage hit an inflection point it became possible to build out wide tables that combine the characteristics of facts and dimensions, and real data modeling took a back seat. That possibility led to bias toward moving quickly with less emphasis on making the data model sustainable. I'm seeing the pendulum start to swing the other way, where the complexity of these scrappy and loosely structured data models is hampering the ability to innovate, and even slowing down the business. The models are often inflexible and hard to maintain with hidden bugs and gotchas.
- bjornasm 3y ago
  Interesting, this makes me think of a process I am going through, where I have a couple of very wide tables where I feel a need to build fact tables simply to get a better understanding of the data and the domain they are from. I have already stumbled a couple of times as the groupings was not as expected at all. Think this was the last push for me to build those fact tables once and for all, to get an overview and also to be able to add my own inferred groups to the data.
  - di456 3y ago
    Sounds like you are on the right track. A non-technical benefit is there won't be so much context to keep in your head when triaging data issues in the future. A database table is very similar to the separation of concerns problem of core software engineering. A table really just is an abstraction for storing and fetching various data points. Finding the right balance is part art and part science.
tfehring 3y ago
A lot of it is dated but the first 3 chapters are still well worth a read for an aspiring DE.

alexleeds 3y ago

I was going to say, ‘Bullsh*t Learn SQL.’ Then I read the post.

quickthrower2 3y ago
Then learn what sequence of actions your data store(s) takes based on that SQL. Relational engines are great but can need tuning (add indexes, de/normailze, materialized views or not) and then key/value ones are going to be doing scans from the entire cluster generally unless you get the cluster key in there and ideally the primary key too.
jldugger 3y ago
At least you're eventually consistent ^_^

phillipcarter 3y ago

Really enjoyed this article. Made me feel a tad inspired to write a similar one, "There is no Product Management roadmap", tailored for the many people looking to break into that discipline. Loooots of parallels.

ghiculescu 3y ago
i think you're probably onto something, what do you think the equivalent to SQL would be in product management?
- kozikow 3y ago
  Jira? Powerpoint?
  - ghiculescu 3y ago
    Nah. Being good at Jira makes you worse at product management.
- phillipcarter 3y ago
  It's a little more loosie goosie, but there's really no single tool. The intuition is that a good PM on a team helps guide things along at all stages such that when anyone looks back, they think "this never would have happened without them driving this". But the details of that aren't easily moldable into some framework or tool you can use for other projects, teams, or organizations.

claytonjy 3y ago

I agree with so much in this article: there's no narrow path, SQL is king, and enough with the damn influencers. But how many entry-level DE jobs will hire folks without programming experience? I've never heard of such a thing in the US, perhaps the UK market is different? If you're new to DE you're unlikely to have any experience managing a database or using common ETL tooling, so your transferable skills are basically SQL and python, and that's what every interview process is going to screen you for.

benrutter 3y ago
I've worked in a big company that had explicit "junior data engineer" roles, but even those would expect some programming experience - either through a bootcamp, degree or related role like data analysis. But basically yeah, as a UK data engineer, I'd say what you've said is pretty much right on- programming language + SQL will get you hired much more than "I've used this ETL tool"

rectang 3y ago

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which when it gets changed breaks in unpredictable ways. And the cultural traditions of testing and validation are underdeveloped because SQL has no core testing story — people count on joins being logically correct, but don’t go to the trouble to prove that true. Vector operations are just hard to get right in any ecosystem and if you don’t stay humble and apply skepticism to your work, it’s going to be wrong a lot. I try to do stuff in pandas or similar — which has its own problems, but at least allows for testable library code. But in the field of data engineering, that’s swimming against the tide.

jerrygenser 3y ago
I've done written heavily tested code in python. But also heavily tested code in SQL. When I was doing data engineering 5 years ago, dbt was a new tool and I used that to write tests, ci/cd, etc. in data engineering. It was a game changer.
jci 3y ago
Can’t speak for SQL _in general_ but in Postgres land pgTAP is wonderful. Invest in writing some factory functions, and you have the ability to write a very expressive test suite. We use Postgres functions to provide a data interface that is thoroughly tested. This allows us to ensure that any migration does not break any contracts.
xmprt 3y ago
> Data engineering can be unsatisfying if you thrive on writing reliable systems This is what I needed for me to stop considering data engineering. Not that I was thinking about it a lot. I feel very happy with my current job but the thought of the code I write being so brittle as to paralyze me from making any changes terrifies me.
- 3y ago

chrismarlow9 3y ago

Good article but a bit short. Anyway two books I can recommend for data engineering: Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB. High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bunch of headaches on the data server side by doing things client side and caching/prefetching more than what the user asked for.

EstesPark 3y ago
To add some additional notes - Designing Data Intensive Applications was a heavy read to dive into for me as a junior engineer. I took a step back after the first couple of chapters and read Understanding Distributed Systems[0] which was a fantastic primer. [0] https://understandingdistributed.systems https://understandingdistributed.systems
SOLAR_FIELDS 3y ago
Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform data and present it to end users for analysis (BI) in such a way that it’s not painful or arduous to consume it, and ensuring that it is clean and accurate. In other words, how well you move the data has always been a secondary point to how people can access and use said data. Note that I’m not in a dedicated data engineering role, but have worked adjacent to it for the better part of a decade and have done a fair amount of data engineering in that time.
- Jarwain 3y ago
  This kinda sounds like higher level concerns where the books cover lower level problems?

markus_zhang 3y ago

Does DevOps ever hire juniors? I applied many times as a DE but never got any reply, not even one interview. I'm preparing to get a RH cert and see if it opens some doors.

wyclif 3y ago
Sounds like a good idea despite how many people diss certs on this forum. I've been trying to get a entry-level DevOps role for over a year and I can't even get a nibble—nothing! No replies, no interviews. I feel like I'm a competent backend dev, even though a lot of times I feel imposter syndrome when I look at the things other developers are doing.

analyte123 3y ago

Can't forget about SQL. It's also worth noting that designing any central data system without a SQL interface is probably a mistake. The people making it may be more comfortable with a REST interface or Python API, but data users will show up and demand that data be SQL-accessible. So now you get to hack it in, or help them develop their own shadow database that is poorly replicated from the main system.

idinnoaname 3y ago
The author stated that the only logical starting point is to learn to solve problems in a dialect of SQL then approach problems of the same type in other dialects of SQL. You’re entirely right, and the author seems to agree to an extent.

iblaine 3y ago

The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst. SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.

woleium 3y ago
for a moment there I thought you said "what makes data engineers is excel"
- jeremyjh 3y ago
  do not speaks its name
sanderjd 3y ago
The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.
- _bk90 3y ago
  A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.
  - sanderjd 3y ago
    I would certainly prefer it if that were the case, but it largely is not, in my experience.
    - _bk90 3y ago
      It seems like your experience doesn't overlap with data engineering, in that case.

dikei 3y ago

One the most important thing about SQL is learn to read the output of EXPLAIN queries: it's the key to understand why your queries are not running as fast as you'd like.

wokwokwok 3y ago

I think there's a fair caveat here: In a context free situation, sure, just learn SQL. It's the workhorse, and you'll definitely use it. If you have to start somewhere, start there. ...buuuut, obviously look at job ads. If all the junior jobs say "experience with SnowPanda" or "Working knowledge of FancyCloudPlatform" or whatever the name of the latest hotness is, you'd be daft not to invest some time and effort into whatever everyone is actually asking for to. You don't need someone to tell you what those things are; literally look at the bullet points on job ads... but don't ignore them.

victor9000 3y ago
This also works when reviewing resumes. If you see enough candidates list xyz as a skill, then it's probably worth looking into.

moandcompany 3y ago

But how else will the influencers get their riches if not by selling bootcamps and courses? They've learned like the others to sell shovels during a gold rush.

wirthjason 3y ago

There’s long been framework fatigue in frontend JS work. I feel data engineering has a lot of that now. There’s different schedulers and orchestrators. Databases, ORMs, and wrappers. Cloud providers and hosting platforms. The list goes on. It’s overwhelming.

Sterm 3y ago

Telling data engineers to learn sql because it's what's used in BoringCorp is a bit like telling front end developers in 2003 to learn VB because it's what most user interfaces are written in. Data engineering which pays well happens exactly where databases fail and you need bespoke solutions.

jmye 3y ago
They’re speaking to entry-level people, who will not be designing bespoke solutions. SQL is exactly the right recommendation.
- Sterm 3y ago
  In the same way that VB was to front end developers 20 years ago.
  - jmye 3y ago
    Sure, the language that has underpinned data work for the last few decades and has relevance regardless of what shiny framework you put on top of it is just like VB twenty years ago. Good comparison.
    - Sterm 3y ago
      [dead]

Solvency 3y ago

Am I the only one who finds this to be nearly unreadable on mobile? The full width spaced out text, font, contrast...

sdairs 3y ago
Hey, it's my blog - apologies! I am really bad at frontend and this was the first iteration of me trying to build my blog from scratch doing everything myself. It's definitely not perfect! Appreciate the feedback and I'll try to make it more readable on mobile, sorry!

xwdv 3y ago

Data engineer sounds like a great second remote job to pick up. If you already have a remote software engineering job, picking up an entry level data engineer role on the side will allow you to build up that resume in parallel, without having to worry about the lower salary. It’s an investment and gives you a second career branch to eventually fall back on if you get tired of software engineer, or to eventually just have two different six figure jobs with plenty of skill crossover.

slotrans 3y ago

This person is essentially correct.

benrutter 3y ago

I think the idea of a data engineering roadmap is flawed because there isn't a single cookie cutter "data engineer" role to aim for, just like if wouldn't make sense to talk about a single route into becoming a "software engineer". I'm a data engineer but spend most of my time writing code in dask, and use fairly little of SQL. I've been in roles before though where long and complex SQL routines was the main focus. My advice that nobody asked for: focus on 1) general programming skills which are transferable anywhere and 2) do what you personally find interesting. Like SQL? Sweet plenty if people will pay to do that. Obsessed with using scala and spark in combo? Someone'll probably pay you to do that too. Don't focus on being a "data engineer" or any other label- just enjoy programming and find roles that have more of what makes you happy.

Simon_O_Rourke 3y ago

Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. The issues I encountered working as a data engineer many moons ago, is that in a business that's either agnostic to data quality or doesn't invest in reliability, the data engineers tend to become the face of failure. If a Kafka queue begins acting up, or a data ingestion jobs fails because of some upstream data source changes, then it's usually up to the data engineer to... 1. Notice that something's gone wrong. 2. Figure out who in the organization is going to be affected and let them know. 3. Run any triage and comms with the business keeping them in the loop until it's fixed. In my experience, like the guy who only gives good news to the dictator, certain execs/leadership would quite happily let data engineers take the lead for outages, only swooping back in once everything was up and running again to demand a report on "how this happened" and some glib advice to "take time to stop this happening again". While it can be technically rewarding as a career move, it's got to be highlighted that it's only fun and games until you hit your first major outage.

makeitdouble 3y ago
TBF, in most companies that have a traditional core business (let's say, selling pharma products), IT is a cost center and your run of the mill engineer is seen as the plumber who's suppose to make everything seamless, as you describe. Short of solving a pain point in a spectacular way, the whole IT dept will be the face of stuff crashing, security issues and budget burning.
scary-size 3y ago
This hits home. I communicate a lot more with up and downstream stakeholders as a data engineer. Be it for designing future changes and how those affect others. Or, as you said, triaging outages/delays, figuring out where the missing dependency is. Less problem solving by coding. The SQL thing the article mentions is also true. Even if you work with higher level tech like Spark. SQL gives you a good feel for joins. I also like the fast feedback cycles you get from SQL compared to something Scala/Python based.

xg15 3y ago

Additional useful tech skills besides SQL: grep/sort/uniq, basic bash, jq, python, numpy/scipy.

baazaa 3y ago

The lack of a roadmap is disastrous in terms of professional quality IMO. A lot of orgs have data scientists who are good at coding and maths who can't do their work because the data engineers, who got their jobs because they kinda knew SQL a decade ago, are incompetent. I'm sure this is less of a problem in tech companies which are skimming the cream of the crop, but in other organisations it's apparent that a third-rate data scientist or software engineer is probably much better at their job than a second-rate data engineer. And indeed if I were running an IT dept I'd probably just fill the data engineering roles with decent coders and tell them to reskill a bit.

samplatt 3y ago
>the data engineers, who got their jobs because they kinda knew SQL a decade ago, are incompetent. I don't think I've ever been called out more in my life.
xmcqdpt2 3y ago
I've been in such an org. The problem is that few decent coders want to do data engineering work, because they'd rather build software and because data engineering pays less and is kind of a dead end in a lot of org. You end up with a lot of DBAs that self-style as Data Engineers.
- baazaa 3y ago
  I'm in Australia and data engineers here are overpaid, afaict slightly better paid on average than software engineers who are better paid than data people. So you end up with bizarro stuff where a $150k AUD data engineer who basically can't code or even set-up FTP telling some maths PhD data scientist on $100k what he can or can't do. Managers, unable to gauge technical ability, assume the data engineer must be highly skilled because they're getting paid so much. The wages and lack of skill I think are symptomatic of there being no real entry-point in data eng. There's a very slow osmosis of people from genuinely skilled areas into data eng, the main drawback being that the entire culture of data eng is defective with so many unskilled hacks.
  - 3y ago

TrackerFF 3y ago

I work as an (data) analyst, and I'll echo that many data engineers just end up in the engineering role after having worked as a traditional analyst. You start doing the traditional analysis work, and then slowly get more involved on the tooling and pipeline side. Then you become to "go-to" guy on some specific tech, and slowly find yourself doing more and more engineering work, than writing traditional reports. You get offered more money to do the engineering work, due to engineering positions having higher salary caps than the analyst positions. But it is a natural evolution. When you work with analysis, you obviously work very close with the data. You work close with the engineers (if your org / company has them), or become the engineer if there aren't any.

a_c 3y ago

Software always involve data. There is now a tendency for large companies to announce a slice of engineer's day to day to be a standalone position, and others will follow suit thinking they also need a dedicated position/team to handle such operation. And management training indoctrinates you that a team needs objective, measurable result, and sometimes a roadmap. Somewhere down the road, you realised that whatever your goal is, is 2 degrees detached from you business. Data consistency, mean time to recovery, sprint velocity, DORA, SPACE, what else? It is as if one opens a ramen shop to find keeping the light bulbs on all the time the most important job. What about the ramen? In micro service, there is this grains of sand anti-pattern where a service is too granular[1]. Zoom in a bit, you got left-pad as a library. Zoom out a bit, you got identity crisis of granular team. Don't forget to preach communication is important. How else are you going to coordinate all these services, I mean, teams, otherwise [1] https://www.oreilly.com/content/microservices-antipatterns-and-pitfalls/#grains_of_sand_pitfall https://www.oreilly.com/content/microservices-antipatterns-a...

nevi-me 3y ago

I view data engineering as a hybrid of SQL and programming. I'd go as far as saying that it's a mixture of various disciplines. In the end it depends on what you'd be doing as a data engineer. My experience transitioning into the role/ecosystem in the last decade (I'm only 33) + hiring data analysts + scientists + engineers has left me with this impression: Data engineering isn't so much an entry-level career, similar to data science if you don't have some core stat+math background. You could find a job that requires you to write some data pipelines, and it requires just SQL knowledge, and that's fine. However consider what similarity you have with an ETL developer (think Informatica, Oracle, SSIS). I'm probably one of those gatekeepers because I do think there's some roadmap. I agree that it almost always starts with SQL. It's however important to acquire a broad toolkit though, because some people who've gone into data eng roles straight from uni often find themselves only knowing SQL or some bespoke tool that they're using at work, and end up struggling when they encounter the Sparks and glitters of the fiels. Of the hiring processes that I've been involved with (say 20 candidates so far), I've found software engineers to transition better into the role, because most of the time candidates who come from that discipline have more experience debugging performance issues, which translate well into SQL optimisation and generally figuring one's way around tools. Lastly, I'm self-taught. I'm professionally an accountant who started programming while in uni. I was fortunate to move into an analytics team at work and build my foundations from there. The analysts often had Math backgrounds, and I realised that I wouldn't cut it, so I focused on the tooling and giving them data fast and reliably. It was a lot of night-time and weekend tinkering to get to where I am.

parpfish 3y ago
> It was a lot of night-time and weekend tinkering to get to where I am. I think that this gets at the difficult part of hiring junior candidates for any role -- you want the people that are capable of learning a new thing if they're given some time to learn, but it's hard to find a signal for 'capable of and willing to learn'.

andelink 3y ago

One of the problems I’ve seen with folks in data engineering is an underdeveloped sense of data modeling and table design. And this article is an example of that. It heavily emphasized SQL, but didn’t mention anything about data modeling. If you aren’t designing your tables thoughtfully, no amount of SQL proficiency will save you.

deleted 3y ago
[deleted]
nevi-me 3y ago
This reminds me of an interesting project when I was still a consultant. Our partner (audit + consultancy firm) sold the idea of a magical "analytical base table" which would consolidate the client's data in a single place for quicker reporting and analysis. Of course all of this was new to me (the ABT thing really obscured the problem). I was going for quality review with another partner at the firm whose bread & butter are data management. Let's say it was a very harsh review. At the end of the review, he recommended that I learn data modelling properly. He gave me a copy of his Kimball book for the December holidays. I went through and completed it, and on the other side of that we were able to build a coherent dimensional data model to address the client's problem efficiently. That knowledge makes me agree with you, merely knowing SQL or your way around a database is often not enough, and knowing some theory about data modelling (regardless of which type you prefer) is a key element in a data engineer's toolkit.

daemond 3y ago

The author believes that SQL is the core competency of Data Engineering. Now that ChatGPT can translate natural language into good SQLs. So Data Engineering is going to be retired?

NortySpock 3y ago
I think it will change or alleviate the tedium of the boring bits of exploring a dataset, figuring out if the dataset can join to another dataset in the way you thought of first, and writing a few of the dashboards. Hopefully it means I can dig into the real problem (correctness, idempotency, and performance) rather than the tedious implementation.

nologic01 3y ago

You cant have a roadmap if you dont have a destination, or at least a direction for travel for the immediate next steps. Remarkably the article doesnt event try to define data engineering, so how you can possibly have a roadmap to it. "SQL" is too vague and jargonistic. Could be just "data analyst". The Engineering aspect is supposed to add a more fundamental data flow approach, less business context driven and more close to the metal. This is the skillset of big data framework (streaming data) nosql etc. We could say it puts more focus on performance, but even that is imprecise because a lot of SQL expertise is also about performant designs and for many people would claim that e.g graph DB's are not essential

blowski 3y ago

(Summary of this article): > Data engineering career roadmaps are a waste of time because they're full of things you don't need to learn. Here instead is an over-simplified data engineering career roadmap. Roadmaps are useful when there's a lot you could learn, you're not sure you have a complete list, a lot of it isn't essential, you don't know what order to do it in, and you don't know what resources to learn from. Many juniors are in that place. To get into a company where they can learn on-the-job or from a mentor is extremely difficult. So a good roadmap is a pretty good alternative to that. Saying "there cannot be a roadmap" or "I didn't need a roadmap, neither do you" is unehlpful at best, damaging at worst. To be an effective junior data engineer needs more than just knowing SQL.

parpfish 3y ago

> People used to say “Software Engineering is not an entry level role”. Something that I ran into myself and have seen other data-people struggle with is that it is hard to get good experience working with data when you don't have access to data. Online SQL tutorials and classes typically have pretty clean/small toy datasets that are only really useful for teaching you the correct syntax. Going from these simple queries that are all ~10 lines to a production system with queries that are 100s of lines is jarring.

pixl97 3y ago
And dealing with database sizes that have millions/hundreds of millions of rows. Moving to a system where your indexes are larger than main memory and some really expensive queries may need to spill to temp tables if you don't keep your data sizes under control is also jarring. Big transactions getting deadlocks, and how to read said deadlock graphs and learn how to deal with said locking when you write your SQL starts to become a full time job.

thenerdhead 3y ago

Data engineering reminds me of the George Carlin bit on how names change, but the things don't. Database management -> Data management -> Data architecture -> Database administration -> Big Data engineering -> Data engineering.

maxlamb 3y ago
Database administration is more like a "sysadmin of database". IMO not equivalent to the other ones. Data Architecture is like software architecture, so more like high level system design. I do agree though that data engineering is equivalent to what data management referred to in the past.

Ensorceled 3y ago

A data engineer's "hammer" may be SQL but the data engineer's JOB is gluing all that SQL together into a process that works efficiently and reliably. You need to know how those other tools work together, whether it's DBT or Airflow or ... or ... I do agree with ignoring "influencers" they are almost always trying to tie you into their stack or product and don't necessarily have your best interests at heart.

yesiamyourdad 3y ago

Maybe it's just the positions I've ended up in, but my DE roles have required a TON of ops as well. You can build all these pipelines, but you will be absolutely buried in operational work if you can't get the automation squared away. With the role I've been in for the last 2 years, the most significant things we've done were done in Terraform, Ansible, and Prometheus. Sure, there's lots of SQL and Python, some Java, some Scala, but the biggest issue has been getting the system to where we get useful and actionable alerts and downstream data consumers get notified about late or missing data without requiring manual intervention from the data engineers.

aradox66 3y ago
Agree with this. I think a good 75% of the value I've delivered in data engineering roles has been on proficiency building out cloud infrastructure, CICD, etc. Teams hire data engineers when the analysts are spending too much time and effort (and failing to succeed at) building data pipelines. Often there is a significant need for engineering and cloud infrastructure improvements beyond fixing up a few wonky python ETLs and broken SQL queries.
di456 3y ago
I'll second this too. As a former manager told me, figure out how to scale yourself. Making the end to end development lifestyle easy for peers and partner teams is one way to do that. It could just as easily be called platform or systems engineering with a touch of DevOps. It's like UX for data people. How to make a cohesive experience among the various tools, scripts, services that people use day-to-day so they can use and maintain datasets efficiently.

waffletower 3y ago

I am a Senior Data Engineer professionally and I simply do not use SQL everyday. Services that I test and invoke everyday do use various dialects of SQL, yes. As an almost joking aside, "(f)or Data Engineering, there is only one skill that is absolutely, non-negotiably the first thing you should learn to get started is" Terraform.

sdairs 3y ago

Author of the post here, I had no idea it got posted here and just noticed a huge traffic spike. So, hello! Something I got asked a lot in response to this post, or maybe berated for, is that there is a lot more to Data Engineering than SQL. FWIW, I agree. There is so, so much more to DE than SQL. The point of this post was purely to cover entry-level DE, and not 'This is the only thing you'll ever need in your career'. In my view, developing your SQL skills teaches you a lot of the fundamentals of working with data, enough that you can start to use it in a professional setting. From there, you will develop a lot more skills than just SQL, but your journey could take you in so many different paths that I find it hard to agree with folks who say you should always learn X,Y,Z specific tools. Some folks will end up in Python shops, or Scala, or Bash, or no-code, Airflow, Dagster, Prefect, Control-M, Spark, Flank, Pandas, Polrs, BigQuery, Snowflake, Redshift, ClickHouse, MSSQL, Hadoop, dbt....Some might even go down the K8S and Terraform road, becoming more of a platform-focused DE... There is just so much potential variance depending on where you work and who you are, that it doesn't sit well with me to shoe-horn everyone into a pattern of "First learn Airflow, then learn BigQuery, then learn Spark" when you could have a successful career and never touch a single one of those tools. So, the reason I focused on SQL here, is that in my career consulting with hundreds of DE teams - the only consistent skill across every single team has been SQL. I do not believe SQL is all you'll ever need, but I do think its all you need to get started. It's the only skill I see being relavent regardless of which team you join, and that will still be relevant in 15+ years time.

nevi-me 3y ago
Out of curiousity, what then makes a difference between a data engineer and an ETL developer? > There is just so much potential variance depending on where you work and who you are, that it doesn't sit well with me to shoe-horn everyone into a pattern of "First learn Airflow, then learn BigQuery, then learn Spark" when you could have a successful career and never touch a single one of those tools. I agree with this. My observation has been that people who start/stay in 1 place for too long end up knowing some tool(s) very well but not others. So surely then we should abstract away the tools and identify what's common about them (and SQL)? When I produced classroom content for the graduate program that we ran at work, I used to focus more what's being done (ETL, data management, perf optimisation) and primitives like HTTP for REST, bits about DevOps, security, analysis. The idea being that if one is fluent in understanding the underlying problem, tooling becomes easier to pick up. So I still think that there's a roadmap, and that roadmap is the underlying concepts that data engineering tries to solve. I've found that you can teach someone enough about: - ETL & data management (incl modelling) - DBA - DevOps - Software (architecture and development) to the point where they're not as good as the best people in those fields (if they're confining themselves so), but to the point where they become good at the parts of those fields that are relevant to data engineering. That's often my roadmap.