3 ms·
I view data engineering as a hybrid of SQL and programming. I'd go as far as saying that it's a mixture of various disciplines. In the end it depends on what y
by nevi-me 3y ago
I view data engineering as a hybrid of SQL and programming. I'd go as far as saying that it's a mixture of various disciplines.
In the end it depends on what you'd be doing as a data engineer. My experience transitioning into the role/ecosystem in the last decade (I'm only 33) + hiring data analysts + scientists + engineers has left me with this impression:
Data engineering isn't so much an entry-level career, similar to data science if you don't have some core stat+math background. You could find a job that requires you to write some data pipelines, and it requires just SQL knowledge, and that's fine. However consider what similarity you have with an ETL developer (think Informatica, Oracle, SSIS).
I'm probably one of those gatekeepers because I do think there's some roadmap. I agree that it almost always starts with SQL.
It's however important to acquire a broad toolkit though, because some people who've gone into data eng roles straight from uni often find themselves only knowing SQL or some bespoke tool that they're using at work, and end up struggling when they encounter the Sparks and glitters of the fiels.
Of the hiring processes that I've been involved with (say 20 candidates so far), I've found software engineers to transition better into the role, because most of the time candidates who come from that discipline have more experience debugging performance issues, which translate well into SQL optimisation and generally figuring one's way around tools.
Lastly, I'm self-taught. I'm professionally an accountant who started programming while in uni. I was fortunate to move into an analytics team at work and build my foundations from there. The analysts often had Math backgrounds, and I realised that I wouldn't cut it, so I focused on the tooling and giving them data fast and reliably.
It was a lot of night-time and weekend tinkering to get to where I am.
- parpfish 3y ago> It was a lot of night-time and weekend tinkering to get to where I am. I think that this gets at the difficult part of hiring junior candidates for any role -- you want the people that are capable of learning a new thing if they're given some time to learn, but it's hard to find a signal for 'capable of and willing to learn'.