3 ms·
I feel like this might be case of data engineers. They're not usually software engineers. They're tool users not tool makers. So they'll cobble things togethe
by Vicinity9635 3y ago
I feel like this might be case of data engineers.
They're not usually software engineers. They're tool users not tool makers.
So they'll cobble things together to accomplish the task, using only available tools and never anything custom that would do it task much more cleanly, because they understand data, not software. They're not computer scientists or programmers, they're just users. And we all know what that means.
- entropicdrifter 3y agoAgreed. I've been "the backend engineer who works with the data engineers" for several years now and I've seen their general trend of re-inventing the wheel the hard way a number of times. I've spent the majority of my career building better tools for data-related tasks, then winning over my users by showing off performance and productivity gains.
- scruple 3y agoI stepped into a Data Engineering Lead role in 2019. Stepped out of it in 2021. My team was the first in the org to really approach data engineering and we were all software engineers. I'm told that the systems we built have largely been replaced by Rube Goldberg machines pieced together by the folks who came after us. Those replacement systems aren't even working, they're failing to deliver on the same simple data pipelines that we had working by the start of 2020. They're cobbled together using a million little AWS pieces and Docker and k8s... I'm glad that I left that role when I did, we were being pushed by a new-hire with a fancy Data Engineering VP title to do all sorts of asinine things. I went and looked just now and I see that he's Senior VP at a different company, he started there this summer. Onward and upward!