2 ms·
In data eng roles I've had to build systems that ingest tons of data quickly and learnt multithreading, used profilers to fine tune perf, learnt how SQL databas
by zaptheimpaler 5y ago
In data eng roles I've had to build systems that ingest tons of data quickly and learnt multithreading, used profilers to fine tune perf, learnt how SQL databases work under the hood and tricked query planners, learnt how to structure map reduce style jobs to exploit parallelism on huge clusters. Admittedly this was a few years ago when the field was more nascent. Maybe today it's less engineering heavy if you can get away with gluing together a few off the shelf solutions like Kafka/Snowflake.
In SWE roles I've had occasional deep dives into tech, but a lot of it is about knowing the business logic and delivering simple things quickly. It's like write SQL query, maybe change a model, write some simple business logic in a hairy codebase. Since it's simple, there is more pressure to deliver quickly as well. Sometimes doing things the “right” way would have been more interesting but also more time consuming, so we decide to just put a bandaid over it. It feels like there is less agency as well, just implementing whatever features/roadmap the PMs hand down.
But I see other people on this thread have had the exact opposite experience.. so I guess it really just comes down the particular role, company and your interests.
- Grimm1 5y agoThis largely mirrors my own experience. The work was just more interesting than regular SWE