3 ms·
I generally agree. Maybe not totally, but software developers will have a very different job role in 10 years. The most obvious is BigQuery, Palantir, Snowflak
by yazaddaruvala 5y ago
I generally agree. Maybe not totally, but software developers will have a very different job role in 10 years.
The most obvious is BigQuery, Palantir, Snowflake, etc. So many data pipelines that currently need 10+ teams of engineers to work on will be done through SparkSQL and SparkML by 1 or 2 teams of Data Engineers (i.e. no distributed systems knowledge) instead. It’s not there yet, but dealing (in an asynchronous context) with data at scale is very close to getting commoditized.
Fortunately, UI applications (synchronous, latency sensitive applications, modeling data input from the real world, making things pretty/unique) will likely always need people to write something kinda like code lol. However, I feel like they will likely have classes/experience with HCI/design, product management/MBAs, in the long-long run probably with law/accounting/medicine/architecture/etc.
That said, there is just so much of the real world that needs to be better modeled digitally that everyone that learns coding-like skills will continue to be highly employable. At least, until the singularity. Then all bets are off.
- ramoz 5y agoCompletely agree that Data Engineering is a bloated skill set. I have devops engineers who can manage/build out complex pipelines and actually prefer they do since we simultaneously manage large scale cloud architectures. For anything overtly complex (actual data parsing challenges) they usually consult with a data scientist on a team. I’m constantly reviewing data engineering resume and am wondering “what exactly is this person going to be doing if it isn’t some outdated architecture and o&m work …. Why would we hire this person vs a cloud native software engineer who costs x2 less” etc