4 ms·
I guess my response is "OK" ? Maybe this is surprising for some people, but... why would it be surprising? Data engineers are usually working in a few ways. 1.
by insanitybit 3y ago
I guess my response is "OK" ? Maybe this is surprising for some people, but... why would it be surprising? Data engineers are usually working in a few ways.
1. With in memory datasets using Python Notebooks
2. With big data storage systems like Hive
3. With something like Spark or maybe even Flink
All of these tools were built for people who, for the most part, only need to know Python and SQL. And they do a bunch of interesting stuff with those tools. Also, I guess maybe it's worth noting that under the hood the implementations are almost always C/C++.
I guess maybe someone could mistakenly think that this is a good thing? Or a bad thing? Is that the point? Sorry I don't have Twitter so I don't know if there are further replies clarifying this. I'm sort of at a loss for the point because this is not surprising at all but maybe this is targeting people really early on in their careers who aren't aware - that's sort of the problem with putting a Twitter post here, it feels like HN is not the right audience, but their followers may be really green and exactly the people who would not be aware.
- antonvs 3y agoI don’t know if you clicked through to the link, but there’s some additional context in the original tweet: > You don’t need to know the high performance languages to make a killing as a data engineer! As such, the tweet seems fairly self-explanatory.
- insanitybit 3y agoI did, yes, my post was in response to the full tweet. I think I'm just way outside of the target audience because I didn't know about this "you have to know high performance languages as a data engineer" conversation.
- threeseed 3y agoIt comes from the Hadoop era of Data Engineering. Which is that you were expected to know Scala or Java in order to build pipelines. That all changed when Spark added Python support.