This has been a bit of an annoying thing for me for quite a while. There is a huge difference between a data engineer and a data scientist. A data scientist is

This has been a bit of an annoying thing for me for quite a while. There is a huge difference between a data engineer and a data scientist. A data scientist is not and should not be a data engineer. These are 2 different specializations that work together. A data engineer is more of what we use to refer to as people who wrote utility processes to process data or do system optimizations. At some point the industry decided to do away with a lot of the things we used to do (desktop apps, distirbuted systems, etc) and moved to REST services only. Then people realized oh wait.. we can't process data on a rest/web app. In typical inexperienced fashion, people tried to cram in there, but it doesn't work. (See Javascript neural networks) What is data engineering? It's all about moving data around efficiently and processing it in a way to is per formant and reactive. A lot of people tie hadoop/spark to being a data engineer. That's a terrible way of going about it. More of the modern approaches to this is using streaming platforms and reacting to events. (Sadly a lot of the ML stuff is tied to either python/tensorflow or spark) At times data engineering is pushed towards data maintenance and pushing all of the data in a bucket. This isn't a very valuable use of effort.. but people want to be buzzword compliant. Note: There are use cases for hadoop and spark.. but those rarer now. (They've better for very large datasets and merging for data that you have a much longer timeframe for the answer).