3 ms·
While all Software Engineers deal with data, dealing with enormous and fast datasets is a legit specialization in Software Engineering. Once you reach a certain
by nyokodo 7y ago
While all Software Engineers deal with data, dealing with enormous and fast datasets is a legit specialization in Software Engineering. Once you reach a certain scale or speed, simple things become quite tricky, so you want people who know how to deal with it.
- closeparen 7y agoBut architecting and implementing scalable systems is a requirement for any backend engineering role in the tech industry; what makes data different?
- squeaky-clean 7y ago99.9% of systems probably don't need to be that scalable.
- nyokodo 7y agoAt FANG companies, as an example, the scale is so great that they don't differentiate types of Software Engineers as everyone has to deal with massive scale. But at most companies in the industry architecting and scaling requirements are very modest even on the backend so they simply do not hit the scaling cliffs that Data Engineers are specialized in solving. Subsequently, Engineers working there do not gain the experience or knowledge to deal with that scale. Additionally, it's not that "data" is different, it's entirely conditional on the nature of the data in question and what a company needs to do with it. Edit: Added clarification.