4 ms·
ML and data-science relies on software to work. Nobody is going to do data-science and ML by hand, so already you have a demand for software engineers and compu
by didibus 5y ago
ML and data-science relies on software to work. Nobody is going to do data-science and ML by hand, so already you have a demand for software engineers and computer scientists to build tools and implement algorithms and information pipelines for the data-scientists and businesses who want to leverage ML.
I think there might be a bit of a misconception of what existing software engineers did. Very rarely were they focused on designing inference models or performing data analysis. Most of the time the business side of things, like a business analysts, or people working on the business problem would perform analysis manually, and come up with business rules and logic. Software engineers could help them leverage more powerful techniques by building them tools that help for data analysis, like BI platforms, or data stores that can do bigger and more complex queries faster. Or they could help them with letting them know of techniques, generally known as old-school AI, such as edit distance algorithms, graph algorithms, logic rule engines, expert systems, etc.
So nothing has really changed, except that there is even more demand from businesses who want software engineers to help them setup tools and pipelines for them to experiment with ML and have data scientists analyze their data.
What it would take to get rid of the demand for software engineers is to be able to automate their work, and ML is far from able to do that. It could one day, but we are talking about a computer that can reprogram itself successfully, and that can also understand the requests of the user in how it should reprogram itself. And we're talking not just reprogramming, but it should also be able to replicate itself to other machines and all that, since most use case today involve distributed system, as no one computer has the compute needed for the scale we operate at. Things like GPT-3 show some promise, but as it stands today, you'd need a software engineer to inspect and review everything it generates, which would take almost as much time if not more.
I think what is far more likely is that the offer increases, that is, that there eventually is just a lot more people with the ability to perform the work of software engineers. This will probably happen as a combination of more people joining the field, and the barrier to entry becoming smaller. The latter could be helped a little bit by ML, if it can deliver better auto-complete for example, but I think it largely is just driven by frameworks, PAAS, SAAS and IAAS, library, and all other type of code reuse we've been building for years now.