4 ms·

> towards non-technical folks trying to get their feet wet in software. Eg: Data scientists or business. A bit tangential to the original post, but where does

by spi 4y ago

> towards non-technical folks trying to get their feet wet in software. Eg: Data scientists or business. A bit tangential to the original post, but where does this belief that data scientists are non-technical folks? I am a data scientist myself, and in my view it's way more technical than most software development. Albeit I wouldn't still call neither data scientists nor software engineers "smarter" than the average. Sure, if you want to train your bread and butter text classifier it just takes 10 lines of boilerplate code. But you don't need an AI-assisted tool for that - you just go to hugging face, copy paste those 10 lines, done, it's certainly faster than getting some AI-assisted code editor work for you. For everything that is a bit more complicated, you need endless adjustments to your code, and it's quite unlikely more than a handful of people before you ever wrote the same code. It is, indeed, a somewhat painful and slow process (because just "testing" your code often takes minutes, if not hours, so finding out bugs becomes annoying). And a somewhat simple, AI-based, error highlight tool might be useful to weed out the most stupid ones and save some time. But I will never trust something like copilot (or Kite, I guess, which I never tried) to write my code for me, as the challenging parts of the work involve long-term connection between different pieces of code (data loader, loss function, model function) that are written independently but must "cooperate" in a very non-trivial way. It is not at all uncommon that I make hours-long screen sharing calls with a colleague, discussing non-trivial mathematical computations, only to end up changing one or two lines of code that don't have an immediate link with the problem we are trying to solve. This kind of things are notoriously hard for AI to grasp, so they can't do any decent job in writing that for me. Add on top that a lot of the code you find freely online is just ridiculously bad or broken, and you might only get unusable models generated by AI engines trained on those. So, what kind of work are you referring to when talking about "data scientists or business" in your comment?

ogarten 4y ago
I found that "being technical" means different things to different people. In the software world people seem to be referred to as technical when they write software systems not as much as singular scripts. Data science is definitely technical but a lot of code work tends to happen in Jupyter notebooks or something similar. The main challenge is in understanding the ML/AI algorithms, the possible choices for your analyses that actually make sense for the problem, ... . Besides that, due to the AI hype, there are so many people in data science who don't know much about coding or software engineering. Therefore, helping these people might be profitable (or not).
mrtranscendence 4y ago
I work at a firm with many data scientists (I am one of them -- though my title has wavered back and forth between data scientist and ML engineer). Whether or not data scientists are "technical" and in what sense could be a difficult question to answer. I can't speak very broadly, but at least for my company most data scientists are not doing the kind of work you describe. There certainly are some folks constructing and training complex machine learning models, but I think the majority work on the level of more basic statistical models and rules of thumb, where a project's final output might be a dashboard or presentation. Arguably some might refer to this as data analysis rather than data science, but none of these terms are particularly well defined. That's not to say they aren't technical in some sense. All of them can and do code to one degree or another (with perhaps the exception of a small number of people who've been in the industry for decades), though not all of them do so with high proficiency or attention to software engineering best practices. That also goes for some of the engineers where I work, admittedly. All in all, the broad level of technical aptitude has grown over the past few years. But not everyone with the title of data scientist is a machine learning specialist, nor are they necessarily skilled at software engineering. Edit: As for Copilot, I found it worse than useless. It miserably failed every test I threw at it, from machine learning to (especially) Spark data pipelines, only redeeming itself with a string handling problem -- for which the solution was still entirely wrong but at least interestingly wrong. I frankly don't see how anyone pays for it, though perhaps it's better for projects with a ton of boilerplate.

by spi 4y ago

ogarten 4y ago

I found that "being technical" means different things to different people. In the software world people seem to be referred to as technical when they write software systems not as much as singular scripts. Data science is definitely technical but a lot of code work tends to happen in Jupyter notebooks or something similar. The main challenge is in understanding the ML/AI algorithms, the possible choices for your analyses that actually make sense for the problem, ... . Besides that, due to the AI hype, there are so many people in data science who don't know much about coding or software engineering. Therefore, helping these people might be profitable (or not).

mrtranscendence 4y ago

I work at a firm with many data scientists (I am one of them -- though my title has wavered back and forth between data scientist and ML engineer). Whether or not data scientists are "technical" and in what sense could be a difficult question to answer. I can't speak very broadly, but at least for my company most data scientists are not doing the kind of work you describe. There certainly are some folks constructing and training complex machine learning models, but I think the majority work on the level of more basic statistical models and rules of thumb, where a project's final output might be a dashboard or presentation. Arguably some might refer to this as data analysis rather than data science, but none of these terms are particularly well defined. That's not to say they aren't technical in some sense. All of them can and do code to one degree or another (with perhaps the exception of a small number of people who've been in the industry for decades), though not all of them do so with high proficiency or attention to software engineering best practices. That also goes for some of the engineers where I work, admittedly. All in all, the broad level of technical aptitude has grown over the past few years. But not everyone with the title of data scientist is a machine learning specialist, nor are they necessarily skilled at software engineering. Edit: As for Copilot, I found it worse than useless. It miserably failed every test I threw at it, from machine learning to (especially) Spark data pipelines, only redeeming itself with a string handling problem -- for which the solution was still entirely wrong but at least interestingly wrong. I frankly don't see how anyone pays for it, though perhaps it's better for projects with a ton of boilerplate.