4 ms·

Do you ever feel like all the noise influences the way you think about your own career? I work as a data scientist and sometimes find the hype so off-putting th

by s1t5 6y ago

Do you ever feel like all the noise influences the way you think about your own career? I work as a data scientist and sometimes find the hype so off-putting that I think that I should look for a role that's related to solving some optimization problems outside of ML, or a software engineering role in some completely different domain and work as a backend developer or something similar.

gas9S9zw3P9c 6y ago
Definitely. I came from a software engineering background into AI 6+ years ago, and I was lucky to be there at the right time. From purely a career perspective, leaving aside that this may be your life's passion, I believe that right now is the worst time to get started in AI. These is so much noise and competition but very few actual jobs or value created. It's just a research PR machine. Just as with Data Science, companies that think they need AI often just need better data collection, pipeline and infrastructure engineering. Good backend/infrastructure/data engineers are so much harder to find these days, and these skills IMO provide much more value than doing some kind of modeling.
- MrBuddyCasino 6y ago
  > Good backend/infrastructure/data engineers are so much harder to find these days, and these skills IMO provide much more value than doing some kind of modeling. After all these years of FOMO on AI, this is music to my ears.
  - ricksharp 6y ago
    Right, I’ve been reading about machine learning for about 5 years, have read hundreds of articles about different techniques, and have often tried to explore ways it could be used. However, I’ve never found a practical use in software engineering. Every time I think I discover something that could use machine learning, I usually don’t have any data to work with or don’t have a clear definition of what the inputs and outputs would be. In the end, I find a way to develop a solution that doesn’t need AI and often makes me realize that AI would not have been able to provide the required reliability. I even participated in a self-driving RC car competition (running on raspi 3). I ended up scrapping the ML solution because it could only run at 10 hertz which was not even close to what was needed. I ended up writing a custom algorithm in C that ran at 200fps and could do “advanced” moves like backing up and course correcting. I’m not saying that ML doesn’t have its place. I’m saying that right now it’s all about images and nlp and has huge costs. In any practical usage, the ML parts that I would need are already available as a service that I can use almost free (like azure cognitive services).
    - KineticLensman 6y ago
      > In the end, I find a way to develop a solution that doesn’t need AI and often makes me realize that AI would not have been able to provide the required reliability. This was brought home to me back in the 1990s at a place where we were trying to popularise expert system technology (a positively prehistoric form of AI). It was one of my first jobs in industry after leaving academia. The goal was to create a system that could predict the speed at which a unit of military vehicles could move so that a contact report expert system could decide whether two groups of vehicles could have feasibly moved between two points in a given time. The system could use this to help decide whether two reports referred to two different enemy units or a single unit that had moved from point A to point B. After quite a lot of time cranking out and debugging rules to describe the movement behaviours, I realised that some simple convoy arithmetic would do just as good a job - units are often constrained by the speed of the slowest vehicle, and respect inter-vehicle distances. For most purposes this simple arithmetic was just as good (and orders of magnitude faster) than the complex rule engine.
    - skwb 6y ago
      I think there is a lot of more practical deep learning that are coming out, but it often times working with the people who know the field intimately. For example, in my PhD thesis project, I am making a series of deep learning models to make a cardiac MRI autopilot. I've build a series of deep learning networks to localize the cardiac landmarks that define the cardiac imaging planes. And our group has even made it into a clinical prototype that works within our imaging workflow. I think the field is shifting in that ML technologies are increasingly requiring domain level knowledge in order to make a practical endpoint.
  - apohn 6y ago
    I've been involved in a lot of Data Science projects, and the most successful ones are the ones where the Data Scientist either did or was part of the infrastructure and Data Engineering. Most of the failed projects basically were "Hire a Data Scientist to make a magic model" and nothing else was supported. Basically, "Here's some data, do some magic." I read somewhere that 90% of data is low/zero value, and I'd agree with that. There's absolutely a place for people doing pure statistical or ML/Deep Learning modeling. But the rest of the organization has to support them so they can have that narrow focus. A lot of places want to take a shortcut and not do the data work. 5 years ago the FOMO was from Hadoop. Hadoop Here, Hadoop there...people just forgot you have to actually collect a lot of high quality data to make Hadoop useful. It's AI now.
bumby 6y ago
I’ve found that non-technical management expectations are just too swayed by the AI hype machine. I’ve also found operations research field to be more grounded and still have plenty of research opportunities. There’s something very fulfilling about optimizing a real-world system, particularly when the answer isn’t intuitive.
apohn 6y ago
>I work as a data scientist and sometimes find the hype so off-putting that I think that I should look for a role that's related to... I'm Data Scientist and the hype is making me really jaded about the whole thing. I work in a customer facing role and it's just painful to see how many sensible and solvable problems with ML/Deep Learning get buried and destroyed under the pile of hype of solving unrealistic "sexy" problems with "AI." It was okay when Machine Learning was the hot thing, it's been awful since "AI" became the hot thing.

by s1t5 6y ago

gas9S9zw3P9c 6y ago

Definitely. I came from a software engineering background into AI 6+ years ago, and I was lucky to be there at the right time. From purely a career perspective, leaving aside that this may be your life's passion, I believe that right now is the worst time to get started in AI. These is so much noise and competition but very few actual jobs or value created. It's just a research PR machine. Just as with Data Science, companies that think they need AI often just need better data collection, pipeline and infrastructure engineering. Good backend/infrastructure/data engineers are so much harder to find these days, and these skills IMO provide much more value than doing some kind of modeling.

MrBuddyCasino 6y ago
> Good backend/infrastructure/data engineers are so much harder to find these days, and these skills IMO provide much more value than doing some kind of modeling. After all these years of FOMO on AI, this is music to my ears.
- ricksharp 6y ago
  Right, I’ve been reading about machine learning for about 5 years, have read hundreds of articles about different techniques, and have often tried to explore ways it could be used. However, I’ve never found a practical use in software engineering. Every time I think I discover something that could use machine learning, I usually don’t have any data to work with or don’t have a clear definition of what the inputs and outputs would be. In the end, I find a way to develop a solution that doesn’t need AI and often makes me realize that AI would not have been able to provide the required reliability. I even participated in a self-driving RC car competition (running on raspi 3). I ended up scrapping the ML solution because it could only run at 10 hertz which was not even close to what was needed. I ended up writing a custom algorithm in C that ran at 200fps and could do “advanced” moves like backing up and course correcting. I’m not saying that ML doesn’t have its place. I’m saying that right now it’s all about images and nlp and has huge costs. In any practical usage, the ML parts that I would need are already available as a service that I can use almost free (like azure cognitive services).
  - KineticLensman 6y ago
    > In the end, I find a way to develop a solution that doesn’t need AI and often makes me realize that AI would not have been able to provide the required reliability. This was brought home to me back in the 1990s at a place where we were trying to popularise expert system technology (a positively prehistoric form of AI). It was one of my first jobs in industry after leaving academia. The goal was to create a system that could predict the speed at which a unit of military vehicles could move so that a contact report expert system could decide whether two groups of vehicles could have feasibly moved between two points in a given time. The system could use this to help decide whether two reports referred to two different enemy units or a single unit that had moved from point A to point B. After quite a lot of time cranking out and debugging rules to describe the movement behaviours, I realised that some simple convoy arithmetic would do just as good a job - units are often constrained by the speed of the slowest vehicle, and respect inter-vehicle distances. For most purposes this simple arithmetic was just as good (and orders of magnitude faster) than the complex rule engine.
  - skwb 6y ago
    I think there is a lot of more practical deep learning that are coming out, but it often times working with the people who know the field intimately. For example, in my PhD thesis project, I am making a series of deep learning models to make a cardiac MRI autopilot. I've build a series of deep learning networks to localize the cardiac landmarks that define the cardiac imaging planes. And our group has even made it into a clinical prototype that works within our imaging workflow. I think the field is shifting in that ML technologies are increasingly requiring domain level knowledge in order to make a practical endpoint.
- apohn 6y ago
  I've been involved in a lot of Data Science projects, and the most successful ones are the ones where the Data Scientist either did or was part of the infrastructure and Data Engineering. Most of the failed projects basically were "Hire a Data Scientist to make a magic model" and nothing else was supported. Basically, "Here's some data, do some magic." I read somewhere that 90% of data is low/zero value, and I'd agree with that. There's absolutely a place for people doing pure statistical or ML/Deep Learning modeling. But the rest of the organization has to support them so they can have that narrow focus. A lot of places want to take a shortcut and not do the data work. 5 years ago the FOMO was from Hadoop. Hadoop Here, Hadoop there...people just forgot you have to actually collect a lot of high quality data to make Hadoop useful. It's AI now.

bumby 6y ago

I’ve found that non-technical management expectations are just too swayed by the AI hype machine. I’ve also found operations research field to be more grounded and still have plenty of research opportunities. There’s something very fulfilling about optimizing a real-world system, particularly when the answer isn’t intuitive.

apohn 6y ago

>I work as a data scientist and sometimes find the hype so off-putting that I think that I should look for a role that's related to... I'm Data Scientist and the hype is making me really jaded about the whole thing. I work in a customer facing role and it's just painful to see how many sensible and solvable problems with ML/Deep Learning get buried and destroyed under the pile of hype of solving unrealistic "sexy" problems with "AI." It was okay when Machine Learning was the hot thing, it's been awful since "AI" became the hot thing.