8 ms·

I work as a data scientist and have some perspective on this. There's no boat to miss, you'll probably be fine. Just keep a couple of things in mind - The fund

by st1x7 6y ago

I work as a data scientist and have some perspective on this. There's no boat to miss, you'll probably be fine. Just keep a couple of things in mind - The fundamental skills that you need are mathematics and software engineering. Depending on your background it might take years of additional studying. - There is a big oversupply of people for the junior-mid level data science jobs. There are more people who want to get in the field than there are jobs. If you drastically switch careers, you'll take yourself out of a field where your skillset is incredibly rare and your competition is limited and put yourself in a place where everyone else wants the same job. - The fact that you have a PhD is going to help you. Personally, I don't think that a PhD in a field other than mathematics/computer science is that relevant but employers tend to favor applicants with PhDs mostly because there are too many candidates for any given job and asking for a PhD is just a strong initial filter. There are also research jobs within data science for which a PhD requirement (in a relevant discipline) makes more sense but these are a small proportion of all the data science jobs. - If you're already employed with your agriculture PhD, there must be a number of opportunities for you apply the techniques that you're currently learning wihout leaving the industry. That's probably the path that I would suggest - it would allow you to expand your skillset without taking big risks and you'll have more options in the future. Use the career capital that you already have and explore your options instead of making a sharp turn in your career direction that might leave you disappointed.

jimbob45 6y ago
> There is a big oversupply of people for the junior-mid level data science jobs. Honest question is there any field where that isn’t true right now? Seems like there are no junior jobs in any sector.
- mettamage 6y ago
  I live in Amsterdam. What I'm seeing here is that the security field is lagging behind the data science / web dev field. It's not easy to find junior jobs, but a lot easier than data science / web dev. The most fun example that I currently have is that my LinkedIn profile is super wide. I market myself as a generalist software dev, because that's what I am. I'm not a specialist and I've dabbled in almost everything. Naturally, no recruiter contacts me because of that. But you know who does? A European company looking for a reverse engineer. I found it quite surprising that they were capable of reading that I have some IDA Pro experience, because it's in the fine print of my profile and not at all readable when you skim it.
  - kranner 6y ago
    > I found it quite surprising that they were capable of reading that I have some IDA Pro experience, because it's in the fine print of my profile and not at all readable when you skim it. Wouldn't searching profiles for "IDA Pro" find it?
    - mettamage 6y ago
      I guess, but I wouldn't be on the top of the list as I am mentioning it once and there's also a lot more mobile and web buzz words on my profile.
      - jlokier 6y ago
        They don't read your profile at first. Sometimes they don't even read it at all before contacting you. They search for keywords, and rely on the search algorithm prioritisation and a good looking one-line summary of your profile to pick you out of pages of search results. They'll find your "IDA Pro" if that's in the list of terms they are looking for, no matter how deeply it's buried in the description of your past jobs. If they don't read your profile they might not even know that it's buried.
- deng 6y ago
  German here. From my perspective it's hard to find - embedded systems programmers, meaning people who know real-time systems, have good C/C++ knowledge, know their way around the Linux kernel and are also able to do basic things with a scope. - good(!) C++ programmers in general - people who know devops and software development infrastructure C++ might not be sexy, but there's a vast amount of legacy software out there which is not going away anytime soon.
  - jeffreygoesto 6y ago
    That is true. I found that for embedded C++ to be successful, you need to have some good (excellent sometimes was harmful, but that's a different topic) C++ _as well as_ good domain and optimization skills. It's a combination that is more and more scarce. C++ tends to be used in domains where size and speed matter. You need to know what the compiler and machine will do with your code to be fast. And what's more, only the domain knowledge will let you pick the right data structures to be fast and compact in the first place. And there are a lot of foot guns around... ;-)
    - wallscratch 6y ago
      > I found that for embedded C++ to be successful, you need to have some good (excellent sometimes was harmful, but that's a different topic) Why were excellent C++ skills sometimes harmful?
      - jeffreygoesto 6y ago
        First of all, it leaves the team behind if you use features that are too advanced. Makes collaboration hard if you have wildly diverging skills (in both ways). Also, it did in our case take you away so far from the bare metal, that the code was elegant, but slow. It did not play well with our static allocators and allocated/deallocated way too much, especially temporaries. You might i.e. check out talks like [0]. [0] https://m.youtube.com/watch?v=NH1Tta7purM https://m.youtube.com/watch?v=NH1Tta7purM
        6y ago
- kylestlb 6y ago
  one factor here is that 'senior' doesn't really mean anything anymore other than 3+ years of experience.
yilugurlu 6y ago
> - If you're already employed with your agriculture PhD, there must be a number of opportunities for you apply the techniques that you're currently learning wihout leaving the industry. That's probably the path that I would suggest - it would allow you to expand your skillset without taking big risks and you'll have more options in the future. Use the career capital that you already have and explore your options instead of making a sharp turn in your career direction that might leave you disappointed. This is real gold. If you have existing knowledge about some area than you can learn and apply those things, thus you'll get the traction that you want earlier, and it would be more rewarding in the end. If there is a chance, I won't miss that opportunity. Good luck.
- ghaff 6y ago
  In general, domain knowledge is incredibly valuable and will generally trump someone who may have some more technical skills but doesn't know the field. I'd add that a lot of larger companies--heard someone from Shell talking about this a couple months ago--are training up people who have knowledge of the business with "citizen data scientist" skill sets. Without knowing all the details of your situation, it seems at least a much lower risk path to acquire some data science skills--maybe your company will even pay for it--that you can pair with your existing domain knowledge.
  - NoOneNew 6y ago
    Your comment will forever be under rated due to how spot on it is. It's a fundamental issue in tech that people don't appreciate. While yes, some problems are universal and general tech can solve them without industry experience, I'm pretty sure most people agree that those problems are either solved or there's an army of devs already on it. These days it's not really enough to just be a dev or data sci. You need to know a field and apply your tech knowledge in better ways than the uninitiated would never imagine.
    - ethbr0 6y ago
      The problem is that real world, physical business data is taken from what one can get, not what one would want. And usually "what the business was collecting for (unrelated purpose)." This means it has nearly infinite caveats and assumptions. A specifications doc or readout will never sufficiently express all of these. Especially if humans were involved in the data generated. Consequently, the most useful data products are going to turn on whether of not you (did this small thing) to (correct for this obvious bias or flaw that anyone familiar with the industry knows).
birdsbirdsbirds 6y ago
>There is a big oversupply of people for the junior-mid level data science jobs. Which shouldn't be too much of a problem because they have the skills to analyse the market and find a local optimum that suits them.
junon 6y ago
As a software engineer who has worked intimately with data scientists in Big Tech companies, this comment sums it up exactly. One minor thing to add is that not only is the data science field overflowing with junior/mid candidates, but the entire computer science field as a whole.
- dehrmann 6y ago
  What happens in seven years when those candidates become experienced? Will they have dropped out? Will there be a glut of experienced software developers?
  - junon 6y ago
    I sincerely doubt a large portion of them are cut-out for the sector at all. That's the issue.
lmilcin 6y ago
I don't exactly agree on why employers prefer PhDs even if not exactly in the field. Having PhD shows that you have committed to one field hard enough and long enough to be granted the title even if it is PhD in philosophy. Also, if you switch from another career try to figure out how you can use your past experience to differentiate yourself from other people. For example, I have studied theoretical math and I have spent 20 years as software developer working on varied projects from embedded to algorithmic trading to backend to frontend applications. I am currently switching to electronics and will use my knowledge to work on projects that require high level of both math and software engineering skills. Think in terms of realtime control systems, building reliable and performant stuff (reliability to differentiate from competition), etc. Never think of moving as rebooting your career. Always start with mindset of building on top of what you know -- always move forward, not backward. A little change in mindset goes a long way.
- st1x7 6y ago
  I actually think that we agree on the PhD requirement. > Having PhD shows that you have committed to one field hard enough and long enough to be granted the title even if it is PhD in philosophy. Yes, exactly. That's what I meant by "strong initial filter".
- MakeUsersWant 6y ago
  Could you please say more about how you find maths-intense software engineering niches and projects? What niches are there? I too am a pure maths PhD with a background in software, but I am somewhat disappointed with the data science world.
camphero 6y ago
The data science professionals my company has had on staff have been a joke. It’s similar to the wave of inexperienced developers hired in the late 1990s; employers didn’t have the skills to prove that the contractors didn’t know what they were doing, so a ton of money was poured in with big dreams. Part of that is that they don’t understand what we do and weren’t trained. They other part is that the business was just told they needed data people. What businesses need today are to understand what they’re interested in first, and then hire people with proven experience and knowledge to accomplish those goals. Having a PhD in any applicable subject in addition to sufficient practical knowledge of data science, machine learning, and AI would be a substantial plus for a niche position in the industry, a non-profit, or even in finance. Don’t give up.
- ethbr0 6y ago
  > practical knowledge of data science This is key. What do companies actually want when they hire a data scientist? Actionable products that make their business better. What does it take to produce actionable products? A data strategy (collection, ingress, normalize, enrich, store, expose), a compute provisioning strategy, data engineering (pull from source system(s), land in target stores in a reliable, available, automated manner), data science, and data application (reporting, integration with target systems, app development). Which of those components do they typically have? A data scientist. Because they just hired one. Career-wise, the more unifaceted your skillset is, the more you're limited to employers that already have all the other pieces in place. Which effectively limits you to very large enterprise (~T100). Start to be able to fill some of the other roles yourself, and you can compete for and succeed in smaller and more interesting opportunities.
mihaela 6y ago
This reply should be made into billboard and placed in front of every "Data science bootcamp" out there.
CinchWrench 6y ago
>the fundamental skills that you need are mathematics and software engineering So much this. If I have to interview another junior-level DS who has a MNIST project in their github and still somehow can't manage fizzbuzz or a fibonacci function I'm probably going to take up religious asceticism. EDIT: I said junior, but I meant Senior. We're talking people with PhD's who claim to have done extensive software engineering in previous roles.
- kordlessagain 6y ago
  Try filtering them on their specific ability to visualize things. My neighbor was 80+ year old math guy who could see graphs of equations in his head. My hypothesis is that good stats people probably don't visualize (Aphant) or visualize very specific types of data in a unique way. Without visualization, people tend to fall back to logical thinking - or emotional thinking, depending. For example, I have a friend who can look at 2D seismic data and see what the underground formation looks like in his mind, in 3D.
  - whoisburbansky 6y ago
    The person you are replying to seemed to be complaining about their lack of programming ability, not statistical ability, so how would this help in their situation?
    - CinchWrench 6y ago
      Yup, I think I and that child are talking past each other. :-( It's good though because they raise some really valid points about the importance of intuition. I joke with my colleagues that all we're doing is encoding data as a hilbert space and slapping an algorithm on it, but that elides the fact that intuition like the child is talking about is important for knowing how to build that hilbert space.
- 6gvONxR4sf7o 6y ago
  If I have to interview another whose only ML tools are GLMs, random forests and boosted trees (only ever with one hot encoding, of course) I’m going to do the same.
asdffdsa 6y ago
What kind of math? It takes me a while to learn math so any focused area would help cut down the time taken immensely (not that I don't enjoy learning tidbits like supremum/infimum)
- deleted 6y ago
  [deleted]
- st1x7 6y ago
  Multivariable calculus, differential equations, linear algebra, probability theory, statistics.
  - randcraw 6y ago
    Ranked from most to least essential, I'd suggest: probability, statistics, linear algebra, basic calculus, operations research, multivariate and partial and vector calculus, diff eq.
    - st1x7 6y ago
      True but order of importance is different from order of study. You should really start with calculus because you can't study probability without it.
alexashka 6y ago
> Depending on your background it might take years of additional studying. Excellent summary but this one bit is trying to gloss over reality. It will most certainly take multiple years of additional studying when you come in contact with anything related to software engineering, because that's just what it is. Of course there is no shortage of people who learn one language, one framework, one tool and sit on their ass depending on the people they work with to pick up the slack. If that's you, none of the things being said in this thread matter - just go get your job, make your money and live your life :)
6gvONxR4sf7o 6y ago
I slightly disagree about your reasoning on the PhD thing, for some roles. Having a PhD in an experimental science is a (weak) indicator that you’ve learned how to design, conduct, and analyze an experiment on your own.
giantg2 6y ago
"There is a big oversupply of people for the junior-mid level data science jobs." I feel this is true for most tech jobs. I see tons of senior job postings, but relatively very few entry-level or mid-level postings.

by st1x7 6y ago

jimbob45 6y ago

> There is a big oversupply of people for the junior-mid level data science jobs. Honest question is there any field where that isn’t true right now? Seems like there are no junior jobs in any sector.

mettamage 6y ago
I live in Amsterdam. What I'm seeing here is that the security field is lagging behind the data science / web dev field. It's not easy to find junior jobs, but a lot easier than data science / web dev. The most fun example that I currently have is that my LinkedIn profile is super wide. I market myself as a generalist software dev, because that's what I am. I'm not a specialist and I've dabbled in almost everything. Naturally, no recruiter contacts me because of that. But you know who does? A European company looking for a reverse engineer. I found it quite surprising that they were capable of reading that I have some IDA Pro experience, because it's in the fine print of my profile and not at all readable when you skim it.
- kranner 6y ago
  > I found it quite surprising that they were capable of reading that I have some IDA Pro experience, because it's in the fine print of my profile and not at all readable when you skim it. Wouldn't searching profiles for "IDA Pro" find it?
  - mettamage 6y ago
    I guess, but I wouldn't be on the top of the list as I am mentioning it once and there's also a lot more mobile and web buzz words on my profile.
    - jlokier 6y ago
      They don't read your profile at first. Sometimes they don't even read it at all before contacting you. They search for keywords, and rely on the search algorithm prioritisation and a good looking one-line summary of your profile to pick you out of pages of search results. They'll find your "IDA Pro" if that's in the list of terms they are looking for, no matter how deeply it's buried in the description of your past jobs. If they don't read your profile they might not even know that it's buried.
deng 6y ago
German here. From my perspective it's hard to find - embedded systems programmers, meaning people who know real-time systems, have good C/C++ knowledge, know their way around the Linux kernel and are also able to do basic things with a scope. - good(!) C++ programmers in general - people who know devops and software development infrastructure C++ might not be sexy, but there's a vast amount of legacy software out there which is not going away anytime soon.
- jeffreygoesto 6y ago
  That is true. I found that for embedded C++ to be successful, you need to have some good (excellent sometimes was harmful, but that's a different topic) C++ _as well as_ good domain and optimization skills. It's a combination that is more and more scarce. C++ tends to be used in domains where size and speed matter. You need to know what the compiler and machine will do with your code to be fast. And what's more, only the domain knowledge will let you pick the right data structures to be fast and compact in the first place. And there are a lot of foot guns around... ;-)
  - wallscratch 6y ago
    > I found that for embedded C++ to be successful, you need to have some good (excellent sometimes was harmful, but that's a different topic) Why were excellent C++ skills sometimes harmful?
    - jeffreygoesto 6y ago
      First of all, it leaves the team behind if you use features that are too advanced. Makes collaboration hard if you have wildly diverging skills (in both ways). Also, it did in our case take you away so far from the bare metal, that the code was elegant, but slow. It did not play well with our static allocators and allocated/deallocated way too much, especially temporaries. You might i.e. check out talks like [0]. [0] https://m.youtube.com/watch?v=NH1Tta7purM https://m.youtube.com/watch?v=NH1Tta7purM
      - 6y ago
kylestlb 6y ago
one factor here is that 'senior' doesn't really mean anything anymore other than 3+ years of experience.

yilugurlu 6y ago

> - If you're already employed with your agriculture PhD, there must be a number of opportunities for you apply the techniques that you're currently learning wihout leaving the industry. That's probably the path that I would suggest - it would allow you to expand your skillset without taking big risks and you'll have more options in the future. Use the career capital that you already have and explore your options instead of making a sharp turn in your career direction that might leave you disappointed. This is real gold. If you have existing knowledge about some area than you can learn and apply those things, thus you'll get the traction that you want earlier, and it would be more rewarding in the end. If there is a chance, I won't miss that opportunity. Good luck.

ghaff 6y ago
In general, domain knowledge is incredibly valuable and will generally trump someone who may have some more technical skills but doesn't know the field. I'd add that a lot of larger companies--heard someone from Shell talking about this a couple months ago--are training up people who have knowledge of the business with "citizen data scientist" skill sets. Without knowing all the details of your situation, it seems at least a much lower risk path to acquire some data science skills--maybe your company will even pay for it--that you can pair with your existing domain knowledge.
- NoOneNew 6y ago
  Your comment will forever be under rated due to how spot on it is. It's a fundamental issue in tech that people don't appreciate. While yes, some problems are universal and general tech can solve them without industry experience, I'm pretty sure most people agree that those problems are either solved or there's an army of devs already on it. These days it's not really enough to just be a dev or data sci. You need to know a field and apply your tech knowledge in better ways than the uninitiated would never imagine.
  - ethbr0 6y ago
    The problem is that real world, physical business data is taken from what one can get, not what one would want. And usually "what the business was collecting for (unrelated purpose)." This means it has nearly infinite caveats and assumptions. A specifications doc or readout will never sufficiently express all of these. Especially if humans were involved in the data generated. Consequently, the most useful data products are going to turn on whether of not you (did this small thing) to (correct for this obvious bias or flaw that anyone familiar with the industry knows).

birdsbirdsbirds 6y ago

>There is a big oversupply of people for the junior-mid level data science jobs. Which shouldn't be too much of a problem because they have the skills to analyse the market and find a local optimum that suits them.

junon 6y ago

As a software engineer who has worked intimately with data scientists in Big Tech companies, this comment sums it up exactly. One minor thing to add is that not only is the data science field overflowing with junior/mid candidates, but the entire computer science field as a whole.

dehrmann 6y ago
What happens in seven years when those candidates become experienced? Will they have dropped out? Will there be a glut of experienced software developers?
- junon 6y ago
  I sincerely doubt a large portion of them are cut-out for the sector at all. That's the issue.

lmilcin 6y ago

I don't exactly agree on why employers prefer PhDs even if not exactly in the field. Having PhD shows that you have committed to one field hard enough and long enough to be granted the title even if it is PhD in philosophy. Also, if you switch from another career try to figure out how you can use your past experience to differentiate yourself from other people. For example, I have studied theoretical math and I have spent 20 years as software developer working on varied projects from embedded to algorithmic trading to backend to frontend applications. I am currently switching to electronics and will use my knowledge to work on projects that require high level of both math and software engineering skills. Think in terms of realtime control systems, building reliable and performant stuff (reliability to differentiate from competition), etc. Never think of moving as rebooting your career. Always start with mindset of building on top of what you know -- always move forward, not backward. A little change in mindset goes a long way.

st1x7 6y ago
I actually think that we agree on the PhD requirement. > Having PhD shows that you have committed to one field hard enough and long enough to be granted the title even if it is PhD in philosophy. Yes, exactly. That's what I meant by "strong initial filter".
MakeUsersWant 6y ago
Could you please say more about how you find maths-intense software engineering niches and projects? What niches are there? I too am a pure maths PhD with a background in software, but I am somewhat disappointed with the data science world.

camphero 6y ago

The data science professionals my company has had on staff have been a joke. It’s similar to the wave of inexperienced developers hired in the late 1990s; employers didn’t have the skills to prove that the contractors didn’t know what they were doing, so a ton of money was poured in with big dreams. Part of that is that they don’t understand what we do and weren’t trained. They other part is that the business was just told they needed data people. What businesses need today are to understand what they’re interested in first, and then hire people with proven experience and knowledge to accomplish those goals. Having a PhD in any applicable subject in addition to sufficient practical knowledge of data science, machine learning, and AI would be a substantial plus for a niche position in the industry, a non-profit, or even in finance. Don’t give up.

ethbr0 6y ago
> practical knowledge of data science This is key. What do companies actually want when they hire a data scientist? Actionable products that make their business better. What does it take to produce actionable products? A data strategy (collection, ingress, normalize, enrich, store, expose), a compute provisioning strategy, data engineering (pull from source system(s), land in target stores in a reliable, available, automated manner), data science, and data application (reporting, integration with target systems, app development). Which of those components do they typically have? A data scientist. Because they just hired one. Career-wise, the more unifaceted your skillset is, the more you're limited to employers that already have all the other pieces in place. Which effectively limits you to very large enterprise (~T100). Start to be able to fill some of the other roles yourself, and you can compete for and succeed in smaller and more interesting opportunities.

mihaela 6y ago

This reply should be made into billboard and placed in front of every "Data science bootcamp" out there.

CinchWrench 6y ago

>the fundamental skills that you need are mathematics and software engineering So much this. If I have to interview another junior-level DS who has a MNIST project in their github and still somehow can't manage fizzbuzz or a fibonacci function I'm probably going to take up religious asceticism. EDIT: I said junior, but I meant Senior. We're talking people with PhD's who claim to have done extensive software engineering in previous roles.

kordlessagain 6y ago
Try filtering them on their specific ability to visualize things. My neighbor was 80+ year old math guy who could see graphs of equations in his head. My hypothesis is that good stats people probably don't visualize (Aphant) or visualize very specific types of data in a unique way. Without visualization, people tend to fall back to logical thinking - or emotional thinking, depending. For example, I have a friend who can look at 2D seismic data and see what the underground formation looks like in his mind, in 3D.
- whoisburbansky 6y ago
  The person you are replying to seemed to be complaining about their lack of programming ability, not statistical ability, so how would this help in their situation?
  - CinchWrench 6y ago
    Yup, I think I and that child are talking past each other. :-( It's good though because they raise some really valid points about the importance of intuition. I joke with my colleagues that all we're doing is encoding data as a hilbert space and slapping an algorithm on it, but that elides the fact that intuition like the child is talking about is important for knowing how to build that hilbert space.
6gvONxR4sf7o 6y ago
If I have to interview another whose only ML tools are GLMs, random forests and boosted trees (only ever with one hot encoding, of course) I’m going to do the same.

asdffdsa 6y ago

What kind of math? It takes me a while to learn math so any focused area would help cut down the time taken immensely (not that I don't enjoy learning tidbits like supremum/infimum)

deleted 6y ago
[deleted]
st1x7 6y ago
Multivariable calculus, differential equations, linear algebra, probability theory, statistics.
- randcraw 6y ago
  Ranked from most to least essential, I'd suggest: probability, statistics, linear algebra, basic calculus, operations research, multivariate and partial and vector calculus, diff eq.
  - st1x7 6y ago
    True but order of importance is different from order of study. You should really start with calculus because you can't study probability without it.

alexashka 6y ago

> Depending on your background it might take years of additional studying. Excellent summary but this one bit is trying to gloss over reality. It will most certainly take multiple years of additional studying when you come in contact with anything related to software engineering, because that's just what it is. Of course there is no shortage of people who learn one language, one framework, one tool and sit on their ass depending on the people they work with to pick up the slack. If that's you, none of the things being said in this thread matter - just go get your job, make your money and live your life :)

6gvONxR4sf7o 6y ago

I slightly disagree about your reasoning on the PhD thing, for some roles. Having a PhD in an experimental science is a (weak) indicator that you’ve learned how to design, conduct, and analyze an experiment on your own.

giantg2 6y ago

"There is a big oversupply of people for the junior-mid level data science jobs." I feel this is true for most tech jobs. I see tons of senior job postings, but relatively very few entry-level or mid-level postings.