5 ms·

When I was last interviewing candidates for a data scientist position, I can't count how many physicists applied for the job when they didn't even understand wh

by fmajid 8y ago

When I was last interviewing candidates for a data scientist position, I can't count how many physicists applied for the job when they didn't even understand what overfitting was. There is a definite sense of "physics is hard, therefore everything else must be easy" groupthink, combined with Dunning-Kruger syndrome.

noobermin 8y ago
To be fair, OP seems to have actually successfully transitioned, so they don't seem like the other physicists you seem to have a glut of.
ritoune 8y ago
That's your chance for a cheap hire? They're probably not going to get hired anywhere else and you can see in-interview if they are able to understand quickly! There's not much value for junior positions to "know concept X", so what you want to know is the derivative...
analog31 8y ago
That's sad because overfitting predates ML by a long shot, and most experimental physicists of my generation (PhD 1990) knew what it was, even if we didn't use that term. In my view it's not so much that "everything else must be easy," but that physicists are expected to take a multidisciplinary approach to our work. In the process of getting my thesis project to work, I had to teach myself electronics and programming, not to mention a variety of mechanical skills such as machining and vacuum plumbing. Naturally, the job market situation encourages physicists to spread our wings beyond physics research, and to claim that we can do things. ;-) Most physicists are pretty good at math. This actually gives us a head start in quantitative branches of other disciplines such as engineering. Today, when the engineers at my workplace run into a hard quantitative problem, they go looking for a physicist. If math is what makes a subject seem hard to most people, you might discover a few physicists trying to sneak in.
- thrmsforbfast 8y ago
  > (PhD 1990) The title of the post is extremely misleading. The author of this post graduated from an undergraduate program in 2018. By "transitioned from academia", the author means "graduated with a B.S. (after a few research-y summer internships) and got a job". That first job is always an accomplishment, but in this cae was definitely not a "transition from academia" in the usual sense. As a result, the post generally contains irrelevant advice for people actually transitioning (i.e., professors, post-docs, and late-stage graduate students).
tibyat 8y ago
not knowing a term is not a good reflection of their ability.
jsweojtj 8y ago
I'm really curious now. How were you testing their understanding of overfitting? Like what was the setup/questions?
acdc4life 8y ago
If you are hiring physicist to do Machine Learning, then you are doing it all wrong. Physicist are excellent at mathematical modeling, especially those in their 50s+. Data science from my experience is vastly inferior to mathematical modeling. The OP that wrote the article seems young, and doesn't seem like they've seen much. In the world of quantitative finance, physicists and mathematicians reign supreme. These guys get paid millions in salary and bonuses, because they earn 10x that for the firms. Their entire jobs are to mathematically model things. There are very successful hedge funds that only hire physicist and mathematicians. The current data science and machine learning is child's play compared to the stuff these quant funds do. Physicist, especially ones that have mathematical maturity are incredible assets, they have a lethal skillset that isn't really there in silicon valley. The majority of data science posts I see on HN are mediocre at best. I think age and experience are highly under valued. The physicists and mathematicians that Draper, Raytheon, DARPA, Renaissance Tech, Black Rock or Bridge Water (I would include SpaceX and Tesla in this list) have are superior to any data scientist that you will find in Silicon Valley. Mathematical modeling is incredibly hard, seasoned physicists are very good at it (I can't emphasize how difficult and long this process can be, nor can I articulate the reward for doing things this way). > There is a definite sense of "physics is hard, therefore everything else must be easy" groupthink, combined with Dunning-Kruger syndrome. You are severely misinformed, and you severely lack a breadth of industry experience. Physicists (especially the experienced ones)have something called mathematical maturity. Having mathematical maturity is a skillset, you are able to make precise formulations of some phenomenon you want to understand things about, and prove with rigor that your ideas are correct, or that they converge to what you want. This is a lethal mind to have, it's far superior to machine learning or data science. You aren't relying on heuristics like "overfitting". These concepts aren't well defined, and are very subjective. I can tell you this with 100% confidence, AlphaGo, Duplex, Siri etc. are all gimmicks, no different from Watson or Deep Blue. They are just applying statistical brute force to narrow domains. I'd love to see a machine learning system that consistently matches the profits that the successful quantitative funds do. I know no tech company can do this, they don't have a fundamental understanding of mathematical analysis, they don't hire the right people. The right people are at DARPA, Draper, Tesla, SpaceX, any organization working on the hard sciences. Go figure; pretty much all of the major important discoveries came from either military or laborites investigating the hard science. Data science/ML, is great for simple data sets (there is a mathematical definition of this, not heuristic). For many industrial problems like what SAP, or McKinsey does it's perfect. The available cloud compute power and a basic Statistics + CS degree will disrupt these businesses. But do know this: they are severely limited. Machine learning will very quickly start showing its cracks.
deleted 8y ago
[deleted]
internet555 8y ago
Sometimes I wish physicists or mathematicians or whoever will actually try these engineering or whatever problems. There may be certain aspects of them that your training will make you very able to handle, but these problems are actually very hard and you need a lot of their specific knowledge for them.
- ummonk 8y ago
  Was physicist. Did engineering problems. Didn’t have much trouble (especially compared to those with more specialized educations in the relevant fields).
godelski 8y ago
That's really interesting because in my undergrad (which I hold a BS in Physics) overfitting and error were big topics in all labs. In fact, most physicists I know complain about overfitting in current ML models and how people are making these models and just blindly accepting results.

by fmajid 8y ago

noobermin 8y ago

To be fair, OP seems to have actually successfully transitioned, so they don't seem like the other physicists you seem to have a glut of.

ritoune 8y ago

That's your chance for a cheap hire? They're probably not going to get hired anywhere else and you can see in-interview if they are able to understand quickly! There's not much value for junior positions to "know concept X", so what you want to know is the derivative...

analog31 8y ago

That's sad because overfitting predates ML by a long shot, and most experimental physicists of my generation (PhD 1990) knew what it was, even if we didn't use that term. In my view it's not so much that "everything else must be easy," but that physicists are expected to take a multidisciplinary approach to our work. In the process of getting my thesis project to work, I had to teach myself electronics and programming, not to mention a variety of mechanical skills such as machining and vacuum plumbing. Naturally, the job market situation encourages physicists to spread our wings beyond physics research, and to claim that we can do things. ;-) Most physicists are pretty good at math. This actually gives us a head start in quantitative branches of other disciplines such as engineering. Today, when the engineers at my workplace run into a hard quantitative problem, they go looking for a physicist. If math is what makes a subject seem hard to most people, you might discover a few physicists trying to sneak in.

thrmsforbfast 8y ago
> (PhD 1990) The title of the post is extremely misleading. The author of this post graduated from an undergraduate program in 2018. By "transitioned from academia", the author means "graduated with a B.S. (after a few research-y summer internships) and got a job". That first job is always an accomplishment, but in this cae was definitely not a "transition from academia" in the usual sense. As a result, the post generally contains irrelevant advice for people actually transitioning (i.e., professors, post-docs, and late-stage graduate students).

tibyat 8y ago

not knowing a term is not a good reflection of their ability.

jsweojtj 8y ago

I'm really curious now. How were you testing their understanding of overfitting? Like what was the setup/questions?

acdc4life 8y ago

If you are hiring physicist to do Machine Learning, then you are doing it all wrong. Physicist are excellent at mathematical modeling, especially those in their 50s+. Data science from my experience is vastly inferior to mathematical modeling. The OP that wrote the article seems young, and doesn't seem like they've seen much. In the world of quantitative finance, physicists and mathematicians reign supreme. These guys get paid millions in salary and bonuses, because they earn 10x that for the firms. Their entire jobs are to mathematically model things. There are very successful hedge funds that only hire physicist and mathematicians. The current data science and machine learning is child's play compared to the stuff these quant funds do. Physicist, especially ones that have mathematical maturity are incredible assets, they have a lethal skillset that isn't really there in silicon valley. The majority of data science posts I see on HN are mediocre at best. I think age and experience are highly under valued. The physicists and mathematicians that Draper, Raytheon, DARPA, Renaissance Tech, Black Rock or Bridge Water (I would include SpaceX and Tesla in this list) have are superior to any data scientist that you will find in Silicon Valley. Mathematical modeling is incredibly hard, seasoned physicists are very good at it (I can't emphasize how difficult and long this process can be, nor can I articulate the reward for doing things this way). > There is a definite sense of "physics is hard, therefore everything else must be easy" groupthink, combined with Dunning-Kruger syndrome. You are severely misinformed, and you severely lack a breadth of industry experience. Physicists (especially the experienced ones)have something called mathematical maturity. Having mathematical maturity is a skillset, you are able to make precise formulations of some phenomenon you want to understand things about, and prove with rigor that your ideas are correct, or that they converge to what you want. This is a lethal mind to have, it's far superior to machine learning or data science. You aren't relying on heuristics like "overfitting". These concepts aren't well defined, and are very subjective. I can tell you this with 100% confidence, AlphaGo, Duplex, Siri etc. are all gimmicks, no different from Watson or Deep Blue. They are just applying statistical brute force to narrow domains. I'd love to see a machine learning system that consistently matches the profits that the successful quantitative funds do. I know no tech company can do this, they don't have a fundamental understanding of mathematical analysis, they don't hire the right people. The right people are at DARPA, Draper, Tesla, SpaceX, any organization working on the hard sciences. Go figure; pretty much all of the major important discoveries came from either military or laborites investigating the hard science. Data science/ML, is great for simple data sets (there is a mathematical definition of this, not heuristic). For many industrial problems like what SAP, or McKinsey does it's perfect. The available cloud compute power and a basic Statistics + CS degree will disrupt these businesses. But do know this: they are severely limited. Machine learning will very quickly start showing its cracks.

deleted 8y ago

[deleted]

internet555 8y ago

Sometimes I wish physicists or mathematicians or whoever will actually try these engineering or whatever problems. There may be certain aspects of them that your training will make you very able to handle, but these problems are actually very hard and you need a lot of their specific knowledge for them.

ummonk 8y ago
Was physicist. Did engineering problems. Didn’t have much trouble (especially compared to those with more specialized educations in the relevant fields).

godelski 8y ago

That's really interesting because in my undergrad (which I hold a BS in Physics) overfitting and error were big topics in all labs. In fact, most physicists I know complain about overfitting in current ML models and how people are making these models and just blindly accepting results.