4 ms·

I often make this point (if you don't have a grad-level understanding..). And people get pissy. Same with statistics ("if you don't have a PHD in stats; you

I often make this point (if you don't have a grad-level understanding..). And people get pissy. Same with statistics ("if you don't have a PHD in stats; you don't understand stats"). -- I had someone contact me for a "consulting" position, to replace their rule-based insurance claim denial system, with "ML". Contract-work, nobody in-house to review its performance, no insight to the problems of the domain and they wanted to throw ML at it to "solve their problem". When I refused, and pointed out that it would be highly irresponsible, to "contract-out" this type of work, particularly given its life-or-death implications; the CEO got angry. Told me; its "post-claims processing" and it isn't life or death. To which I told him bullshit; your denial of claims is going to directly influence how doctors practice their treatment. There is a direct feedback cycle. The fact that you don't see it, makes it even more dangerous. They simply, didn't understand, how HIGHLY inappropriate it was to just throw random ML at a problem, particularly as a one-off consulting project (and no in-house expertise). ^ thats real-world. Personally I don't think you should be allowed anywhere near ML, UNLESS you have that PHD in computer science. I don't even think you should be allowed to HIRE people for ML until you fully understand the hazards with letting a computer control critical decisions. So yeah, with respect to your distinction of "ML research" and "applied ML". "NO."

FranzFerdiNaN 8y ago
> Personally I don't think you should be allowed anywhere near ML, UNLESS you have that PHD in computer science. I rather have people have a PhD in logic or ethics, so they hopefully wont make for example racist programs without even thinking about it. Unfortunately, so far i dont have a lot of faith in computer science as a field when it comes to ethics, as it's all about the technical challenge. That something can be build does not mean it should be build.
mindcrime 8y ago
So yeah, with respect to your distinction of "ML research" and "applied ML". "NO." Interesting. I don't see anything in the story above that supports that position. There's nothing about the scenario you described that would be affected by having, or not having, a Phd in CS. If anything, as somebody else pointed out, this is more of question of philosophy or ethics. It's also a pretty niche example, which is not representative of the kinds of things that ML can be used for. If you want somebody who work on pricing optimization or a recommender system for your e-commerce site, you really don't need somebody who's doing cutting edge ML research.
- paulgrant999 8y ago
  > There's nothing about the scenario you described that would be affected by having, or not having, a Phd in CS. For starters, understanding there is a basic feedback cycle, where your ML in rejecting claims, directly affects patient treatment, wouldn't escape the attention of a PHD. Part of doing research, is anticipating questions you might not ordinarily think of, in connection with your research i.e. to examine the literature for known unknowns. Guess what? You think he looked, even after being directly apprised in VERY clear terminology? > It's also a pretty niche example, which is not representative of the kinds of things that ML can be used for. Fraud detection, niche? Thats what he is doing. Looking for "patterns" of fraud. He's just lazy and doesn't actually want to look at his data. Which again, means he shouldn't be running ML++. ML trains bias built in; and doesn't handle drift particularly well (isn't robust). You can do all sorts of things to try and address it. But thats only if you know about it (and care to pay for it). Some things, ML isn't suited for. Its like that woman diabetic who died, because some bureacrat cut off her foodstamps, even though they knew it was a bureacratic error; or that police officer, who let a man with cardiac arythmia die in the back of his car, because he had to "follow procedure". Except, worse. At least these people can be dragged out into the light, and fired. ML? who keeps track of that? ++ actually it means he shouldn't be running an insurance claims "fraud detection" company. but is a larger point outside the topic.
Xcelerate 8y ago
> Personally I don't think you should be allowed anywhere near ML, UNLESS you have that PHD in computer science A lot of ML research is done by PhDs in other fields. I did research that focused on developing compact group invariant features (for neural networks) for predicting local atomic energies in materials science, and a few mathematicians I follow did work on developing convolutional neural networks that utilize Clebsch–Gordan coefficients to generalize the translational invariance to other symmetries. On the contrary, a lot of CS machine learning research is heavily application focused (generate such-and-such new thing using a GAN). If anything, mathematicians are the ones who understand machine learning at the very deepest level of theory. This isn't to say there aren't many theory-focused CS presentations/publications; I'm just refuting your point that highly theoretical machine learning research is purely the domain of CS PhDs.
- jacobolus 8y ago
  This seems like it misses the point. Obviously there are professional numerical analysts, statisticians, mathematical physicists, etc. who have sufficient background and interest to keep up with cutting edge research and do solid work in machine learning. The argument is not that everyone needs a CS degree per se, but rather that you shouldn’t have your excel guy who just went through a machine learning MOOC but has no further training or deeper understanding try to apply machine learning to life-or-death problems.
  - mindcrime 8y ago
    shouldn’t have your excel guy who just went through a machine learning MOOC but has no further training or deeper understanding try to apply machine learning to life-or-death problems. Sure, but most problems aren't life-or-death. They're mundane problems related to improving a business process for a widget manufacturer, etc.
- paulgrant999 8y ago
  well logically, I can't make the statement a PHD is required; because a PHD, doesn't mean you know ML. So you can consider the comp sci a "fuzzy" field denominator indicating a heavy theoretical familiarity with the math behind computer science. > I'm just refuting your point that highly theoretical machine learning research is purely the domain of CS PhDs. actually I don't think ML is enough in and of itself; I also prefer a heavy domain-specific expertise. ;) but that was outside the scope of the original comment. in truth, the PHD doesn't guarantee intelligence; it gives you a track record to see what the person has been up to i.e. it is necessary, but not sufficient. I talked to a PHD doing superconductivity research where I could put together stuff he hadn't even thought of (in seven years of study!). Then I talked to a post-doc doing CFD simulation of certain physical systems, who had discovered a very very neat trick to speeding up computation (by orders of magnitude). The two could not be compared in quality. I would trust the latter (with one-sentence (or for particularly subtle shit, two sentences)) to immediately understand the implications; the former? You could spoon-feed him all night long and he would barely be able to decipher the sentence, much less the implications. Overall arching point: if you think you know ML, and haven't used it/studied it extensively, you don't know ML. You simply don't have the critical knowledge base of edge/corner cases to detect when your ML is giving you crappy results (and why; or more importantly, how to OR IF YOU CAN, remediate). > On the contrary, a lot of CS research on neural networks is heavily application focused Grad-level? its crap. I spent a couple of months refreshing the knowledge base on computing (reading papers), and lord it was like wading through a river of shit. And just when I was thinking this, about two days later I read a paper from some grad who basically said the same exact shit ("enough with the nature-anologies"). Couple of gems every now and then, but few and far between. Mostly derivative shit (where the person who wrote the paper was either desperate to get published; or didn't understand the problem domain sufficiently to see why there solution was computationally-equivalent to a prior solution). PHD's theres a slight higher increase in quality; but where they generally shine, is in the breadth of their knowledge-base. The extra time contemplating shit, makes a vast difference. To be frank, I find the masters from a top school, to be equivalent to a sophomore-junior year bachelors. or possibly a remedial (senior-year). PHD depends on the aptitude of the student, and the selection-quality of their research. You can either rock it, or coast. > If anything, mathematicians are the people who understand machine learning at the very deepest level. Good ones, yeah. Don't know the frequency of'em, because I never took a math degree. -- now compare and contrast this, with the positions held by the "applied ML" crowd. - sorry I couldn't reply. "posting too fast" <rolls eyes>

FranzFerdiNaN 8y ago

> Personally I don't think you should be allowed anywhere near ML, UNLESS you have that PHD in computer science. I rather have people have a PhD in logic or ethics, so they hopefully wont make for example racist programs without even thinking about it. Unfortunately, so far i dont have a lot of faith in computer science as a field when it comes to ethics, as it's all about the technical challenge. That something can be build does not mean it should be build.

mindcrime 8y ago

So yeah, with respect to your distinction of "ML research" and "applied ML". "NO." Interesting. I don't see anything in the story above that supports that position. There's nothing about the scenario you described that would be affected by having, or not having, a Phd in CS. If anything, as somebody else pointed out, this is more of question of philosophy or ethics. It's also a pretty niche example, which is not representative of the kinds of things that ML can be used for. If you want somebody who work on pricing optimization or a recommender system for your e-commerce site, you really don't need somebody who's doing cutting edge ML research.

paulgrant999 8y ago
> There's nothing about the scenario you described that would be affected by having, or not having, a Phd in CS. For starters, understanding there is a basic feedback cycle, where your ML in rejecting claims, directly affects patient treatment, wouldn't escape the attention of a PHD. Part of doing research, is anticipating questions you might not ordinarily think of, in connection with your research i.e. to examine the literature for known unknowns. Guess what? You think he looked, even after being directly apprised in VERY clear terminology? > It's also a pretty niche example, which is not representative of the kinds of things that ML can be used for. Fraud detection, niche? Thats what he is doing. Looking for "patterns" of fraud. He's just lazy and doesn't actually want to look at his data. Which again, means he shouldn't be running ML++. ML trains bias built in; and doesn't handle drift particularly well (isn't robust). You can do all sorts of things to try and address it. But thats only if you know about it (and care to pay for it). Some things, ML isn't suited for. Its like that woman diabetic who died, because some bureacrat cut off her foodstamps, even though they knew it was a bureacratic error; or that police officer, who let a man with cardiac arythmia die in the back of his car, because he had to "follow procedure". Except, worse. At least these people can be dragged out into the light, and fired. ML? who keeps track of that? ++ actually it means he shouldn't be running an insurance claims "fraud detection" company. but is a larger point outside the topic.

Xcelerate 8y ago

> Personally I don't think you should be allowed anywhere near ML, UNLESS you have that PHD in computer science A lot of ML research is done by PhDs in other fields. I did research that focused on developing compact group invariant features (for neural networks) for predicting local atomic energies in materials science, and a few mathematicians I follow did work on developing convolutional neural networks that utilize Clebsch–Gordan coefficients to generalize the translational invariance to other symmetries. On the contrary, a lot of CS machine learning research is heavily application focused (generate such-and-such new thing using a GAN). If anything, mathematicians are the ones who understand machine learning at the very deepest level of theory. This isn't to say there aren't many theory-focused CS presentations/publications; I'm just refuting your point that highly theoretical machine learning research is purely the domain of CS PhDs.

jacobolus 8y ago
This seems like it misses the point. Obviously there are professional numerical analysts, statisticians, mathematical physicists, etc. who have sufficient background and interest to keep up with cutting edge research and do solid work in machine learning. The argument is not that everyone needs a CS degree per se, but rather that you shouldn’t have your excel guy who just went through a machine learning MOOC but has no further training or deeper understanding try to apply machine learning to life-or-death problems.
- mindcrime 8y ago
  shouldn’t have your excel guy who just went through a machine learning MOOC but has no further training or deeper understanding try to apply machine learning to life-or-death problems. Sure, but most problems aren't life-or-death. They're mundane problems related to improving a business process for a widget manufacturer, etc.
paulgrant999 8y ago
well logically, I can't make the statement a PHD is required; because a PHD, doesn't mean you know ML. So you can consider the comp sci a "fuzzy" field denominator indicating a heavy theoretical familiarity with the math behind computer science. > I'm just refuting your point that highly theoretical machine learning research is purely the domain of CS PhDs. actually I don't think ML is enough in and of itself; I also prefer a heavy domain-specific expertise. ;) but that was outside the scope of the original comment. in truth, the PHD doesn't guarantee intelligence; it gives you a track record to see what the person has been up to i.e. it is necessary, but not sufficient. I talked to a PHD doing superconductivity research where I could put together stuff he hadn't even thought of (in seven years of study!). Then I talked to a post-doc doing CFD simulation of certain physical systems, who had discovered a very very neat trick to speeding up computation (by orders of magnitude). The two could not be compared in quality. I would trust the latter (with one-sentence (or for particularly subtle shit, two sentences)) to immediately understand the implications; the former? You could spoon-feed him all night long and he would barely be able to decipher the sentence, much less the implications. Overall arching point: if you think you know ML, and haven't used it/studied it extensively, you don't know ML. You simply don't have the critical knowledge base of edge/corner cases to detect when your ML is giving you crappy results (and why; or more importantly, how to OR IF YOU CAN, remediate). > On the contrary, a lot of CS research on neural networks is heavily application focused Grad-level? its crap. I spent a couple of months refreshing the knowledge base on computing (reading papers), and lord it was like wading through a river of shit. And just when I was thinking this, about two days later I read a paper from some grad who basically said the same exact shit ("enough with the nature-anologies"). Couple of gems every now and then, but few and far between. Mostly derivative shit (where the person who wrote the paper was either desperate to get published; or didn't understand the problem domain sufficiently to see why there solution was computationally-equivalent to a prior solution). PHD's theres a slight higher increase in quality; but where they generally shine, is in the breadth of their knowledge-base. The extra time contemplating shit, makes a vast difference. To be frank, I find the masters from a top school, to be equivalent to a sophomore-junior year bachelors. or possibly a remedial (senior-year). PHD depends on the aptitude of the student, and the selection-quality of their research. You can either rock it, or coast. > If anything, mathematicians are the people who understand machine learning at the very deepest level. Good ones, yeah. Don't know the frequency of'em, because I never took a math degree. -- now compare and contrast this, with the positions held by the "applied ML" crowd. - sorry I couldn't reply. "posting too fast" <rolls eyes>