4 ms·

The author says "AI is already at or beyond human accuracy in all the tasks on this slide and is continuing to get better rapidly" and one of his examples is "

The author says "AI is already at or beyond human accuracy in all the tasks on this slide and is continuing to get better rapidly" and one of his examples is "Medical diagnosis from scans". That is an example of precisely the sort of snake oil hype he's berating in the social prediction category. In an extremely narrow sense of pattern recognition of some "image features", i.e. 5% of what a radiologist actually does, he's probably right. But context is the other 95%, and AI is nowhere close to being able to approach expert accuracy in that. It's a goal as far away from reality as AGI. "AI" tools will probably improve the productivity of radiologists, and there are statistical learning tools that already kind of do that (usually not actually widely used in medical practice, you can say yet, I can say who knows but nice prototype). But actual diagnosis, like the part where an MD makes a judgement call and the part which malpractice insurance is for? Not in any of our lifetimes. A radiologist friend complains that it's been 10+ years since they've been using speech recognition instead of a human transcriptionist, and all the systems out there are still really bad. Recognizing medical lingo is something you can probably achieve with more training data, but the software that sometimes drops "not" from a scan report is a cost-cutting measure, not a productivity tool. It makes the radiologist worse off because he's got to waste his time proofreading the hell out of it, but the hospital saves money.

randomwalker 7y ago
Author here. I appreciate your criticism. What I had in mind was more along the lines of Google's claims around diabetic retinopathy. I received feedback very similar to yours, i.e. that those claims are based on an extremely narrow problem formulation: https://twitter.com/MaxALittle/status/1196957870853627904 https://twitter.com/MaxALittle/status/1196957870853627904 I will correct this in future versions of the talk and paper.
- Balgair 7y ago
  Thanks for the .pdf and the research in general, great stuff! One thing I'd love is a look at 'noise' in these systems, specifically injecting noise into them. Addons like Noiszy [0] and trackmenot [1] claim to help, but I'd imagine that doing so with your GPS location is a bit tougher. I'd love to know more on such tactics, as it seems that opt-ing out of tracking isn't super feasible anymore (despite the effectiveness of the tracking). Again, great work, please keep it up! [0] https://noiszy.com/ https://noiszy.com/ [1] https://trackmenot.io/ https://trackmenot.io/
- jl2718 7y ago
  Then I shall write to you directly. I don’t know how you can make the claim that automated essay grading is anything but a shockingly mendacious academic abuse of student’s time and brainpower. To me, this seems far worse than job applicant filtering, firstly because hiring is fundamentally predictive, and secondly because many jobs have a component of legitimately rigid qualifications. An essay is a tool to affect the thoughts of a human. It is not predictive of some hidden factor; it stands alone. It must be original to have value; a learned pattern of ideas is the anti-pattern for novelty. If the grading of an essay can be, in any way, assisted by an algorithm, it is probably not worth human effort to produce. If you personally use essay grading software, or know of anybody at Princeton that does, you have an absolute obligation to disclose this to all of your students and prospective applicants. They are paying for humans to help them become better humans.
ACow_Adonis 7y ago
FYI, I too looked at his categorisation of what was/was advancing/was not snake oil and didn't exactly agree with all his decisions either. Medical imaging diagnosis was one of them. Speech recognition/transcription was another. I don't know if it's my accent or my speech patterns(though foreigners regularly compliment my wife and myself on our pronunciation), but the tech hasn't gotten noticeably better for me since the days of Dragon natural speaking, and that was, what...10 years ago? Sure, I can "hey Google/siri/alexa" a handful of predefined commands, but I still have to talk in a sort of stoccato "I am talking to a computer" voice, it still only gets it right 90% of the time, and God help you if you try anything new/natural not in the form of "writing Boolean logic programs with my voice".
- dahart 7y ago
  I feel like it’s gotten loads better. You can watch as the voice recognition on your phone changes the words it recognizes to match the context in the sentence. Sometimes it gets a word wrong and fixes it after a half second. Google translate does magical things recognizing common phrasing constructions and bad accents, stuff that Dragon could never do. I built a lipsync pipeline for video games based on Dragon a decade ago, and it most definitely was not as good as what I have on my phone today.
  - ACow_Adonis 7y ago
    Again, just anecdotally, I don't know if its just me, but most of my experience is of google/apple translate and auto-corrects going from the word I want to a wrong/incorrect one. It is one of my most frustrating everyday software experiences. Not only is it not getting better, its actually getting worse, because before I at least had the correct sentence. Now my correct sentence is mangled as it tries to force corrections/substitutions, and I have to continually go back and manually-correct the auto-correct. It seems to work for me on short pre-formed sentences and toy examples (if you communicate using pre-formed phrases and use well-worn cliches in your writing, it seems to pick up and predict for them). I wonder whether the "increased accuracy" of modern solutions aren't just functionally having access to a larger library of lookup rules of stored common/popular phrases and direct translations (a huge part of practical 'AI' advancement has been on the scaling-infrastructure/collection of new-scales-of-data rather than the AI techniques themselves IMO) effectively mined from its training data, but the moment I try to write or dictate anything new, original, or lengthy and it turns absolutely pear-shaped.

randomwalker 7y ago

Author here. I appreciate your criticism. What I had in mind was more along the lines of Google's claims around diabetic retinopathy. I received feedback very similar to yours, i.e. that those claims are based on an extremely narrow problem formulation: https://twitter.com/MaxALittle/status/1196957870853627904 https://twitter.com/MaxALittle/status/1196957870853627904 I will correct this in future versions of the talk and paper.

Balgair 7y ago
Thanks for the .pdf and the research in general, great stuff! One thing I'd love is a look at 'noise' in these systems, specifically injecting noise into them. Addons like Noiszy [0] and trackmenot [1] claim to help, but I'd imagine that doing so with your GPS location is a bit tougher. I'd love to know more on such tactics, as it seems that opt-ing out of tracking isn't super feasible anymore (despite the effectiveness of the tracking). Again, great work, please keep it up! [0] https://noiszy.com/ https://noiszy.com/ [1] https://trackmenot.io/ https://trackmenot.io/
jl2718 7y ago
Then I shall write to you directly. I don’t know how you can make the claim that automated essay grading is anything but a shockingly mendacious academic abuse of student’s time and brainpower. To me, this seems far worse than job applicant filtering, firstly because hiring is fundamentally predictive, and secondly because many jobs have a component of legitimately rigid qualifications. An essay is a tool to affect the thoughts of a human. It is not predictive of some hidden factor; it stands alone. It must be original to have value; a learned pattern of ideas is the anti-pattern for novelty. If the grading of an essay can be, in any way, assisted by an algorithm, it is probably not worth human effort to produce. If you personally use essay grading software, or know of anybody at Princeton that does, you have an absolute obligation to disclose this to all of your students and prospective applicants. They are paying for humans to help them become better humans.

ACow_Adonis 7y ago

FYI, I too looked at his categorisation of what was/was advancing/was not snake oil and didn't exactly agree with all his decisions either. Medical imaging diagnosis was one of them. Speech recognition/transcription was another. I don't know if it's my accent or my speech patterns(though foreigners regularly compliment my wife and myself on our pronunciation), but the tech hasn't gotten noticeably better for me since the days of Dragon natural speaking, and that was, what...10 years ago? Sure, I can "hey Google/siri/alexa" a handful of predefined commands, but I still have to talk in a sort of stoccato "I am talking to a computer" voice, it still only gets it right 90% of the time, and God help you if you try anything new/natural not in the form of "writing Boolean logic programs with my voice".

dahart 7y ago
I feel like it’s gotten loads better. You can watch as the voice recognition on your phone changes the words it recognizes to match the context in the sentence. Sometimes it gets a word wrong and fixes it after a half second. Google translate does magical things recognizing common phrasing constructions and bad accents, stuff that Dragon could never do. I built a lipsync pipeline for video games based on Dragon a decade ago, and it most definitely was not as good as what I have on my phone today.
- ACow_Adonis 7y ago
  Again, just anecdotally, I don't know if its just me, but most of my experience is of google/apple translate and auto-corrects going from the word I want to a wrong/incorrect one. It is one of my most frustrating everyday software experiences. Not only is it not getting better, its actually getting worse, because before I at least had the correct sentence. Now my correct sentence is mangled as it tries to force corrections/substitutions, and I have to continually go back and manually-correct the auto-correct. It seems to work for me on short pre-formed sentences and toy examples (if you communicate using pre-formed phrases and use well-worn cliches in your writing, it seems to pick up and predict for them). I wonder whether the "increased accuracy" of modern solutions aren't just functionally having access to a larger library of lookup rules of stored common/popular phrases and direct translations (a huge part of practical 'AI' advancement has been on the scaling-infrastructure/collection of new-scales-of-data rather than the AI techniques themselves IMO) effectively mined from its training data, but the moment I try to write or dictate anything new, original, or lengthy and it turns absolutely pear-shaped.