4 ms·

Title seems somewhat misleading. He said ML often performs poorly on out of sample inputs. Seems different from being “a long way from real world use.” I don’

Title seems somewhat misleading. He said ML often performs poorly on out of sample inputs. Seems different from being “a long way from real world use.” I don’t think anyone would argue ML is not being used in the real world!

tharkun__ 5y ago
Yes and no. Let's quote a bit more: > “All of AI, not just healthcare, has a proof-of-concept-to-production gap,” he says. “The full cycle of a machine learning project is not just modeling. It is finding the right data, deploying it, monitoring it, feeding data back [into the model], showing safety—doing all the things that need to be done [for a model] to be deployed Healthcare has some special needs in regards to what "real world use" means. Especially the "showing safety" part he mentions. That's way different from some recommendation engine application, where it doesn't really matter, whether your ML approach just creates a bunch of bad feedback loops and people get sent into rabbit holes of bad music. No lives are at stake in that sense but the recommendation engine still "performs poorly on out of sample inputs" and is so to speak, "a long way from real world use". It's just that either nobody notices or even if they do, again, no lives are at stake and so it's OK that we're getting banana software (i.e. software that ripens in the hands of customers).
- tkgally 5y ago
  An intermediate case between life-and-death healthcare and who-cares music recommendations is machine translation. Thanks to advances in AI and to companies like Google and DeepL providing translation services for free, MT is now being used widely in the real world. Sometimes it performs miraculously, enabling effective communication and cooperation among people without a common language, and sometimes it fails horribly.

4 ms·

Title seems somewhat misleading. He said ML often performs poorly on out of sample inputs. Seems different from being “a long way from real world use.” I don’

by sesuximo 5y ago

tharkun__ 5y ago
Yes and no. Let's quote a bit more: > “All of AI, not just healthcare, has a proof-of-concept-to-production gap,” he says. “The full cycle of a machine learning project is not just modeling. It is finding the right data, deploying it, monitoring it, feeding data back [into the model], showing safety—doing all the things that need to be done [for a model] to be deployed Healthcare has some special needs in regards to what "real world use" means. Especially the "showing safety" part he mentions. That's way different from some recommendation engine application, where it doesn't really matter, whether your ML approach just creates a bunch of bad feedback loops and people get sent into rabbit holes of bad music. No lives are at stake in that sense but the recommendation engine still "performs poorly on out of sample inputs" and is so to speak, "a long way from real world use". It's just that either nobody notices or even if they do, again, no lives are at stake and so it's OK that we're getting banana software (i.e. software that ripens in the hands of customers).
- tkgally 5y ago
  An intermediate case between life-and-death healthcare and who-cares music recommendations is machine translation. Thanks to advances in AI and to companies like Google and DeepL providing translation services for free, MT is now being used widely in the real world. Sometimes it performs miraculously, enabling effective communication and cooperation among people without a common language, and sometimes it fails horribly.