9 ms·

Machine learning’s crumbling foundations

208 points by Anon84 5y ago

howmayiannoyyou 5y ago
Really depends on the domain and the engine. OpenAI code generation is staggering (https://www.youtube.com/watch?v=SGUCcjHTmGY&t=1214s https://www.youtube.com/watch?v=SGUCcjHTmGY&t=1214s), its summarization and classification is still very much a work in progress.
- safog 5y ago
  I agree with this - when there's an economic incentive to get clean data, you get clean data. For instance, there's a lot of manual clean up work put into things like training data sets for speech recognition because there has been a lot of investment there. Same with self driving I assume because so much $$$ got invested there. Radiology scans or cough based COVID detectors or medical claims on the other hand? I wouldn't expect it. It's just researchers trying to get a quick paper without adequate funding.
- Orou 5y ago
  Most ML requires collecting, cleaning, and transforming datasets into something that a model can train on for a specific domain. Codex and Copilot aren't good examples of this because they are training on terabytes of public code repos - meaning that there is no code cleaning step. It's relying on the sheer volume of data that is being processed to try and filter the 'unclean' data (think buggy code written by a human) out of the model. These are really the exception rather than the rule when it comes to collecting data for ML/AI applications.
thaumasiotes 5y ago
> The disdain for the qualitative expertise of domain experts who produce data is a well-understood guilty secret within ML circles, embodied in Frederick Jelinek’s ironic talk, "Every time I fire a linguist, the performance of the speech recognizer goes up." This reminds me of how the chimp sign language studies got much better results from hearing evaluators than from deaf ones.
- pjscott 5y ago
  Doctorow seems to be missing the meaning of that quote (which is also not the title of a talk, ironic or otherwise). It was specifically a comment on the usefulness of computer language models created manually based on linguistic theories of grammar versus ones in the same model family created automatically from real-world data -- the latter tended to work better. These days I usually hear it quoted more broadly as a warning about the danger of encoding too much possibly-wrong domain knowledge in an ML system when a more generic model and the training data are sufficient to learn the useful parts on their own. Neither of those translates into disdain for qualitative understanding of the underlying reality behind the data set, which is one of those things that everyone knows is important. The problem is that such understanding is actually hard, and easy to mess up even when you're trying.
blululu 5y ago
This seems like a re-hashing of Michael Jordan's essay on the subject: https://medium.com/@mijordan3/artificial-intelligence-the-revolution-hasnt-happened-yet-5e1d5812e1e7 https://medium.com/@mijordan3/artificial-intelligence-the-re...
- dang 5y ago
  Discussed here: Artificial Intelligence – The Revolution Hasn’t Happened Yet (2018) - https://news.ycombinator.com/item?id=25530178 https://news.ycombinator.com/item?id=25530178 - Dec 2020 (120 comments) The AI Revolution Hasn’t Happened Yet - https://news.ycombinator.com/item?id=16873778 https://news.ycombinator.com/item?id=16873778 - April 2018 (161 comments)
thaumasiotes 5y ago
> Ethnic groups whose surnames were assigned in recent history for tax-collection purposes (Ashkenazi Jews, Han Chinese, Koreans, etc) have a relatively small pool of surnames and a slightly larger pool of first names. This is... not accurate. The reason the Chinese have a small pool of surnames is that their surnames are much less recent than ours, not more recent. And I don't think the Ashkenazi surnames are particularly more recent than the surnames of the Europeans they lived among. Rather, they have concentrated surnames mostly because they were occupationally concentrated.
- simonw 5y ago
  https://www.familysearch.org/wiki/en/Jewish_Personal_Names https://www.familysearch.org/wiki/en/Jewish_Personal_Names has more information about compulsory adoption of surnames amongst European Jews for taxation purposes in the 18th century.
  - taeric 5y ago
    I think the assertion was more that that is when everyone was forced to take surnames?
    - thaumasiotes 5y ago
      From https://en.wikipedia.org/wiki/Surname#History https://en.wikipedia.org/wiki/Surname#History : > By 1400, most English and some Scottish people used surnames, but many Scottish and Welsh people did not adopt surnames until the 17th century, or later. > During the modern era, many cultures around the world adopted family names, particularly for administrative reasons, especially during the age of European expansion and particularly since 1600. Notable examples include the Netherlands (1795–1811), Japan (1870s), Thailand (1920), and Turkey (1934). So that would put Ashkenazic surnames at healthily older than e.g. Dutch surnames.
- ilamont 5y ago
  Yeah, that was an odd claim about Han Chinese surnames. Many have been around for thousands of years (https://www.chinadaily.com.cn/ezine/2007-07/20/content_5441241.htm https://www.chinadaily.com.cn/ezine/2007-07/20/content_54412...) and almost all are single-character surnames based on a limited set of possible sounds (~400 in Mandarin, IIRC)
ErikAugust 5y ago
“Everyone wants to do the model work, not the data work”
- civilized 5y ago
  Which is sad because data work can lead to real domain knowledge, while fitting a grab bag of generic models teaches you nothing by itself (wooo, this thing has 0.0003 higher AUC than that thing!) Fitting generic data science predictive models is such a rote task these days that there's a crowd of start-ups begging you to pay them to automate it for you.
- mcswell 5y ago
  I don't know how it is in other fields; I'm a linguist, who made the transition to computational linguistics back when you had to be a linguist to be a computational linguist (the 1980s). Slow forward to statistical (and now neural) comp ling; I find it incredibly boring. But the data work still needs to be done, and there are still linguists. And even more than me, they find computational linguistics (of whatever type) less interesting that "real" linguistics. So they will do data work, and willingly.
- ip26 5y ago
  Sad part is while there are still people who prefer the data work in some fields, it’s not valued, since the model people have decided the data is a commodity! Result: they too have to move to model work!
simonw 5y ago
URL should be changed to https://pluralistic.net/2021/08/19/failure-cascades/ https://pluralistic.net/2021/08/19/failure-cascades/ - same content on the author's site, without having to navigate around the Medium paywall.
- dang 5y ago
  Ok, changed from https://doctorow.medium.com/machine-learnings-crumbling-foundations-bd11efa22b0 https://doctorow.medium.com/machine-learnings-crumbling-foun.... Thanks!
alecco 5y ago
https://archive.is/SoCQN https://archive.is/SoCQN
axelroze 5y ago
It's a structural issue caused by the way wealth creation works for majority of people in tech. Job hopping, trendy frameworks in CV, "high-impact" projects done ASAP, etc. No one wants to do boring, slow pace work with lots of planning, reflection and introspection. And why would they do it? These kind of jobs are usually worst paid. We, the practitioners, have every economic incentive to go the other route. The problem goes far wider in tech than just ML. And unless the society collectively learns to appreciate patience and long-term thinking, as virtues above all else, it won't go away any time soon. What can be done is to discourage use of ML systems if an explainable deterministic system can be used (even one developed in a rush). For example credit scoring. Rules are good while black box artificial neural network isn't, even if the NN has some % more accuracy. Then if the rules are not good then can be amended and in special cases customer support could also override the rules based on human (hopefully unbiased) judgement. The problem mentioned in the article of COVID-19 detection based on radiology scans is an example of a system which needs ANNs due to the nature of image processing (very difficult problem for rules AI). While techniques such as ShAP could be helpful a radiologist still needs to check because ANNs learn a lot of useless noise very often and the prediction can be nonsensical. Here it would be best to use PCR tests, serology or any more traditional and "boring" tool as it works. Luckily that is the case and shit CNN models start and end their lives in some useless paper.
- deleted 5y ago
  [deleted]
- xyzzy21 5y ago
  Sadly PCR tests for COVID also test positive for flu and half a dozen other causes. That's why CDC/FDA are seeking proposals for a new test that actually works! https://www.cdc.gov/csels/dls/locs/2021/07-21-2021-lab-alert-Changes_CDC_RT-PCR_SARS-CoV-2_Testing_1.html https://www.cdc.gov/csels/dls/locs/2021/07-21-2021-lab-alert...
  - maxerickson 5y ago
    You've fallen for the internet. Please restart and try again. https://www.reuters.com/article/factcheck-covid19-pcr-test-idUSL1N2P42U5 https://www.reuters.com/article/factcheck-covid19-pcr-test-i...
simonw 5y ago
My favourite example of bad data in for machine learning is the tragic tale of Scots Wikipedia: https://www.theguardian.com/uk-news/2020/aug/26/shock-an-aw-us-teenager-wrote-huge-slice-of-scots-wikipedia https://www.theguardian.com/uk-news/2020/aug/26/shock-an-aw-... It turned out an enthusiastic but misguided US teenager who didn't actually know the Scots language was responsible for most of the entries on it... and a bunch of natural language machine learning models had already been trained on it.
- qweqwweqwe-90i 5y ago
  Scots pretty much is a dialect of English that is phonetically spelt out - it's not surprising that a US teenager could write it.
  - m-i-l 5y ago
    >"Scots pretty much is a dialect of English that is phonetically spelt out - it's not surprising that a US teenager could write it." No, there are a number of distinct linguistic features of Scots, and it has its own regional dialects, e.g. Doric, Orcadian, Shetland (which is also in part based on the extinct Norn language). See e.g. https://dsl.ac.uk/about-scots/history-of-scots/ https://dsl.ac.uk/about-scots/history-of-scots/ (and sub-pages such as https://dsl.ac.uk/about-scots/history-of-scots/grammar/ https://dsl.ac.uk/about-scots/history-of-scots/grammar/ ) for further information. Simply doing a dictionary-lookup word-replacement completely misses all of this nuance.
- amelius 5y ago
  Reminds me of: https://en.wikipedia.org/wiki/English_as_She_Is_Spoke https://en.wikipedia.org/wiki/English_as_She_Is_Spoke
simonw 5y ago
I've seen this in a corporate setting: a machine learning model trained to automatically apply categories to new content based on user-select categories for existing content... that failed to take into account that the category list itself was poorly chosen, so the user-selected categories didn't have a particularly strong relationship to the content they were classifying.
notanzaiiswear 5y ago
It sounds like cherry picking bad examples to me. Likewise you could say "programming's foundations are crumbling" by citing all sorts of programming projects that use bad or faulty code. Meanwhile, speech recognition seems to work extremely well by now (I am a little bit older, so I remember when it didn't work so well). I am also not aware of any real world cases of AI being used to detect Corona, so that seems to be an example in favour of AI. People tried to use AI, but it didn't work out. So it isn't being used for that purpose.
- wnoise 5y ago
  > programming's foundations are crumbling That's also correct, and has been for some time (it got worse on each tech boom). This may just be a special case of that.
  - notanzaiiswear 5y ago
    What do you mean? At least from the point of view of the end user, apps seem to become better over time.
    - AlbertCory 5y ago
      I was tempted to just downvote this, but I thought I'd reply instead: No, they do not. An existing version of an app may get better over time, but unfortunately it then gets replaced with a different version, which starts from the position of extreme bugginess. In the case of Microsoft Office apps, for instance, one could easily argue that they are steadily getting worse as more and more features are added. Google Chrome is pretty clearly getting worse in terms of the amount of memory it uses. I could go on.
      - notanzaiiswear 5y ago
        So why not go back to some old version of it? I don't think "memory consumption" is necessarily a good indicator, because sometimes using more memory is a sign of good optimization. Also how is the memory consumption if you turn off all modern features?
taeric 5y ago
This hits so many of the greatest hits for how to speak to emotion and play on existing feelings more than following the data. Starts off with an appeal to so called technical debt. A nebulous concept that more plays on debt being bad then it shows anything to actually do. It then moves to comparing to other engineering, with the implicit idea that they have it together in ways that we don't. Oh, and I skipped the part of statistical abuse. Because, what? Turns out special cases abound in data driven efforts. Instead of looking for ways out, we are looking to blame those that tried? That... Doesn't seem productive. I also don't buy some of the argument. Focusing on voter purging as if that is a data science problem seems willfully ignorant. That is a blatant power grab that is just hiding behind data jargon.
exo-pla-net 5y ago
TLDR: Many ML models in production are terrible, because they were trained on terrible data. These bad models are being used in high stakes situations, such as COVID-19 detection. ML engineers need professional ethos/regulation, analogous to how civic engineers seeking to build a bridge don't screw around. My take: Yep, if the model is used a high stakes situation, this is absolutely the case. The model should be required to undergo rigorous testing / peer review before it's released into the wild. In a high stakes situations, we have to ensure that a model is good before people get their hands on it, because people can be reliably depended on to treat the model as an oracle. The metaphor of a "crumbling foundation" is a bad one, though. It's just unregulated; models aren't leaning on one another, and there isn't a risk of wholesale collapse.
data4lyfe 5y ago
The garbage in garbage out cascading failure generally seems to crash pretty fast. Given the U.S. is a capitalistic society the companies / institutions that do this and don't achieve their goals through data science should be apparent and then fail accordingly. Am I missing something here?
- wffurr 5y ago
  The trail of devastation left by this process, in financial and human terms, when medical systems go awry or vendors to state judicial systems wrongly convict innocent people.
  - mcswell 5y ago
    I agree about your latter example, but about your first example: isn't it the case that these faulty AI systems for medical diagnosis have been rejected? Doctors don't like them because they don't want to be replaced or one-upped, and because they just don't trust them (rightly so, as it turns out). So the systems, which were put out for use on a trial basis, don't get used.
unbroken 5y ago
There's a nice substack I found that is precisely about this problem and wider variations of it, that is, the problem of figuring out numbers that actually tell you something about the universe: https://desystemize.substack.com https://desystemize.substack.com
6gvONxR4sf7o 5y ago
> One common failure mode? Treating data that was known to be of poor quality as if it was reliable because good data was not available… they also use the poor quality data to assess the resulting models. This drives me nuts. Spend $10k getting high quality data and throw a simple model at it? Nah, let’s spend a month of time from someone making $400k/yr for less trustworthy results. And on the blogosphere it’s even worse. ‘This is the best data available so here goes’ justifies so much worse-than-worthless BS. And don’t even get me started on the ‘better than human’ headlines that result.
- zmmmmm 5y ago
  I feel like its even worse than just resorting to bad data when good data isn't available: the field of deep learning has cultivated the perception that it's robust to bad data as one of its hallmarks. That is, you can pump relatively raw data into it and it will self-select features and then self-regulate their use so therefore most of the initial steps of data cleaning, feature selection etc are not necessary, or require less expertise. This is now spilling over into general ML so that when quacks assert that their model just magically overcomes these things and people actually believe it.
  - nerdponx 5y ago
    The irony is that there are techniques for dealing with noisy/mislabeled/bad data (e.g. gold loss correction [0], errors-in-variables models [1]), but that stuff isn't "sexy" and not enough practitioners know about it. 0: https://arxiv.org/abs/1802.05300 https://arxiv.org/abs/1802.05300 1: https://en.m.wikipedia.org/wiki/Errors-in-variables_models https://en.m.wikipedia.org/wiki/Errors-in-variables_models
    - topranks 5y ago
      Is this any more than fancy outlier detection (genuinely asking, this is not my field). i.e. if the majority of data fed to a system is bad, will it work?
      - 5y ago
dmix 5y ago
You could use this article's underlying thesis to explain why a lot of tech companies fail as well. Google Health is a good example of failing to appreciate specialization and domain-expertise. Trying to draw value from broad generic data collection when IRL it requires vertical-focused domain-oriented collection and analysis to really draw value. Funnelling everything into a giant pool of data only had so much value - reducing it to just a proprietary API integrations platform in exchange for valuable data. This AI analogy extends to healthcare in real life: The job of any generalist doctor is largely just triaging you to the specialists. You reach the limits of their care pretty quickly for anything serious. AI is much the same way, the generic multipurpose tools tend to quickly lose value after surface level stuff before requiring heavy specialization. Google's search engine is full of custom vertical categorization, where simple Pagerank wasn't enough. This is why startups can be very useful to society as they get forced to focus on smaller issues early on, out of pure practicality, or quickly die off if they try to bite a bigger problems than they can chew. Almost every major multi-faceted business started off with a few 'whales' on which they built their business. Most of the biggest startup flops have been the ones that took VC really early before doing the dirty hard work of truly finding and understanding the problems they are trying to solve.
- iamstupidsimple 5y ago
  > Google Health is a good example of failing to appreciate specialization and domain-expertise. Trying to draw value from broad generic data collection when IRL it requires vertical-focused domain-oriented collection and analysis to really draw value. I'm not sure I agree with this statement. From what I've heard, Google Health employed a huge team of doctors and they were included through the entire feature development lifecycle, similar to how the product org functions in other software companies.
  - dmix 5y ago
    Hiring a broad set of domain experts != a domain/vertical focused business. ‘Doctors’ can cover a massive disparate field of study. My point is they did it backwards, they should have found real world healthcare problems to solve then built the common ground between them. Building a generic API platform or cloud database turned out to not be the problem anyone needed help solving. Most companies who did the integration to Health did it for marketing, not because it was essential to any business value. How many companies have done “AI” merely for marketing too? Google search ranked websites better than anyone, they zeroed in on that one problem and removed all the cruft, while Yahoo and others were jamming as much crap into their ‘portals’ as possible. Google seemed to have forgot that lesson. Waymo fell for this too. They built an entirely new type of car and gambled on a whole new taxi service (among other promises) that would entirely disrupt transportation - as the starting point. Innovation rarely ever jumps ten steps ahead like that. They chose to solve a thousand problems at once while the rest of the world with actual delivered products are struggling to solve even assisted highway driving in high-end luxury cars.. cars people were going to buy anyway.
FridayoLeary 5y ago
This can explain why pollsters get things wrong so often.
aledalgrande 5y ago
Seems to me that the foundations are not crumbling, but there should be a way to formally determine how good is a model going to be in the wild before it is used, especially in certain industries. Which I think it's where research is focused on these days? White box models, Bayesian distributions etc.?
clircle 5y ago
I've always thought it was very sad and unfortunate that core data classes like sampling design and experimental design have fallen out of academic style.
killjoywashere 5y ago
I ask the following as someone who builds and tests models and also annotates data as a domain expert. Is labeling really undervalued by society? Or just by VCs? I mean, if society depends more on the labeler (e.g. radiologists) why should society reward people for trying to replace the radiologists, regardless of the data quality? From a societal perspective where human factors scientists tell us that we need people to actually be employed to achieve a sense of self-worth and happiness, shouldn’t we punish labelers who might otherwise only enrich the capitalists and undermine the health of the nation’s workforces, and thus the wellbeing of the nation as a whole? Did we learn nothing from the underemployed, disaffected, demoralized, suicidally depressed Trump electorate? The Trump presidency may be a hot mess from which the country may never recover, but are these not the lessons that we ostensibly learned, that were the topic of millions of gallons of ink between 2016 and 2018?
nixpulvis 5y ago
Why is prose in monospace!? This style needs to die.
skybrian 5y ago
The article isn’t very clear about when harm has been done. It’s unclear which machine learning models researched production and whether human safety was on the line, like it would be for a bridge or a driverless car. For example: > Hundreds of ML teams built models to automate covid detection, and every single one was useless or worse. That’s bad, but it doesn’t seem to mean there were hundreds that made it to production use? Drilling down, there is this bit from Technology Review [1] > That hasn’t stopped some of these tools from being rushed into clinical practice. Wynants says it isn’t clear which ones are being used or how. Hospitals will sometimes say that they are using a tool only for research purposes, which makes it hard to assess how much doctors are relying on them. “There’s a lot of secrecy,” she says. So clearly there was a lot of research that wasn’t immediately useful, but in the end it’s not clear how much reached production, whether it was critical to any health decisions, or whether people were harmed by it. [1] https://www.technologyreview.com/2021/07/30/1030329/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic/ https://www.technologyreview.com/2021/07/30/1030329/machine-...
thesz 5y ago
"Some of the data-cleaning workers are atomized pieceworkers, such as those who work for Amazon's Mechanical Turk, who lack both the context in which the data was gathered and the context for how it will be used." The knowledge of the goal makes for yet another bias. "Need to know basis" in the intelligence "community" at the same time protects someone and tries to alleviate sources' biases.
AlexandrB 5y ago
> In the early 2000s, there was a movement to produce tools and training that would let domain experts produce their own tools – rather than delivering "requirements" to a programmer, a bookstore clerk or nurse or librarian could just make their own tools using Visual Basic. This is something interesting that I hadn't noticed. "RAD" tools like VB that I remember from when I was a teenager seem to have ceased to exist - replaced with either complex IDEs and languages that almost require a CS degree or with dumbed-down "glue some stuff together" automation like IFTTT/Shortcuts. The last bastion of application development for "normies" is probably Excel?
- jdlshore 5y ago
  It goes through cycles. In the 90s, it was called 4GLs ("fourth generation programming language"). Then RAD tools. (Or maybe RAD then 4GL? I don't remember.) The latest evolution of this cycle seems to be low-code/no-code.
- tonyedgecombe 5y ago
  It's interesting that tools like VB were derided by many and the practitioners looked down on. Meanwhile they were productive tools, they did live up to their name.
yandie 5y ago
This is why we've been trying to encourage people to think about lightweight data logging as a mitigation for data quality problems. Similar to how we monitor applications with Prometheus, we should approach ML monitoring with the same rigor. Disclaimer: I'm one of the authors. We spend a lot of effort to build the standard for data logging here: https://github.com/whylabs/whylogs https://github.com/whylabs/whylogs. It's meant to be a lightweight and open standard for collecting statistical signatures of your data without having to run SQL/expensive analysis.
Salgat 5y ago
The academics is rife with intentionally misleading results and findings that aren't actually innovative or informative. Worse yet, providing source code and a way to repeat the results is rare, and what good is a research paper with results that aren't repeatable? ML has a bright future, but it's very hard for me to take the academic research seriously. That's why it seems like few care about anything beyond making new models and bolstering their resume by fudging the results.
wolfium3 5y ago
I work in a company where ML has made a considerable difference to our bottom line (search component of an e-commerce site). When I joined the company, search was so bad, it was easier to just use google and include 'inurl:' to actually find products on the site. Now, years later, the builtin search actually gives you what you're looking for better than google does. (This is important because if you can't find something quickly you're more likely to shop elsewhere). If you've seen "ML done right", you wouldn't use the word "crumbling". That said - I won't deny ML is over-hyped. It works for very specific problems and in many cases the best solution is a non-ML one. Knowing when NOT to use ML is just as important as knowing when to use it.
- lloydatkinson 5y ago
  Does ML really work for you here vs elastic search or other full text search?
  - wolfium3 5y ago
    I'm not 100% familiar with the details - there's a ML component that calculates some stuff to narrow the ES search down (like categories), then at the end there's another ML system that re-ranks things. (ES alone doesn't give the best results)
EVa5I7bHFq9mnYK 5y ago
Labeled data is just much more expensive than both the computing time and model builders' time. Only a handful of rich corporations can afford to hire all the human labelers needed. Or use billions of labelers for free, as with google captcha. All the other teams are trying to do with whatever data crumbs are available.
mulcyber 5y ago
Hot take: there is no "bad" data. It's a term we often hear, that implies there is "good" and "bad" data. A dataset can have errors in labeling, be very small, be unbalanced, but all that can be managed with the proper methods. THE biggest problem is when you training data does not correspond to the production use-case. It's not that the dataset is "bad", it's just that the problem you're solving with your ML algorithm trained on that data does not correspond to the problem you're trying to solve. The most "perfect" ML algorithm trained on the most "perfect" dataset for self-driving cars for example (for detection, segmentation of objects or whatever) made the US will have problems when the cars drive in an other country. Your MNIST-trained NN will have problems in a country where numbers are written slightly differently. Some people will put pictures of cats in your car model classification software. Pictures taken on a smartphone by your users will be different than your dataset scrapped on the web. There is no bad data, just badly used data. And most of the work (and the most interesting part IMO) in ML is to identify, quantify and neutralize biases in models and differences between the data you have and the data the production system will work with.

208 points by Anon84 5y ago

Really depends on the domain and the engine. OpenAI code generation is staggering (https://www.youtube.com/watch?v=SGUCcjHTmGY&t=1214s https://www.youtube.com/watch?v=SGUCcjHTmGY&t=1214s), its summarization and classification is still very much a work in progress.

safog 5y ago
I agree with this - when there's an economic incentive to get clean data, you get clean data. For instance, there's a lot of manual clean up work put into things like training data sets for speech recognition because there has been a lot of investment there. Same with self driving I assume because so much $$$ got invested there. Radiology scans or cough based COVID detectors or medical claims on the other hand? I wouldn't expect it. It's just researchers trying to get a quick paper without adequate funding.
Orou 5y ago
Most ML requires collecting, cleaning, and transforming datasets into something that a model can train on for a specific domain. Codex and Copilot aren't good examples of this because they are training on terabytes of public code repos - meaning that there is no code cleaning step. It's relying on the sheer volume of data that is being processed to try and filter the 'unclean' data (think buggy code written by a human) out of the model. These are really the exception rather than the rule when it comes to collecting data for ML/AI applications.

thaumasiotes 5y ago

> The disdain for the qualitative expertise of domain experts who produce data is a well-understood guilty secret within ML circles, embodied in Frederick Jelinek’s ironic talk, "Every time I fire a linguist, the performance of the speech recognizer goes up." This reminds me of how the chimp sign language studies got much better results from hearing evaluators than from deaf ones.

pjscott 5y ago
Doctorow seems to be missing the meaning of that quote (which is also not the title of a talk, ironic or otherwise). It was specifically a comment on the usefulness of computer language models created manually based on linguistic theories of grammar versus ones in the same model family created automatically from real-world data -- the latter tended to work better. These days I usually hear it quoted more broadly as a warning about the danger of encoding too much possibly-wrong domain knowledge in an ML system when a more generic model and the training data are sufficient to learn the useful parts on their own. Neither of those translates into disdain for qualitative understanding of the underlying reality behind the data set, which is one of those things that everyone knows is important. The problem is that such understanding is actually hard, and easy to mess up even when you're trying.

blululu 5y ago

This seems like a re-hashing of Michael Jordan's essay on the subject: https://medium.com/@mijordan3/artificial-intelligence-the-revolution-hasnt-happened-yet-5e1d5812e1e7 https://medium.com/@mijordan3/artificial-intelligence-the-re...

dang 5y ago
Discussed here: Artificial Intelligence – The Revolution Hasn’t Happened Yet (2018) - https://news.ycombinator.com/item?id=25530178 https://news.ycombinator.com/item?id=25530178 - Dec 2020 (120 comments) The AI Revolution Hasn’t Happened Yet - https://news.ycombinator.com/item?id=16873778 https://news.ycombinator.com/item?id=16873778 - April 2018 (161 comments)

thaumasiotes 5y ago

> Ethnic groups whose surnames were assigned in recent history for tax-collection purposes (Ashkenazi Jews, Han Chinese, Koreans, etc) have a relatively small pool of surnames and a slightly larger pool of first names. This is... not accurate. The reason the Chinese have a small pool of surnames is that their surnames are much less recent than ours, not more recent. And I don't think the Ashkenazi surnames are particularly more recent than the surnames of the Europeans they lived among. Rather, they have concentrated surnames mostly because they were occupationally concentrated.

simonw 5y ago
https://www.familysearch.org/wiki/en/Jewish_Personal_Names https://www.familysearch.org/wiki/en/Jewish_Personal_Names has more information about compulsory adoption of surnames amongst European Jews for taxation purposes in the 18th century.
- taeric 5y ago
  I think the assertion was more that that is when everyone was forced to take surnames?
  - thaumasiotes 5y ago
    From https://en.wikipedia.org/wiki/Surname#History https://en.wikipedia.org/wiki/Surname#History : > By 1400, most English and some Scottish people used surnames, but many Scottish and Welsh people did not adopt surnames until the 17th century, or later. > During the modern era, many cultures around the world adopted family names, particularly for administrative reasons, especially during the age of European expansion and particularly since 1600. Notable examples include the Netherlands (1795–1811), Japan (1870s), Thailand (1920), and Turkey (1934). So that would put Ashkenazic surnames at healthily older than e.g. Dutch surnames.
ilamont 5y ago
Yeah, that was an odd claim about Han Chinese surnames. Many have been around for thousands of years (https://www.chinadaily.com.cn/ezine/2007-07/20/content_5441241.htm https://www.chinadaily.com.cn/ezine/2007-07/20/content_54412...) and almost all are single-character surnames based on a limited set of possible sounds (~400 in Mandarin, IIRC)

ErikAugust 5y ago

“Everyone wants to do the model work, not the data work”

civilized 5y ago
Which is sad because data work can lead to real domain knowledge, while fitting a grab bag of generic models teaches you nothing by itself (wooo, this thing has 0.0003 higher AUC than that thing!) Fitting generic data science predictive models is such a rote task these days that there's a crowd of start-ups begging you to pay them to automate it for you.
mcswell 5y ago
I don't know how it is in other fields; I'm a linguist, who made the transition to computational linguistics back when you had to be a linguist to be a computational linguist (the 1980s). Slow forward to statistical (and now neural) comp ling; I find it incredibly boring. But the data work still needs to be done, and there are still linguists. And even more than me, they find computational linguistics (of whatever type) less interesting that "real" linguistics. So they will do data work, and willingly.
ip26 5y ago
Sad part is while there are still people who prefer the data work in some fields, it’s not valued, since the model people have decided the data is a commodity! Result: they too have to move to model work!

simonw 5y ago

URL should be changed to https://pluralistic.net/2021/08/19/failure-cascades/ https://pluralistic.net/2021/08/19/failure-cascades/ - same content on the author's site, without having to navigate around the Medium paywall.

dang 5y ago
Ok, changed from https://doctorow.medium.com/machine-learnings-crumbling-foundations-bd11efa22b0 https://doctorow.medium.com/machine-learnings-crumbling-foun.... Thanks!

alecco 5y ago

https://archive.is/SoCQN https://archive.is/SoCQN

axelroze 5y ago

It's a structural issue caused by the way wealth creation works for majority of people in tech. Job hopping, trendy frameworks in CV, "high-impact" projects done ASAP, etc. No one wants to do boring, slow pace work with lots of planning, reflection and introspection. And why would they do it? These kind of jobs are usually worst paid. We, the practitioners, have every economic incentive to go the other route. The problem goes far wider in tech than just ML. And unless the society collectively learns to appreciate patience and long-term thinking, as virtues above all else, it won't go away any time soon. What can be done is to discourage use of ML systems if an explainable deterministic system can be used (even one developed in a rush). For example credit scoring. Rules are good while black box artificial neural network isn't, even if the NN has some % more accuracy. Then if the rules are not good then can be amended and in special cases customer support could also override the rules based on human (hopefully unbiased) judgement. The problem mentioned in the article of COVID-19 detection based on radiology scans is an example of a system which needs ANNs due to the nature of image processing (very difficult problem for rules AI). While techniques such as ShAP could be helpful a radiologist still needs to check because ANNs learn a lot of useless noise very often and the prediction can be nonsensical. Here it would be best to use PCR tests, serology or any more traditional and "boring" tool as it works. Luckily that is the case and shit CNN models start and end their lives in some useless paper.

deleted 5y ago
[deleted]
xyzzy21 5y ago
Sadly PCR tests for COVID also test positive for flu and half a dozen other causes. That's why CDC/FDA are seeking proposals for a new test that actually works! https://www.cdc.gov/csels/dls/locs/2021/07-21-2021-lab-alert-Changes_CDC_RT-PCR_SARS-CoV-2_Testing_1.html https://www.cdc.gov/csels/dls/locs/2021/07-21-2021-lab-alert...
- maxerickson 5y ago
  You've fallen for the internet. Please restart and try again. https://www.reuters.com/article/factcheck-covid19-pcr-test-idUSL1N2P42U5 https://www.reuters.com/article/factcheck-covid19-pcr-test-i...

simonw 5y ago

My favourite example of bad data in for machine learning is the tragic tale of Scots Wikipedia: https://www.theguardian.com/uk-news/2020/aug/26/shock-an-aw-us-teenager-wrote-huge-slice-of-scots-wikipedia https://www.theguardian.com/uk-news/2020/aug/26/shock-an-aw-... It turned out an enthusiastic but misguided US teenager who didn't actually know the Scots language was responsible for most of the entries on it... and a bunch of natural language machine learning models had already been trained on it.

qweqwweqwe-90i 5y ago
Scots pretty much is a dialect of English that is phonetically spelt out - it's not surprising that a US teenager could write it.
- m-i-l 5y ago
  >"Scots pretty much is a dialect of English that is phonetically spelt out - it's not surprising that a US teenager could write it." No, there are a number of distinct linguistic features of Scots, and it has its own regional dialects, e.g. Doric, Orcadian, Shetland (which is also in part based on the extinct Norn language). See e.g. https://dsl.ac.uk/about-scots/history-of-scots/ https://dsl.ac.uk/about-scots/history-of-scots/ (and sub-pages such as https://dsl.ac.uk/about-scots/history-of-scots/grammar/ https://dsl.ac.uk/about-scots/history-of-scots/grammar/ ) for further information. Simply doing a dictionary-lookup word-replacement completely misses all of this nuance.
amelius 5y ago
Reminds me of: https://en.wikipedia.org/wiki/English_as_She_Is_Spoke https://en.wikipedia.org/wiki/English_as_She_Is_Spoke

simonw 5y ago

I've seen this in a corporate setting: a machine learning model trained to automatically apply categories to new content based on user-select categories for existing content... that failed to take into account that the category list itself was poorly chosen, so the user-selected categories didn't have a particularly strong relationship to the content they were classifying.

notanzaiiswear 5y ago

It sounds like cherry picking bad examples to me. Likewise you could say "programming's foundations are crumbling" by citing all sorts of programming projects that use bad or faulty code. Meanwhile, speech recognition seems to work extremely well by now (I am a little bit older, so I remember when it didn't work so well). I am also not aware of any real world cases of AI being used to detect Corona, so that seems to be an example in favour of AI. People tried to use AI, but it didn't work out. So it isn't being used for that purpose.

wnoise 5y ago
> programming's foundations are crumbling That's also correct, and has been for some time (it got worse on each tech boom). This may just be a special case of that.
- notanzaiiswear 5y ago
  What do you mean? At least from the point of view of the end user, apps seem to become better over time.
  - AlbertCory 5y ago
    I was tempted to just downvote this, but I thought I'd reply instead: No, they do not. An existing version of an app may get better over time, but unfortunately it then gets replaced with a different version, which starts from the position of extreme bugginess. In the case of Microsoft Office apps, for instance, one could easily argue that they are steadily getting worse as more and more features are added. Google Chrome is pretty clearly getting worse in terms of the amount of memory it uses. I could go on.
    - notanzaiiswear 5y ago
      So why not go back to some old version of it? I don't think "memory consumption" is necessarily a good indicator, because sometimes using more memory is a sign of good optimization. Also how is the memory consumption if you turn off all modern features?

taeric 5y ago

This hits so many of the greatest hits for how to speak to emotion and play on existing feelings more than following the data. Starts off with an appeal to so called technical debt. A nebulous concept that more plays on debt being bad then it shows anything to actually do. It then moves to comparing to other engineering, with the implicit idea that they have it together in ways that we don't. Oh, and I skipped the part of statistical abuse. Because, what? Turns out special cases abound in data driven efforts. Instead of looking for ways out, we are looking to blame those that tried? That... Doesn't seem productive. I also don't buy some of the argument. Focusing on voter purging as if that is a data science problem seems willfully ignorant. That is a blatant power grab that is just hiding behind data jargon.

exo-pla-net 5y ago

TLDR: Many ML models in production are terrible, because they were trained on terrible data. These bad models are being used in high stakes situations, such as COVID-19 detection. ML engineers need professional ethos/regulation, analogous to how civic engineers seeking to build a bridge don't screw around. My take: Yep, if the model is used a high stakes situation, this is absolutely the case. The model should be required to undergo rigorous testing / peer review before it's released into the wild. In a high stakes situations, we have to ensure that a model is good before people get their hands on it, because people can be reliably depended on to treat the model as an oracle. The metaphor of a "crumbling foundation" is a bad one, though. It's just unregulated; models aren't leaning on one another, and there isn't a risk of wholesale collapse.

data4lyfe 5y ago

The garbage in garbage out cascading failure generally seems to crash pretty fast. Given the U.S. is a capitalistic society the companies / institutions that do this and don't achieve their goals through data science should be apparent and then fail accordingly. Am I missing something here?

wffurr 5y ago
The trail of devastation left by this process, in financial and human terms, when medical systems go awry or vendors to state judicial systems wrongly convict innocent people.
- mcswell 5y ago
  I agree about your latter example, but about your first example: isn't it the case that these faulty AI systems for medical diagnosis have been rejected? Doctors don't like them because they don't want to be replaced or one-upped, and because they just don't trust them (rightly so, as it turns out). So the systems, which were put out for use on a trial basis, don't get used.

unbroken 5y ago

There's a nice substack I found that is precisely about this problem and wider variations of it, that is, the problem of figuring out numbers that actually tell you something about the universe: https://desystemize.substack.com https://desystemize.substack.com

6gvONxR4sf7o 5y ago

> One common failure mode? Treating data that was known to be of poor quality as if it was reliable because good data was not available… they also use the poor quality data to assess the resulting models. This drives me nuts. Spend $10k getting high quality data and throw a simple model at it? Nah, let’s spend a month of time from someone making $400k/yr for less trustworthy results. And on the blogosphere it’s even worse. ‘This is the best data available so here goes’ justifies so much worse-than-worthless BS. And don’t even get me started on the ‘better than human’ headlines that result.

zmmmmm 5y ago
I feel like its even worse than just resorting to bad data when good data isn't available: the field of deep learning has cultivated the perception that it's robust to bad data as one of its hallmarks. That is, you can pump relatively raw data into it and it will self-select features and then self-regulate their use so therefore most of the initial steps of data cleaning, feature selection etc are not necessary, or require less expertise. This is now spilling over into general ML so that when quacks assert that their model just magically overcomes these things and people actually believe it.
- nerdponx 5y ago
  The irony is that there are techniques for dealing with noisy/mislabeled/bad data (e.g. gold loss correction [0], errors-in-variables models [1]), but that stuff isn't "sexy" and not enough practitioners know about it. 0: https://arxiv.org/abs/1802.05300 https://arxiv.org/abs/1802.05300 1: https://en.m.wikipedia.org/wiki/Errors-in-variables_models https://en.m.wikipedia.org/wiki/Errors-in-variables_models
  - topranks 5y ago
    Is this any more than fancy outlier detection (genuinely asking, this is not my field). i.e. if the majority of data fed to a system is bad, will it work?
    - 5y ago

dmix 5y ago

You could use this article's underlying thesis to explain why a lot of tech companies fail as well. Google Health is a good example of failing to appreciate specialization and domain-expertise. Trying to draw value from broad generic data collection when IRL it requires vertical-focused domain-oriented collection and analysis to really draw value. Funnelling everything into a giant pool of data only had so much value - reducing it to just a proprietary API integrations platform in exchange for valuable data. This AI analogy extends to healthcare in real life: The job of any generalist doctor is largely just triaging you to the specialists. You reach the limits of their care pretty quickly for anything serious. AI is much the same way, the generic multipurpose tools tend to quickly lose value after surface level stuff before requiring heavy specialization. Google's search engine is full of custom vertical categorization, where simple Pagerank wasn't enough. This is why startups can be very useful to society as they get forced to focus on smaller issues early on, out of pure practicality, or quickly die off if they try to bite a bigger problems than they can chew. Almost every major multi-faceted business started off with a few 'whales' on which they built their business. Most of the biggest startup flops have been the ones that took VC really early before doing the dirty hard work of truly finding and understanding the problems they are trying to solve.

iamstupidsimple 5y ago
> Google Health is a good example of failing to appreciate specialization and domain-expertise. Trying to draw value from broad generic data collection when IRL it requires vertical-focused domain-oriented collection and analysis to really draw value. I'm not sure I agree with this statement. From what I've heard, Google Health employed a huge team of doctors and they were included through the entire feature development lifecycle, similar to how the product org functions in other software companies.
- dmix 5y ago
  Hiring a broad set of domain experts != a domain/vertical focused business. ‘Doctors’ can cover a massive disparate field of study. My point is they did it backwards, they should have found real world healthcare problems to solve then built the common ground between them. Building a generic API platform or cloud database turned out to not be the problem anyone needed help solving. Most companies who did the integration to Health did it for marketing, not because it was essential to any business value. How many companies have done “AI” merely for marketing too? Google search ranked websites better than anyone, they zeroed in on that one problem and removed all the cruft, while Yahoo and others were jamming as much crap into their ‘portals’ as possible. Google seemed to have forgot that lesson. Waymo fell for this too. They built an entirely new type of car and gambled on a whole new taxi service (among other promises) that would entirely disrupt transportation - as the starting point. Innovation rarely ever jumps ten steps ahead like that. They chose to solve a thousand problems at once while the rest of the world with actual delivered products are struggling to solve even assisted highway driving in high-end luxury cars.. cars people were going to buy anyway.

FridayoLeary 5y ago

This can explain why pollsters get things wrong so often.

aledalgrande 5y ago

Seems to me that the foundations are not crumbling, but there should be a way to formally determine how good is a model going to be in the wild before it is used, especially in certain industries. Which I think it's where research is focused on these days? White box models, Bayesian distributions etc.?

clircle 5y ago

I've always thought it was very sad and unfortunate that core data classes like sampling design and experimental design have fallen out of academic style.

killjoywashere 5y ago

I ask the following as someone who builds and tests models and also annotates data as a domain expert. Is labeling really undervalued by society? Or just by VCs? I mean, if society depends more on the labeler (e.g. radiologists) why should society reward people for trying to replace the radiologists, regardless of the data quality? From a societal perspective where human factors scientists tell us that we need people to actually be employed to achieve a sense of self-worth and happiness, shouldn’t we punish labelers who might otherwise only enrich the capitalists and undermine the health of the nation’s workforces, and thus the wellbeing of the nation as a whole? Did we learn nothing from the underemployed, disaffected, demoralized, suicidally depressed Trump electorate? The Trump presidency may be a hot mess from which the country may never recover, but are these not the lessons that we ostensibly learned, that were the topic of millions of gallons of ink between 2016 and 2018?

nixpulvis 5y ago

Why is prose in monospace!? This style needs to die.

skybrian 5y ago

The article isn’t very clear about when harm has been done. It’s unclear which machine learning models researched production and whether human safety was on the line, like it would be for a bridge or a driverless car. For example: > Hundreds of ML teams built models to automate covid detection, and every single one was useless or worse. That’s bad, but it doesn’t seem to mean there were hundreds that made it to production use? Drilling down, there is this bit from Technology Review [1] > That hasn’t stopped some of these tools from being rushed into clinical practice. Wynants says it isn’t clear which ones are being used or how. Hospitals will sometimes say that they are using a tool only for research purposes, which makes it hard to assess how much doctors are relying on them. “There’s a lot of secrecy,” she says. So clearly there was a lot of research that wasn’t immediately useful, but in the end it’s not clear how much reached production, whether it was critical to any health decisions, or whether people were harmed by it. [1] https://www.technologyreview.com/2021/07/30/1030329/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic/ https://www.technologyreview.com/2021/07/30/1030329/machine-...

thesz 5y ago

"Some of the data-cleaning workers are atomized pieceworkers, such as those who work for Amazon's Mechanical Turk, who lack both the context in which the data was gathered and the context for how it will be used." The knowledge of the goal makes for yet another bias. "Need to know basis" in the intelligence "community" at the same time protects someone and tries to alleviate sources' biases.

AlexandrB 5y ago

> In the early 2000s, there was a movement to produce tools and training that would let domain experts produce their own tools – rather than delivering "requirements" to a programmer, a bookstore clerk or nurse or librarian could just make their own tools using Visual Basic. This is something interesting that I hadn't noticed. "RAD" tools like VB that I remember from when I was a teenager seem to have ceased to exist - replaced with either complex IDEs and languages that almost require a CS degree or with dumbed-down "glue some stuff together" automation like IFTTT/Shortcuts. The last bastion of application development for "normies" is probably Excel?

jdlshore 5y ago
It goes through cycles. In the 90s, it was called 4GLs ("fourth generation programming language"). Then RAD tools. (Or maybe RAD then 4GL? I don't remember.) The latest evolution of this cycle seems to be low-code/no-code.
tonyedgecombe 5y ago
It's interesting that tools like VB were derided by many and the practitioners looked down on. Meanwhile they were productive tools, they did live up to their name.

yandie 5y ago

This is why we've been trying to encourage people to think about lightweight data logging as a mitigation for data quality problems. Similar to how we monitor applications with Prometheus, we should approach ML monitoring with the same rigor. Disclaimer: I'm one of the authors. We spend a lot of effort to build the standard for data logging here: https://github.com/whylabs/whylogs https://github.com/whylabs/whylogs. It's meant to be a lightweight and open standard for collecting statistical signatures of your data without having to run SQL/expensive analysis.

Salgat 5y ago

The academics is rife with intentionally misleading results and findings that aren't actually innovative or informative. Worse yet, providing source code and a way to repeat the results is rare, and what good is a research paper with results that aren't repeatable? ML has a bright future, but it's very hard for me to take the academic research seriously. That's why it seems like few care about anything beyond making new models and bolstering their resume by fudging the results.

wolfium3 5y ago

I work in a company where ML has made a considerable difference to our bottom line (search component of an e-commerce site). When I joined the company, search was so bad, it was easier to just use google and include 'inurl:' to actually find products on the site. Now, years later, the builtin search actually gives you what you're looking for better than google does. (This is important because if you can't find something quickly you're more likely to shop elsewhere). If you've seen "ML done right", you wouldn't use the word "crumbling". That said - I won't deny ML is over-hyped. It works for very specific problems and in many cases the best solution is a non-ML one. Knowing when NOT to use ML is just as important as knowing when to use it.

lloydatkinson 5y ago
Does ML really work for you here vs elastic search or other full text search?
- wolfium3 5y ago
  I'm not 100% familiar with the details - there's a ML component that calculates some stuff to narrow the ES search down (like categories), then at the end there's another ML system that re-ranks things. (ES alone doesn't give the best results)

EVa5I7bHFq9mnYK 5y ago

Labeled data is just much more expensive than both the computing time and model builders' time. Only a handful of rich corporations can afford to hire all the human labelers needed. Or use billions of labelers for free, as with google captcha. All the other teams are trying to do with whatever data crumbs are available.

mulcyber 5y ago

Hot take: there is no "bad" data. It's a term we often hear, that implies there is "good" and "bad" data. A dataset can have errors in labeling, be very small, be unbalanced, but all that can be managed with the proper methods. THE biggest problem is when you training data does not correspond to the production use-case. It's not that the dataset is "bad", it's just that the problem you're solving with your ML algorithm trained on that data does not correspond to the problem you're trying to solve. The most "perfect" ML algorithm trained on the most "perfect" dataset for self-driving cars for example (for detection, segmentation of objects or whatever) made the US will have problems when the cars drive in an other country. Your MNIST-trained NN will have problems in a country where numbers are written slightly differently. Some people will put pictures of cats in your car model classification software. Pictures taken on a smartphone by your users will be different than your dataset scrapped on the web. There is no bad data, just badly used data. And most of the work (and the most interesting part IMO) in ML is to identify, quantify and neutralize biases in models and differences between the data you have and the data the production system will work with.