11 ms·
AI tools have been built to catch Covid, but none of them helped
- vitorsr 5y ago> “The models are so similar—they almost all use the same techniques with minor tweaks, the same inputs—and they all make the same mistakes,” says Wynants. “If all these people making new models instead tested models that were already available, maybe we’d have something that could really help in the clinic by now.” Sounds about right.
- briefcomment 5y agoI'm curious what people find is more likely given this evidence: - 100s of models were done incorrectly, and none even remotely correctly - there is no phenomenon to correctly model Both seem extremely unlikely. Not sure what the alternatives are.
- tablespoon 5y agoThat's a false choice. There's also: - Technology not suitable or adequate for this use case. I mean, we've been to the "AI over-promises and under-delivers" rodeo before.
- briefcomment 5y agoYou think a 100% failure rate on the hottest topic in the world right now is likely?
- tablespoon 5y ago> You think a 100% failure rate on the hottest topic in the world right now is likely? Yeah. Attempts at powered flight had a 100% failure rate in the 1800s. It's kind of absurd to think that "AI" (especially in its current incarnation) must be able to solve any "hot" problem that it's thrown at.
- briefcomment 5y agoYou think AI is so primitive that we can't make any headway into the biggest problem we currently have? That seems like a fairly fringe view. Especially on a problem that has well defined data like medical imaging.
- gumby 5y agoMedical diagnosis has been one of the core ai domains domains since the early 1960s so people have been breaking the picks at that coalface for almost 60 years. And those older systems has more intelligence in them. Todays’s “AI” is a small number of tricks aimed at large amounts of data, with an unprecedented and enormous amount of marketing added.
- iamstupidsimple 5y ago> And those older systems has more intelligence in them. Todays’s “AI” is a small number of tricks aimed at large amounts of data, with an unprecedented and enormous amount of marketing added. If you're talking about deep neural networks, I can understand this viewpoint. But generally the last decade has proven widely successful for high quality vision recognition models that just weren't available before then. And with something like transfer learning, you can take a powerful off-the-shelf model and specialize it without a huge dataset. The real barriers IME are around legal and privacy implications. There's also a strong argument about if these models creating enough value in the first place, but they can work on a technical level.
- boxercommemt 5y agoHey alexa win the war on drugs.
- tablespoon 5y ago> You think AI is so primitive that we can't make any headway into the biggest problem we currently have? More or less. It's almost certain if "AI" makes any headway at all, it will fall far short of the hype. > That seems like a fairly fringe view. Especially on a problem that has well defined data like medical imaging. I doubt it.
- RandomLensman 5y agoOh yeah. Just as AI has not made the stock market predictable. There is (maybe) progress in pockets but not in general. Systems that change strongly and react to forecasts are really difficult to handle (and also stuff happens, eg new variants). Also on the wet side, there is very little progress on something like general antiviral drugs. Some stuff is just really hard.
- deleted 5y ago[deleted]
- throwaway2048 5y ago100s of models that all make the same wrong assumptions are going to be equally worthless. You can make all the models about phlogiston you want, none of them are going to accurately explain fire.
- briefcomment 5y agoRight, so is phlogiston to fire as our current understanding of what's happening is to reality?
- new299 5y agoMy guess would be that it's hard to get a clean well labeled dataset where case and control datasets are otherwise similar. Particularly, in a pandemic the people doing the data collection are otherwise occupied. Another problem you have is when you do it right the results look worse than a biased dataset. So there's some incentive to continue working with a biased dataset.
- webel0 5y ago100s of DL groups with little previous experience in this area set out to train similar models at the outset of the pandemic. Many rushed out models. I’m not surprised that they ran into similar pitfalls.
- roenxi 5y ago> there is no phenomenon to correctly model COVID exists, it causes physical changes and affects the world outside the host. 100% it can be modelled somehow in theory. That being said, there are harsh rules against collecting data (for example, it would be extremely illegal to create a sample set by taking a random group of people and infecting half of them with COVID on purpose). So I expect actually getting training data is a huge hill to climb. If most of the models used the same training data I assume there is a reason like that. So in practice, I don't expect a model quickly.
- Barrin92 5y ago>100% it can be modelled somehow in theory. The entire premise of modelling something rests on the fact that the behaviour of what you're trying to model in the future depends on its behaviour in the past. This is not true for all systems, it's not even true for most systems. John Kay differentiates between 'resolvable uncertainty' and 'radical uncertainty'. Modelling a volcano eruption is resolvable because it's a reasonably predictable system and you can make confident stochastic claims. Complex human systems are chaotic, have 'unknown unknowns', are simply subject to chance or non-linear behaviour that completely throws any prediction off. You cannot model something that is subject to forces you cannot even know at the point of making the model, and this is true for any complex human system pretty much. Modelling Covid as if we have access to Hari Seldon's psycho-history was always a terrible idea.
- pas 5y agoWhat? We have to model human health. Find good proxies for it (non invasive scans, bloodwork, PCR, rapid antigen test, whole genome sequencing, anamnesis [medical history]) and there's ample to predict. Of course there are better and worse signals to use.
- Barrin92 5y ago>We have to model human health no, the correct behavior under uncertainty isn't modelling, it's something akin to Taleb's notion of antifragility or robustness. The correct response to a pandemic isn't some sort of Hari Seldon psycho-history which, as the article points out, is futile. The right response is building systems so responsive they can crush a pandemic before it gets to that stage.
- jltsiren 5y agoAs someone with a background in CS (but not AI/ML) and working in bioinformatics: People with a background in methods like solving problems that are easy to solve with the methods they know. Those are rarely the problems people in biology/medicine want to solve. Some common pitfalls include: * Asking the right questions. * Framing the questions correctly. * Finding computational problems that capture the essence of the questions and that can be solved efficiently. * Interpreting the results. * Avoiding systemic biases in rare but important edge cases. Finding the right problems to solve takes time, and the first attempts will probably fail.
- briefcomment 5y ago> first attempts will probably fail First hundreds of attempts? In your experience, is that a reasonable number?
- jltsiren 5y agoThe first attempts made by any particular person are likely to fail. And when many people approach similar problems from similar perspectives, they are likely to make similar mistakes. Give them a few years, and someone will probably figure out what the mistakes were and how to avoid them.
- beerandt 5y agoRight- thousands of people making the same predictable learning curve mistakes isn't thousands of man-hours of new research. It's thousands of (wo)men all duplicating the same few hours of research, which isn't likely to result in any breakthrough. If 10 people hike to Mt Everest's base camp, it's not the equivalent of one person climbing ~10 times further to the peak.
- deleted 5y ago[deleted]
- neolog 5y agoWhat field do you come from where people succeed so often at cutting-edge research?
- LeetHacks 5y agoVery early on in the pandemic one research group opensourced "COVID-Net", using the described flawed dataset. Given the fact that at that time there was still quite a lot of "if we all help we can beat the pandemic" positiveness going around. An easily run-able piece of small code that promised "results" was available and the fact that a lot of organizations wanted positive PR, you get these insultingly low quality "research" practices. I looked at a lot of the initial papers and models and most were forks of COVID-Net or used an already established model like ImageNet and retrained it with the bad dataset. Presto! Paper out PR happy. Very few questioned the dataset or the approach in general. But they loved bragging about it on LinkedIn.
- toast0 5y agoThere's almost certainly something to model, you don't hear stories about hospitals filling up with patients and nobody knows if they're covid patients or how to deal with them; or at least, not anymore. 100s of wrong models doesn't sound extremely unlikely to me. It's really easy to train a model and test that it works on some dataset and then find it doesn't work on a different one. Setting up the inputs for training is difficult because of data privacy, there's intense pressure to publish, models have no expectation of explanability, lots of reasons for a wrong model that looks promising. I've worked adjacent to people trying to solve problems with machine learning and it's rough going. When things work, ok nice, but when they don't, it's much harder to tweak than something based on heuristics (which, of course, don't always apply either).
- beerandt 5y agoThis is similar to people wanting to volunteer after disasters, but get mad when they're told they'd be more in the way than helpful. It's selfishness and narcissism masquerading as philanthropy. Instead of doing an actual useful task, they want credit for the exciting/pioneering front line work. It was the same thing with the ventilator shortage. How many people publicly declared they were "inventing" a new, easy to make ventilator? Design was never the problem, manufacturing was. But few were interested in manufacturing extant (and already FDA approved) designs.
- afurculita 5y agoYep, that's because they didn't use a blockchain. That would have solved everything.
- deleted 5y ago[deleted]
- cjohnson318 5y agoNot sure why this was downvoted.
- mrlanderson 5y agobcuz a lot of hacker news readers have no sense of humor or sarcasm detection
- mrlanderson 5y agolol fuck most of you people are so useless
- tablespoon 5y ago> Yep, that's because they didn't use a blockchain. That would have solved everything. I just bought the COVID-19 NFT and burned it. Pandemic over.
- ffffg 5y agoAll you covid faggots are just that - fucking fascist faggots. You all suck more dick than the entire Castro and Folsom district combined. I hope you all get your throats slit. FUCK YOU COMMIES!
- new299 5y ago> Many unwittingly used a data set that contained chest scans of children who did not have covid as their examples of what non-covid cases looked like. But as a result, the AIs learned to identify kids, not covid. > Driggs’s group trained its own model using a data set that contained a mix of scans taken when patients were lying down and standing up. Because patients scanned while lying down were more likely to be seriously ill, the AI learned wrongly to predict serious covid risk from a person’s position. > Errors like these seem obvious in hindsight. They can also be fixed by adjusting the models, if researchers are aware of them. I wouldn't trust any method of fixing errors in data collection. You need to redo data collection and perform a proper case-control study. You need to select your cases and controls such that the demographics look similar. In general, your goal is not to pick cases out against a background of the general population. It's to pick cases out from an at risk group. And if there's any difference in procedure used for data collection between cases and controls, your data is compromised. Researchers often suggest correcting the dataset (cropping out artifacts etc.). In my opinion this doesn't really work... artifacts (for example in image data) can often cause subtile global differences in illumination which ML will pick up on but not be obvious to someone inspecting the images. The only solution when you have such artifacts (and only one that would be acceptable to me if I was doing DD, or evaluating the research) is to redo data collection in the context of a more reasonable case-control study.
- LeetHacks 5y agoIf properly done, that approach would most likely generate a model that works really well and detects COVID for the right reasons. Put it out in the wild and it breaks down again because all radiologist would have to use the exact same method of scanning patients. Practice shows that scan parameters, position, annotations vastly differ across hospital, scanner and radiologist. A client of mine wanted me to look into this dataset and help creating a model for detecting COVID. One of the doctors they where working with wanted to be able to submit pictures they took (from their point and shoot camera) of the CT scan. Good luck putting that variation in your dataset.
- new299 5y agoIt's a first step I guess. At least if you can get the classification to work with controlled data collection you have reasonable evidence that the approach can work. If all other parameters area randomized between case and control, this is also fine. I'd guess you can also add illumination and other artifacts to the datasets to make the training robust to this. But ultimately in the extreme case (like wanting to submit images from a point and shoot camera) I suspect you'll have a hard time building a system that's robust to that... or robust enough to be used in a diagnostic context... I personally don't think I'd be comfortable working without a standard configuration used by the radiologists and a controlled protocol.
- cscscscsc 5y agoFuck All Branch Covidians. You are fucked. Your ass is dead - you don't even know it yet.
- cscscscsc 5y agoBranch covidians are a fucking cult. You know what happened last time fascism was tried? Oh yeh, we MURDERED their fucking asses.
- sg47 5y agoAI tools cannot catch covid. Only humans can.
- briefcomment 5y agoDon't forget bats.
- js8 5y agoBillions of people have been built to catch Covid. Some of them helped..
- gunshai 5y agoWhat is happening in this thread? "branch covidians"... What the hell is that?!
- ht43q3 5y agoOnly nazi cock-sucking fascists want to talk about covid. I guess everyone here wears the swastika or sucks cock?
- nradov 5y agoWhat ever happened to using AI to diagnose infections based on cough sounds? It seemed like a promising approach. https://news.mit.edu/2020/covid-19-cough-cellphone-detection-1029 https://news.mit.edu/2020/covid-19-cough-cellphone-detection...
- aaaxyz 5y agoBy the time you experience symptoms you've already been contagious for a few days. It's like trying to detect fires by looking for piles of ash.
- nradov 5y agoDid you even read the article? It's about detecting cases without clinical symptoms yet.
- bumby 5y agoI vaguely remember reading more about this elsewhere and I believe they only used COVID positive patients. It would be interesting if there are follow-up studies with mixed training data if true.
- nradov 5y agoNo the researchers tested using both positive and negative patients.
- bumby 5y agoAh, ok. I think this was the article I originally read which implies they only used 2,500 covid positive patients. But rereading it, maybe it’s just unclear reporting. “Across around 2,500 captured cough recordings of people confirmed to have COVID-19, the AI correctly identified 97.1 percent of them – and 100 percent of the asymptomatic cases.” https://www.sciencealert.com/ai-cough-analysis-could-detect-covid-19-even-if-you-re-asymptomatic https://www.sciencealert.com/ai-cough-analysis-could-detect-...
- dontreact 5y agoThe silly thing here is that it’s very easy to train models nowadays, so the medical AI space is very noisy. With the right data and the right application, this technology can definitely be helpful. However, it is very hard to sort out signal from noise since any reasonably competent undergrad can throw a convnet at some images and get an AUC. Actually, I have even seen a high school science fair where they did a medical computer vision application. I think this speaks more to the accessibility and ease of use of the technology that is out there than it does to the inherent limitations of the tools we have. I am reasonably confident that there is a group out there that was more careful and has something that might have been useful, but it’s just impossible for clinicians to know who to trust. I feel that both uncritical, un-nuanced hype (Radiologists will be replaced in 5 years!) and uncritical, un-nuanced closed mindedness (AI will never be clinically useful) is harmful for tackling the biggest problem the field faces right now: trust. Who to trust. When to trust.
- mattnewport 5y ago> I am reasonably confident that there is a group out there that was more careful and has something that might have been useful What is that confidence based on? It seems like the researchers here made a reasonably thorough effort to find "something that might have been useful" and only turned up two that were even worth further investigation: "She and her colleagues have looked at 232 algorithms for diagnosing patients or predicting how sick those with the disease might get. They found that none of them were fit for clinical use. Just two have been singled out as being promising enough for future testing." So are you reasonably confident that these researchers just didn't look hard enough to find the good stuff?
- JohnJamesRambo 5y agoShould have built AI tools to educate the ignorant populace and encourage them to get a vaccine. I know there have to be tons of adboys and girls out there whose only job is to persuade people to do something that could have knocked this out of the park. I see we are still at about 50% vaccination and Walmart is begging people to come get a free vaccine. I’m sorry my tone is negative but this is really getting me frustrated and making me lose hope in the future of humanity really. The stupid is winning. It’s the Asimov quote coming true. :( We’ve got to take real action against this. "Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge."
- jauer 5y ago(also frustrated) re: adboys and girls, the problem (at least in the US) is that vaccines are political, so persuading people to get them is seen as dangerously close to subverting democratic society. Same thing as taking action to counter foreign influence campaigns. see also: the leaks to project veritas about Facebook's counter vaccine hesitancy program.
- kQq9oHeAz6wLLS 5y agoWell, back when Harris and Biden both said they were hesitant to get a vaccine produced under Trump, they kinda set the stage for the dumpster fire we have today.
- selimthegrim 5y agoThe point was if he rushed it. The companies delayed their announcement and strengthened the trials so the current Trumpista favorite argument would in theory lose steam.
- throwawayboise 5y agoNo. They didn't want Trump to have that "win." It was purely political. The vaccines they are pushing now are the very same ones that were developed and tested while Trump was in office.
- wesleywt 5y agoAs a biologist who started working in ML, I see exactly the same issues in the field I am working in. I am spending most of my time fixing bad while others just use the public datasets as is. Maybe I am wasting my time because they are in the top journals and I am not.
- cweill 5y agoThe hardest part about ML is understanding the domain. I agree with Jeremy Howard on this, that it's easier to teach a domain expert to do ML than to teach an ML expert the domain. I think you are on the right path.
- 8note 5y agoDoes a PCR test count as AI? It has some S shaped function to trigger
- covidiot321 5y agoMaybe cause, they are gay fucking pieces of shit. San Francisco should sink into the ocean - bunch of commie liberal socialist cock-sucking ass-fucking pieces of dick.
- yobbo 5y agoThe tools that are inferring the family tree of mutations (for example nextstrain.org) have been helpful. That's also a form of AI.
- aaccount 5y agoMost people working in the AI field are completely ignorant to the underlying mechanisms. They just hack together some code from github and feed some data in. Just because you get an output doesn't mean its even remotely accurate. Anyone who does understand the underlying mechanisms are probably still trying to clean the data sets.
- 29athrowaway 5y agoSuperspreaders do not care about compliance or COVID prevention tools. Superspreaders (antimaskers, antilockdown, antidistancing, antivaxxers) are the people keeping COVID alive.
- Leparamour 5y ago>Superspreaders (antimaskers, antilockdown, antidistancing, antivaxxers) are the people keeping COVID alive. It's not as black and white as you try to paint it. I'm not living in the US but I've encountered a sizable number of people who dropped all caution and safety measures as soon as they got vaccinated even though they still can catch the virus and spread the virus with the added bonus of favoring the formation of immune escaping mutations.
- nradov 5y agoSARS-CoV-2 is now endemic in the worldwide human population, just like several other coronaviruses. It will never go away regardless of what people do. Fortunately the vaccines can limit the death toll.
- disabled 5y agoI guess the AI caught COVID-19.
- ahD5zae7 5y agoYeah, I guess it would be cool to do an X-ray or CT scan of every patient, preferably in repeatable and controlled conditions similar to the AI training set. On the other hand there supposedly were very good results with training dogs to identify covid patients by smell alone. Which sounds faster, cheaper and probably even less error prone (judging by this article). I guess it doesn't scale well and there's not much money to be made, so there's that. I think it may be the case with AI researchers where if the only tool you have is a hammer then every problem starts looking like a nail /jk
- nradov 5y agoX-rays and CT scans involve some expense and radiation exposure. It makes no sense to subject every patient to such imaging studies. We have cheaper and safer tests to diagnose most cases of COVID-19.
- sgent 5y agoWe have good tools for diagnosing COVID-19, and have had them from almost the start. If your looking for medical AI problems, start with the 12-lead EKG which fits on one piece of 8x10 piece of paper, is 2D, and current formulaic algorithms misread about 50% of the time when compared to a paramedic or nurse. Much less a physician, cardiologist, or electrophysiologist. If this algorithm was better it would absolutely save lives and make everyday life easier for clinicians.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]