23 ms·

All models are wrong, but some are completely wrong

305 points by magoghm 7y ago

smitty1e 7y ago
One feels that, irrespective of the models, the data in the covid-19 case may be unusually bad. It may be time to add a third error category[1] I. False positive II. False negative III. Deliberately skewed off the map for propaganda reasons. [1] https://en.m.wikipedia.org/wiki/Type_I_and_type_II_errors#Type_II_error https://en.m.wikipedia.org/wiki/Type_I_and_type_II_errors#Ty...
- YZF 7y ago
  This is driving me crazy ;) the poor quality of data. You'd want health organizations around the world to be publishing every possible detail (anonymized) so that the disease can be better understood. Yet three months in, with over a million cases worldwide, we still have experts disagreeing about things like asymptomatic transmission, use of masks, droplets vs. aerosol, how much distance one should stand from another, viability on surfaces, etc. etc. Even for treatment options rather than insisting on randomized double blind trials start by using the natural experiments that are already happening. We should have the data to answer a lot of these questions (or at least draw out some probability distributions), or at least someone has it. This stuff is going to be critical in informing exit strategies.
  - magoghm 7y ago
    My experience with data is: you never get good data. Always, the closer you look at it the more problems you find in how it was collected.
  - thaumasiotes 7y ago
    > You'd want health organizations around the world to be publishing every possible detail (anonymized) If you're publishing every possible detail, then your data isn't anonymized. Anonymization consists of the removal of almost all of the details.
    - smitty1e 7y ago
      Or even if the data are reasonably tidy, they might expose more when juxtaposed with another dataset.
martingoodson 7y ago
Author here: happy to take comments or criticism
- dwohnitmok 7y ago
  This is a hard problem, mainly because of the problem of motivated reasoning as mentioned by another comment. You're requiring cooperation from multiple different parties here (scientists, journalists, policy makers, readers, etc.) and any of these parties can warp the results in any number of ways regardless of the cooperation of other parties. Climate science still hasn't solved this problem despite trying to implement what you're talking about. It's an age-old problem, if a priori you're looking for something hard enough you're bound to find it.
  - martingoodson 7y ago
    I think, at the least, if journalists get some quotes from other scientists before publishing a piece on a new model it would be a major win.
    - dwohnitmok 7y ago
      That's fair. I mean asking folks to be more circumspect in general is probably a good thing. I prefer though to look at problems through the lens of incentive structures (keeping in mind humans generally heavily time discount incentives and what incentivizes people is not always obvious! Death isn't always much of a disincentive beyond a rather short time horizon). And here I'm having a hard time seeing easy ways to tweak the incentive structure.
    - thaumasiotes 7y ago
      Often they do, but that doesn't affect the piece they write. It's not at all difficult to find people complaining "I was interviewed for this article, most of what I said was left out, and to the extent I am quoted, it's to give the impression that my beliefs are the exact opposite of what I explained to the journalist at length".
    - 7y ago
scribu 7y ago
Reading the examples, where people jumped on a single mistake to discredit an entire report points to some sad conclusions: 1. Scientific literacy is super low in the general population. 2. Motivated reasoning is rampant. People will believe anything that enables them to do what they wanted to do anyway.
- xg15 7y ago
  > 2. Motivated reasoning is rampant. This is an important point. Scientific scrutiny is extremely important, but there is still a difference between a judge that is stern but fair - and one that actively wants you to fail. Motivated reasoners have no problems holding opposing parties to impossibly high standards while accepting claims without any evidence as valid arguments for their side. Today, climate scientists have learned the lessons and improved communication and modeling considerably, even to the point where we now how "attribution science" we we can discuss climate change in the context of particular weather events. We also start seeing changes in weather patterns that are hard to ignore even for laymen. Nevertheless we are still having the same discussions as before.
  - lopmotr 7y ago
    > We also start seeing changes in weather patterns that are hard to ignore even for laymen. Hard to ignore by motivated reasoners! Changes in weather patterns observed by individuals using their own experience are not evidence of global climate change. If there's science showing that, then yes, but personal experience doesn't add any value to the conclusion, it just reinforces whatever the person already wants to believe.
- deleted 7y ago
  [deleted]
- chrisco255 7y ago
  I think the author oversimplifies the problems with climate models. They've had numerous mistakes and extremely critical ones. Not least of which is that they haven't factored for Multi-decadal changes in cloud cover albedo. The climate system is incredibly complex and our models do not have good models for all the subsystems that make it up, including the oceanic oscillations like Atlantic Multi-decadal Oscillation, PDO, etc. We can't even predict many of these subsystems with any degree of accuracy so we are hopelessly inaccurate at the higher level.
cassiet 7y ago
The headline really cuts off the points nose.
- martingoodson 7y ago
  Yeah - I was trying to play off the George Box quote but maybe it's too obscure https://en.wikipedia.org/wiki/All_models_are_wrong https://en.wikipedia.org/wiki/All_models_are_wrong
  - magoghm 7y ago
    The reference to the George Box quote in the title is what originally got my attention.
  - KerryJones 7y ago
    Interesting, I did _not_ know the reference (I'm a SWE, not in scientific community), and I thought it was a clickbait headline. Obviously this differs from the person above me ^ who clicked the link _because_ of the reference. Just perspective, and I'm very happy to know the reference now.
  - cassiet 7y ago
    It’s less the obscurity, and more that to a layman, you are saying in the title that modelling doesn't work, or it really doesn’t work. Understanding your audience and all that. Someone should write an article about it so we aren’t misrepresenting our point by trying to be clever and assume the layman understands that which has not been explained to them...
grawprog 7y ago
What I've noticed about models, or at least when people are talking about them or trying to prove a point about them, is that people forget models are a simplified version of a specific part of reality, much the same as models of airplanes or something. No matter how many variables you include, you can never capture the utterly massive and unpredictable amount of variables that exist in reality. But they're not supposed to. They're tools that are supposed to be used to help get an idea about how something might happen, that's marginally better than just guessing, due again to models always being incomplete, despite any amount of best efforts. Like any tools, using models incorrectly, fucking around with them until you just get what you want to see or using the wrong one for the wrong task can be potentially dangerous, especially when those models are used to guide large decisions with serious consequences and impacts. Reasoning and common sense need to be used alongside models and if the models don't seem to reflect reality, the problem is with the model and not reality and the model needs to be adjusted or thrown out otherwise continuing to use it will just lead to lousier and lousier decisions.
- hinkley 7y ago
  It’s funny, but when you point out that a bad analogy is actually pretty accurate if you actually know anything about the other concept, people don’t want to talk about it any more. We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hill climbing algorithm, from task management to performance tuning. My preferred strategy for optimization more closely resembles how a fruit grower would pick fruit, and the term, “low-hanging fruit” galls me because you would go bankrupt using that strategy. And probably lose most of your trees to disease. It could be a quite good analogy, if the lessons taken from it weren’t just so childishly naive.
  - edmundsauto 7y ago
    Can you elaborate on your preferred strategy?
    - 7y ago
puranjay 7y ago
I fail to understand how you can reasonably model an unprecedented event in modern history. We have no data on how people will act under a weeks, even months long lockdown. Will they stay indoors and follow guidelines? Maybe. Will they watch their livelihoods get affected, their mental health deteriorate, or get careless over time and break quarantine? Maybe. We just don't know because we just don't have the data or heck, even anecdotal evidence from similar events in the past
- ajuc 7y ago
  There's very little unprecedented events in modern history, and pandemic certainly isn't one.
  - puranjay 7y ago
    Complete lockdowns across multiple countries and cessation of all economic activity certainly is unprecedented.
    - not2b 7y ago
      We aren't ceasing all economic activity, fortunately.
- martingoodson 7y ago
  If you're interested in this specific area, here is a useful summary of what's known: https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30460-8/fulltext https://www.thelancet.com/journals/lancet/article/PIIS0140-6... tl;dr not much
- veddox 7y ago
  On the contrary, epidemics are reasonably easy to model (at least at a basic level), and modelling them has a long academic tradition. As with all modelling, it all depends on what kind of detail you want to get out of your analysis/predictions. If you‘re looking at a simple population infection model, lockdown efficacy is just a factor affecting the contact rate among uninfected actors. You run multiple scenarios for multiple levels of this factor, and see where that gets you. Or did you mean something else? (Source: I‘m not an epidemiologist, but as an ecological modeller I work with very similar tools.)
DoofusOfDeath 7y ago
Regarding the "all models are wrong" maxim... Is that statement 100% true for the low-level models that physicists use and develop? In particular, I'm curious if quantum-physics models are 100% right, just not 100% precise.
- tomrod 7y ago
  I keep this distinction in mind: - Models are deliberate simplifications of reality, in order to guide thinking and otherwise pull in only important information - Formulations (formalizations) are encapsulation of principles into a mathematical framework While there is significant overlap, the two categories do not overlap 100%. I see formalizations of physics as the latter, and we use the former to help keep our understanding of the latter clear.
  - deleted 7y ago
    [deleted]
- Jabbles 7y ago
  Some predictions of Quantum Electrodynamics have been verified to within 1 part in 100,000,000. https://en.wikipedia.org/wiki/Precision_tests_of_QED https://en.wikipedia.org/wiki/Precision_tests_of_QED However, we know it cannot be completely accurate, as it has no way of explaining gravity.
- deleted 7y ago
  [deleted]
- carapace 7y ago
  The Standard Model has no known contradictions in the real world. https://en.wikipedia.org/wiki/Standard_Model https://en.wikipedia.org/wiki/Standard_Model In "QED" Feynman states that the predictions are as precise as being able to measure the distance between (points in) New York and Los Angeles accurately to within the width of a human hair. https://en.wikipedia.org/wiki/QED:_The_Strange_Theory_of_Light_and_Matter https://en.wikipedia.org/wiki/QED:_The_Strange_Theory_of_Lig...
paulsutter 7y ago
"the best material model of a cat is another, or preferably the same, cat” - Norbert Weiner
- LittlePeter 7y ago
  Wiener
firefoxd 7y ago
This is a good time for schools to go back to teaching about Ignorance: https://www.nytimes.com/2015/08/24/opinion/the-case-for-teaching-ignorance.html https://www.nytimes.com/2015/08/24/opinion/the-case-for-teac...
kgwgk 7y ago
> The FT chose to run with an inflammatory headline, assuming an extreme value of ρ that most researchers consider highly implausible. The headline "Coronavirus may have infected half of UK population — Oxford study” is not really out of line with the preprint the article talks about. What's questionable is that it was a good idea to write about the preprint at all. "Importantly, the results we present here suggest the ongoing epidemics in the UK and Italy started at least a month before the first reported death and have already led to the accumulation of significant levels of herd immunity in both countries." "Our overall approach rests on the assumption that only a very small proportion of the population is at risk of hospitalisable illness. [...] Three different scenarios under which the model closely reproduces the reported death counts in the UK up to 19/03/2020 are presented in Figure 1 . [...] [In two of those scenearios] By 19/03/2020, approximately 36% (R 0 =2.25) and 40% (R 0 =2.75) of the population would have already been exposed to SARS-CoV-2. [...] [The third scenario] suggests that 68% would have been infected by 19/03/2020." The secondary headline “New epidemiological model shows vast majority of people suffer little or no illness” was much worse as that's the assumption in the model and not the result. It was changed to "New epidemiological model shows urgent need for large-scale testing" in the amended article. > Since its publication, hundreds of scientists have attacked the work, forcing the original authors to state publicly that they were not trying to make a forecast at all. "Attacked" sounds as if the criticism was unwarranted.
galaxyLogic 7y ago
Maybe the best thing about models is not the numbers they produce but the insight into how different variables affect each other, how steeply. But this information is already contained in the model itself. Therefore people should be reading the models, not their results. The models should of course be verified by comparing their results to empirical data. But that does not often exist with global things like pandemics and climtae change.
rossdavidh 7y ago
So, I basically agree with everything this article says, but it seems to miss a basic point. If journalists do what this paper suggests, they make less money. Journalists, and the newsmedia corporations and organizations that employ them, don't run with the most inflammatory headline possible as an accidental fluke of a mistake that they were too careless to catch. Even public sector newsmedia organizations use measures of how widely read their articles are, as a figure of merit to how well they are performing. Private sector newsmedia are rewarded financially in more or less direct proportion to how widely read (or at least clicked on) their articles are, not how well informed the reader is after they're done reading it (if they even do read past the headline). If there is one less to be learned from this whole Covid-19 debacle (and I'm sure there are several), it is that our entire news ecosystem, public and private, is fundamentally structured wrong for doing what is supposed to be its purpose, which is to make people better informed. It's not bad at it by mistake, it's bad at it as an inevitable consequence of its design.
- jbreckmckye 7y ago
  A small (but important) correction: generally, journalists do not write headlines; sub-editors do. Most subs have been journalists but most journalists do not become subs.
  - marcus_holmes 7y ago
    I think the distinction is moot. They all work in the newsroom, and are that side of the "journalistic integrity" divide. Subs are not part of the management team, or commercially motivated.
- chris_f 7y ago
  Exactly. I stated calling systems that work this way "Bad by Design." Sometimes the design is deliberately setup that way by an individual or group with something to gain, but other times it is just because the system contains different parties with competing interests. Anytime I see a clearly inefficient or ineffective system the first question I try to figure out is whether it works that way by design. In many cases once I learn more about it, the answer is yes.
IBCNU 7y ago
All models are wrong, some are deadly.
cousin_it 7y ago
> Journalists must get quotes from other experts before publishing No, this isn't enough. This whole way of thinking isn't enough. It's a big part of the reason for the current situation. Journalists should report what's true, not what Tom, Dick or Harry said. If a journalist isn't qualified to make object-level claims on a given topic, don't write on that topic. For example, if Bob says there's a forest fire, then instead of publishing "Bob says there's a forest fire", you must do enough legwork to tell your readers "There's a forest fire" or "There isn't". I allow myself to ignore all journalism that don't follow that guideline, and it makes me happier.
- acqq 7y ago
  > Journalists should report what is true, not what Tom, Dick or Harry said. Especially they should not give 33% plausibility to Tom, 33% to Dick and 33% to Harry. Which is what they typically do, and call that "professional journalism." If Tom represents 95% of scientists and Dick and Harry the fringe 5% also financed by (let say) tobacco industry or oil corporations, or Boeing, or those paid by the CIA, they should not even mention Dick and Harry in the same article (or TV a show), especially not in anything worth a major headline. They should appear with the smallest possible note in some smallest possible corner and with the title like "oil corporations paid these persons to support them again." Sadly, but that sounds like a dream. The world would be very different then.
  - thu2111 7y ago
    What you describe here is a deeply problematic fallacy and in fact what journalists already do. They are very, very far from your hypothetical "33% to Tom, 33% to Dick, 33% to Harry" scenario and it's one of the factors that undermines trust in journalism. What actually happens is this: 1. Journalists have a story they want to write. They probably already decided what the message is going to be, but let's be generous and assume they didn't. 2. They consult a rolodex of, almost exclusively, government funded academics. This is true even if the story they're writing is about activity or science that takes place only in the private sector and for which the academics in question have no actual experience. They may consult two or three grant funded academics if they want their story to seem especially robust. If they give the private sector a chance to reply at all (often not) they will cite one or two sentences of a brief phone call in which the person being talked to doesn't know what they're trying to defend themselves against and is probably just confused. It's not a real interview with an actual in-house expert. 3. This story is then published as "Experts say, ..." even if what the chosen experts say flatly contradicts common sense or things that can be checked with 10 minutes and a search engine. 4. Readers comment below the line, pointing out the flaws in the story. If a specific company is involved, they may do a blog post explaining their side of the story which the journalists will either completely ignore, or if they think they can get away with it, selectively quote one or two sentences in a misleading way. Industrial scientists/engineers are sometimes assumed to be inherently evil and untrustworthy. But the whole idea that journalists are untrusted because they very rarely quote people in industry is a saw of the left; it's not true, which is why the examples given always seem so curiously weird and out of date. Tobacco industry? when was the last time you even read a newspaper article about them? Oil corporations? Those firms that have spent the last 15 years rebranding themselves as energy companies because they now make solar panels too? There's a lot of really good analysis of the trouble journalism has got itself in for, and a significant part of the blame is laid at the feet of journalists uncritically reporting anything academics say as the Whole Truth and Nothing But The Truth, when in fact they routinely contradict each other, make up statistics, report obvious common sense as "findings", are hugely over-confident in their own predictive abilities, mis-use statistics and so on. A good book to read on the topic is "Wrong" by David Freedman: https://www.goodreads.com/book/show/8134625-wrong https://www.goodreads.com/book/show/8134625-wrong He's a former journalist and so has direct experience of this problem. He cites many examples where expert testimony caused misleading or wrong stories to be published, but IIRC none of them involved corporate scientists. Mostly academics like nutritionists, psychologists and so on.
nitrogen 7y ago
Would we encourage an epidemiologist to apply ‘fresh thinking’ to the design of an electrical substation? Yes, absolutely. If an epidemiologist identifies and models a trend in human disease around substations, or a trend in failures of substations, or a new way of modeling the ways electrical demand can change over time and influence demand in other times and places, etc., then their input should absolutely be considered. Just as it's annoying when a total outsider claims to know everything about a field, it is equally problematic when insiders refuse to acknowledge anyone on the outside.
- salty_biscuits 7y ago
  That rubbed me up the wrong way too. Applied mathematics is really quite transferable across disciplines, you just usually don't have the domain knowledge to stop you chasing dead ends or reinventing the wheel.
  - Avamander 7y ago
    Not to mention that sometimes experts in their fields miss common knowledge in other areas. Like the biologist that reinvented integration, Riemann sum - http://care.diabetesjournals.org/cgi/content/abstract/17/2/152 http://care.diabetesjournals.org/cgi/content/abstract/17/2/1... (https://fliptomato.wordpress.com/2007/03/19/medical-researcher-discovers-integration-gets-75-citations/ https://fliptomato.wordpress.com/2007/03/19/medical-research...).
- noobermin 7y ago
  The easy medium is talking to other scientists, may be getting them on as a co-author.
m0zg 7y ago
I'd like to point out that we _still_ don't know if that "high R0" model was right or not. And we won't know that until we randomly test a sufficiently large random sample of the UK population for antibodies. They imposed containment _very_ late, and they go to pubs _all the time_. It is not implausible that the majority of their population already had COVID19 without even knowing what it was. What we do know is that "doomsday style" models from IHME that just last week were predicting 50K beds needed in NY are off by a factor of 3-4, and hospitalization are starting to flatten out already. You can guess the direction they were wrong in. And before you start an uninformed argument, yes, these models assume the current isolation measures. In the meanwhile NY hoarded the ventilators and medical supplies because it anticipated this prediction to be true. To be clear, I don't blame NY - they used the best information they had, which turned out to be bullshit. Better be safe than sorry. These are not harmless errors. When this is over, someone should study these fiascos and estimate the death toll just from bad models alone. https://twitter.com/AlexBerenson/status/1246465515704463360 https://twitter.com/AlexBerenson/status/1246465515704463360
dham 7y ago
All I can think about with these models in general is the book: Jurassic Park. A lot of talk about chaos theory in the book(kind of skimmed over in the movie). I don't know how accurate all of it is, but it seems trying to model how a dinosaur park would work in the modern day could be a lot like modeling what a virus will do(although probably easier to model a Dinosaur park)
strenholme 7y ago
I am working on trying to make an accurate model for predicting COVID-19 growth based on the per-county figures we have with COVID-19 (courtesy The New York Times). To say the data is noisy would be an understatement. What I have found, so far, is that if we look at current daily growth (averaged over seven days) and use exponentiation to predict future growth based on the previous week’s figures, the numbers are too high (usually by a factor of two, but the error amount is all over the place). Point being, we’re seeing a more complicated growth model than simple exponential growth; the actual growth is lower. My work so far is on GitHub: https://github.com/samboy/covid-19-html https://github.com/samboy/covid-19-html This is a work in progress and I’m nowhere near being able to make a simple easy to read graph showing a reasonable projection of COVID-19 growth in the United States.
- justanotherc 7y ago
  It could only be exponential if the population was infinite. It has to be more of a bell curve, because as infection grows, there are fewer hosts to infect, and therefore growth would start to decline.
  - strenholme 7y ago
    Even in places where 0.05% of the population has confirmed cases of the virus, I’m seeing the curve flatten. Whether that’s from quarantine of from the virus hitting its limit, I can not say. But the curve is flattening.
    - justanotherc 7y ago
      Yes that's good news. However when you say the curve is flattening, I believe you mean we have reached peak daily new cases, meaning the number of new cases we see is going to start declining from here. Collectively many people seem to be referring to that as "curve flattening", but my understanding is that flattening the curve means slowing the growth rate overall, so that it takes us longer to reach peak daily new cases. It is not intended to indicate a particular point along the x axis. In fact if we are actually flattening the curve, it will take us LONGER to reach our peak. Also, its difficult to measure whether we have been successful or not, because the only thing we have to measure against would be hypothetical worse case scenarios.
lazyjones 7y ago
Regarding rule 2: I stumbled across https://www.sciencemediacentre.org/working-with-us/for-journalists/roundups-for-journalists/ https://www.sciencemediacentre.org/working-with-us/for-journ... which does a decent job of aggregating expert reactions to questions that pop up in the media. I have many more issues with current journalism than the author of the blog post, rooted in their "fire & forget" nature of publications (no visible revisions, no corrections, almost all currently accessible articles are too old to be useful or even correct).
golemiprague 7y ago
The author tries to make it as if the scientific models are mainly correct except from one odd mistake which is used to discredit the whole research but this is not exactly the situation. Models are a predictive tools not a scientific fact and we have to remember that. Nobody can predict how this Corona situation will pan out, different countries take different approaches and the results don't seem to have a conclusive pattern or model, no model can explain the differences between Sweden, Japan, Italy and why they exists. People can sense that the models are just not good enough and it is not because of an odd mistake, the odd mistake is just a scapegoat to a very strong and probably correct gut feeling or common sense about the models margin of error. The same thing goes for climate change, it is not about the details, it is a deep scepticism about the ability to create a model for futuristic problems of such a complicated system, let alone of the solution to the problems. If we can't even model properly something that is happening right now how can we expect to model the future? We all remember the modelling of the ozone layer issue, a modelling that in hindsight was wrong as the hole is closing now and nobody can explain why even though with the rise of China the use of the very things that are causing it like aerosols just increased over all and yet the hole is closing. So I am not saying models are useless as a tool and not that should be entirely ignored, but a bit of scepticism about the ability of scientists to predict the future is pretty healthy.
6510 7y ago
I think making medical claims is quite punishable if one doesn't have a license. This could easily be extended to include journalists and politicians as well as other areas of science (if the stakes are high enough) It would work like this: For each field there has to be a committee similar to the ones medical practitioners already have. If the behavior is not up to standards they do an inquiry and if need be the credentials are stripped. Someone with credentials may sign-off on articles written by journalists and must do so for laws made by law makers. Such articles and laws stay active for as long as the credentials are valid. If the person dies a new one has to sign off on it. If that doesn't happen the law is abolished automatically and the archived news article will have to clearly state at the top that such validation is missing as-of [say] march 2043. It may also solicit such review. We can make a convenient api that allows people with credentials to stick our their neck to approve a publication. The list of professionals endorsing the perspective must be made available from the article or law.
- cryptonector 7y ago
  > I think making medical claims is quite punishable if one doesn't have a license. Well, it's tricky. If you say "I don't believe covid-19 is a serious problem", that can't be punishable as there is a large spectrum of legitimate thinking as to its severity and what trade-offs we should want to make. Such a statement is not remotely like practicing medicine without a license, but some will argue that it is if it could help them shut up the speaker. While saying that "chloroquine phosphate is a good prophylactic for covid-19" to someone who would believe it certainly should be punishable (as attempted murder perhaps! and regardless of whether the speaker is a licensed physician!). > This could easily be extended to include journalists and politicians as well as other areas of science (if the stakes are high enough) Journalists? Eh, maybe, but government officers generally have privileges and immunities -- good luck getting them to let those go. Anyways, the rest of your comment reads like 1984. Quis custodiet ipsos custodes and all that. Regulatory capture and all that. There's no Objectivity. There's no way to set up a mechanism that yields objectively-correct results. All systems will be susceptible to collective delusions and other failures. There's no silver bullet here, and free speech should be part of the mix. Reactionary thinking is fun for the angry, but not good for society.
johndoe42377 7y ago
It is so strange - when I write exactly the same assertions I am getting downvoted into oblivion lol.
tallgiraffe 7y ago
Absolutely true. The amount of bogus science being shared is astonishing. Yet, almost no one seems to notice, care or demand better. All that’s said in the article should be, but personally I’ve lost all hope. We might be living through a generation that values TikTok And click bait more than truth, and it won’t end well, but that’s where we are heading.
carlmr 7y ago
>Would we encourage an epidemiologist to apply ‘fresh thinking’ to the design of an electrical substation? Perhaps we should treat with caution the predictions of electrical engineers about pandemic disease outbreaks. I kind of disagree with this point. The models we see are mostly statistical models. Anybody with a statistics background (mathematician, physicist, chemist, biologist, engineer, computer scientist, epidemiologist) can have enough of an understanding to make valid points. Discarding opinions based on somebody's background is an argumentative fallacy in and of itself. You should of course check if somebody is trustworthy, but epidemiologists should be scrutinized under this aspect like everybody else.
- jeegsy 7y ago
  wouldn't you need to be an SME to interpret the results properly and put them in their proper context? Not to mention what inputs, variable would actually make sense in the real world?
timkam 7y ago
An observation from inside academia: there are some (potentially many) researchers who use the opportunity to push obscure models and simulations of their research sub-subdomains, even if they have no knowledge about public health, epidemiology, or even strong past real-world success stories of the methods they are advertising. Sometimes, this happens with a "talk to the media first" approach (before subjection of their work to the scrutiny of actual domain experts or even any form of peer review), which is certainly dangerous.
jeegsy 7y ago
> Perhaps we should treat with caution the predictions of electrical engineers about pandemic disease outbreaks Or former software company CEOs for that matter

305 points by magoghm 7y ago

One feels that, irrespective of the models, the data in the covid-19 case may be unusually bad. It may be time to add a third error category[1] I. False positive II. False negative III. Deliberately skewed off the map for propaganda reasons. [1] https://en.m.wikipedia.org/wiki/Type_I_and_type_II_errors#Type_II_error https://en.m.wikipedia.org/wiki/Type_I_and_type_II_errors#Ty...

YZF 7y ago
This is driving me crazy ;) the poor quality of data. You'd want health organizations around the world to be publishing every possible detail (anonymized) so that the disease can be better understood. Yet three months in, with over a million cases worldwide, we still have experts disagreeing about things like asymptomatic transmission, use of masks, droplets vs. aerosol, how much distance one should stand from another, viability on surfaces, etc. etc. Even for treatment options rather than insisting on randomized double blind trials start by using the natural experiments that are already happening. We should have the data to answer a lot of these questions (or at least draw out some probability distributions), or at least someone has it. This stuff is going to be critical in informing exit strategies.
- magoghm 7y ago
  My experience with data is: you never get good data. Always, the closer you look at it the more problems you find in how it was collected.
- thaumasiotes 7y ago
  > You'd want health organizations around the world to be publishing every possible detail (anonymized) If you're publishing every possible detail, then your data isn't anonymized. Anonymization consists of the removal of almost all of the details.
  - smitty1e 7y ago
    Or even if the data are reasonably tidy, they might expose more when juxtaposed with another dataset.

martingoodson 7y ago

Author here: happy to take comments or criticism

dwohnitmok 7y ago
This is a hard problem, mainly because of the problem of motivated reasoning as mentioned by another comment. You're requiring cooperation from multiple different parties here (scientists, journalists, policy makers, readers, etc.) and any of these parties can warp the results in any number of ways regardless of the cooperation of other parties. Climate science still hasn't solved this problem despite trying to implement what you're talking about. It's an age-old problem, if a priori you're looking for something hard enough you're bound to find it.
- martingoodson 7y ago
  I think, at the least, if journalists get some quotes from other scientists before publishing a piece on a new model it would be a major win.
  - dwohnitmok 7y ago
    That's fair. I mean asking folks to be more circumspect in general is probably a good thing. I prefer though to look at problems through the lens of incentive structures (keeping in mind humans generally heavily time discount incentives and what incentivizes people is not always obvious! Death isn't always much of a disincentive beyond a rather short time horizon). And here I'm having a hard time seeing easy ways to tweak the incentive structure.
  - thaumasiotes 7y ago
    Often they do, but that doesn't affect the piece they write. It's not at all difficult to find people complaining "I was interviewed for this article, most of what I said was left out, and to the extent I am quoted, it's to give the impression that my beliefs are the exact opposite of what I explained to the journalist at length".
  - 7y ago

scribu 7y ago

Reading the examples, where people jumped on a single mistake to discredit an entire report points to some sad conclusions: 1. Scientific literacy is super low in the general population. 2. Motivated reasoning is rampant. People will believe anything that enables them to do what they wanted to do anyway.

xg15 7y ago
> 2. Motivated reasoning is rampant. This is an important point. Scientific scrutiny is extremely important, but there is still a difference between a judge that is stern but fair - and one that actively wants you to fail. Motivated reasoners have no problems holding opposing parties to impossibly high standards while accepting claims without any evidence as valid arguments for their side. Today, climate scientists have learned the lessons and improved communication and modeling considerably, even to the point where we now how "attribution science" we we can discuss climate change in the context of particular weather events. We also start seeing changes in weather patterns that are hard to ignore even for laymen. Nevertheless we are still having the same discussions as before.
- lopmotr 7y ago
  > We also start seeing changes in weather patterns that are hard to ignore even for laymen. Hard to ignore by motivated reasoners! Changes in weather patterns observed by individuals using their own experience are not evidence of global climate change. If there's science showing that, then yes, but personal experience doesn't add any value to the conclusion, it just reinforces whatever the person already wants to believe.
deleted 7y ago
[deleted]
chrisco255 7y ago
I think the author oversimplifies the problems with climate models. They've had numerous mistakes and extremely critical ones. Not least of which is that they haven't factored for Multi-decadal changes in cloud cover albedo. The climate system is incredibly complex and our models do not have good models for all the subsystems that make it up, including the oceanic oscillations like Atlantic Multi-decadal Oscillation, PDO, etc. We can't even predict many of these subsystems with any degree of accuracy so we are hopelessly inaccurate at the higher level.

cassiet 7y ago

The headline really cuts off the points nose.

martingoodson 7y ago
Yeah - I was trying to play off the George Box quote but maybe it's too obscure https://en.wikipedia.org/wiki/All_models_are_wrong https://en.wikipedia.org/wiki/All_models_are_wrong
- magoghm 7y ago
  The reference to the George Box quote in the title is what originally got my attention.
- KerryJones 7y ago
  Interesting, I did _not_ know the reference (I'm a SWE, not in scientific community), and I thought it was a clickbait headline. Obviously this differs from the person above me ^ who clicked the link _because_ of the reference. Just perspective, and I'm very happy to know the reference now.
- cassiet 7y ago
  It’s less the obscurity, and more that to a layman, you are saying in the title that modelling doesn't work, or it really doesn’t work. Understanding your audience and all that. Someone should write an article about it so we aren’t misrepresenting our point by trying to be clever and assume the layman understands that which has not been explained to them...

grawprog 7y ago

What I've noticed about models, or at least when people are talking about them or trying to prove a point about them, is that people forget models are a simplified version of a specific part of reality, much the same as models of airplanes or something. No matter how many variables you include, you can never capture the utterly massive and unpredictable amount of variables that exist in reality. But they're not supposed to. They're tools that are supposed to be used to help get an idea about how something might happen, that's marginally better than just guessing, due again to models always being incomplete, despite any amount of best efforts. Like any tools, using models incorrectly, fucking around with them until you just get what you want to see or using the wrong one for the wrong task can be potentially dangerous, especially when those models are used to guide large decisions with serious consequences and impacts. Reasoning and common sense need to be used alongside models and if the models don't seem to reflect reality, the problem is with the model and not reality and the model needs to be adjusted or thrown out otherwise continuing to use it will just lead to lousier and lousier decisions.

hinkley 7y ago
It’s funny, but when you point out that a bad analogy is actually pretty accurate if you actually know anything about the other concept, people don’t want to talk about it any more. We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hill climbing algorithm, from task management to performance tuning. My preferred strategy for optimization more closely resembles how a fruit grower would pick fruit, and the term, “low-hanging fruit” galls me because you would go bankrupt using that strategy. And probably lose most of your trees to disease. It could be a quite good analogy, if the lessons taken from it weren’t just so childishly naive.
- edmundsauto 7y ago
  Can you elaborate on your preferred strategy?
  - 7y ago

puranjay 7y ago

I fail to understand how you can reasonably model an unprecedented event in modern history. We have no data on how people will act under a weeks, even months long lockdown. Will they stay indoors and follow guidelines? Maybe. Will they watch their livelihoods get affected, their mental health deteriorate, or get careless over time and break quarantine? Maybe. We just don't know because we just don't have the data or heck, even anecdotal evidence from similar events in the past

ajuc 7y ago
There's very little unprecedented events in modern history, and pandemic certainly isn't one.
- puranjay 7y ago
  Complete lockdowns across multiple countries and cessation of all economic activity certainly is unprecedented.
  - not2b 7y ago
    We aren't ceasing all economic activity, fortunately.
martingoodson 7y ago
If you're interested in this specific area, here is a useful summary of what's known: https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30460-8/fulltext https://www.thelancet.com/journals/lancet/article/PIIS0140-6... tl;dr not much
veddox 7y ago
On the contrary, epidemics are reasonably easy to model (at least at a basic level), and modelling them has a long academic tradition. As with all modelling, it all depends on what kind of detail you want to get out of your analysis/predictions. If you‘re looking at a simple population infection model, lockdown efficacy is just a factor affecting the contact rate among uninfected actors. You run multiple scenarios for multiple levels of this factor, and see where that gets you. Or did you mean something else? (Source: I‘m not an epidemiologist, but as an ecological modeller I work with very similar tools.)

DoofusOfDeath 7y ago

Regarding the "all models are wrong" maxim... Is that statement 100% true for the low-level models that physicists use and develop? In particular, I'm curious if quantum-physics models are 100% right, just not 100% precise.

tomrod 7y ago
I keep this distinction in mind: - Models are deliberate simplifications of reality, in order to guide thinking and otherwise pull in only important information - Formulations (formalizations) are encapsulation of principles into a mathematical framework While there is significant overlap, the two categories do not overlap 100%. I see formalizations of physics as the latter, and we use the former to help keep our understanding of the latter clear.
- deleted 7y ago
  [deleted]
Jabbles 7y ago
Some predictions of Quantum Electrodynamics have been verified to within 1 part in 100,000,000. https://en.wikipedia.org/wiki/Precision_tests_of_QED https://en.wikipedia.org/wiki/Precision_tests_of_QED However, we know it cannot be completely accurate, as it has no way of explaining gravity.
deleted 7y ago
[deleted]
carapace 7y ago
The Standard Model has no known contradictions in the real world. https://en.wikipedia.org/wiki/Standard_Model https://en.wikipedia.org/wiki/Standard_Model In "QED" Feynman states that the predictions are as precise as being able to measure the distance between (points in) New York and Los Angeles accurately to within the width of a human hair. https://en.wikipedia.org/wiki/QED:_The_Strange_Theory_of_Light_and_Matter https://en.wikipedia.org/wiki/QED:_The_Strange_Theory_of_Lig...

paulsutter 7y ago

"the best material model of a cat is another, or preferably the same, cat” - Norbert Weiner

LittlePeter 7y ago
Wiener

firefoxd 7y ago

This is a good time for schools to go back to teaching about Ignorance: https://www.nytimes.com/2015/08/24/opinion/the-case-for-teaching-ignorance.html https://www.nytimes.com/2015/08/24/opinion/the-case-for-teac...

kgwgk 7y ago

> The FT chose to run with an inflammatory headline, assuming an extreme value of ρ that most researchers consider highly implausible. The headline "Coronavirus may have infected half of UK population — Oxford study” is not really out of line with the preprint the article talks about. What's questionable is that it was a good idea to write about the preprint at all. "Importantly, the results we present here suggest the ongoing epidemics in the UK and Italy started at least a month before the first reported death and have already led to the accumulation of significant levels of herd immunity in both countries." "Our overall approach rests on the assumption that only a very small proportion of the population is at risk of hospitalisable illness. [...] Three different scenarios under which the model closely reproduces the reported death counts in the UK up to 19/03/2020 are presented in Figure 1 . [...] [In two of those scenearios] By 19/03/2020, approximately 36% (R 0 =2.25) and 40% (R 0 =2.75) of the population would have already been exposed to SARS-CoV-2. [...] [The third scenario] suggests that 68% would have been infected by 19/03/2020." The secondary headline “New epidemiological model shows vast majority of people suffer little or no illness” was much worse as that's the assumption in the model and not the result. It was changed to "New epidemiological model shows urgent need for large-scale testing" in the amended article. > Since its publication, hundreds of scientists have attacked the work, forcing the original authors to state publicly that they were not trying to make a forecast at all. "Attacked" sounds as if the criticism was unwarranted.

galaxyLogic 7y ago

Maybe the best thing about models is not the numbers they produce but the insight into how different variables affect each other, how steeply. But this information is already contained in the model itself. Therefore people should be reading the models, not their results. The models should of course be verified by comparing their results to empirical data. But that does not often exist with global things like pandemics and climtae change.

rossdavidh 7y ago

So, I basically agree with everything this article says, but it seems to miss a basic point. If journalists do what this paper suggests, they make less money. Journalists, and the newsmedia corporations and organizations that employ them, don't run with the most inflammatory headline possible as an accidental fluke of a mistake that they were too careless to catch. Even public sector newsmedia organizations use measures of how widely read their articles are, as a figure of merit to how well they are performing. Private sector newsmedia are rewarded financially in more or less direct proportion to how widely read (or at least clicked on) their articles are, not how well informed the reader is after they're done reading it (if they even do read past the headline). If there is one less to be learned from this whole Covid-19 debacle (and I'm sure there are several), it is that our entire news ecosystem, public and private, is fundamentally structured wrong for doing what is supposed to be its purpose, which is to make people better informed. It's not bad at it by mistake, it's bad at it as an inevitable consequence of its design.

jbreckmckye 7y ago
A small (but important) correction: generally, journalists do not write headlines; sub-editors do. Most subs have been journalists but most journalists do not become subs.
- marcus_holmes 7y ago
  I think the distinction is moot. They all work in the newsroom, and are that side of the "journalistic integrity" divide. Subs are not part of the management team, or commercially motivated.
chris_f 7y ago
Exactly. I stated calling systems that work this way "Bad by Design." Sometimes the design is deliberately setup that way by an individual or group with something to gain, but other times it is just because the system contains different parties with competing interests. Anytime I see a clearly inefficient or ineffective system the first question I try to figure out is whether it works that way by design. In many cases once I learn more about it, the answer is yes.

IBCNU 7y ago

All models are wrong, some are deadly.

cousin_it 7y ago

> Journalists must get quotes from other experts before publishing No, this isn't enough. This whole way of thinking isn't enough. It's a big part of the reason for the current situation. Journalists should report what's true, not what Tom, Dick or Harry said. If a journalist isn't qualified to make object-level claims on a given topic, don't write on that topic. For example, if Bob says there's a forest fire, then instead of publishing "Bob says there's a forest fire", you must do enough legwork to tell your readers "There's a forest fire" or "There isn't". I allow myself to ignore all journalism that don't follow that guideline, and it makes me happier.

acqq 7y ago
> Journalists should report what is true, not what Tom, Dick or Harry said. Especially they should not give 33% plausibility to Tom, 33% to Dick and 33% to Harry. Which is what they typically do, and call that "professional journalism." If Tom represents 95% of scientists and Dick and Harry the fringe 5% also financed by (let say) tobacco industry or oil corporations, or Boeing, or those paid by the CIA, they should not even mention Dick and Harry in the same article (or TV a show), especially not in anything worth a major headline. They should appear with the smallest possible note in some smallest possible corner and with the title like "oil corporations paid these persons to support them again." Sadly, but that sounds like a dream. The world would be very different then.
- thu2111 7y ago
  What you describe here is a deeply problematic fallacy and in fact what journalists already do. They are very, very far from your hypothetical "33% to Tom, 33% to Dick, 33% to Harry" scenario and it's one of the factors that undermines trust in journalism. What actually happens is this: 1. Journalists have a story they want to write. They probably already decided what the message is going to be, but let's be generous and assume they didn't. 2. They consult a rolodex of, almost exclusively, government funded academics. This is true even if the story they're writing is about activity or science that takes place only in the private sector and for which the academics in question have no actual experience. They may consult two or three grant funded academics if they want their story to seem especially robust. If they give the private sector a chance to reply at all (often not) they will cite one or two sentences of a brief phone call in which the person being talked to doesn't know what they're trying to defend themselves against and is probably just confused. It's not a real interview with an actual in-house expert. 3. This story is then published as "Experts say, ..." even if what the chosen experts say flatly contradicts common sense or things that can be checked with 10 minutes and a search engine. 4. Readers comment below the line, pointing out the flaws in the story. If a specific company is involved, they may do a blog post explaining their side of the story which the journalists will either completely ignore, or if they think they can get away with it, selectively quote one or two sentences in a misleading way. Industrial scientists/engineers are sometimes assumed to be inherently evil and untrustworthy. But the whole idea that journalists are untrusted because they very rarely quote people in industry is a saw of the left; it's not true, which is why the examples given always seem so curiously weird and out of date. Tobacco industry? when was the last time you even read a newspaper article about them? Oil corporations? Those firms that have spent the last 15 years rebranding themselves as energy companies because they now make solar panels too? There's a lot of really good analysis of the trouble journalism has got itself in for, and a significant part of the blame is laid at the feet of journalists uncritically reporting anything academics say as the Whole Truth and Nothing But The Truth, when in fact they routinely contradict each other, make up statistics, report obvious common sense as "findings", are hugely over-confident in their own predictive abilities, mis-use statistics and so on. A good book to read on the topic is "Wrong" by David Freedman: https://www.goodreads.com/book/show/8134625-wrong https://www.goodreads.com/book/show/8134625-wrong He's a former journalist and so has direct experience of this problem. He cites many examples where expert testimony caused misleading or wrong stories to be published, but IIRC none of them involved corporate scientists. Mostly academics like nutritionists, psychologists and so on.

nitrogen 7y ago

Would we encourage an epidemiologist to apply ‘fresh thinking’ to the design of an electrical substation? Yes, absolutely. If an epidemiologist identifies and models a trend in human disease around substations, or a trend in failures of substations, or a new way of modeling the ways electrical demand can change over time and influence demand in other times and places, etc., then their input should absolutely be considered. Just as it's annoying when a total outsider claims to know everything about a field, it is equally problematic when insiders refuse to acknowledge anyone on the outside.

salty_biscuits 7y ago
That rubbed me up the wrong way too. Applied mathematics is really quite transferable across disciplines, you just usually don't have the domain knowledge to stop you chasing dead ends or reinventing the wheel.
- Avamander 7y ago
  Not to mention that sometimes experts in their fields miss common knowledge in other areas. Like the biologist that reinvented integration, Riemann sum - http://care.diabetesjournals.org/cgi/content/abstract/17/2/152 http://care.diabetesjournals.org/cgi/content/abstract/17/2/1... (https://fliptomato.wordpress.com/2007/03/19/medical-researcher-discovers-integration-gets-75-citations/ https://fliptomato.wordpress.com/2007/03/19/medical-research...).
noobermin 7y ago
The easy medium is talking to other scientists, may be getting them on as a co-author.

m0zg 7y ago

I'd like to point out that we _still_ don't know if that "high R0" model was right or not. And we won't know that until we randomly test a sufficiently large random sample of the UK population for antibodies. They imposed containment _very_ late, and they go to pubs _all the time_. It is not implausible that the majority of their population already had COVID19 without even knowing what it was. What we do know is that "doomsday style" models from IHME that just last week were predicting 50K beds needed in NY are off by a factor of 3-4, and hospitalization are starting to flatten out already. You can guess the direction they were wrong in. And before you start an uninformed argument, yes, these models assume the current isolation measures. In the meanwhile NY hoarded the ventilators and medical supplies because it anticipated this prediction to be true. To be clear, I don't blame NY - they used the best information they had, which turned out to be bullshit. Better be safe than sorry. These are not harmless errors. When this is over, someone should study these fiascos and estimate the death toll just from bad models alone. https://twitter.com/AlexBerenson/status/1246465515704463360 https://twitter.com/AlexBerenson/status/1246465515704463360

dham 7y ago

All I can think about with these models in general is the book: Jurassic Park. A lot of talk about chaos theory in the book(kind of skimmed over in the movie). I don't know how accurate all of it is, but it seems trying to model how a dinosaur park would work in the modern day could be a lot like modeling what a virus will do(although probably easier to model a Dinosaur park)

strenholme 7y ago

I am working on trying to make an accurate model for predicting COVID-19 growth based on the per-county figures we have with COVID-19 (courtesy The New York Times). To say the data is noisy would be an understatement. What I have found, so far, is that if we look at current daily growth (averaged over seven days) and use exponentiation to predict future growth based on the previous week’s figures, the numbers are too high (usually by a factor of two, but the error amount is all over the place). Point being, we’re seeing a more complicated growth model than simple exponential growth; the actual growth is lower. My work so far is on GitHub: https://github.com/samboy/covid-19-html https://github.com/samboy/covid-19-html This is a work in progress and I’m nowhere near being able to make a simple easy to read graph showing a reasonable projection of COVID-19 growth in the United States.

justanotherc 7y ago
It could only be exponential if the population was infinite. It has to be more of a bell curve, because as infection grows, there are fewer hosts to infect, and therefore growth would start to decline.
- strenholme 7y ago
  Even in places where 0.05% of the population has confirmed cases of the virus, I’m seeing the curve flatten. Whether that’s from quarantine of from the virus hitting its limit, I can not say. But the curve is flattening.
  - justanotherc 7y ago
    Yes that's good news. However when you say the curve is flattening, I believe you mean we have reached peak daily new cases, meaning the number of new cases we see is going to start declining from here. Collectively many people seem to be referring to that as "curve flattening", but my understanding is that flattening the curve means slowing the growth rate overall, so that it takes us longer to reach peak daily new cases. It is not intended to indicate a particular point along the x axis. In fact if we are actually flattening the curve, it will take us LONGER to reach our peak. Also, its difficult to measure whether we have been successful or not, because the only thing we have to measure against would be hypothetical worse case scenarios.

lazyjones 7y ago

Regarding rule 2: I stumbled across https://www.sciencemediacentre.org/working-with-us/for-journalists/roundups-for-journalists/ https://www.sciencemediacentre.org/working-with-us/for-journ... which does a decent job of aggregating expert reactions to questions that pop up in the media. I have many more issues with current journalism than the author of the blog post, rooted in their "fire & forget" nature of publications (no visible revisions, no corrections, almost all currently accessible articles are too old to be useful or even correct).

golemiprague 7y ago

The author tries to make it as if the scientific models are mainly correct except from one odd mistake which is used to discredit the whole research but this is not exactly the situation. Models are a predictive tools not a scientific fact and we have to remember that. Nobody can predict how this Corona situation will pan out, different countries take different approaches and the results don't seem to have a conclusive pattern or model, no model can explain the differences between Sweden, Japan, Italy and why they exists. People can sense that the models are just not good enough and it is not because of an odd mistake, the odd mistake is just a scapegoat to a very strong and probably correct gut feeling or common sense about the models margin of error. The same thing goes for climate change, it is not about the details, it is a deep scepticism about the ability to create a model for futuristic problems of such a complicated system, let alone of the solution to the problems. If we can't even model properly something that is happening right now how can we expect to model the future? We all remember the modelling of the ozone layer issue, a modelling that in hindsight was wrong as the hole is closing now and nobody can explain why even though with the rise of China the use of the very things that are causing it like aerosols just increased over all and yet the hole is closing. So I am not saying models are useless as a tool and not that should be entirely ignored, but a bit of scepticism about the ability of scientists to predict the future is pretty healthy.

6510 7y ago

I think making medical claims is quite punishable if one doesn't have a license. This could easily be extended to include journalists and politicians as well as other areas of science (if the stakes are high enough) It would work like this: For each field there has to be a committee similar to the ones medical practitioners already have. If the behavior is not up to standards they do an inquiry and if need be the credentials are stripped. Someone with credentials may sign-off on articles written by journalists and must do so for laws made by law makers. Such articles and laws stay active for as long as the credentials are valid. If the person dies a new one has to sign off on it. If that doesn't happen the law is abolished automatically and the archived news article will have to clearly state at the top that such validation is missing as-of [say] march 2043. It may also solicit such review. We can make a convenient api that allows people with credentials to stick our their neck to approve a publication. The list of professionals endorsing the perspective must be made available from the article or law.

cryptonector 7y ago
> I think making medical claims is quite punishable if one doesn't have a license. Well, it's tricky. If you say "I don't believe covid-19 is a serious problem", that can't be punishable as there is a large spectrum of legitimate thinking as to its severity and what trade-offs we should want to make. Such a statement is not remotely like practicing medicine without a license, but some will argue that it is if it could help them shut up the speaker. While saying that "chloroquine phosphate is a good prophylactic for covid-19" to someone who would believe it certainly should be punishable (as attempted murder perhaps! and regardless of whether the speaker is a licensed physician!). > This could easily be extended to include journalists and politicians as well as other areas of science (if the stakes are high enough) Journalists? Eh, maybe, but government officers generally have privileges and immunities -- good luck getting them to let those go. Anyways, the rest of your comment reads like 1984. Quis custodiet ipsos custodes and all that. Regulatory capture and all that. There's no Objectivity. There's no way to set up a mechanism that yields objectively-correct results. All systems will be susceptible to collective delusions and other failures. There's no silver bullet here, and free speech should be part of the mix. Reactionary thinking is fun for the angry, but not good for society.

johndoe42377 7y ago

It is so strange - when I write exactly the same assertions I am getting downvoted into oblivion lol.

tallgiraffe 7y ago

Absolutely true. The amount of bogus science being shared is astonishing. Yet, almost no one seems to notice, care or demand better. All that’s said in the article should be, but personally I’ve lost all hope. We might be living through a generation that values TikTok And click bait more than truth, and it won’t end well, but that’s where we are heading.

carlmr 7y ago

>Would we encourage an epidemiologist to apply ‘fresh thinking’ to the design of an electrical substation? Perhaps we should treat with caution the predictions of electrical engineers about pandemic disease outbreaks. I kind of disagree with this point. The models we see are mostly statistical models. Anybody with a statistics background (mathematician, physicist, chemist, biologist, engineer, computer scientist, epidemiologist) can have enough of an understanding to make valid points. Discarding opinions based on somebody's background is an argumentative fallacy in and of itself. You should of course check if somebody is trustworthy, but epidemiologists should be scrutinized under this aspect like everybody else.

jeegsy 7y ago
wouldn't you need to be an SME to interpret the results properly and put them in their proper context? Not to mention what inputs, variable would actually make sense in the real world?

timkam 7y ago

An observation from inside academia: there are some (potentially many) researchers who use the opportunity to push obscure models and simulations of their research sub-subdomains, even if they have no knowledge about public health, epidemiology, or even strong past real-world success stories of the methods they are advertising. Sometimes, this happens with a "talk to the media first" approach (before subjection of their work to the scrutiny of actual domain experts or even any form of peer review), which is certainly dangerous.

jeegsy 7y ago

> Perhaps we should treat with caution the predictions of electrical engineers about pandemic disease outbreaks Or former software company CEOs for that matter