13 ms·

The Economist's excess deaths model

243 points by raldu 5y ago

physPop 5y ago
Forgive the naive question: but is code like this typical for "data science"? It seems more like something out of a masters project... Huge script files with zero testing? Theres a huge amount of data reshaping, mutating, and general wrangling going on here, how can one be confident without even a known input->output integration type test?
- ethanbond 5y ago
  In my experience, yes. Data science is highly creative/improvisational and the resulting artifacts of work reflect that. FWIW, normal science is pretty similar. The clear step 1, step 2, ..., you see in scientific publications is effectively retcon done for the benefit of the reader's understanding.
- nerdponx 5y ago
  Yes, sadly. Testing is hard. The researchers who write this code typically don't have the necessary hands-on experience to write good tests, even if they had enough time in a day/week to actually do it. Edit: Also the code tends to be very "high level", chaining lots of high-level API functions together, and even coming up with assertions to test tends to be a bit of a challenge. Testing such code turns out to be surprisingly difficult; you might end up just rewriting big chunks of your code in the test suite. In my data science work, I've focused on writing tests for the complicated sections (e.g. lower-level string processing routines) and just trying to focus on the keep the other stuff very clean and readable.
  - StavrosK 5y ago
    What would "good tests" look like for something like this?
    - physPop 5y ago
      Generate synthetic data -> run model -> check expected outputs. Yes it a lot of work, but you're reaching millions of people with this model and correctness is paramount! Similarly, even such simple test harnesses help when yourself or other go to modify the code. Having flags for "Did I break something" is very important.
      - StavrosK 5y ago
        The deliverable here isn't the code, it's the output. If "check expected outputs" is a subjective step anyway, why not just check that the output of the code when run on the data is as expected and skip the test altogether?
        stonemetal12 5y ago
        > If "check expected outputs" is a subjective step anyway It isn't. It is the evidence that calculations in the code were done correctly. It is like the problems in high school math class. The teacher controls the inputs so that the outputs are known. The teacher then runs the test problem past the student (aka the code). If the known correct answer isn't generated then we know the student didn't do it right.
        StavrosK 5y ago
        How do you tell that the calculations were done correctly? Presumably you have some way of doing that. Then, why don't you apply that way to the output? You don't need tests if you only need to do it once.
        pigeonhole123 5y ago
        Checking that the code outputs what you think it does is a pretty low bar to clear. The fact that you can think of higher bars doesn't invalidate the value of checking this.
        5y ago
- alexpetralia 5y ago
  In general I think data science code is more "research-oriented", meaning that it is not run continuously like a software app and its requirements often evolve as more information is learned from the data. Research code produces results maybe once or twice a day (manually triggered) while software apps potentially service hundreds of thousands of requests a day. Research code, because its requirements are not fixed and it is not frequently run, doesn't need to be as "stable" as a bonafide app. For a bonafide app, requirements do not change as often and the app is run virtually 24/7. Once the research code becomes "productionized" however - i.e. it is deployed in an online system where uptime & accuracy matter - then I think absolutely it becomes more engineering-heavy and looks quite a bit less like this code. Would be curious to hear others' thoughts on this distinction between research vs. production code however.
  - admissionsguy 5y ago
    Many of the "good practices" in software development are primarily meant to reduce the cognitive effort on the part of the person reading the code. I suspect it will be a downvotably unpopular opinion, but a typical researcher has a significantly larger cognitive capacity than a typical software developer. Consequently, a piece of code of certain complexity will look simpler, and be easier to manipulate for a researcher than for an average software developer.
    - patates 5y ago
      It's always easy if you're the only one writing/maintaining the code. Also, if your job is writing code, I'd bet you'd have less difficulty manipulating spaghetti code.
    - codyb 5y ago
      Huh? A researcher has greater cognitive capacity? That's a funny take. My impression was -> Software developers main focus is developing software so they spend a HUGE amount of time developing software, maintaining software, noticing patterns and bugs and pitfalls in software and thus they get pretty decent at writing software. Data scientists main focus is developing models, so they spend a HUGE amount of time developing models, tweaking models, finding data, and cleaning data, and write basic software to achieve some of those goals. I wouldn't expect Albert Einstein, Mozart, or Beyoncé to be some fantastic software developer just cause they're smart individuals. I'd expect people who spend a lot of time writing software to generally be the ones who write decent software.
- admissionsguy 5y ago
  This code is above average by the standards of academia I've dealt with. You've got comments announcing what each part of the code is doing, and there is only one copy of each script. Normally, you would expect to have multiple nested directories with various variations of the same code, plus multiple variations of each script with cryptic suffixes.
- fullshark 5y ago
  Yes, data scientists frequently are working alone and don't care about readability so much as results. Also the code isn't going into production so they don't care about optimizing it by and large.
  - checker 5y ago
    Sure, but physPop's concerns still stands - how can one be confident about the results? Is it just eyeballing "this looks right"? If so, how many anomalies are missed due to handwaving, and how many results are inaccurate? I'm genuinely curious because I understand the need to move fast but is accuracy a necessary sacrifice? (or is there a trick I don't know about)
    - fullshark 5y ago
      There's definitely a greater risk of a bug leading to misleading results. There's no real unique trick other than possibly someone else trying to replicate the results and catching an error, or trying to use the code on another data set and catching a mistake.
    - alexpetralia 5y ago
      If I understand correctly, tests are for what the code should do. There is some business logic and you are asserting that the business logic does what it should do (the test). If you manipulate the business logic, and it no longer does what the test says the code should do, the test fails. Here it is not as clear what the code should do. What should the amount of excess deaths be? In what ways would we change the logic such that the test case would break? If the input data set is static, isn't it more like a mock anyway? I think for this reason you often see more sanity checks in research code because the should case is not as clearly defined.
      - stonemetal12 5y ago
        >What should the amount of excess deaths be? With several fake known inputs and there associated outputs we should be able to determine if the calculation is right. The result on the real world data is not known but when calculating a statistic you should be able to figure out if you are calculating the right statistic or returning 42 for all inputs.
- kspacewalk2 5y ago
  As already pointed out, this isn't an engineering product. They've released the source, did some sanity checks, and moved on with their busy lives. If you find an error - cool, go ahead and report it.
  - Glavnokoman 5y ago
    By sanity check you mean shows the numbers I wanted to see? Not saying there's necessarily something wrong in this particular code but I know first hands how little effort is put into validating the scientific codes and how much of produces just random crap.
    - kspacewalk2 5y ago
      Which is why releasing their code is critical. They lay it all out for you, with their pride/reputation on the line if it produces 'just random crap'. Writing exhaustive unit tests is probably not something they are good at, nor is it a requirement for them, nor is it the best use of their time. I too know first hand that students and even academics who aren't properly trained and given the right tooling will write prototype code that's not production ready. Occasionally there's even a bona fide mistake, though usually it's just very un-generalizable. It's part of my job to make some of it more production ready. That's the best use of my time and training. Conversely, I'm shit at generating useful research ideas or writing papers. Just don't have the right combination of intuition/training/experience. Good thing my academic institution has the resources to employ them and me.
    - epistasis 5y ago
      How can you know which numbers you want to see? With tests it is often best practice to write the tests before writing the code that gets tested. But what if you don't know ahead of time, and can't possibly ever know ahead of time? Writing "tests" for this would be about adding stuff afterwards, like "the max and min values of this column were X and Y". But that is expected to break, if anything changes, because it's not testing anything useful. My question is: what is one concrete test here that would be useful and actually provide confidence that the code is doing what it should be doing, and how does that test provide better sanity checking than inspecting the tables at each step of the process?
- MattGaiser 5y ago
  For something like this, the code is more an aid to analysis rather than the product itself.
- mistrial9 5y ago
  in R particularly, there is a wide range of code quality, by developer standards, yes It should be said that the statistics themselves do make for some 'guide rails' .. the stats are quite demanding, require a lot of upper-division training to use correctly, and visual feedback can course-correct as the results are iteratively found. As said in other comments, the content is often associated with a research goal that is weighted more than code-quality. In contrast, a general purpose development language has very broad application, and probably can go wrong in very broad ways. The coder is getting feedback on results, but often lots of code review and expectation of professional results. I picked out an R ETL sequence from an article last year, and I still pull it out once in a while, to see the extensive, clever and (to my eye) really hard to read R data ingestion and manipulation. Personally I think it is a fair tradeoff to say that the expertise to use this environment is a bit of a filter, and the rigor that the results (therefore intermediate results) demand, does move the expectations on code quality .. As for tests, there is probably no defense on not having tests also.. it is probably going to be more common as the field of R and data analysis inevitably grows.
- da39a3ee 5y ago
  Yes, you can see the divide in the popularity of interactive notebooks like jupyter and observable. To programmers who use a standard git-based workflow, the interactive notebooks are anathema because you can't use version control to compare code state and monitor progress, and because of the global variables. But to many quantitative researchers for whom software version control isn't a large part of their professional worldview, it seems normal to just keep adding to the same document until you hit a state that seems to work, without much emphasis on checkpointing and navigation between checkpoints, and testing at each checkpoint.
- omginternets 5y ago
  This is pretty good by data-science (and other research) standards.
- lanevorockz 5y ago
  That is quite common for R language, it is a scripting language that you are suppose to explore as you use. So if you want to test it, it's very simple ... Just run the script until the part you are interested and then plot the hell out of it. It's widely known on the "theory of testing" that data oriented systems can't be tested without immense amount of effort. If you are interested there are plenty of academic talks about the subject.
- physPop 5y ago
  Would love some commentary on why this is an unpopular question. Good software practices don't only have to be for "production" or CI or large teams of coders. Testing for correctness could be seen as part of the work of delivering a high quality product/graph/prediction/model.
narush 5y ago
No matter the quality of the code presented here, thanks to The Economist for posting this! We all really appreciate it, even if we don't always seem like it :) Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well. So then we end up with the same code, but now no one can see it - which is the worst of all worlds. Let's support / work towards a culture of sharing our code before anything else. And if there's something you think can be improved, maybe consider opening a PR with improvements to really _show the value_ to all these parties of posting their code!
- Taek 5y ago
  Amen! When we have the code, we can all work together to improve the model, add the tests, find the bugs. And honestly, research shouldn't be considered valid if all of the code and data isn't available. Reproducibility is one of the pillars of science, and that's not possible without code and data.
- jarenmf 5y ago
  The code is not even that bad. It's a simple script generating a plot. I think it's too much to expect tests and documentation for it.
  - stephc_int13 5y ago
    I took a quick look, and I found it readable and easy to follow, contrary to 98% of the codebase I see on github.
- rich_sasha 5y ago
  > Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well. I agree that code should be published alongside papers, and great that the Economist, being a newspaper, publishes their code too. But when it is the most reputable sources (researchers, universities etc) publishing code of quality so low cannot possibly be correct (looking at you Imperial), it totally should be scrutinised and critiqued. I think the root cause is, where research is code-based (I guess 90% of science), it should pass the most basic correctness tests before a paper is accepted, or indeed turned into national policy. There are many angles to bite this apple from, but what remains unacceptable is shit code being taken at face value.
da39a3ee 5y ago
Is this estimating the right quantity? Wouldn't it be more relevant to estimate the total number of years of human life lost due to covid, rather than the number of deaths attributable to covid?
- ceejayoz 5y ago
  I’d imagine that’s a lot more difficult.
  - falcor84 5y ago
    It would be more difficult, but I don't think it has to be "a lot more difficult". The code could make a lookup in the actuarial life tables for each country to get the expected remaining years of life for a person of that age, and then just aggregate these, no?
    - klmadfejno 5y ago
      Ballpark yes, but that's likely a huge overestimate, as even normalized to age, its people with co-morbidities that have the lion's share of deaths.
      - falcor84 5y ago
        Yeah, you can of course keep approximating it. The next adjustment would then probably be to use something like quality-adjusted-life-years (QALY) based on the persons co-morbidity, and then (if you want to) your could also take it the other way and reduce QALY for survivors with long-covid.
  - codyb 5y ago
    Definitely an interesting problem though. You'd probably want to gather life expectancy rates for individuals once they reached five years of age (since infant mortality necessarily lowers life expectancy but after a cliff people who make it tend to live longer than the average with a bunch of 0s, 1s, and 2s in it) for each country in question. Then you'd need stats for the age of the dead in each country which I guess you don't really have in most cases since the deaths aren't aggregated anywhere since they haven't been attributed to COVID. Thanks to the Economist for publishing their model. Would be neat to see more stuff like this. I haven't delved too deeply into the code (several 1000 line scripts which would take some sussing out) but it's nice people will have real world projects they can delve into.
fortran77 5y ago
It would be interesting if there are fewer deaths than expected in the next few years.
- JshWright 5y ago
  If it "averages out", that still means millions of life-years lost.
- froh 5y ago
  the point with death is that once you're dead it's over. and it won't "average out" in the sense of life expectancy. it gives the life expectancy growth a dent. it will only average out in the sense that everybody alife will die some day.
  - sumtechguy 5y ago
    I wonder if they mean like the thing like what people say about DST causing deaths. But if you look on the large scale that blip does not exist. Standupmaths did a segment on it a few years ago. Basically the idea was 'yes you were going to die, and DST seemed to make it happen' but it seemed to only make it happen a few days sooner than it normally would have. Maybe that is what they are talking about? I personally know a couple of people who were marked as dying from covid. But the reality was they were going to die very soon. They both had very advanced stages of Alzheimer's.
    - froh 5y ago
      as long as it's very.clear that the age distribution of surplus mortality in times of covid is heavily drawing from other than Alzheimer's dead-soon-anyhow; as long as it's very clear that covid-19 kills otherwise healthy diabetics, overweights, heart patients, cant-afford-healthcare people, was-misguided-about-indoor-events people, multiple sclerosis patients, the list goes on, and on and on, .... as long as that is crystal clear, that "two old dead soon anyhow Alzheimer's patients" completely misrepresent the 500.000+ dead US citizens... ... such an anecdote is perfectly fine ...
    - 5y ago
lanevorockz 5y ago
Why they feel the need to fabricate numbers ? Covid Death numbers counted even people that tested negative which means that even the first numbers were already inflated. This this is just getting way out of hand.
- eckmLJE 5y ago
  The purpose of the excess death model is to measure the real number of deaths without having to argue about whether they were caused by covid infection or not. It is simply the difference between the typical number of deaths in a given time and the real number of deaths -- unless you're saying that death certificates are being fabricated altogether?
  - lanevorockz 5y ago
    You are being silly here ... We know for a matter of FACT that the death count was exaggerated in order to make sure the pandemic could be tackled. For example, people that died and had direct contact with someone with covid are AUTOMATICALLY covid deaths. I know that The Economist is not an honest source but we should still be rational to think about these. If we succumb to eternal politicisation, there is no hope for this society anymore.
    - scld 5y ago
      I haven't heard any stories about entire death certificates being fabricated.....you'd either have hundreds of living people who were considered dead by the state, or you'd have entirely fabricated people showing up as "dead".
    - bitexploder 5y ago
      You completely ignored their point. We don't care about COVID-19 in this discussion. We care about how many people were dying from any cause. Then you attack the economist as a dishonest source when they literally provided their data and made it very easy to audit and verify their claims. Join the actual conversation here, please.
splithalf 5y ago
Some muddled thinking up in this thread. Testing is good for code implementation but the risk for code such as this mostly lies in untestable aspects like the assumptions built into certain types of statistical routines and measurement/definition problems. The only answer for these issues is independent replication. Software thinking emphasizes reusing code, but science should want the opposite. Replication really matters, even if we don’t want to accept that inconvenient fact. Great work with perfect code will fail to replicate and not because there were problems with the initial work, it’s statistically certain.
evilpotatoes 5y ago
Isn't this based on a rather questionable assumption that all excess deaths in this time period were due to undiagnosed COVID ? There's been a large spike in drug overdoses, and some increases in suicides that have been seen. Would it not be reasonable to assume that a number of these deaths were caused by lockdowns, and various other heavy-handed measures authorities used to try to control outcomes ?
- thebruce87m 5y ago
  All the data on suicides I have seen so far shows that there are no significant changes to the suicide rates outside existing trends. For your other point about excess mortality being caused by lockdowns, you can view charts of excess mortality here: https://www.euromomo.eu/graphs-and-maps/ https://www.euromomo.eu/graphs-and-maps/ You can see that some countries, e.g. Norway had lockdowns but did not incur any severe excess mortality - but note that this only proves that you can have lockdowns with no excess, not that every lockdown is equal.
  - petertodd 5y ago
    "Suicide attempt admissions have increased by 100 per cent on average during the pandemic," the release says. "Admissions for substance-use disorders have increased by 200 per cent." https://ottawa.ctvnews.ca/cheo-joins-other-children-s-hospitals-in-declaring-mental-health-crisis-among-youth-as-pandemic-drags-on-1.5433469 https://ottawa.ctvnews.ca/cheo-joins-other-children-s-hospit... While that's not the same as saying actual deaths from suicide have increased, there is a serious problem, at least here in Canada. edit: Attempts up 90% in Colorado as well: https://coloradosun.com/2021/05/25/mental-health-emergency-children-teen-colorado/ https://coloradosun.com/2021/05/25/mental-health-emergency-c...
    - thebruce87m 5y ago
      Sure, I was aware of the increase in ideation and attempts but we were talking about excess deaths. I don’t like it when articles use percentages and omit the absolute numbers which are also important to give a sense of scale. Suicide is a serious issue and shouldn’t be ignored. We should also apply the same logic to suicides as people are applying to covid - would these people have tried suicide anyway but covid/lockdown just brought it forward? Will we therefore have a drop in attempts now to mirror the spike up?
dang 5y ago
Recent and related: There have been 7M-13M excess deaths worldwide during the pandemic - https://news.ycombinator.com/item?id=27177503 https://news.ycombinator.com/item?id=27177503 - May 2021 (457 comments)
jimmygrapes 5y ago
I haven't gone through the data myself, and maybe someone else already mentioned this, but I am not sure whether "excess deaths" takes into account lifespan. It has been roughly 70 to 80 years since "baby boomers" were born; based on common lifespans, wouldn't it be expected to have a "death boom" around this time?
- skybrian 5y ago
  You can estimate gradual changes like that by looking at a long-term graph. Here's a graph from before the pandemic, including a (now out of date) projection: https://www.macrotrends.net/countries/USA/united-states/death-rate https://www.macrotrends.net/countries/USA/united-states/deat... (Note that the slope looks exaggerated because the baseline isn't zero.)
- Cogito 5y ago
  The general way to do that correction is to look at actuarial tables for deaths by cohort. These models are obviously not 100% accurate, but they do take into account many factors, and are incentivised to be as accurate as possible as they determine the profitavility of life insurance. In this context, excess deaths are variations from these tables - more people from a cohort dying in a certain time period than expected.
jwilber 5y ago
Lots of comments around code quality, primarily as it relates to testing. The real code quality sin here is that R, a vectorized-by-default language, is being used to employ for- and while- loops everywhere. It’s strange to see given the absolute plethora of R resources available aimed at avoiding just that. But props to the economist for releasing the code, anyway. I’m a happy subscriber and will continue to be one.
- Redoubts 5y ago
  Well, they only started using R a year or so ago, so they can be forgiven. > Until fairly recently, we were less comfortable with statistical software (like R) that allows more sophisticated visualisations. https://medium.economist.com/mistakes-weve-drawn-a-few-8cdd8a42d368 https://medium.economist.com/mistakes-weve-drawn-a-few-8cdd8...
  - jwilber 5y ago
    That makes it even more strange, as all modern (ie within last 5 years) resources push the tidyverse.
User23 5y ago
2017 had more excess deaths than 2020[1]. The DALY loss was much greater too. What happened in 2017 that dwarfed the pandemic? [1] https://www.pnas.org/content/118/16/e2024850118 https://www.pnas.org/content/118/16/e2024850118
- ericcholis 5y ago
  Flu? That's my completely uneducated guess
- HWR_14 5y ago
  What happened in 2017 that was completely unresponded to that dwarfed the pandemic when the world shut down.
- unpolloloco 5y ago
  If I'm reading this correctly, this paper is defining excess deaths as the number of people who died in the US who wouldn't have were those people living in Europe instead of the US at that point in time? Which points to structural issues in the US system, but not to some event in 2017 that resulted in extra deaths vs. any surrounding years (just that the death rate in the US has dropped much slower than in Europe over the study timeline 2000-2017). The US death rate didn't appreciably increase (~3 per 100k) in 2017 vs. 2016[1] and has generally been improving. That's about 10k extra deaths, not 400k, like the paper says (because the paper's usage of that term is very different than The Economist's!) https://www.cdc.gov/nchs/data-visualization/mortality-trends/ https://www.cdc.gov/nchs/data-visualization/mortality-trends...
  - User23 5y ago
    > but not to some event in 2017 that resulted in extra deaths vs. any surrounding years But that's exactly what the paper shows. It's reasonable to ask the question why was the difference in deaths far greater in 2017 than in the surrounding years and why was the mean age of the persons excessively dying so relatively young? There most likely are structural differences causing excess US deaths, but the claim that those structural differences caused a huge relative spike in one year for no particular reason isn't very satisfying.
fuckyah 5y ago
Check out USmortality.com
fuckyah 5y ago
Check out https://www.USMortality.com https://www.USMortality.com

243 points by raldu 5y ago

Forgive the naive question: but is code like this typical for "data science"? It seems more like something out of a masters project... Huge script files with zero testing? Theres a huge amount of data reshaping, mutating, and general wrangling going on here, how can one be confident without even a known input->output integration type test?

ethanbond 5y ago
In my experience, yes. Data science is highly creative/improvisational and the resulting artifacts of work reflect that. FWIW, normal science is pretty similar. The clear step 1, step 2, ..., you see in scientific publications is effectively retcon done for the benefit of the reader's understanding.
nerdponx 5y ago
Yes, sadly. Testing is hard. The researchers who write this code typically don't have the necessary hands-on experience to write good tests, even if they had enough time in a day/week to actually do it. Edit: Also the code tends to be very "high level", chaining lots of high-level API functions together, and even coming up with assertions to test tends to be a bit of a challenge. Testing such code turns out to be surprisingly difficult; you might end up just rewriting big chunks of your code in the test suite. In my data science work, I've focused on writing tests for the complicated sections (e.g. lower-level string processing routines) and just trying to focus on the keep the other stuff very clean and readable.
- StavrosK 5y ago
  What would "good tests" look like for something like this?
  - physPop 5y ago
    Generate synthetic data -> run model -> check expected outputs. Yes it a lot of work, but you're reaching millions of people with this model and correctness is paramount! Similarly, even such simple test harnesses help when yourself or other go to modify the code. Having flags for "Did I break something" is very important.
    - StavrosK 5y ago
      The deliverable here isn't the code, it's the output. If "check expected outputs" is a subjective step anyway, why not just check that the output of the code when run on the data is as expected and skip the test altogether?
      - stonemetal12 5y ago
        > If "check expected outputs" is a subjective step anyway It isn't. It is the evidence that calculations in the code were done correctly. It is like the problems in high school math class. The teacher controls the inputs so that the outputs are known. The teacher then runs the test problem past the student (aka the code). If the known correct answer isn't generated then we know the student didn't do it right.
        StavrosK 5y ago
        How do you tell that the calculations were done correctly? Presumably you have some way of doing that. Then, why don't you apply that way to the output? You don't need tests if you only need to do it once.
        pigeonhole123 5y ago
        Checking that the code outputs what you think it does is a pretty low bar to clear. The fact that you can think of higher bars doesn't invalidate the value of checking this.
        5y ago
alexpetralia 5y ago
In general I think data science code is more "research-oriented", meaning that it is not run continuously like a software app and its requirements often evolve as more information is learned from the data. Research code produces results maybe once or twice a day (manually triggered) while software apps potentially service hundreds of thousands of requests a day. Research code, because its requirements are not fixed and it is not frequently run, doesn't need to be as "stable" as a bonafide app. For a bonafide app, requirements do not change as often and the app is run virtually 24/7. Once the research code becomes "productionized" however - i.e. it is deployed in an online system where uptime & accuracy matter - then I think absolutely it becomes more engineering-heavy and looks quite a bit less like this code. Would be curious to hear others' thoughts on this distinction between research vs. production code however.
- admissionsguy 5y ago
  Many of the "good practices" in software development are primarily meant to reduce the cognitive effort on the part of the person reading the code. I suspect it will be a downvotably unpopular opinion, but a typical researcher has a significantly larger cognitive capacity than a typical software developer. Consequently, a piece of code of certain complexity will look simpler, and be easier to manipulate for a researcher than for an average software developer.
  - patates 5y ago
    It's always easy if you're the only one writing/maintaining the code. Also, if your job is writing code, I'd bet you'd have less difficulty manipulating spaghetti code.
  - codyb 5y ago
    Huh? A researcher has greater cognitive capacity? That's a funny take. My impression was -> Software developers main focus is developing software so they spend a HUGE amount of time developing software, maintaining software, noticing patterns and bugs and pitfalls in software and thus they get pretty decent at writing software. Data scientists main focus is developing models, so they spend a HUGE amount of time developing models, tweaking models, finding data, and cleaning data, and write basic software to achieve some of those goals. I wouldn't expect Albert Einstein, Mozart, or Beyoncé to be some fantastic software developer just cause they're smart individuals. I'd expect people who spend a lot of time writing software to generally be the ones who write decent software.
admissionsguy 5y ago
This code is above average by the standards of academia I've dealt with. You've got comments announcing what each part of the code is doing, and there is only one copy of each script. Normally, you would expect to have multiple nested directories with various variations of the same code, plus multiple variations of each script with cryptic suffixes.
fullshark 5y ago
Yes, data scientists frequently are working alone and don't care about readability so much as results. Also the code isn't going into production so they don't care about optimizing it by and large.
- checker 5y ago
  Sure, but physPop's concerns still stands - how can one be confident about the results? Is it just eyeballing "this looks right"? If so, how many anomalies are missed due to handwaving, and how many results are inaccurate? I'm genuinely curious because I understand the need to move fast but is accuracy a necessary sacrifice? (or is there a trick I don't know about)
  - fullshark 5y ago
    There's definitely a greater risk of a bug leading to misleading results. There's no real unique trick other than possibly someone else trying to replicate the results and catching an error, or trying to use the code on another data set and catching a mistake.
  - alexpetralia 5y ago
    If I understand correctly, tests are for what the code should do. There is some business logic and you are asserting that the business logic does what it should do (the test). If you manipulate the business logic, and it no longer does what the test says the code should do, the test fails. Here it is not as clear what the code should do. What should the amount of excess deaths be? In what ways would we change the logic such that the test case would break? If the input data set is static, isn't it more like a mock anyway? I think for this reason you often see more sanity checks in research code because the should case is not as clearly defined.
    - stonemetal12 5y ago
      >What should the amount of excess deaths be? With several fake known inputs and there associated outputs we should be able to determine if the calculation is right. The result on the real world data is not known but when calculating a statistic you should be able to figure out if you are calculating the right statistic or returning 42 for all inputs.
kspacewalk2 5y ago
As already pointed out, this isn't an engineering product. They've released the source, did some sanity checks, and moved on with their busy lives. If you find an error - cool, go ahead and report it.
- Glavnokoman 5y ago
  By sanity check you mean shows the numbers I wanted to see? Not saying there's necessarily something wrong in this particular code but I know first hands how little effort is put into validating the scientific codes and how much of produces just random crap.
  - kspacewalk2 5y ago
    Which is why releasing their code is critical. They lay it all out for you, with their pride/reputation on the line if it produces 'just random crap'. Writing exhaustive unit tests is probably not something they are good at, nor is it a requirement for them, nor is it the best use of their time. I too know first hand that students and even academics who aren't properly trained and given the right tooling will write prototype code that's not production ready. Occasionally there's even a bona fide mistake, though usually it's just very un-generalizable. It's part of my job to make some of it more production ready. That's the best use of my time and training. Conversely, I'm shit at generating useful research ideas or writing papers. Just don't have the right combination of intuition/training/experience. Good thing my academic institution has the resources to employ them and me.
  - epistasis 5y ago
    How can you know which numbers you want to see? With tests it is often best practice to write the tests before writing the code that gets tested. But what if you don't know ahead of time, and can't possibly ever know ahead of time? Writing "tests" for this would be about adding stuff afterwards, like "the max and min values of this column were X and Y". But that is expected to break, if anything changes, because it's not testing anything useful. My question is: what is one concrete test here that would be useful and actually provide confidence that the code is doing what it should be doing, and how does that test provide better sanity checking than inspecting the tables at each step of the process?
MattGaiser 5y ago
For something like this, the code is more an aid to analysis rather than the product itself.
mistrial9 5y ago
in R particularly, there is a wide range of code quality, by developer standards, yes It should be said that the statistics themselves do make for some 'guide rails' .. the stats are quite demanding, require a lot of upper-division training to use correctly, and visual feedback can course-correct as the results are iteratively found. As said in other comments, the content is often associated with a research goal that is weighted more than code-quality. In contrast, a general purpose development language has very broad application, and probably can go wrong in very broad ways. The coder is getting feedback on results, but often lots of code review and expectation of professional results. I picked out an R ETL sequence from an article last year, and I still pull it out once in a while, to see the extensive, clever and (to my eye) really hard to read R data ingestion and manipulation. Personally I think it is a fair tradeoff to say that the expertise to use this environment is a bit of a filter, and the rigor that the results (therefore intermediate results) demand, does move the expectations on code quality .. As for tests, there is probably no defense on not having tests also.. it is probably going to be more common as the field of R and data analysis inevitably grows.
da39a3ee 5y ago
Yes, you can see the divide in the popularity of interactive notebooks like jupyter and observable. To programmers who use a standard git-based workflow, the interactive notebooks are anathema because you can't use version control to compare code state and monitor progress, and because of the global variables. But to many quantitative researchers for whom software version control isn't a large part of their professional worldview, it seems normal to just keep adding to the same document until you hit a state that seems to work, without much emphasis on checkpointing and navigation between checkpoints, and testing at each checkpoint.
omginternets 5y ago
This is pretty good by data-science (and other research) standards.
lanevorockz 5y ago
That is quite common for R language, it is a scripting language that you are suppose to explore as you use. So if you want to test it, it's very simple ... Just run the script until the part you are interested and then plot the hell out of it. It's widely known on the "theory of testing" that data oriented systems can't be tested without immense amount of effort. If you are interested there are plenty of academic talks about the subject.
physPop 5y ago
Would love some commentary on why this is an unpopular question. Good software practices don't only have to be for "production" or CI or large teams of coders. Testing for correctness could be seen as part of the work of delivering a high quality product/graph/prediction/model.

narush 5y ago

No matter the quality of the code presented here, thanks to The Economist for posting this! We all really appreciate it, even if we don't always seem like it :) Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well. So then we end up with the same code, but now no one can see it - which is the worst of all worlds. Let's support / work towards a culture of sharing our code before anything else. And if there's something you think can be improved, maybe consider opening a PR with improvements to really _show the value_ to all these parties of posting their code!

Taek 5y ago
Amen! When we have the code, we can all work together to improve the model, add the tests, find the bugs. And honestly, research shouldn't be considered valid if all of the code and data isn't available. Reproducibility is one of the pillars of science, and that's not possible without code and data.
jarenmf 5y ago
The code is not even that bad. It's a simple script generating a plot. I think it's too much to expect tests and documentation for it.
- stephc_int13 5y ago
  I took a quick look, and I found it readable and easy to follow, contrary to 98% of the codebase I see on github.
rich_sasha 5y ago
> Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well. I agree that code should be published alongside papers, and great that the Economist, being a newspaper, publishes their code too. But when it is the most reputable sources (researchers, universities etc) publishing code of quality so low cannot possibly be correct (looking at you Imperial), it totally should be scrutinised and critiqued. I think the root cause is, where research is code-based (I guess 90% of science), it should pass the most basic correctness tests before a paper is accepted, or indeed turned into national policy. There are many angles to bite this apple from, but what remains unacceptable is shit code being taken at face value.

da39a3ee 5y ago

Is this estimating the right quantity? Wouldn't it be more relevant to estimate the total number of years of human life lost due to covid, rather than the number of deaths attributable to covid?

ceejayoz 5y ago
I’d imagine that’s a lot more difficult.
- falcor84 5y ago
  It would be more difficult, but I don't think it has to be "a lot more difficult". The code could make a lookup in the actuarial life tables for each country to get the expected remaining years of life for a person of that age, and then just aggregate these, no?
  - klmadfejno 5y ago
    Ballpark yes, but that's likely a huge overestimate, as even normalized to age, its people with co-morbidities that have the lion's share of deaths.
    - falcor84 5y ago
      Yeah, you can of course keep approximating it. The next adjustment would then probably be to use something like quality-adjusted-life-years (QALY) based on the persons co-morbidity, and then (if you want to) your could also take it the other way and reduce QALY for survivors with long-covid.
- codyb 5y ago
  Definitely an interesting problem though. You'd probably want to gather life expectancy rates for individuals once they reached five years of age (since infant mortality necessarily lowers life expectancy but after a cliff people who make it tend to live longer than the average with a bunch of 0s, 1s, and 2s in it) for each country in question. Then you'd need stats for the age of the dead in each country which I guess you don't really have in most cases since the deaths aren't aggregated anywhere since they haven't been attributed to COVID. Thanks to the Economist for publishing their model. Would be neat to see more stuff like this. I haven't delved too deeply into the code (several 1000 line scripts which would take some sussing out) but it's nice people will have real world projects they can delve into.

fortran77 5y ago

It would be interesting if there are fewer deaths than expected in the next few years.

JshWright 5y ago
If it "averages out", that still means millions of life-years lost.
froh 5y ago
the point with death is that once you're dead it's over. and it won't "average out" in the sense of life expectancy. it gives the life expectancy growth a dent. it will only average out in the sense that everybody alife will die some day.
- sumtechguy 5y ago
  I wonder if they mean like the thing like what people say about DST causing deaths. But if you look on the large scale that blip does not exist. Standupmaths did a segment on it a few years ago. Basically the idea was 'yes you were going to die, and DST seemed to make it happen' but it seemed to only make it happen a few days sooner than it normally would have. Maybe that is what they are talking about? I personally know a couple of people who were marked as dying from covid. But the reality was they were going to die very soon. They both had very advanced stages of Alzheimer's.
  - froh 5y ago
    as long as it's very.clear that the age distribution of surplus mortality in times of covid is heavily drawing from other than Alzheimer's dead-soon-anyhow; as long as it's very clear that covid-19 kills otherwise healthy diabetics, overweights, heart patients, cant-afford-healthcare people, was-misguided-about-indoor-events people, multiple sclerosis patients, the list goes on, and on and on, .... as long as that is crystal clear, that "two old dead soon anyhow Alzheimer's patients" completely misrepresent the 500.000+ dead US citizens... ... such an anecdote is perfectly fine ...
  - 5y ago

lanevorockz 5y ago

Why they feel the need to fabricate numbers ? Covid Death numbers counted even people that tested negative which means that even the first numbers were already inflated. This this is just getting way out of hand.

eckmLJE 5y ago
The purpose of the excess death model is to measure the real number of deaths without having to argue about whether they were caused by covid infection or not. It is simply the difference between the typical number of deaths in a given time and the real number of deaths -- unless you're saying that death certificates are being fabricated altogether?
- lanevorockz 5y ago
  You are being silly here ... We know for a matter of FACT that the death count was exaggerated in order to make sure the pandemic could be tackled. For example, people that died and had direct contact with someone with covid are AUTOMATICALLY covid deaths. I know that The Economist is not an honest source but we should still be rational to think about these. If we succumb to eternal politicisation, there is no hope for this society anymore.
  - scld 5y ago
    I haven't heard any stories about entire death certificates being fabricated.....you'd either have hundreds of living people who were considered dead by the state, or you'd have entirely fabricated people showing up as "dead".
  - bitexploder 5y ago
    You completely ignored their point. We don't care about COVID-19 in this discussion. We care about how many people were dying from any cause. Then you attack the economist as a dishonest source when they literally provided their data and made it very easy to audit and verify their claims. Join the actual conversation here, please.

splithalf 5y ago

Some muddled thinking up in this thread. Testing is good for code implementation but the risk for code such as this mostly lies in untestable aspects like the assumptions built into certain types of statistical routines and measurement/definition problems. The only answer for these issues is independent replication. Software thinking emphasizes reusing code, but science should want the opposite. Replication really matters, even if we don’t want to accept that inconvenient fact. Great work with perfect code will fail to replicate and not because there were problems with the initial work, it’s statistically certain.

evilpotatoes 5y ago

Isn't this based on a rather questionable assumption that all excess deaths in this time period were due to undiagnosed COVID ? There's been a large spike in drug overdoses, and some increases in suicides that have been seen. Would it not be reasonable to assume that a number of these deaths were caused by lockdowns, and various other heavy-handed measures authorities used to try to control outcomes ?

thebruce87m 5y ago
All the data on suicides I have seen so far shows that there are no significant changes to the suicide rates outside existing trends. For your other point about excess mortality being caused by lockdowns, you can view charts of excess mortality here: https://www.euromomo.eu/graphs-and-maps/ https://www.euromomo.eu/graphs-and-maps/ You can see that some countries, e.g. Norway had lockdowns but did not incur any severe excess mortality - but note that this only proves that you can have lockdowns with no excess, not that every lockdown is equal.
- petertodd 5y ago
  "Suicide attempt admissions have increased by 100 per cent on average during the pandemic," the release says. "Admissions for substance-use disorders have increased by 200 per cent." https://ottawa.ctvnews.ca/cheo-joins-other-children-s-hospitals-in-declaring-mental-health-crisis-among-youth-as-pandemic-drags-on-1.5433469 https://ottawa.ctvnews.ca/cheo-joins-other-children-s-hospit... While that's not the same as saying actual deaths from suicide have increased, there is a serious problem, at least here in Canada. edit: Attempts up 90% in Colorado as well: https://coloradosun.com/2021/05/25/mental-health-emergency-children-teen-colorado/ https://coloradosun.com/2021/05/25/mental-health-emergency-c...
  - thebruce87m 5y ago
    Sure, I was aware of the increase in ideation and attempts but we were talking about excess deaths. I don’t like it when articles use percentages and omit the absolute numbers which are also important to give a sense of scale. Suicide is a serious issue and shouldn’t be ignored. We should also apply the same logic to suicides as people are applying to covid - would these people have tried suicide anyway but covid/lockdown just brought it forward? Will we therefore have a drop in attempts now to mirror the spike up?

dang 5y ago

Recent and related: There have been 7M-13M excess deaths worldwide during the pandemic - https://news.ycombinator.com/item?id=27177503 https://news.ycombinator.com/item?id=27177503 - May 2021 (457 comments)

jimmygrapes 5y ago

I haven't gone through the data myself, and maybe someone else already mentioned this, but I am not sure whether "excess deaths" takes into account lifespan. It has been roughly 70 to 80 years since "baby boomers" were born; based on common lifespans, wouldn't it be expected to have a "death boom" around this time?

skybrian 5y ago
You can estimate gradual changes like that by looking at a long-term graph. Here's a graph from before the pandemic, including a (now out of date) projection: https://www.macrotrends.net/countries/USA/united-states/death-rate https://www.macrotrends.net/countries/USA/united-states/deat... (Note that the slope looks exaggerated because the baseline isn't zero.)
Cogito 5y ago
The general way to do that correction is to look at actuarial tables for deaths by cohort. These models are obviously not 100% accurate, but they do take into account many factors, and are incentivised to be as accurate as possible as they determine the profitavility of life insurance. In this context, excess deaths are variations from these tables - more people from a cohort dying in a certain time period than expected.

jwilber 5y ago

Lots of comments around code quality, primarily as it relates to testing. The real code quality sin here is that R, a vectorized-by-default language, is being used to employ for- and while- loops everywhere. It’s strange to see given the absolute plethora of R resources available aimed at avoiding just that. But props to the economist for releasing the code, anyway. I’m a happy subscriber and will continue to be one.

Redoubts 5y ago
Well, they only started using R a year or so ago, so they can be forgiven. > Until fairly recently, we were less comfortable with statistical software (like R) that allows more sophisticated visualisations. https://medium.economist.com/mistakes-weve-drawn-a-few-8cdd8a42d368 https://medium.economist.com/mistakes-weve-drawn-a-few-8cdd8...
- jwilber 5y ago
  That makes it even more strange, as all modern (ie within last 5 years) resources push the tidyverse.

User23 5y ago

2017 had more excess deaths than 2020[1]. The DALY loss was much greater too. What happened in 2017 that dwarfed the pandemic? [1] https://www.pnas.org/content/118/16/e2024850118 https://www.pnas.org/content/118/16/e2024850118

ericcholis 5y ago
Flu? That's my completely uneducated guess
HWR_14 5y ago
What happened in 2017 that was completely unresponded to that dwarfed the pandemic when the world shut down.
unpolloloco 5y ago
If I'm reading this correctly, this paper is defining excess deaths as the number of people who died in the US who wouldn't have were those people living in Europe instead of the US at that point in time? Which points to structural issues in the US system, but not to some event in 2017 that resulted in extra deaths vs. any surrounding years (just that the death rate in the US has dropped much slower than in Europe over the study timeline 2000-2017). The US death rate didn't appreciably increase (~3 per 100k) in 2017 vs. 2016[1] and has generally been improving. That's about 10k extra deaths, not 400k, like the paper says (because the paper's usage of that term is very different than The Economist's!) https://www.cdc.gov/nchs/data-visualization/mortality-trends/ https://www.cdc.gov/nchs/data-visualization/mortality-trends...
- User23 5y ago
  > but not to some event in 2017 that resulted in extra deaths vs. any surrounding years But that's exactly what the paper shows. It's reasonable to ask the question why was the difference in deaths far greater in 2017 than in the surrounding years and why was the mean age of the persons excessively dying so relatively young? There most likely are structural differences causing excess US deaths, but the claim that those structural differences caused a huge relative spike in one year for no particular reason isn't very satisfying.

fuckyah 5y ago

Check out USmortality.com

fuckyah 5y ago

Check out https://www.USMortality.com https://www.USMortality.com