6 ms·

No matter the quality of the code presented here, thanks to The Economist for posting this! We all really appreciate it, even if we don't always seem like it :)

by narush 5y ago

No matter the quality of the code presented here, thanks to The Economist for posting this! We all really appreciate it, even if we don't always seem like it :) Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well. So then we end up with the same code, but now no one can see it - which is the worst of all worlds. Let's support / work towards a culture of sharing our code before anything else. And if there's something you think can be improved, maybe consider opening a PR with improvements to really _show the value_ to all these parties of posting their code!

Taek 5y ago
Amen! When we have the code, we can all work together to improve the model, add the tests, find the bugs. And honestly, research shouldn't be considered valid if all of the code and data isn't available. Reproducibility is one of the pillars of science, and that's not possible without code and data.
jarenmf 5y ago
The code is not even that bad. It's a simple script generating a plot. I think it's too much to expect tests and documentation for it.
- stephc_int13 5y ago
  I took a quick look, and I found it readable and easy to follow, contrary to 98% of the codebase I see on github.
rich_sasha 5y ago
> Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well. I agree that code should be published alongside papers, and great that the Economist, being a newspaper, publishes their code too. But when it is the most reputable sources (researchers, universities etc) publishing code of quality so low cannot possibly be correct (looking at you Imperial), it totally should be scrutinised and critiqued. I think the root cause is, where research is code-based (I guess 90% of science), it should pass the most basic correctness tests before a paper is accepted, or indeed turned into national policy. There are many angles to bite this apple from, but what remains unacceptable is shit code being taken at face value.
- monkey_monkey 5y ago
  > publishing code of quality so low cannot possibly be correct (looking at you Imperial) If this is referring to the quality of the model developed by Neil Ferguson @ Imperial College, John Carmack audited the code and didn't have a big problem with it: https://twitter.com/ID_AA_Carmack/status/1254872368763277313 https://twitter.com/ID_AA_Carmack/status/1254872368763277313
epistasis 5y ago
I would love to see the tests that people come up for this... Tests can be helpful and informative, or they can be make-work boilerplate that will never reveal any flaws. And in this type of code base, it's hard to imagine the test that's written in code that will be informative. When dealing with data, "testing" is most often done visually, with plotting. The process of exploratory data analysis is not anything at all like writing code to an architected specification, and the same practices are not as effective for different types of work. Insisting that there should be automated tests for something like this reminds me of the cultural phenomenon of extreme table-phobia in the 2000s that happened as a reaction to using tables for layout. I had trouble getting people to believe that is was OK to use tables for tabular data, it was extremely frustrating. Or in the last 5-10 years, with design trends preferring spare layouts with lots of empty space, it was super hard to get information dense designs into production, because they were so averse to the idea that people may need information density for some things. I'm also reminded of this video about the deficiencies of notebooks: https://youtu.be/7jiPeIFXb6U https://youtu.be/7jiPeIFXb6U Yes, there are huge deficiencies to notebooks. But using them is an entirely different task than the type of code that one would write in an IDE, for code that gets put into a library and will be run by others. I have two work modes: 1) IDE (actually Emacs) where I write tests and the code is meant to be reusable, even if it's a one-off script, then 2) exploratory data analysis and processing, in notebooks. Though both of these use code, they do not and should not share much else in the way of practices. We develop best practices in fields to work against our worst instincts. And once we establish best practices, we become strict about them to make sure we don't fall back into bad practices. But when approaching a different field it's good to reevaluate these best practices to see if they still fit.
- jfoutz 5y ago
  I think it always makes sense to have an oracle, or a few. Nothing super fancy, just a few values you _know_ have a certain outcome. f(4,5) == 6 is good enough. just, whatever you type in to test as you develop the code, write it down and save it for later. I mean, yeah, you can go nuts with boilerplate. But, I always seem to regret not having written down a handful of cases. If nothing else, it helps me remember what the heck this function is for, and a few samples of expected input and output, along with source are more immediately useful (to me) than comments. Opinions differ. This isn't a hill I'll die on - as evidenced by my lack of tests from time to time. But I've never regretted the few minutes to provide them, and I have regretted not providing them
lanevorockz 5y ago
Data should only be taken seriously when the full context is shared for reference. Somehow internet created a generation of TLDRs or ELI5 which just is very sensible to lies and manipulation. Even though I can build trust to my sources, I still think it is mandatory to have access to the raw data.
- mikepurvis 5y ago
  The Economist has been publishing since 1843. I'd imagine that for the vast majority of that time, readers have trusted them with either no details at all about methodology or little more than a high level overview of what was done— certainly there was no expectation of being able to reproduce and iterate on the full analysis locally, even thirty years ago, much less in 19th century. Don't get me wrong— this is a good direction to be pushing in. But it's something that's been very much enabled by the internet, which is why I'm finding it weird to be blaming the prior status quo on internet culture attention span memes.
- IAmEveryone 5y ago
  > Somehow internet created a generation of TLDRs or ELI5 No, it has created a generation. of conspiracy-mongering freaks. Or, more precisely, it has allowed that particularly seedy underbelly of society to become more noticeable.
nojito 5y ago
The code doesn’t even matter in this instance. The discussion should be on the model design and whether it’s sound. Leave it to HN to start criticizing code whenever anything is posted.
- CrazyStat 5y ago
  The code absolutely does matter, as we need to be confident that it is a correct implementation of the model. To turn your first sentence around, if the code is incorrect than the model doesn't even matter. I've done my share of implementing statistical models in code and have seen plenty of examples of incorrect code failing to implement the model correctly.
  - nojito 5y ago
    Your second paragraph doesn’t match the first. Simply clone the repo, download the data, and verify the results. The discussion needs to not be on the code but the quality of the assumptions in their model. Which you can read here. https://www.economist.com/graphic-detail/2021/05/13/how-we-estimated-the-true-death-toll-of-the-pandemic https://www.economist.com/graphic-detail/2021/05/13/how-we-e...
    - CrazyStat 5y ago
      Why should I care about the assumptions of a model if that's not the model that is implemented, because the code contains errors? Obviously the assumptions of the model are important. But I was responding to a claim that the quality of the code is not important, which is also obviously false. Neither discussion alone is sufficient.
pkilgore 5y ago
Code like this is like (not) caring about memory leaks in a Missile Control module where the explosive is the garbage collector: When code is one shot, accomplishes its goal, and does not need to be extended or relied upon in the future, why do all these things? Is there any goal above that is not also serviced, at least in significant part, by open-sourcing it? Bravo to The Economist for releasing it. This is unabashedly a Good Thing.

by narush 5y ago

Taek 5y ago

Amen! When we have the code, we can all work together to improve the model, add the tests, find the bugs. And honestly, research shouldn't be considered valid if all of the code and data isn't available. Reproducibility is one of the pillars of science, and that's not possible without code and data.

jarenmf 5y ago

The code is not even that bad. It's a simple script generating a plot. I think it's too much to expect tests and documentation for it.

stephc_int13 5y ago
I took a quick look, and I found it readable and easy to follow, contrary to 98% of the codebase I see on github.

rich_sasha 5y ago

> Even if it seems justified to shit on the code as is (bad style, lack of comments, no tests, whatever), all it does is discourage similar companies / researchers / folks in academia from posting their code as well. I agree that code should be published alongside papers, and great that the Economist, being a newspaper, publishes their code too. But when it is the most reputable sources (researchers, universities etc) publishing code of quality so low cannot possibly be correct (looking at you Imperial), it totally should be scrutinised and critiqued. I think the root cause is, where research is code-based (I guess 90% of science), it should pass the most basic correctness tests before a paper is accepted, or indeed turned into national policy. There are many angles to bite this apple from, but what remains unacceptable is shit code being taken at face value.

monkey_monkey 5y ago
> publishing code of quality so low cannot possibly be correct (looking at you Imperial) If this is referring to the quality of the model developed by Neil Ferguson @ Imperial College, John Carmack audited the code and didn't have a big problem with it: https://twitter.com/ID_AA_Carmack/status/1254872368763277313 https://twitter.com/ID_AA_Carmack/status/1254872368763277313

epistasis 5y ago

I would love to see the tests that people come up for this... Tests can be helpful and informative, or they can be make-work boilerplate that will never reveal any flaws. And in this type of code base, it's hard to imagine the test that's written in code that will be informative. When dealing with data, "testing" is most often done visually, with plotting. The process of exploratory data analysis is not anything at all like writing code to an architected specification, and the same practices are not as effective for different types of work. Insisting that there should be automated tests for something like this reminds me of the cultural phenomenon of extreme table-phobia in the 2000s that happened as a reaction to using tables for layout. I had trouble getting people to believe that is was OK to use tables for tabular data, it was extremely frustrating. Or in the last 5-10 years, with design trends preferring spare layouts with lots of empty space, it was super hard to get information dense designs into production, because they were so averse to the idea that people may need information density for some things. I'm also reminded of this video about the deficiencies of notebooks: https://youtu.be/7jiPeIFXb6U https://youtu.be/7jiPeIFXb6U Yes, there are huge deficiencies to notebooks. But using them is an entirely different task than the type of code that one would write in an IDE, for code that gets put into a library and will be run by others. I have two work modes: 1) IDE (actually Emacs) where I write tests and the code is meant to be reusable, even if it's a one-off script, then 2) exploratory data analysis and processing, in notebooks. Though both of these use code, they do not and should not share much else in the way of practices. We develop best practices in fields to work against our worst instincts. And once we establish best practices, we become strict about them to make sure we don't fall back into bad practices. But when approaching a different field it's good to reevaluate these best practices to see if they still fit.

jfoutz 5y ago
I think it always makes sense to have an oracle, or a few. Nothing super fancy, just a few values you _know_ have a certain outcome. f(4,5) == 6 is good enough. just, whatever you type in to test as you develop the code, write it down and save it for later. I mean, yeah, you can go nuts with boilerplate. But, I always seem to regret not having written down a handful of cases. If nothing else, it helps me remember what the heck this function is for, and a few samples of expected input and output, along with source are more immediately useful (to me) than comments. Opinions differ. This isn't a hill I'll die on - as evidenced by my lack of tests from time to time. But I've never regretted the few minutes to provide them, and I have regretted not providing them

lanevorockz 5y ago

Data should only be taken seriously when the full context is shared for reference. Somehow internet created a generation of TLDRs or ELI5 which just is very sensible to lies and manipulation. Even though I can build trust to my sources, I still think it is mandatory to have access to the raw data.

mikepurvis 5y ago
The Economist has been publishing since 1843. I'd imagine that for the vast majority of that time, readers have trusted them with either no details at all about methodology or little more than a high level overview of what was done— certainly there was no expectation of being able to reproduce and iterate on the full analysis locally, even thirty years ago, much less in 19th century. Don't get me wrong— this is a good direction to be pushing in. But it's something that's been very much enabled by the internet, which is why I'm finding it weird to be blaming the prior status quo on internet culture attention span memes.
IAmEveryone 5y ago
> Somehow internet created a generation of TLDRs or ELI5 No, it has created a generation. of conspiracy-mongering freaks. Or, more precisely, it has allowed that particularly seedy underbelly of society to become more noticeable.

nojito 5y ago

The code doesn’t even matter in this instance. The discussion should be on the model design and whether it’s sound. Leave it to HN to start criticizing code whenever anything is posted.

CrazyStat 5y ago
The code absolutely does matter, as we need to be confident that it is a correct implementation of the model. To turn your first sentence around, if the code is incorrect than the model doesn't even matter. I've done my share of implementing statistical models in code and have seen plenty of examples of incorrect code failing to implement the model correctly.
- nojito 5y ago
  Your second paragraph doesn’t match the first. Simply clone the repo, download the data, and verify the results. The discussion needs to not be on the code but the quality of the assumptions in their model. Which you can read here. https://www.economist.com/graphic-detail/2021/05/13/how-we-estimated-the-true-death-toll-of-the-pandemic https://www.economist.com/graphic-detail/2021/05/13/how-we-e...
  - CrazyStat 5y ago
    Why should I care about the assumptions of a model if that's not the model that is implemented, because the code contains errors? Obviously the assumptions of the model are important. But I was responding to a claim that the quality of the code is not important, which is also obviously false. Neither discussion alone is sufficient.

pkilgore 5y ago

Code like this is like (not) caring about memory leaks in a Missile Control module where the explosive is the garbage collector: When code is one shot, accomplishes its goal, and does not need to be extended or relied upon in the future, why do all these things? Is there any goal above that is not also serviced, at least in significant part, by open-sourcing it? Bravo to The Economist for releasing it. This is unabashedly a Good Thing.