3 ms·
I would love to see the tests that people come up for this... Tests can be helpful and informative, or they can be make-work boilerplate that will never reveal
by epistasis 5y ago
I would love to see the tests that people come up for this...
Tests can be helpful and informative, or they can be make-work boilerplate that will never reveal any flaws. And in this type of code base, it's hard to imagine the test that's written in code that will be informative.
When dealing with data, "testing" is most often done visually, with plotting.
The process of exploratory data analysis is not anything at all like writing code to an architected specification, and the same practices are not as effective for different types of work. Insisting that there should be automated tests for something like this reminds me of the cultural phenomenon of extreme table-phobia in the 2000s that happened as a reaction to using tables for layout. I had trouble getting people to believe that is was OK to use tables for tabular data, it was extremely frustrating. Or in the last 5-10 years, with design trends preferring spare layouts with lots of empty space, it was super hard to get information dense designs into production, because they were so averse to the idea that people may need information density for some things.
I'm also reminded of this video about the deficiencies of notebooks:
https://youtu.be/7jiPeIFXb6U https://youtu.be/7jiPeIFXb6U
Yes, there are huge deficiencies to notebooks. But using them is an entirely different task than the type of code that one would write in an IDE, for code that gets put into a library and will be run by others. I have two work modes: 1) IDE (actually Emacs) where I write tests and the code is meant to be reusable, even if it's a one-off script, then 2) exploratory data analysis and processing, in notebooks. Though both of these use code, they do not and should not share much else in the way of practices.
We develop best practices in fields to work against our worst instincts. And once we establish best practices, we become strict about them to make sure we don't fall back into bad practices. But when approaching a different field it's good to reevaluate these best practices to see if they still fit.
- jfoutz 5y agoI think it always makes sense to have an oracle, or a few. Nothing super fancy, just a few values you _know_ have a certain outcome. f(4,5) == 6 is good enough. just, whatever you type in to test as you develop the code, write it down and save it for later. I mean, yeah, you can go nuts with boilerplate. But, I always seem to regret not having written down a handful of cases. If nothing else, it helps me remember what the heck this function is for, and a few samples of expected input and output, along with source are more immediately useful (to me) than comments. Opinions differ. This isn't a hill I'll die on - as evidenced by my lack of tests from time to time. But I've never regretted the few minutes to provide them, and I have regretted not providing them
- epistasis 5y agoThat sort of test doesn't really apply here in any meaningful way. Suppose you want to validate that a certain column is in the file. You could place your assumptions around that explicitly in the read_tsv function call and have it generate the error for you on reading the file, or it can generate an error when you try to use that column. Either way, there's no "test" yet the intention is clearly stated in the code, and the code will simply fail out rather than proceed if it encounters input that can't be meaningfully used. This code is in many ways a declarative use of other code, describing the data, doing some joins and filtering, and plugging it into well tested libraries of code. The data structures that store the data and the library functions that deal with the data are designed in a way that bugs often result in errors immediately. In many ways it's like writing a SQL statement. One could make test data tables and make sure that the SQL statement does what one expects, but the SQL statement already embodies intent in the same way that a test of code would embody intent.
- jfoutz 5y agoSo, I'm not an R guy, but I pawed through some of the code. I get what you're saying, there's an environment, it needs to be just so, or it won't work. that's fine. And yeah, it does feel a lot like SQL. I would point out this guy - https://github.com/TheEconomist/covid-19-the-economist-global-excess-deaths-model/blob/025cb1921500fa43da5ea56a877b1069833b7824/scripts/1_excess_deaths_global_estimates_script.R#L714 https://github.com/TheEconomist/covid-19-the-economist-globa... that sort of has the look of something that took a pass or two to get right. And I think, would be nice to have a test case (sample call). Again, not a hill I'm going to die on. Overall, I don't think we have a big difference of opinion, yeah, it looks like gluing together libraries - as a non, native R speaker, I think I can make sense of the project, it's cool they put it out there, and I don't think there's anything _wrong_ with what they've got.
- nojito 5y agoThat function is pretty standard. If you want to run a loess, it runs the loess function if not it runs a windowed average. What exactly would you test it for?
- eyelidlessness 5y agoI haven’t looked at the code but this stands out to me: > When dealing with data, "testing" is most often done visually, with plotting. Even if your validation process is entirely manual, you still can benefit from automated testing. Snapshot tests are completely agnostic from whatever serialization format. If you eyeball check something and say “yup this is good”, that’s something you could snapshot and never test again.