4 ms·
Always use checked pre- and post-conditions in a data pipeline. This simple post-condition would have caught this issue: The sheet after merge operation must h
by jononor 6y ago
Always use checked pre- and post-conditions in a data pipeline.
This simple post-condition would have caught this issue:
The sheet after merge operation must have a number of rows equal to the sum of number of rows for all merged sheets.
Assuming this is a merge of a standardized input, then another post-condition might be:
The number of columns in output shall equal the number of columns in the input.
Might want to check header names, and order as well.
Thinking in terms of universal properties, and putting the checks into production, is better than unit-testing.
- jen729w 6y agoEven just within my own sheet I always build in some sort of idiot-check, the assumption being that I’m the idiot. Do a SUM and match it against a COUNTIF on another column, or something. It doesn’t really matter what it is but if the data is important at all, I always scatter little checks throughout. The case described in this article sounds like someone who just doesn’t know how to use Excel. I mean why in the name of VLOOKUP would you override the default and choose to save as .xls? That was a conscious decision. Anyone worth their salt knows that’s stupid. Excel is not to blame here.
- dkarp 6y agoI’d guess they wrote a script to merge the csv file and export in xls. Xls might have been the default of the library they used in a rush. I think this may be the case with some python libraries that generate excel files as xlsx is a more complicated format.
- deleted 6y ago[deleted]
- 1_player 6y ago> Always use checked pre- and post-conditions in a data pipeline. [A] simple post-condition would have caught this issue I'm not sure they teach that in medical school.
- avibhu 6y agoWhich I feel might be indicative of a larger problem. If you are going to deal with data (and its processing), it only makes sense to teach those skills in school itself.
- swiley 6y agoThey spend so much time memorizing random trivia maybe a course in handling data wouldn't be a bad idea.
- ArnoVW 6y agoGive them a break. They don't have the luxury of a highly-structured field, purposefuly constructed by humans (engineers & mathematiciens on top of that) with only 50 years of legacy. They have to deal with a field that was haphasardly constructed by nature over the course of 4 billion years. Imagine having to retro-document a code base with 4 billion years of history, created solely by junior developers fresh out of college..
- jussij 6y agoThis quote from the article says this work was being done be a developer(s) and if that is the case they should have know better. The problem is that PHE's own developers picked an old file format to do this - known as XLS.
- Closi 6y agoIt's unlikely this was done by a developer in reality, and it's more likely to be a data analyst that was called a developer in the press release.