4 ms·

What kind of cursed date is 32/1/2017? If you're starting with that kind of data-quality, what confidence can you have in the rest of it, even if the rest "app

What kind of cursed date is 32/1/2017? If you're starting with that kind of data-quality, what confidence can you have in the rest of it, even if the rest "appears" more normal? To expand my thoughts further: Data quality is like water-quality. If you find something bad in your water source, you're not going to think that you'll be okay just picking it out and carrying on with the rest. You need to have a proper process to treat it and more importantly, need to investigate the upstream source of the bad data.

ziddoap 2y ago
The date you've noticed is put there purposefully in this example Excel sheet by the original author. >I then use the function pointblank::test_col_vals_in_set from the pointblank package to detect if there are any issues by setting no_issue to be either TRUE or FALSE. >If there are any issues, I will isolate and display them. >I will then correct the invalid dates accordingly and proceed. We just assume that the date was supposed to be 31-01-2017. https://jeremy-selva.netlify.app/blog/2024-02-15-tackling-formatted-cell-data/ https://jeremy-selva.netlify.app/blog/2024-02-15-tackling-fo...
- xnorswap 2y ago
  Yes, but the point I'm trying to make is that I think "isolation" is the wrong approach for data quality issues. That's like fishing out a turd. You need to quarantine the lot until you fix the upstream. Automating that process (invalidation followed by filtering) is even worse, it will merely mask data quality issues when you want the opposite, to grind everything to a halt until you get workable and realistic data. Perhaps I'm not "realistic" enough about real world data to work in data analytics. The frustration from ignoring bad data was a large part of why I quit data analysis for software development around 15 years ago.
  - ziddoap 2y ago
    For sure, you'd generally want to address the source of the issue rather than band-aid it, but that isn't always possible (or not possible immediately) and you just have to work with what you've got. Most of the time your boss isn't going to let you "grind everything to a halt". But, more importantly, I think the blog is focused more on the practical "here's how to do x with some code", and less about the theory of data science.
- luisdva14 2y ago
  post author here, thanks to both of these commenters for pointing out this issues with the intentionally bad date. I'll make note of this and update the post.
darkhorn 2y ago
When you make a survey surveyor might write the data incorrectly mistakenly, it happens. If the data is written down mistakenly the basic statistics like median, mean, standard deviation etc won't change much. After all it is just a survey and you already sample from the population. So, even if the data in the sample was correct you will not get the exact population data. But of course if inconsistency in the data is found it is further analyzed whether if it was a small mistake intentional wrong doing.

ziddoap 2y ago

The date you've noticed is put there purposefully in this example Excel sheet by the original author. >I then use the function pointblank::test_col_vals_in_set from the pointblank package to detect if there are any issues by setting no_issue to be either TRUE or FALSE. >If there are any issues, I will isolate and display them. >I will then correct the invalid dates accordingly and proceed. We just assume that the date was supposed to be 31-01-2017. https://jeremy-selva.netlify.app/blog/2024-02-15-tackling-formatted-cell-data/ https://jeremy-selva.netlify.app/blog/2024-02-15-tackling-fo...

xnorswap 2y ago
Yes, but the point I'm trying to make is that I think "isolation" is the wrong approach for data quality issues. That's like fishing out a turd. You need to quarantine the lot until you fix the upstream. Automating that process (invalidation followed by filtering) is even worse, it will merely mask data quality issues when you want the opposite, to grind everything to a halt until you get workable and realistic data. Perhaps I'm not "realistic" enough about real world data to work in data analytics. The frustration from ignoring bad data was a large part of why I quit data analysis for software development around 15 years ago.
- ziddoap 2y ago
  For sure, you'd generally want to address the source of the issue rather than band-aid it, but that isn't always possible (or not possible immediately) and you just have to work with what you've got. Most of the time your boss isn't going to let you "grind everything to a halt". But, more importantly, I think the blog is focused more on the practical "here's how to do x with some code", and less about the theory of data science.
luisdva14 2y ago
post author here, thanks to both of these commenters for pointing out this issues with the intentionally bad date. I'll make note of this and update the post.

darkhorn 2y ago

When you make a survey surveyor might write the data incorrectly mistakenly, it happens. If the data is written down mistakenly the basic statistics like median, mean, standard deviation etc won't change much. After all it is just a survey and you already sample from the population. So, even if the data in the sample was correct you will not get the exact population data. But of course if inconsistency in the data is found it is further analyzed whether if it was a small mistake intentional wrong doing.