4 ms·

What Do You Do When Reliable Sources Publish Unreliable Data?

6 points by data_dan_ 5y ago

data_dan_ 5y ago
(I wrote this article) We recently wrote an article (https://l.bit.io/o-cop26 https://l.bit.io/o-cop26) about methane emissions and the COP26 commitment to cut emissions. During the writing of that article, we found some serious inconsistencies in some of the data sources. Discussions of data quality and validation in data science tend to end with recommendations for a few data validation checks, such as making sure data come from trusted sources; handling missing values; and investigating outliers. These sorts of checks are important, but they won't save an analysis from perfectly-formatted data from a trusted source that happens to be wrong for reasons that can't be found in the dataset itself. Even data of apparently good quality can lead to faulty conclusions. This article delves into this question by exploring a case study. The U.N. publishes greenhouse gas emissions data supplied each year by parties to the UNFCCC (United Nations Framework Convention on Climate Change). The data are consistent, up-to-date, and well formatted, and the U.N. is a reliable source of official data. However, there is good reason to believe the data submitted by some countries is not accurate. There are other trusted data sources that show startlingly large differences from the U.N. data. In particular, we found that Russia's Methane emissions data were highly inconsistent with the World Resources Institute (WRI) Climate Analysis Indicators Tool (CAIT) data, even though these data were quite similar to the U.N. data for other countries.
rurban 5y ago
Change the label to unreliable source then.
- otacust 5y ago
  I agree. Trust has a very limited place when dealing with data, and I don't think you can extend the benefit of the doubt to sources that are not consistently reliable. Though in this case, it looks like the U.N. is aggregating self-reported country-level emissions data, so I guess the situation is a little different.
  - rurban 5y ago
    Looking at the data I would also be very sceptical on the other given source, could be politically influenced. The outlier is huge. But the self-given data looks also suspicious, given the history.
    - data_dan_ 5y ago
      The Washington Post did some really great work on generating a variety of comparison datasets: https://www.washingtonpost.com/climate-environment/interactive/2021/greenhouse-gas-emissions-pledges-data/methodology/?itid=lk_inline_manual_53 https://www.washingtonpost.com/climate-environment/interacti.... You're right, though -- it's really hard to avoid the issue of political influence in climate data. None of the data can exist in a vacuum; it all has (geo)political implications.
      - rurban 5y ago
        Well, the Wapo is really one of the most untrusted sources ever. It's merely a CIA propaganda outlet. Scientific research would be a try, but propaganda outlets for sure not.
otacust 5y ago
The Washington Post article referenced in this post is really interesting: https://www.washingtonpost.com/climate-environment/interactive/2021/russia-greenhouse-gas-emissions/ https://www.washingtonpost.com/climate-environment/interacti... The yearly revisions of the GHG emissions estimates are really striking.

6 points by data_dan_ 5y ago

(I wrote this article) We recently wrote an article (https://l.bit.io/o-cop26 https://l.bit.io/o-cop26) about methane emissions and the COP26 commitment to cut emissions. During the writing of that article, we found some serious inconsistencies in some of the data sources. Discussions of data quality and validation in data science tend to end with recommendations for a few data validation checks, such as making sure data come from trusted sources; handling missing values; and investigating outliers. These sorts of checks are important, but they won't save an analysis from perfectly-formatted data from a trusted source that happens to be wrong for reasons that can't be found in the dataset itself. Even data of apparently good quality can lead to faulty conclusions. This article delves into this question by exploring a case study. The U.N. publishes greenhouse gas emissions data supplied each year by parties to the UNFCCC (United Nations Framework Convention on Climate Change). The data are consistent, up-to-date, and well formatted, and the U.N. is a reliable source of official data. However, there is good reason to believe the data submitted by some countries is not accurate. There are other trusted data sources that show startlingly large differences from the U.N. data. In particular, we found that Russia's Methane emissions data were highly inconsistent with the World Resources Institute (WRI) Climate Analysis Indicators Tool (CAIT) data, even though these data were quite similar to the U.N. data for other countries.

rurban 5y ago

Change the label to unreliable source then.

otacust 5y ago
I agree. Trust has a very limited place when dealing with data, and I don't think you can extend the benefit of the doubt to sources that are not consistently reliable. Though in this case, it looks like the U.N. is aggregating self-reported country-level emissions data, so I guess the situation is a little different.
- rurban 5y ago
  Looking at the data I would also be very sceptical on the other given source, could be politically influenced. The outlier is huge. But the self-given data looks also suspicious, given the history.
  - data_dan_ 5y ago
    The Washington Post did some really great work on generating a variety of comparison datasets: https://www.washingtonpost.com/climate-environment/interactive/2021/greenhouse-gas-emissions-pledges-data/methodology/?itid=lk_inline_manual_53 https://www.washingtonpost.com/climate-environment/interacti.... You're right, though -- it's really hard to avoid the issue of political influence in climate data. None of the data can exist in a vacuum; it all has (geo)political implications.
    - rurban 5y ago
      Well, the Wapo is really one of the most untrusted sources ever. It's merely a CIA propaganda outlet. Scientific research would be a try, but propaganda outlets for sure not.

otacust 5y ago

The Washington Post article referenced in this post is really interesting: https://www.washingtonpost.com/climate-environment/interactive/2021/russia-greenhouse-gas-emissions/ https://www.washingtonpost.com/climate-environment/interacti... The yearly revisions of the GHG emissions estimates are really striking.