6 ms·

I do work with "open data" on a near-obsessive basis and -- friends, please do not trust "open data" portals to reflect reality accurately. The datasets are oft

by chaps 2y ago

I do work with "open data" on a near-obsessive basis and -- friends, please do not trust "open data" portals to reflect reality accurately. The datasets are often curated, categories changed during the ETL processes, rows missing, and things like that. For example, Chicago's "crimes" dataset intentionally doesn't include all homicides. Can't remember the exact dataset, but I once had a conversation with Chicago's head of open data who told me that they intentionally removed many rows because they were concerned that the public was going to misinterpret the results... but didn't make it clear that rows were missing. So I guess everybody gets the opportunity to misinterpret the results! FOIA is the better alternative because it gives you the original, pre-cleaned data. Open data is a lie.

kalendos 2y ago
I can only imagine. Many ETLs are already messy in companies with better tooling and processes. Would love to read more about your experience with Open Data. Any place where I can reach out?
- chaps 2y ago
  Here's something about shotspotter data in Chicago: https://x.com/foiachap/status/1775296597850480663 https://x.com/foiachap/status/1775296597850480663 And this one makes some rounds: https://mchap.io/that-time-the-city-of-seattle-accidentally-gave-me-32m-emails-for-40-dollars4997.html https://mchap.io/that-time-the-city-of-seattle-accidentally-... Feel free to reach out!
pjot 2y ago
This is super true. For my city’s portal as well. I’ve found one way around this by versioning the dataset - that is, committing the diffs in git. Credit to Simon Willison’s git-scraping technique. I do this with my power company’s outage map: https://github.com/patricktrainer/entergy-outages https://github.com/patricktrainer/entergy-outages 67k commits! https://simonwillison.net/2020/Oct/9/git-scraping/ https://simonwillison.net/2020/Oct/9/git-scraping/
- chaps 2y ago
  That's a really freaking neat trick. Thanks!
bshep 2y ago
Where I grew up the data for murders is curated in such a way that anybody that dies 24 after being attacked is not considered a ‘murder’. Tehy do this to reduce the statistical murder rate.
- whoiscroberts 2y ago
  Well now we know why crime is down
- chaps 2y ago
  Can you say more about this?
IanCal 2y ago
Although pre-cleaned data is often not reflective of reality and requires careful work to use, often requiring a lot more knowledge of the field.
gordon_freeman 2y ago
But even if dataset is incomplete or not accurate, do you think we could at least get directionally right insights from such datasets?
- chaps 2y ago
  Yes, of course there can be. But I cannot ignore the harms in doing so, by misrepresenting the data in a way that disallows others to understand what is or isn't there -- it happens regularly. These datasets are often used as a political tool and contracted with local universities to show that they're providing data... though not actually providing the accurate data. Simultaneously though, people who don't know data will champion the data as accurate because it comes from a university program. Sometimes what can happen is that somebody inexperienced will try to make some assessment of the data and come to the exact wrong conclusion because they didn't know what not to trust. But it gets on the news anyway and damage is done. We can do better than that.
stevage 2y ago
I worked in open data for quite a few years. This is a very weird take. Open data portals generally have data is useful form. FOI probably gives you PDFs.
- chaps 2y ago
  "FOI probably gives you PDFs." Having submitted thousands of FOIA requests, I get the impression that you haven't, actually, submitted many FOIA requests. I've received many, many, many, many non-PDF FOIA responses. Share me some of the open data you've worked with and I'd love to poke at it and tell you where it's wrong and where assumptions about its data is wrong.
  - 0cf8612b2e1e 2y ago
    Thousands?! Do you have a public list on everything? Have you had to fight a lot of malicious compliance which balloons up your request count? Or do they typically require an incredibly narrow request that you have up submit N entries per topic?
  - stevage 2y ago
    What an unappealing offer. No thanks.
    - chaps 2y ago
      Not any different from being red-team'd, but you do you. But thanks for your input -- it makes more sense that your apparent reluctance to be challenged makes it clear why you think my take is odd. Even still, I challenge you to challenge yourself to understand where your blind spots are. I've done it many times and have found significant problems with the open datasets I've worked with. If you think my take is weird, it's only because you're not looking or the data you're looking at is inconsequential. To me, this stuff is literal life and death. If we make mistakes in our analysis because of misinformation from the source, then the lives and deaths of people we're trying to understand becomes tarnished. We can treat our neighbors better than that.
amy-petrik-214 2y ago
Hah that's classic politics "Hello John Q. Public, here's all our data! It speaks for itself" John Q. Public: "Wow, you really improved last few years homicide-wise" "And so you see, a third party unrelated to us has just confirmed what a great job we're doing with simple empirical, evidence-based governance!" So that means what you want to do is specialize in identifying bias in these datasets and finding the smoking gun. Such a task can be an ugly business but necessary for the public good, pushing data sharers to either share good data, or not share, but not share tricksy data in this unethical way.

by chaps 2y ago

kalendos 2y ago

I can only imagine. Many ETLs are already messy in companies with better tooling and processes. Would love to read more about your experience with Open Data. Any place where I can reach out?

chaps 2y ago
Here's something about shotspotter data in Chicago: https://x.com/foiachap/status/1775296597850480663 https://x.com/foiachap/status/1775296597850480663 And this one makes some rounds: https://mchap.io/that-time-the-city-of-seattle-accidentally-gave-me-32m-emails-for-40-dollars4997.html https://mchap.io/that-time-the-city-of-seattle-accidentally-... Feel free to reach out!

pjot 2y ago

This is super true. For my city’s portal as well. I’ve found one way around this by versioning the dataset - that is, committing the diffs in git. Credit to Simon Willison’s git-scraping technique. I do this with my power company’s outage map: https://github.com/patricktrainer/entergy-outages https://github.com/patricktrainer/entergy-outages 67k commits! https://simonwillison.net/2020/Oct/9/git-scraping/ https://simonwillison.net/2020/Oct/9/git-scraping/

chaps 2y ago
That's a really freaking neat trick. Thanks!

bshep 2y ago

Where I grew up the data for murders is curated in such a way that anybody that dies 24 after being attacked is not considered a ‘murder’. Tehy do this to reduce the statistical murder rate.

whoiscroberts 2y ago
Well now we know why crime is down
chaps 2y ago
Can you say more about this?

IanCal 2y ago

Although pre-cleaned data is often not reflective of reality and requires careful work to use, often requiring a lot more knowledge of the field.

gordon_freeman 2y ago

But even if dataset is incomplete or not accurate, do you think we could at least get directionally right insights from such datasets?

chaps 2y ago
Yes, of course there can be. But I cannot ignore the harms in doing so, by misrepresenting the data in a way that disallows others to understand what is or isn't there -- it happens regularly. These datasets are often used as a political tool and contracted with local universities to show that they're providing data... though not actually providing the accurate data. Simultaneously though, people who don't know data will champion the data as accurate because it comes from a university program. Sometimes what can happen is that somebody inexperienced will try to make some assessment of the data and come to the exact wrong conclusion because they didn't know what not to trust. But it gets on the news anyway and damage is done. We can do better than that.

stevage 2y ago

I worked in open data for quite a few years. This is a very weird take. Open data portals generally have data is useful form. FOI probably gives you PDFs.

chaps 2y ago
"FOI probably gives you PDFs." Having submitted thousands of FOIA requests, I get the impression that you haven't, actually, submitted many FOIA requests. I've received many, many, many, many non-PDF FOIA responses. Share me some of the open data you've worked with and I'd love to poke at it and tell you where it's wrong and where assumptions about its data is wrong.
- 0cf8612b2e1e 2y ago
  Thousands?! Do you have a public list on everything? Have you had to fight a lot of malicious compliance which balloons up your request count? Or do they typically require an incredibly narrow request that you have up submit N entries per topic?
- stevage 2y ago
  What an unappealing offer. No thanks.
  - chaps 2y ago
    Not any different from being red-team'd, but you do you. But thanks for your input -- it makes more sense that your apparent reluctance to be challenged makes it clear why you think my take is odd. Even still, I challenge you to challenge yourself to understand where your blind spots are. I've done it many times and have found significant problems with the open datasets I've worked with. If you think my take is weird, it's only because you're not looking or the data you're looking at is inconsequential. To me, this stuff is literal life and death. If we make mistakes in our analysis because of misinformation from the source, then the lives and deaths of people we're trying to understand becomes tarnished. We can treat our neighbors better than that.

amy-petrik-214 2y ago

Hah that's classic politics "Hello John Q. Public, here's all our data! It speaks for itself" John Q. Public: "Wow, you really improved last few years homicide-wise" "And so you see, a third party unrelated to us has just confirmed what a great job we're doing with simple empirical, evidence-based governance!" So that means what you want to do is specialize in identifying bias in these datasets and finding the smoking gun. Such a task can be an ugly business but necessary for the public good, pushing data sharers to either share good data, or not share, but not share tricksy data in this unethical way.