5 ms·

The US government's open data on Data.gov is currently being scrubbed

240 points by xbmcuser 2y ago

forgetfreeman 2y ago
Do we have any idea what datasets have gone missing? More details please from anyone who has them.
- epakai 2y ago
  You can look at catalog.data.gov it shows totals. I'm comparing January 14th to today. The biggest loss I see is Organizations - Department of Energy, 5473 to 3647. I also see under Bureaus - Energy Programs, 4347 to 2521. These are overlapping categories (-1826 on both). There are others, but they seem smaller, a few hundred at most.
  - bayindirh 2y ago
    Maybe its related to climate change and advantages of renewables? I know DoE does "other things", but I don't expect these to be public anyway.
    - epakai 2y ago
      I can't tell from the Internet Archive. The datasets that went away seem to have been added quite recently, between January 8th and 14th. There isn't a suitable capture between January 8th and 25th to see which tags or categories changed within the DoE datasets.
      - ccgreg 2y ago
        Common Crawl's latest crawl was Jan 12th-25th, and the index is available.
  - deleted 2y ago
    [deleted]
  - epakai 2y ago
    Looking closer, there was a major increase quite recently before this decrease. On January 8th there were 3617 DoE datasets. On the 14th it was up to 5473. Now we're back down to just 30 more than on the 8th. Could it have been a publishing mistake, or some order to undo recent publications for review by the new admin?
    - refurb 2y ago
      This seems like a super important observation. If the number of dataset saw a massive jump (50%), then back down a week later, that seems more like the correction of an error.
- labster 2y ago
  [flagged]
Terr_ 2y ago
[flagged]
- bilekas 2y ago
  I'm not sure if it's nefarious or not, but I would have to lean on the side of "If the goverment is quietly removing information then probably it's not for the benefit of the public. Also from their own site at data.gov > The United States Government’s open data site is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government. The part " to inform decisions by the public and policymakers" in particular stands out. Anyway data sets is not exactly expensive to keep hosted for a government body so there's not really any excuse to scrub them unless they're polluted in some way ?
  - bayindirh 2y ago
    Even if it's polluted, you can mark it as is, with reasoning and keep it up, so you can learn what dirty/bad data is, and build quality detectors with the information contained in/with the "dirty" or "low quality" data set.
- bayindirh 2y ago
  Open Data initiatives generally build upon the premise of immutability. Data Sets can be alive, i.e. new data can be streamed into them, but they won't be purged, shortened or slimmed down. You can expect more historical data to be opened as the digitization and organization continues, and don't expect this data to be randomly being removed from archives. Even going one step further, if this data sets are refined and formatted, they are versioned, so you can always see the previous versions (e.g.: https://www.zenodo.org https://www.zenodo.org). If you're starting to remove any data set from a data pool w/o any explanation and announcement, this means there's a new "Department of Truth" in the works, 99% of the time.
  - specialist 2y ago
epakai 2y ago
It seems like some groups already expected this from efforts during the previous Trump admin. This article mentions a few, and more recent preparations: https://www.scientificamerican.com/article/scientists-scramble-to-save-climate-data-from-trump-again/ https://www.scientificamerican.com/article/scientists-scramb...
Symbiote 2y ago
I found https://uc3.cdlib.org/data-mirror/ https://uc3.cdlib.org/data-mirror/ which was a mirror a few years ago, but it's been taken down due to a lack of ongoing funding. It would be a great start to have a mirror of just the metadata, to at least see what is being removed.
trelliscoded 2y ago
That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not. I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration. I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.
- duxup 2y ago
  First Trump administration had a very strong pattern of talk and maybe a few actions… and then they seem to forget what they were doing and the result is little to no action. Trump loves to talk about trade but when it came to China he put up some token tariffs and announced “I didn’t say it would be easy” and then did nothing the rest of his term.
  - m3047 2y ago
    Trump killed the Trans Pacific Partnership. (This was a surprise to me. I'm grateful that I'm not at risk of being extradited to rot in a chinese prison for reverse engineering some IoT POS.)
    - tzs 2y ago
      How would TPP have put you at risk for Chinese prison?
      - immibis 2y ago
        Because of the copyright enforcement section of the TPP?
1shooner 2y ago
Is there any frame of reference to compare this to? I.e what is the normal fluctuation/movement of that 'datasets available' number? For instance, it dropped by 18k between 9/13 and 10/13.
- alphan0n 2y ago
  The API might be a good place to start. I’m on mobile at the moment so I haven’t been able to dig too deep, but here’s [0] a query that returns the metadata for everything modified since Jan 1st (add more rows at the end if it’s cut off). I don’t think it shows deletes directly, but it can be compared to the current catalog. API documentation here [1] [0] https://catalog.data.gov/api/3/action/package_search?q=metadata_modified:[2025-01-01T00:00:00Z%20TO%202025-01-31T23:59:59Z]&rows=200 https://catalog.data.gov/api/3/action/package_search?q=metad... [1] https://data.gov/developers/apis/ https://data.gov/developers/apis/ *fixed first url
khaki54 2y ago
Complete nothingburger. The malicious action would be to subtly change key datasets to promote some kind of biased narrative. It wouldn't be the removal of less than 1% of the datasets. Comical to think the Trump admin would be the ones "scrubbing" data when they passed the OPEN Government Data Act as part of the Foundations for Evidence Based Policymaking Act at the beginning of 2019. The law that requires agencies to publish their data to the site! Trump must really be playing 4d chess if one of the first actions in office is to "scrub" a few obscure and yet unidentified datasets from data.gov.
- deleted 2y ago
  [deleted]
- nojito 2y ago
  Why would they remove any datasets? Also calling SVI obscure is quite hilarious.
diamondfist25 2y ago
TDS going hard here

240 points by xbmcuser 2y ago

forgetfreeman 2y ago

Do we have any idea what datasets have gone missing? More details please from anyone who has them.

epakai 2y ago
You can look at catalog.data.gov it shows totals. I'm comparing January 14th to today. The biggest loss I see is Organizations - Department of Energy, 5473 to 3647. I also see under Bureaus - Energy Programs, 4347 to 2521. These are overlapping categories (-1826 on both). There are others, but they seem smaller, a few hundred at most.
- bayindirh 2y ago
  Maybe its related to climate change and advantages of renewables? I know DoE does "other things", but I don't expect these to be public anyway.
  - epakai 2y ago
    I can't tell from the Internet Archive. The datasets that went away seem to have been added quite recently, between January 8th and 14th. There isn't a suitable capture between January 8th and 25th to see which tags or categories changed within the DoE datasets.
    - ccgreg 2y ago
      Common Crawl's latest crawl was Jan 12th-25th, and the index is available.
- deleted 2y ago
  [deleted]
- epakai 2y ago
  Looking closer, there was a major increase quite recently before this decrease. On January 8th there were 3617 DoE datasets. On the 14th it was up to 5473. Now we're back down to just 30 more than on the 8th. Could it have been a publishing mistake, or some order to undo recent publications for review by the new admin?
  - refurb 2y ago
    This seems like a super important observation. If the number of dataset saw a massive jump (50%), then back down a week later, that seems more like the correction of an error.
labster 2y ago
[flagged]

Terr_ 2y ago

[flagged]

bilekas 2y ago
I'm not sure if it's nefarious or not, but I would have to lean on the side of "If the goverment is quietly removing information then probably it's not for the benefit of the public. Also from their own site at data.gov > The United States Government’s open data site is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government. The part " to inform decisions by the public and policymakers" in particular stands out. Anyway data sets is not exactly expensive to keep hosted for a government body so there's not really any excuse to scrub them unless they're polluted in some way ?
- bayindirh 2y ago
  Even if it's polluted, you can mark it as is, with reasoning and keep it up, so you can learn what dirty/bad data is, and build quality detectors with the information contained in/with the "dirty" or "low quality" data set.
bayindirh 2y ago
Open Data initiatives generally build upon the premise of immutability. Data Sets can be alive, i.e. new data can be streamed into them, but they won't be purged, shortened or slimmed down. You can expect more historical data to be opened as the digitization and organization continues, and don't expect this data to be randomly being removed from archives. Even going one step further, if this data sets are refined and formatted, they are versioned, so you can always see the previous versions (e.g.: https://www.zenodo.org https://www.zenodo.org). If you're starting to remove any data set from a data pool w/o any explanation and announcement, this means there's a new "Department of Truth" in the works, 99% of the time.
- specialist 2y ago

epakai 2y ago

It seems like some groups already expected this from efforts during the previous Trump admin. This article mentions a few, and more recent preparations: https://www.scientificamerican.com/article/scientists-scramble-to-save-climate-data-from-trump-again/ https://www.scientificamerican.com/article/scientists-scramb...

Symbiote 2y ago

I found https://uc3.cdlib.org/data-mirror/ https://uc3.cdlib.org/data-mirror/ which was a mirror a few years ago, but it's been taken down due to a lack of ongoing funding. It would be a great start to have a mirror of just the metadata, to at least see what is being removed.

trelliscoded 2y ago

That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not. I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration. I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.

duxup 2y ago
First Trump administration had a very strong pattern of talk and maybe a few actions… and then they seem to forget what they were doing and the result is little to no action. Trump loves to talk about trade but when it came to China he put up some token tariffs and announced “I didn’t say it would be easy” and then did nothing the rest of his term.
- m3047 2y ago
  Trump killed the Trans Pacific Partnership. (This was a surprise to me. I'm grateful that I'm not at risk of being extradited to rot in a chinese prison for reverse engineering some IoT POS.)
  - tzs 2y ago
    How would TPP have put you at risk for Chinese prison?
    - immibis 2y ago
      Because of the copyright enforcement section of the TPP?

1shooner 2y ago

Is there any frame of reference to compare this to? I.e what is the normal fluctuation/movement of that 'datasets available' number? For instance, it dropped by 18k between 9/13 and 10/13.

alphan0n 2y ago
The API might be a good place to start. I’m on mobile at the moment so I haven’t been able to dig too deep, but here’s [0] a query that returns the metadata for everything modified since Jan 1st (add more rows at the end if it’s cut off). I don’t think it shows deletes directly, but it can be compared to the current catalog. API documentation here [1] [0] https://catalog.data.gov/api/3/action/package_search?q=metadata_modified:[2025-01-01T00:00:00Z%20TO%202025-01-31T23:59:59Z]&rows=200 https://catalog.data.gov/api/3/action/package_search?q=metad... [1] https://data.gov/developers/apis/ https://data.gov/developers/apis/ *fixed first url

khaki54 2y ago

Complete nothingburger. The malicious action would be to subtly change key datasets to promote some kind of biased narrative. It wouldn't be the removal of less than 1% of the datasets. Comical to think the Trump admin would be the ones "scrubbing" data when they passed the OPEN Government Data Act as part of the Foundations for Evidence Based Policymaking Act at the beginning of 2019. The law that requires agencies to publish their data to the site! Trump must really be playing 4d chess if one of the first actions in office is to "scrub" a few obscure and yet unidentified datasets from data.gov.

deleted 2y ago
[deleted]
nojito 2y ago
Why would they remove any datasets? Also calling SVI obscure is quite hilarious.

diamondfist25 2y ago

TDS going hard here