9 ms·

White House Open Data Unavailable

170 points by runesoerensen 10y ago

runesoerensen 10y ago
Related tweet: "Today Trump removed all open data (9GB) from the White House https://open.whitehouse.gov/browse https://open.whitehouse.gov/browse but I grabbed it all Jan 20! Will distribute soon" https://twitter.com/denormalize/status/831581871230193664 https://twitter.com/denormalize/status/831581871230193664
- weaksauce 10y ago
  Why did trump do this? What was in the data? Could this be an error?
  - deleted 10y ago
    [deleted]
  - ajones 10y ago
    It isn't an error. If you click into one of the individual data sources (links available on the home page), you see messages like this: > This page is being updated. It will post records of White House visitors on an ongoing basis, once they become available. > This page is being updated. It will post records of White House staff salaries as the data becomes available for the Trump Administration.
    - huydotnet 10y ago
      No request sent from the browser
  - sizzzzlerz 10y ago
    I think the Russians were concerned about its availability.
  - OedipusRex 10y ago
    I'd argue that Trump probably doesn't have a clue this even exist, someone else dictated it's archival/removal.
    - wry_discontent 10y ago
      President Bannon's orders
- lucaspiller 10y ago
  If anyone wants to help out with these sorts of preservation projects https://www.reddit.com/r/Archiveteam/ https://www.reddit.com/r/Archiveteam/ and https://www.reddit.com/r/DataHoarder/ https://www.reddit.com/r/DataHoarder/ are good places to start.
  - toomuchtodo 10y ago
    #archiveteam on efnet irc EDIT: Looks like Jason Scott is aware of the WH.gov dump and has offered to ingest at the Internet Archive: https://twitter.com/textfiles/status/831585297003458560 https://twitter.com/textfiles/status/831585297003458560
glbrew 10y ago
What a douche
- sctb 10y ago
  Please comment civilly and substantively on HN or not at all. https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - glbrew 10y ago
    Ok sorry, but I actually don't think I violated any of the guidelines. As per the "civil" rule this is consistent with how I would respond in real life and given the circumstances the negativity is not gratuitous.
    - sctb 10y ago
      > When disagreeing, please reply to the argument instead of calling names. E.g. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
grzm 10y ago
Is there more to this than a page of empty results? Looking at this from a troubleshooting perspective, there are a number of things that could be causing this page besides the data itself being gone. I agree, it's something that should be looked into, but before proclaiming a purge, I'd want to know more about the situation.
- runesoerensen 10y ago
  I did consider using "No Longer Available" rather than "Purged" in the title, but all things considered I think there's overwhelming reason to believe this was intentional. Hopefully we'll soon have more information if someone looks further into this.
  - grzm 10y ago
    Some other things I'd like to know, partly from being unfamiliar with the data that would otherwise be here. How far back did the existing data go? Was there anything similar between the Bush and Obama administrations? It sounds like you might be familiar with the data that's usually there. Providing this additional context would be really helpful. but all things considered I think there's overwhelming reason to believe this was intentional. I understand where you're coming from. I also know that this kind of hyperbole plays right into the narrative that's being constructed to discredit anyone who raises concerns about the Trump administration.
  - mattdeboard 10y ago
    You considered using a less inflammatory, more accurate version of your title, but decided to go the other way.
  - massysett 10y ago
    So you recklessly decided to post something inflammatory just because you think there's "overwhelming reason to believe" it--so overwhelming that you haven't cited it? Meanwhile "hopefully" we'll have more information "if someone looks further into" it? When Trump does this the New York Times calls it a lie. He often says we "need to figure out what the hell is going on," just like your "if someone looks further into this."
problems 10y ago
Is there any reason to believe this won't just be moved to the Obama admin archive or something? Seems like it could just be part of the website change process.
- idiot_stick 10y ago
  >Seems like it could just be part of the website change process. It might be. But don't worry, if that's the case I'm sure there will be a story right at the top of HN that says, "Whoops! Trump didn't delete anything". Because we hate fake news and are not hypocrites...right?
  - jrochkind1 10y ago
    it is the top-voted comment right here before you even made your post.
    - idiot_stick 10y ago
      I shouldn't have to read the comments to determine if the headline is a lie or not.
      - jrochkind1 10y ago
        Indeed you shouldn't, I was responding to your suggestion that the HN community would ignore it if the article content or headline were determined to be inaccurate. (I _think_ that's what you were suggesting through the sarcasm?) The community already hadn't ignored it before you even made that suggestion. So the aspersions against HN's ethical or intellectual consistency were unwarranted.
sunraa 10y ago
From Jan 26th: http://web.archive.org/web/20170126072506/https://open.whitehouse.gov/browse http://web.archive.org/web/20170126072506/https://open.white... Bulk of them seem to be reports to Congress regarding staff with the odd climate change adaptation task force.
privong 10y ago
All the data appear to still be available here: https://open.obamawhitehouse.archives.gov/ https://open.obamawhitehouse.archives.gov/ I think that might just be the usual revamping of the White House website for the new administration. I spot-checked a few datasets and it looked like they are still available at the link above. Mods, a better title for this might be "White House Open Data Page Archived"? And the link changed to the above link?
- runesoerensen 10y ago
  Thanks for sharing this! The notion of archiving "Open Data" due to a change in government still seems odd considering this is factual, time-stamped data.
  - jdoliner 10y ago
    All digital properties go to a clean slate when the Presidency changes hands. There's nothing odd about this. It's similar to the physical aspects of the government, such as cabinet posts, that are directly under the Presidents purview. It being factual really has no bearing on this, although given that for data it just gets archived rather than completely purged it seems to be an incredibly reasonable treatment. The title of this post, on the other hand, is, dare I say, an alternative fact.
  - larma 10y ago
    Apparently they changed the software used to store the data, in the bottom on https://open.obamawhitehouse.archives.gov/ https://open.obamawhitehouse.archives.gov/ it says "powered by dkan", on https://open.whitehouse.gov/ https://open.whitehouse.gov/ it says "powered by socrata". Maybe it was not easily possible to migrate the data and/or it is planned to be done later.
- problems 10y ago
  Thanks, this is exactly what I suspected would happen. Same thing as the last panic when they switched the web server and pages on major Obama policies vanished. Just an entirely new webapp moved in. Defend your policies, whatever you stand for, but please don't panic over a website changing hands.
dj-wonk 10y ago
I would like to get good answers on this question: What are the U.S. Government data retention policies and rules? > As I understand it, the most applicable overarching law is the Federal Records Act. More information can be found on the Record Management FAQ. Additionally, I've found the NARA Management Guide. ... Question posted at: http://opendata.stackexchange.com/questions/10446/cross-cutting-u-s-government-data-retention-policies-and-rules http://opendata.stackexchange.com/questions/10446/cross-cutt...
rodionos 10y ago
It would actually be better for the White House datasets to be cataloged and managed via catalog.data.gov. They posted some speeches and videos back in 2014 but have been running their own show since then. https://www.data.gov/metrics https://www.data.gov/metrics Notice that White House is listed as having a total of 4 datasets.
- bb88 10y ago
  Those pretty much have press releases and other specific videos.
LyalinDotCom 10y ago
We as a community need to invest into people who are willing to take the time and duplicate all data that is "public and open source" in truly independent repo's. Its simple as that, we know from history that as times change, government changes, political policies change, the truth becomes a victim.
69mlgsniperdad 10y ago
um.. clickbait? misleading for certain.

170 points by runesoerensen 10y ago

Related tweet: "Today Trump removed all open data (9GB) from the White House https://open.whitehouse.gov/browse https://open.whitehouse.gov/browse but I grabbed it all Jan 20! Will distribute soon" https://twitter.com/denormalize/status/831581871230193664 https://twitter.com/denormalize/status/831581871230193664

weaksauce 10y ago
Why did trump do this? What was in the data? Could this be an error?
- deleted 10y ago
  [deleted]
- ajones 10y ago
  It isn't an error. If you click into one of the individual data sources (links available on the home page), you see messages like this: > This page is being updated. It will post records of White House visitors on an ongoing basis, once they become available. > This page is being updated. It will post records of White House staff salaries as the data becomes available for the Trump Administration.
  - huydotnet 10y ago
    No request sent from the browser
- sizzzzlerz 10y ago
  I think the Russians were concerned about its availability.
- OedipusRex 10y ago
  I'd argue that Trump probably doesn't have a clue this even exist, someone else dictated it's archival/removal.
  - wry_discontent 10y ago
    President Bannon's orders
lucaspiller 10y ago
If anyone wants to help out with these sorts of preservation projects https://www.reddit.com/r/Archiveteam/ https://www.reddit.com/r/Archiveteam/ and https://www.reddit.com/r/DataHoarder/ https://www.reddit.com/r/DataHoarder/ are good places to start.
- toomuchtodo 10y ago
  #archiveteam on efnet irc EDIT: Looks like Jason Scott is aware of the WH.gov dump and has offered to ingest at the Internet Archive: https://twitter.com/textfiles/status/831585297003458560 https://twitter.com/textfiles/status/831585297003458560

glbrew 10y ago

What a douche

sctb 10y ago
Please comment civilly and substantively on HN or not at all. https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- glbrew 10y ago
  Ok sorry, but I actually don't think I violated any of the guidelines. As per the "civil" rule this is consistent with how I would respond in real life and given the circumstances the negativity is not gratuitous.
  - sctb 10y ago
    > When disagreeing, please reply to the argument instead of calling names. E.g. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."

grzm 10y ago

Is there more to this than a page of empty results? Looking at this from a troubleshooting perspective, there are a number of things that could be causing this page besides the data itself being gone. I agree, it's something that should be looked into, but before proclaiming a purge, I'd want to know more about the situation.

runesoerensen 10y ago
I did consider using "No Longer Available" rather than "Purged" in the title, but all things considered I think there's overwhelming reason to believe this was intentional. Hopefully we'll soon have more information if someone looks further into this.
- grzm 10y ago
  Some other things I'd like to know, partly from being unfamiliar with the data that would otherwise be here. How far back did the existing data go? Was there anything similar between the Bush and Obama administrations? It sounds like you might be familiar with the data that's usually there. Providing this additional context would be really helpful. but all things considered I think there's overwhelming reason to believe this was intentional. I understand where you're coming from. I also know that this kind of hyperbole plays right into the narrative that's being constructed to discredit anyone who raises concerns about the Trump administration.
- mattdeboard 10y ago
  You considered using a less inflammatory, more accurate version of your title, but decided to go the other way.
- massysett 10y ago
  So you recklessly decided to post something inflammatory just because you think there's "overwhelming reason to believe" it--so overwhelming that you haven't cited it? Meanwhile "hopefully" we'll have more information "if someone looks further into" it? When Trump does this the New York Times calls it a lie. He often says we "need to figure out what the hell is going on," just like your "if someone looks further into this."

problems 10y ago

Is there any reason to believe this won't just be moved to the Obama admin archive or something? Seems like it could just be part of the website change process.

idiot_stick 10y ago
>Seems like it could just be part of the website change process. It might be. But don't worry, if that's the case I'm sure there will be a story right at the top of HN that says, "Whoops! Trump didn't delete anything". Because we hate fake news and are not hypocrites...right?
- jrochkind1 10y ago
  it is the top-voted comment right here before you even made your post.
  - idiot_stick 10y ago
    I shouldn't have to read the comments to determine if the headline is a lie or not.
    - jrochkind1 10y ago
      Indeed you shouldn't, I was responding to your suggestion that the HN community would ignore it if the article content or headline were determined to be inaccurate. (I _think_ that's what you were suggesting through the sarcasm?) The community already hadn't ignored it before you even made that suggestion. So the aspersions against HN's ethical or intellectual consistency were unwarranted.

sunraa 10y ago

From Jan 26th: http://web.archive.org/web/20170126072506/https://open.whitehouse.gov/browse http://web.archive.org/web/20170126072506/https://open.white... Bulk of them seem to be reports to Congress regarding staff with the odd climate change adaptation task force.

privong 10y ago

All the data appear to still be available here: https://open.obamawhitehouse.archives.gov/ https://open.obamawhitehouse.archives.gov/ I think that might just be the usual revamping of the White House website for the new administration. I spot-checked a few datasets and it looked like they are still available at the link above. Mods, a better title for this might be "White House Open Data Page Archived"? And the link changed to the above link?

runesoerensen 10y ago
Thanks for sharing this! The notion of archiving "Open Data" due to a change in government still seems odd considering this is factual, time-stamped data.
- jdoliner 10y ago
  All digital properties go to a clean slate when the Presidency changes hands. There's nothing odd about this. It's similar to the physical aspects of the government, such as cabinet posts, that are directly under the Presidents purview. It being factual really has no bearing on this, although given that for data it just gets archived rather than completely purged it seems to be an incredibly reasonable treatment. The title of this post, on the other hand, is, dare I say, an alternative fact.
- larma 10y ago
  Apparently they changed the software used to store the data, in the bottom on https://open.obamawhitehouse.archives.gov/ https://open.obamawhitehouse.archives.gov/ it says "powered by dkan", on https://open.whitehouse.gov/ https://open.whitehouse.gov/ it says "powered by socrata". Maybe it was not easily possible to migrate the data and/or it is planned to be done later.
problems 10y ago
Thanks, this is exactly what I suspected would happen. Same thing as the last panic when they switched the web server and pages on major Obama policies vanished. Just an entirely new webapp moved in. Defend your policies, whatever you stand for, but please don't panic over a website changing hands.

dj-wonk 10y ago

I would like to get good answers on this question: What are the U.S. Government data retention policies and rules? > As I understand it, the most applicable overarching law is the Federal Records Act. More information can be found on the Record Management FAQ. Additionally, I've found the NARA Management Guide. ... Question posted at: http://opendata.stackexchange.com/questions/10446/cross-cutting-u-s-government-data-retention-policies-and-rules http://opendata.stackexchange.com/questions/10446/cross-cutt...

rodionos 10y ago

It would actually be better for the White House datasets to be cataloged and managed via catalog.data.gov. They posted some speeches and videos back in 2014 but have been running their own show since then. https://www.data.gov/metrics https://www.data.gov/metrics Notice that White House is listed as having a total of 4 datasets.

bb88 10y ago
Those pretty much have press releases and other specific videos.

LyalinDotCom 10y ago

We as a community need to invest into people who are willing to take the time and duplicate all data that is "public and open source" in truly independent repo's. Its simple as that, we know from history that as times change, government changes, political policies change, the truth becomes a victim.

69mlgsniperdad 10y ago

um.. clickbait? misleading for certain.