5 ms·

U.S. government sets record for failures to find files when asked

142 points by ZoeZoeBee 11y ago

HillaryBriss 11y ago
From the article: "In some high-profile instances, usually after news organizations filed expensive federal lawsuits, the Obama administration found tens of thousands of pages after it previously said it couldn't find any." One explanation is that the administration is being dishonest. Another is that the administration is just inefficient, sloppy and slow at handling these kinds of requests. Obama has taken a "civil rights action hero" approach to being president. A different president might have taken an "efficient administrator and process reformer emphasizing high quality service" approach to being president. But that sounds pretty boring.
- sgift 11y ago
  > Another is that the administration is just inefficient, sloppy and slow at handling these kinds of requests. Which hints at a profound lack of preparation on their part. Basically, they act like a company that when asked before a merger to provide their paperwork for due dilligance answers "Um, oh .. eh .. we didn't think that you need this. We will go and find all that we need and then get back to you, okay?" with the difference that the company in question has a real incentive to get all the papers, but still could have avoided this situation if their initial "paper creation" process included "archive papers so that they can be retrieved without hassle" I wonder if forcing/automating such an archiving at creation/update of documentation could solve this problem.
  - mkhpalm 11y ago
    It certainly couldn't hurt... but I believe it would still be hard to find certain things. An example of that is taxes, accounting, or just general budgeting. Depends where the incentives are as to how easily findable anything will ever be.
- ams6110 11y ago
  A different president might have taken an "efficient administrator and process reformer emphasizing high quality service" approach to being president. But that sounds pretty boring. Also sounds a lot closer to what the job of an executive really is.
- sehugg 11y ago
  There were also a record number of FOIA lawsuits last year: http://dailycaller.com/2016/01/06/record-number-of-foia-lawsuits-filed-against-obama/ http://dailycaller.com/2016/01/06/record-number-of-foia-laws...
tyingq 11y ago
I would guess some of it would be the result of either vague, or overly broad requests as well. I found an amusing response to a request like that here: http://www.mackinac.org/media/images/2009/7FOIAb.jpg http://www.mackinac.org/media/images/2009/7FOIAb.jpg To me, that's a reasonable response to an overly broad request.
- Spooky23 11y ago
  Not at all, it's pure bullshit designed to intimidate the request or. State agencies, especially police agencies, are laser focused on external funding sources and track federal grant monies very carefully. Likewise, the Feds require exacting tracking of assets purchased. Usually there is an office of 6-12 people dedicated to those grants.
  - tyingq 11y ago
    I see we disagree, but in this case, the language of the request seems very broad to me. It clearly pulls in related documents that may not even mention a grant directly. That language is quoted in the reply.
    - ZoeZoeBee 11y ago
      >The website Gawker sued the State Department last year after it said it couldn't find any emails that Philippe Reines, an aide to Hillary Clinton and former deputy assistant secretary of state, had sent to journalists. After the lawsuit, the agency said it found 90,000 documents about correspondence between Reines and reporters. Broadness of Scope is the Only way to deal with Agencies attempting to obfuscate data and documents. --tyingq: My Apologies if it was not you, I'd be interested in knowing how people believe pertinent documents can be researched if not for a broad net of related documents, when the succinct question to be asked is unable to be known due to completely hidden documents and "missing information"
      - tyingq 11y ago
        "You could respond instead of downvoting" I did not downvote. In fact, I don't think you can downvote direct responses to your own comments, can you? Edit: >> I'd be interested in knowing how people believe pertinent documents can be researched if not for a broad net of related documents Well, there's broad (90,000 documents, your example) and really broad (2 million documents, my example).
dbg31415 11y ago
https://www.youtube.com/watch?v=CU0m6Rxm9vU https://www.youtube.com/watch?v=CU0m6Rxm9vU
danso 11y ago
Obama's administration is rightfully getting increased scrutiny over FOIA...the day after his inaugural address, he made a very clear statement that FOIA "should be administered with a clear presumption: In the face of doubt, openness prevails." [0] (ProPublica recently published a new retrospective here [0a]) In terms of being data-oriented, Obama's administration has been very forward thinking, IMO...it's one thing to release White House visitor logs under threat of lawsuit, it's another to go the extra mile and throw it on a data portal where it can be easily searched and downloaded as bulk data, along with other data such as staff salaries [1]. You could argue that that's just the way things would be for any administration due to the advent of the Web...but only if you didn't look at how the Senate distributes their required disclosures. But being more open-minded to releasing machine-readable data and responding to FOIAs are a different thing...just as Obama can make an argument that he's been very pro-civil-liberties, and yet his administration has prosecuted more federal whistleblowers than all other administrations combined [3]. Maybe I missed it...but it seems like the AP goes out of its way to avoid any use of percentages for comparison between administrations...Maybe this was not tracked in the same way during the Bush administration? That said, I wouldn't be surprised if Obama led in both raw numbers and percentages of denials. Though it's still difficult to know how apples to apples that is. I had to Google around quite a bit to find any reference to Bush numbers...here's an old AP article I found linked to from an old Slashdot thread [4]: > - In all, the agencies reviewed by the AP reported getting 444,924 FOIA requests in fiscal 2009, compared with 493,610 in fiscal 2008. Besides those cases, they started each year with FOIA requests pending: 145,162 as the 2009 budget year began last October; 156,611 as the 2008 fiscal period started. > - The agencies processed 501,158 FOIA requests in the 2009 period, compared with 504,110 the previous year. > Much of the Obama administration’s early effort on FOIA seems to have been aimed at clearing out the backlog of old cases: The number of requests still sitting around past the time limits spelled out in the open-records law fell from 124,019 in budget year 2008 to 67,764 at the end of the most recent budget year over the 17 agencies, the AP’s review found > - The agencies reviewed provided everything sought in FOIA requests in at least 162,205 cases last fiscal year, compared with 196,776 the previous year. > - They denied FOIA requests in their entirety based on exemptions 20,005 times last fiscal year, compared with 21,057 times the previous year. [0] https://www.propublica.org/article/has-obama-kept-his-open-government-pledge https://www.propublica.org/article/has-obama-kept-his-open-g... [0a] https://www.propublica.org/article/trying-to-get-records-from-most-transparent-administration-ever https://www.propublica.org/article/trying-to-get-records-fro... [1] https://open.whitehouse.gov/browse https://open.whitehouse.gov/browse [2] https://sunlightfoundation.com/blog/2014/08/05/now-its-easier-to-account-for-how-the-senate-spends-your-money/ https://sunlightfoundation.com/blog/2014/08/05/now-its-easie... [3] http://www.politifact.com/punditfact/statements/2014/jan/10/jake-tapper/cnns-tapper-obama-has-used-espionage-act-more-all-/ http://www.politifact.com/punditfact/statements/2014/jan/10/... [4] http://www.seattletimes.com/seattle-news/politics/ap-analysis-of-obama-foia-record-at-a-glance/ http://www.seattletimes.com/seattle-news/politics/ap-analysi...
Spooky23 11y ago
The government at all levels is terrified of allowing employees to talk to anyone outside of their agency. If you look at news reports at the federal, state, local levels of government from 1990 to today, you'll see that quotes that would come from a local official in command in 1990 are now issued from a PR flack at HQ. I witnessed one example first hand where a school principal, terrified of being quoted about a book sale, frantically directed a person who turned out to be a reporter to the school district public affairs officer. FOIA is similar -- these searches used to be distributed but in many cases are now all conducted by counsel at HQ, and they limit the scope of their search to the official repositories that may or may not be relevant. If you want actual information from a FOIA process, focus your search on email, which is usually centralized, and jump off on addition requests from there.
- x5n1 11y ago
  It's censorship plain and simple. Instead of denying information, they simply deny access. And if they do provide access, they provide it through P/R who knows pretty much nothing so can't really say anything. If they ever do provide access you better behave yourself or they will never talk to you again. It's scummy totalitarian control of information while at the same time trying to extort as much information out of the public as possible... in the name of terrorism. Terrorism that their shit policies are at least partially responsible for.
  - deleted 11y ago
    [deleted]
- istionsmer 11y ago
  Based on 3+ years of employment with a large federal agency, what you are saying ins't correct. I obviously cannot speak for every federal agency. However, at my organization FOIAs are routinely distributed to individuals/divisions that are likely to have pertinent records. The main thing that we are required to search are e-mail inboxes (to include archived e-mails) and local paper/digital files (e.g., paper documents and electronic folders). If there are other databases/systems that an organization owns the records for, then those must be searched as well. There are three main problems with the way FOIA requests are fulfilled. 1) They are incredibly time consuming to do properly, 2) Very few additional resources are given in order to alleviate the time consuming process, and 3) the process for tasking and responding to requests is grossly inefficient. The requests are literally sent out via e-mail and responses back are sent via e-mail attachment. To paint the picture a little more clearly, let's say someone submits a FOIA request for all records related to the "Boston Bombing". For a broad search like that, hundreds if not thousands of people will be searching through their records. Those hundreds/thousands of people each find five documents which they have to send back to the agency's FOIA office via e-mail attachments. Most of the records are duplicates (e.g., an e-mail sent out to a distribution list), but everyone still has to send their record in. Let's just say it is a very frustrating process for everyone involved. I hope it doesn't sound as though I am making excuses - quite the opposite. Merely pointing out that while there are definitely issues that affect accuracy, they might not be the issues everyone assumes. Perhaps this is a sector that is in need of some innovation/disruption =).
skywhopper 11y ago
I'll be the first to agree that the Obama administration most definitely went against the spirit of the 2008 campaign with its lockdown of certain secrets. That said, I find the headline number to be completely useless without comparison data. What was the ratio under Bush, Clinton, Bush, Reagan, Carter, and Ford? Also need to see trends of number and type of requests over that time. And who's making the requests and what are they for? Are we talking about requests for troop locations (reasonably rejected), Obama's Kenyan birth certificate (spurious and nonexistent), or investigation results in re torture and the financial crisis (legitimate and embarrassing)?
- Shivetya 11y ago
  I am certainly not one to defend this Administration but it is very easy to declare something while running for office as well as when you initially take office. Then reality hits. Pretty much your ideals if they did exist will be buried under the bureaucracy. It isn't as if he gets to decide what is released and what isn't. Some functionary is making that determination and likely going the safe route each time. Even if he did make a decision prior agreements with others can put a block in. BTW, a wonderful site http://www.foia.gov/data.html http://www.foia.gov/data.html
  - mindslight 11y ago
    What a charitable way of stating that campaigns are empty marketing, and that after the election only serving the status quo matters. I just wish people had long term memory, so they'd stop serving as enabling patsies.
    - bdcravens 11y ago
      Perhaps this is important to remember in the coming months. For example, if you look at Cruz's first day promises, for example, it's ridiculous. The scary thing is that even though he's a racist hate-mongerer, Donald Trump might be the most honest one on either side.
      - 11y ago
dang 11y ago
It breaks the HN guidelines to rewrite titles to editorialize them, so please don't do that. (Submitted title was "Obama's Lack of Transprency: Censored or Denied 77% of FOIA Requests".)
hsnewman 11y ago
Well, at least we are #1, nobody's worst...

142 points by ZoeZoeBee 11y ago

From the article: "In some high-profile instances, usually after news organizations filed expensive federal lawsuits, the Obama administration found tens of thousands of pages after it previously said it couldn't find any." One explanation is that the administration is being dishonest. Another is that the administration is just inefficient, sloppy and slow at handling these kinds of requests. Obama has taken a "civil rights action hero" approach to being president. A different president might have taken an "efficient administrator and process reformer emphasizing high quality service" approach to being president. But that sounds pretty boring.

sgift 11y ago
> Another is that the administration is just inefficient, sloppy and slow at handling these kinds of requests. Which hints at a profound lack of preparation on their part. Basically, they act like a company that when asked before a merger to provide their paperwork for due dilligance answers "Um, oh .. eh .. we didn't think that you need this. We will go and find all that we need and then get back to you, okay?" with the difference that the company in question has a real incentive to get all the papers, but still could have avoided this situation if their initial "paper creation" process included "archive papers so that they can be retrieved without hassle" I wonder if forcing/automating such an archiving at creation/update of documentation could solve this problem.
- mkhpalm 11y ago
  It certainly couldn't hurt... but I believe it would still be hard to find certain things. An example of that is taxes, accounting, or just general budgeting. Depends where the incentives are as to how easily findable anything will ever be.
ams6110 11y ago
A different president might have taken an "efficient administrator and process reformer emphasizing high quality service" approach to being president. But that sounds pretty boring. Also sounds a lot closer to what the job of an executive really is.
sehugg 11y ago
There were also a record number of FOIA lawsuits last year: http://dailycaller.com/2016/01/06/record-number-of-foia-lawsuits-filed-against-obama/ http://dailycaller.com/2016/01/06/record-number-of-foia-laws...

tyingq 11y ago

I would guess some of it would be the result of either vague, or overly broad requests as well. I found an amusing response to a request like that here: http://www.mackinac.org/media/images/2009/7FOIAb.jpg http://www.mackinac.org/media/images/2009/7FOIAb.jpg To me, that's a reasonable response to an overly broad request.

Spooky23 11y ago
Not at all, it's pure bullshit designed to intimidate the request or. State agencies, especially police agencies, are laser focused on external funding sources and track federal grant monies very carefully. Likewise, the Feds require exacting tracking of assets purchased. Usually there is an office of 6-12 people dedicated to those grants.
- tyingq 11y ago
  I see we disagree, but in this case, the language of the request seems very broad to me. It clearly pulls in related documents that may not even mention a grant directly. That language is quoted in the reply.
  - ZoeZoeBee 11y ago
    >The website Gawker sued the State Department last year after it said it couldn't find any emails that Philippe Reines, an aide to Hillary Clinton and former deputy assistant secretary of state, had sent to journalists. After the lawsuit, the agency said it found 90,000 documents about correspondence between Reines and reporters. Broadness of Scope is the Only way to deal with Agencies attempting to obfuscate data and documents. --tyingq: My Apologies if it was not you, I'd be interested in knowing how people believe pertinent documents can be researched if not for a broad net of related documents, when the succinct question to be asked is unable to be known due to completely hidden documents and "missing information"
    - tyingq 11y ago
      "You could respond instead of downvoting" I did not downvote. In fact, I don't think you can downvote direct responses to your own comments, can you? Edit: >> I'd be interested in knowing how people believe pertinent documents can be researched if not for a broad net of related documents Well, there's broad (90,000 documents, your example) and really broad (2 million documents, my example).

dbg31415 11y ago

https://www.youtube.com/watch?v=CU0m6Rxm9vU https://www.youtube.com/watch?v=CU0m6Rxm9vU

danso 11y ago

Obama's administration is rightfully getting increased scrutiny over FOIA...the day after his inaugural address, he made a very clear statement that FOIA "should be administered with a clear presumption: In the face of doubt, openness prevails." [0] (ProPublica recently published a new retrospective here [0a]) In terms of being data-oriented, Obama's administration has been very forward thinking, IMO...it's one thing to release White House visitor logs under threat of lawsuit, it's another to go the extra mile and throw it on a data portal where it can be easily searched and downloaded as bulk data, along with other data such as staff salaries [1]. You could argue that that's just the way things would be for any administration due to the advent of the Web...but only if you didn't look at how the Senate distributes their required disclosures. But being more open-minded to releasing machine-readable data and responding to FOIAs are a different thing...just as Obama can make an argument that he's been very pro-civil-liberties, and yet his administration has prosecuted more federal whistleblowers than all other administrations combined [3]. Maybe I missed it...but it seems like the AP goes out of its way to avoid any use of percentages for comparison between administrations...Maybe this was not tracked in the same way during the Bush administration? That said, I wouldn't be surprised if Obama led in both raw numbers and percentages of denials. Though it's still difficult to know how apples to apples that is. I had to Google around quite a bit to find any reference to Bush numbers...here's an old AP article I found linked to from an old Slashdot thread [4]: > - In all, the agencies reviewed by the AP reported getting 444,924 FOIA requests in fiscal 2009, compared with 493,610 in fiscal 2008. Besides those cases, they started each year with FOIA requests pending: 145,162 as the 2009 budget year began last October; 156,611 as the 2008 fiscal period started. > - The agencies processed 501,158 FOIA requests in the 2009 period, compared with 504,110 the previous year. > Much of the Obama administration’s early effort on FOIA seems to have been aimed at clearing out the backlog of old cases: The number of requests still sitting around past the time limits spelled out in the open-records law fell from 124,019 in budget year 2008 to 67,764 at the end of the most recent budget year over the 17 agencies, the AP’s review found > - The agencies reviewed provided everything sought in FOIA requests in at least 162,205 cases last fiscal year, compared with 196,776 the previous year. > - They denied FOIA requests in their entirety based on exemptions 20,005 times last fiscal year, compared with 21,057 times the previous year. [0] https://www.propublica.org/article/has-obama-kept-his-open-government-pledge https://www.propublica.org/article/has-obama-kept-his-open-g... [0a] https://www.propublica.org/article/trying-to-get-records-from-most-transparent-administration-ever https://www.propublica.org/article/trying-to-get-records-fro... [1] https://open.whitehouse.gov/browse https://open.whitehouse.gov/browse [2] https://sunlightfoundation.com/blog/2014/08/05/now-its-easier-to-account-for-how-the-senate-spends-your-money/ https://sunlightfoundation.com/blog/2014/08/05/now-its-easie... [3] http://www.politifact.com/punditfact/statements/2014/jan/10/jake-tapper/cnns-tapper-obama-has-used-espionage-act-more-all-/ http://www.politifact.com/punditfact/statements/2014/jan/10/... [4] http://www.seattletimes.com/seattle-news/politics/ap-analysis-of-obama-foia-record-at-a-glance/ http://www.seattletimes.com/seattle-news/politics/ap-analysi...

Spooky23 11y ago

The government at all levels is terrified of allowing employees to talk to anyone outside of their agency. If you look at news reports at the federal, state, local levels of government from 1990 to today, you'll see that quotes that would come from a local official in command in 1990 are now issued from a PR flack at HQ. I witnessed one example first hand where a school principal, terrified of being quoted about a book sale, frantically directed a person who turned out to be a reporter to the school district public affairs officer. FOIA is similar -- these searches used to be distributed but in many cases are now all conducted by counsel at HQ, and they limit the scope of their search to the official repositories that may or may not be relevant. If you want actual information from a FOIA process, focus your search on email, which is usually centralized, and jump off on addition requests from there.

x5n1 11y ago
It's censorship plain and simple. Instead of denying information, they simply deny access. And if they do provide access, they provide it through P/R who knows pretty much nothing so can't really say anything. If they ever do provide access you better behave yourself or they will never talk to you again. It's scummy totalitarian control of information while at the same time trying to extort as much information out of the public as possible... in the name of terrorism. Terrorism that their shit policies are at least partially responsible for.
- deleted 11y ago
  [deleted]
istionsmer 11y ago
Based on 3+ years of employment with a large federal agency, what you are saying ins't correct. I obviously cannot speak for every federal agency. However, at my organization FOIAs are routinely distributed to individuals/divisions that are likely to have pertinent records. The main thing that we are required to search are e-mail inboxes (to include archived e-mails) and local paper/digital files (e.g., paper documents and electronic folders). If there are other databases/systems that an organization owns the records for, then those must be searched as well. There are three main problems with the way FOIA requests are fulfilled. 1) They are incredibly time consuming to do properly, 2) Very few additional resources are given in order to alleviate the time consuming process, and 3) the process for tasking and responding to requests is grossly inefficient. The requests are literally sent out via e-mail and responses back are sent via e-mail attachment. To paint the picture a little more clearly, let's say someone submits a FOIA request for all records related to the "Boston Bombing". For a broad search like that, hundreds if not thousands of people will be searching through their records. Those hundreds/thousands of people each find five documents which they have to send back to the agency's FOIA office via e-mail attachments. Most of the records are duplicates (e.g., an e-mail sent out to a distribution list), but everyone still has to send their record in. Let's just say it is a very frustrating process for everyone involved. I hope it doesn't sound as though I am making excuses - quite the opposite. Merely pointing out that while there are definitely issues that affect accuracy, they might not be the issues everyone assumes. Perhaps this is a sector that is in need of some innovation/disruption =).

skywhopper 11y ago

I'll be the first to agree that the Obama administration most definitely went against the spirit of the 2008 campaign with its lockdown of certain secrets. That said, I find the headline number to be completely useless without comparison data. What was the ratio under Bush, Clinton, Bush, Reagan, Carter, and Ford? Also need to see trends of number and type of requests over that time. And who's making the requests and what are they for? Are we talking about requests for troop locations (reasonably rejected), Obama's Kenyan birth certificate (spurious and nonexistent), or investigation results in re torture and the financial crisis (legitimate and embarrassing)?

Shivetya 11y ago
I am certainly not one to defend this Administration but it is very easy to declare something while running for office as well as when you initially take office. Then reality hits. Pretty much your ideals if they did exist will be buried under the bureaucracy. It isn't as if he gets to decide what is released and what isn't. Some functionary is making that determination and likely going the safe route each time. Even if he did make a decision prior agreements with others can put a block in. BTW, a wonderful site http://www.foia.gov/data.html http://www.foia.gov/data.html
- mindslight 11y ago
  What a charitable way of stating that campaigns are empty marketing, and that after the election only serving the status quo matters. I just wish people had long term memory, so they'd stop serving as enabling patsies.
  - bdcravens 11y ago
    Perhaps this is important to remember in the coming months. For example, if you look at Cruz's first day promises, for example, it's ridiculous. The scary thing is that even though he's a racist hate-mongerer, Donald Trump might be the most honest one on either side.
    - 11y ago

dang 11y ago

It breaks the HN guidelines to rewrite titles to editorialize them, so please don't do that. (Submitted title was "Obama's Lack of Transprency: Censored or Denied 77% of FOIA Requests".)

hsnewman 11y ago

Well, at least we are #1, nobody's worst...