5 ms·

I can't think of anything more obtuse than releasing a database as a hundred plus page PDF. It seems like the bureaucrats responding to these requests are spec

I can't think of anything more obtuse than releasing a database as a hundred plus page PDF. It seems like the bureaucrats responding to these requests are specifically trained to not release information.

incision 12y ago
>'It seems like the bureaucrats responding to these requests are specifically trained to not release information.' They are. I have witnessed someone reprimanded specifically for 'making it too easy' by providing an Excel sheet instead of printing the same to a PDF.
- shard972 12y ago
  What can be done about this? Can we call our congressmen or something?
  - TheCoelacanth 12y ago
    This is about the UK, so you should probably call your MP instead.
anigbrowl 12y ago
They want to minimize the amount of time they spend on the FOIA requests, because it comes out of their operating budget. So they select the releasable records and then print them to pdf, where they can do any necessary redaction before release. Exporting them into another database where they can be manipulated freely is beyond the skills of most database users, and requires an expensive database administrator, who will have to have meetings with the administrator of the original database, which might result in travel requests, expenses...at which point most organizations will invoke the clause in FOIA rules (at least in America) of charging the requestor the reasonable costs of the data retrieval, which in a case like this will be substantial - at least a few thousand. Bear in mind that that FOIA requests are rarely so broad in scope as to justify the release of an entire database; most requests are for a subset of information about particular individuals, timeframes, political administrations or suchlike. I often dig through stuff like this in PDF format (because I enjoy reading court cases), and while I can't say I like it, if I really need to get the data out I set up some sort of scraping macro. Most people want something in human readable form and I suspect that if they were given an optical disc with an SQL dtabase on it, or even an XML file, they'd complain about the impenetrability of it. Perhaps you could suggest some good candidates for release formats that met the criteria of being easily accessible/convertible as well as being manipulable and easy to produce.
- jon-wood 12y ago
  > Perhaps you could suggest some good candidates for release formats that met the criteria of being easily accessible/convertible as well as being manipulable and easy to produce. The entire case here was around the requestor specifying he'd like the data in Excel format (although presumably CSV would also be acceptable). I can't think of many databases which can't export query results to CSV, or indeed many civil servants who are so completely inept with a computer that they couldn't redact information as (or more) easily in Excel than they could with a PDF.
  - incision 12y ago
    >'I can't think of many databases which can't export query results to CSV...' This is pretty much irrelevant as these requests are not going to end up in the hands of a DBA, or anyone with anything approaching direct access to the databases in question. It's going to end up with someone who has an adversarial relationship with their own IT department as a 'maker of work'. Bureaucracy isn't something these agencies foist on the public, it just as bad if not worse inside. The request will go up and down through a series of managers, directors and deputies before being put on the schedule for a change control board meeting. In that meeting, a representative of the database administrators will explain that this request is not only complex, but potentially quite dangerous. He will point out that required data is actually completely accessible via a clunky mainframe interface wrapped in Java. Some plebe in the receiving office of the request will end up clicking through that Java interface and printing/exporting the requested data one page at a time. This process will repeat itself until such requests are a weekly if not daily occurrence at which point the plebe might be given a dedicated interface for exporting such requests. >'...or indeed many civil servants who are so completely inept with a computer that they couldn't redact information as (or more) easily in Excel than they could with a PDF.' I take it you've not met many civil servants then? In any case, the mechanics aren't really the issue. It's the game of hot potato behind the scenes and the eschewing of responsibility that takes place at all levels. I can virtually guarantee that even if all the stars aligned such that someone did obtain and redact a fully electronic copy of such data it would be rejected for release due to the 'unfamiliar' format. Someone in the chain would fear that they couldn't be sure of what they were releasing because it's not in the specific format they're used to.
eli 12y ago
FWIW, in some US states you can request documents be provided in electronic formats. The Federal FOIA law is fairly weak.
allegory 12y ago
In the 1980s they did source escrow on paper. This meant printing out several boxes of fanfold paper for a medium sized project and shipping them to the escrow company. Imagine typing all that back in...
vetinari 12y ago
I can imagine something even more obtuse: have the pdfs contain bitmap renderings of the pages, so you would need not to parse them, but OCR them... Unfortunately, this is not my wild imagination, but this happened in this reality. Sad, I know.
- PeterisP 12y ago
  A classic practice is to archive pdf's of printed and scanned documents, instead the source digital documents - based on the concept that you need to have any signatures, approval stamps or notes/amendments as they might be on the actual document. So instead of an excel worksheet you get a scanned picture that's possibly even unusable for OCR if someone has a wide signature and some line(s) of it go inside the actual table.
DanBC 12y ago
I requested all my data from a department of my local council. Someone print-screened all the screens of their database with my information. I had several hundred pages of badly printed black and white pages of generic enterprise database screens. > It seems like the bureaucrats responding to these requests are specifically trained to not release information. I kind of wish that were true, because maliciousness is somehow more comforting than incompetence.

incision 12y ago

>'It seems like the bureaucrats responding to these requests are specifically trained to not release information.' They are. I have witnessed someone reprimanded specifically for 'making it too easy' by providing an Excel sheet instead of printing the same to a PDF.

shard972 12y ago
What can be done about this? Can we call our congressmen or something?
- TheCoelacanth 12y ago
  This is about the UK, so you should probably call your MP instead.

anigbrowl 12y ago

They want to minimize the amount of time they spend on the FOIA requests, because it comes out of their operating budget. So they select the releasable records and then print them to pdf, where they can do any necessary redaction before release. Exporting them into another database where they can be manipulated freely is beyond the skills of most database users, and requires an expensive database administrator, who will have to have meetings with the administrator of the original database, which might result in travel requests, expenses...at which point most organizations will invoke the clause in FOIA rules (at least in America) of charging the requestor the reasonable costs of the data retrieval, which in a case like this will be substantial - at least a few thousand. Bear in mind that that FOIA requests are rarely so broad in scope as to justify the release of an entire database; most requests are for a subset of information about particular individuals, timeframes, political administrations or suchlike. I often dig through stuff like this in PDF format (because I enjoy reading court cases), and while I can't say I like it, if I really need to get the data out I set up some sort of scraping macro. Most people want something in human readable form and I suspect that if they were given an optical disc with an SQL dtabase on it, or even an XML file, they'd complain about the impenetrability of it. Perhaps you could suggest some good candidates for release formats that met the criteria of being easily accessible/convertible as well as being manipulable and easy to produce.

jon-wood 12y ago
> Perhaps you could suggest some good candidates for release formats that met the criteria of being easily accessible/convertible as well as being manipulable and easy to produce. The entire case here was around the requestor specifying he'd like the data in Excel format (although presumably CSV would also be acceptable). I can't think of many databases which can't export query results to CSV, or indeed many civil servants who are so completely inept with a computer that they couldn't redact information as (or more) easily in Excel than they could with a PDF.
- incision 12y ago
  >'I can't think of many databases which can't export query results to CSV...' This is pretty much irrelevant as these requests are not going to end up in the hands of a DBA, or anyone with anything approaching direct access to the databases in question. It's going to end up with someone who has an adversarial relationship with their own IT department as a 'maker of work'. Bureaucracy isn't something these agencies foist on the public, it just as bad if not worse inside. The request will go up and down through a series of managers, directors and deputies before being put on the schedule for a change control board meeting. In that meeting, a representative of the database administrators will explain that this request is not only complex, but potentially quite dangerous. He will point out that required data is actually completely accessible via a clunky mainframe interface wrapped in Java. Some plebe in the receiving office of the request will end up clicking through that Java interface and printing/exporting the requested data one page at a time. This process will repeat itself until such requests are a weekly if not daily occurrence at which point the plebe might be given a dedicated interface for exporting such requests. >'...or indeed many civil servants who are so completely inept with a computer that they couldn't redact information as (or more) easily in Excel than they could with a PDF.' I take it you've not met many civil servants then? In any case, the mechanics aren't really the issue. It's the game of hot potato behind the scenes and the eschewing of responsibility that takes place at all levels. I can virtually guarantee that even if all the stars aligned such that someone did obtain and redact a fully electronic copy of such data it would be rejected for release due to the 'unfamiliar' format. Someone in the chain would fear that they couldn't be sure of what they were releasing because it's not in the specific format they're used to.

eli 12y ago

FWIW, in some US states you can request documents be provided in electronic formats. The Federal FOIA law is fairly weak.

allegory 12y ago

In the 1980s they did source escrow on paper. This meant printing out several boxes of fanfold paper for a medium sized project and shipping them to the escrow company. Imagine typing all that back in...

vetinari 12y ago

I can imagine something even more obtuse: have the pdfs contain bitmap renderings of the pages, so you would need not to parse them, but OCR them... Unfortunately, this is not my wild imagination, but this happened in this reality. Sad, I know.

PeterisP 12y ago
A classic practice is to archive pdf's of printed and scanned documents, instead the source digital documents - based on the concept that you need to have any signatures, approval stamps or notes/amendments as they might be on the actual document. So instead of an excel worksheet you get a scanned picture that's possibly even unusable for OCR if someone has a wide signature and some line(s) of it go inside the actual table.

DanBC 12y ago

I requested all my data from a department of my local council. Someone print-screened all the screens of their database with my information. I had several hundred pages of badly printed black and white pages of generic enterprise database screens. > It seems like the bureaucrats responding to these requests are specifically trained to not release information. I kind of wish that were true, because maliciousness is somehow more comforting than incompetence.