26 ms·

(disclaimer: I own a non-profit for submitting requests, education, and advocating the release of digital records) I seriously recommend that everyone submit

(disclaimer: I own a non-profit for submitting requests, education, and advocating the release of digital records) I seriously recommend that everyone submit a FOIA request for information that they're interested in and make it public. Either locally or federally. You'll learn a lot about the interface between you and your government. Something to think about - Illinois FOIA allows the submission of code within a FOIA request. For example, as long as you know the schema of a database, you can submit SQL for a government owned/contracted database. The schema is often included in contract specs, but more often requires a request to figure it out. My nfp is working on some projects (through muckrock) to build out a list of schemas across all states' big cities. It's possible, but there's still a lot of work to do!

paulie_a 8y ago
Chicago also has a open data initieve with tons of large datasets that is publicly available.
- bpchaps 8y ago
  Yep [1]! It's pretty damn impressive, but it's by no means a complete list of the data Chicago stores. Which is where FOIA comes into the picture. According to Chicago's (ex) Chief Data Officer, they also have a fairly routine process of scraping through Chicago's FOIA requests for gauging interest for new datasets. In other (somewhat naive) words - the more data-like FOIA requests that Chicago receives, the more open data will be made available. On a similar note, federal FOIA has some built-in "proactive disclosure" language that makes information publicly available after it's been requested more than three times, or is likely to be requested often or with interest. Not an expert on this piece by any means, but it's something to consider when thinking about open data portals and the like. Also, check out Data USA's (exceptionally bloated) data portal [2], which includes a lot of data throughout portals throughout the US. [1] https://data.cityofchicago.org https://data.cityofchicago.org [2] https://datausa.io/ https://datausa.io/
danso 8y ago
> you can submit SQL for a government owned/contracted database I’ve heard of journalists working with government IT folks to craft a SQL query, but not sending code in the records request. Do you by chance have an example letter that uses code?
- bpchaps 8y ago
  I just made sent one: https://www.muckrock.com/foi/chicago-169/canvas-database-schema-60175/ https://www.muckrock.com/foi/chicago-169/canvas-database-sch... The idea comes from [1], where like you say, a journalist/requester can work with the government IT folk to craft a SQL query. A big problem comes, though, from the burden and staffing required to write the query. To add to that, FOIA often requires cost estimates prior to starting a request - which, in the case of writing code, can muddle estimations... which have to be paid on the onset. My position is that if the requester understands the schema, then the burden required to write the query is essentially nullified. That way, the only cost would then be the review of the query, which is trivial in comparison to writing one from scratch. There's a lot of prereq knowledge needed before submitting a request for code. That, and there's generally a potential for penalty against a public body that doesn't run a full search. So in many ways, there isn't really a need for a requester to submit their own SQL from the onset. [1] http://illinoiscourts.gov/Opinions/AppellateCourt/2018/2ndDistrict/2170617.pdf http://illinoiscourts.gov/Opinions/AppellateCourt/2018/2ndDi... (not a laywer, etc)
  - danso 8y ago
    Thanks!
  - toomuchtodo 8y ago
    Is it not possible to request an electronic copy of the database dump in its entirety?
    - exikyut 8y ago
      Nooo! Think of the untold millions of potential FOIA requests you're eliminating! At least that's the ideal case scenario. Practically speaking, that would require an in-depth, contextualized review of every single row of every single column in every single schema. So, the answer is probably yes, pursuant to the "associated costs" field of the request being at least 5 figures, and probably 6.
scroot 8y ago
You all hiring?
- bpchaps 8y ago
  Maybe one day. :)
bradleyjg 8y ago
I submitted a FOIA request five years ago to the State Department. About six months ago I received a partial response which was redacted to uselessness. I’m still waiting for the rest of the responsive documents. My take away from this experience is that if you aren’t prepared to sue ($450 absolute minimum cost) don’t bother.
- bpchaps 8y ago
  Yeah - that's one of the main reasons why I haven't really focused on federal FOIA that much. It just takes too damn long. You aught to try something local - you're much more likely have a more open conversation about the records. Also - lots of pro bono FOIA lawyers exist to help ease that burden. FOIA covers legal costs if the suit goes in your favor.
bpchaps 8y ago
I should clarify by "allows", I really mean "doesn't disallow". Similar, but ultimately very different.
userbinator 8y ago
you can submit SQL for a government owned/contracted database I do hope they sanitise queries first or at least only give read-only access, right?
- bpchaps 8y ago
  When I say "submit", I mean "submit a FOIA request over email with some SQL, then prey that the FOIA officer will google 'SQL' before flatout rejecting your request". :)

paulie_a 8y ago

Chicago also has a open data initieve with tons of large datasets that is publicly available.

bpchaps 8y ago
Yep [1]! It's pretty damn impressive, but it's by no means a complete list of the data Chicago stores. Which is where FOIA comes into the picture. According to Chicago's (ex) Chief Data Officer, they also have a fairly routine process of scraping through Chicago's FOIA requests for gauging interest for new datasets. In other (somewhat naive) words - the more data-like FOIA requests that Chicago receives, the more open data will be made available. On a similar note, federal FOIA has some built-in "proactive disclosure" language that makes information publicly available after it's been requested more than three times, or is likely to be requested often or with interest. Not an expert on this piece by any means, but it's something to consider when thinking about open data portals and the like. Also, check out Data USA's (exceptionally bloated) data portal [2], which includes a lot of data throughout portals throughout the US. [1] https://data.cityofchicago.org https://data.cityofchicago.org [2] https://datausa.io/ https://datausa.io/

danso 8y ago

> you can submit SQL for a government owned/contracted database I’ve heard of journalists working with government IT folks to craft a SQL query, but not sending code in the records request. Do you by chance have an example letter that uses code?

bpchaps 8y ago
I just made sent one: https://www.muckrock.com/foi/chicago-169/canvas-database-schema-60175/ https://www.muckrock.com/foi/chicago-169/canvas-database-sch... The idea comes from [1], where like you say, a journalist/requester can work with the government IT folk to craft a SQL query. A big problem comes, though, from the burden and staffing required to write the query. To add to that, FOIA often requires cost estimates prior to starting a request - which, in the case of writing code, can muddle estimations... which have to be paid on the onset. My position is that if the requester understands the schema, then the burden required to write the query is essentially nullified. That way, the only cost would then be the review of the query, which is trivial in comparison to writing one from scratch. There's a lot of prereq knowledge needed before submitting a request for code. That, and there's generally a potential for penalty against a public body that doesn't run a full search. So in many ways, there isn't really a need for a requester to submit their own SQL from the onset. [1] http://illinoiscourts.gov/Opinions/AppellateCourt/2018/2ndDistrict/2170617.pdf http://illinoiscourts.gov/Opinions/AppellateCourt/2018/2ndDi... (not a laywer, etc)
- danso 8y ago
  Thanks!
- toomuchtodo 8y ago
  Is it not possible to request an electronic copy of the database dump in its entirety?
  - exikyut 8y ago
    Nooo! Think of the untold millions of potential FOIA requests you're eliminating! At least that's the ideal case scenario. Practically speaking, that would require an in-depth, contextualized review of every single row of every single column in every single schema. So, the answer is probably yes, pursuant to the "associated costs" field of the request being at least 5 figures, and probably 6.

scroot 8y ago

You all hiring?

bpchaps 8y ago
Maybe one day. :)

bradleyjg 8y ago

I submitted a FOIA request five years ago to the State Department. About six months ago I received a partial response which was redacted to uselessness. I’m still waiting for the rest of the responsive documents. My take away from this experience is that if you aren’t prepared to sue ($450 absolute minimum cost) don’t bother.

bpchaps 8y ago
Yeah - that's one of the main reasons why I haven't really focused on federal FOIA that much. It just takes too damn long. You aught to try something local - you're much more likely have a more open conversation about the records. Also - lots of pro bono FOIA lawyers exist to help ease that burden. FOIA covers legal costs if the suit goes in your favor.

bpchaps 8y ago

I should clarify by "allows", I really mean "doesn't disallow". Similar, but ultimately very different.

userbinator 8y ago

you can submit SQL for a government owned/contracted database I do hope they sanitise queries first or at least only give read-only access, right?

bpchaps 8y ago
When I say "submit", I mean "submit a FOIA request over email with some SQL, then prey that the FOIA officer will google 'SQL' before flatout rejecting your request". :)