4 ms·

Digitizing 55,000 pages of civic meetings

60 points by djha-skin 4y ago

jeffbee 4y ago
Hrmm, Berkeley resident here. It's clear that the city possesses the original digital copies of the last several decades of these documents, and the fact that their system of record produces them as images is just a weird quirk. Instead of OCRing them, wouldn't it be better to just get the city to fix their system? I'm not the only one who thinks so. Berkeleyside recently wrote about it: https://www.berkeleyside.org/2022/08/12/new-city-website-limits-access-to-vast-archive-of-berkeley-records https://www.berkeleyside.org/2022/08/12/new-city-website-lim...
- toomuchtodo 4y ago
  FOIA request for the entire corpus on digital media and upload to the Internet Archive as a collection? The archive OCRs PDFs as part of derive operations when items are uploaded. Could even crowdsource the FOIA request using Muckrock.com.
  - jeffbee 4y ago
    Ok but the originals were available on the city's website until literally a few weeks ago. I don't think disappearing them was intended, it was just the inevitable result of hiring some consultant clowns to redo the site. It's a very open city, with a full-time staffed archive at the central library who will find whatever you want to read.
    - toomuchtodo 4y ago
      I was not aware. Hopefully these docs can be returned to their previously publicly available glory in short order. Putting them in the Internet Archive ensures access in perpetuity.
  - gvkhna 4y ago
    I’m not too clear on the intricacies of FOIA requests but some data cities have, they charge for. Would FOIA be able to get that as well? Otherwise I feel this type of request could be denied for being too large/expensive in nature to fulfill. For instance many counties provide historical tax records for a fee, companies like Zillow pay for this data. Curious if you know more for the purpose of freeing more data.
defrost 4y ago
Digitised full text search is a start. The next level up is full linked cross referencing; bio files on all people that appear in the minutes, the ability to trace their interactions through time, the disclosure of what public ownership records they have for land and companies in the immediate and adjacent regions, what boards they have positions on. If they are there to (say) lobby against local public transportation then who are they affiliated with that is also lobbying against the same in other county areas | states, etc. Disclaimer: we did this some years ago for global mineral exploration and project development, linking public records from companies, stock exchanges, land bureaus, etc. and eventually onsold the data and the process [1]. It can be a grind to set up and get swinging, but ultimately worthwhile and can be funded via a subscription model. [1] https://www.spglobal.com/marketintelligence/en/campaigns/metals-mining https://www.spglobal.com/marketintelligence/en/campaigns/met...
- jeffbee 4y ago
  You must get accused of “doxxing” a lot. Bad actors seem to believe they have a right to appear at every council meeting completely free from whatever they’ve said or done in the past.
  - defrost 4y ago
    I have no knowledge of any dox'ing accusations from the first mid 1970's incarnation through until ~2008 (the last time I had substantial inside access to the mineral intelligence database). Mineral rights leases were all public record, board positions for traded companies are required disclosures to stock exchnages, chaining subsidiaries and linking them to shell ownership of capital | leases etc related to projects is all fair game, it's just peeling back the obfuscation. We could list all mineral and energy projects, companies, board members, leases, etc related to (say) the Koch brothers by degrees of interest (ownership) and produce a chrnology of aquisition .. but at no point did we ever pursue or pass on their home address, social calender, personal photographs, etc beyond what was already released to a stock exchange or in a prospectus. WRT City and Suburban councils we still do a similar thing here in the state where I live, peeling back connections to real estate companies, shopping | parking moguls, pipe laying companies etc. If people are going to have their rates go toward some project put forward in council (road kerbing, for example), it's good to know that the person pushing this is related to a contractor that will directly benefit (and might be paying a kickback for votes).
thadk 4y ago
There are several USA aggregator platforms that help with these kind of public meetings, but they are a bit regional. Original city marked as * https://www.documenters.org https://www.documenters.org / https://www.citybureau.org https://www.citybureau.org covers Chicago*, Atlanta, Cleveland, Detroit, Fresno, Minneapolis, Omaha (backed by MuckRock products) and helps activate volunteers. NYC* has an aggregator around their many Community Boards somewhere but it is not turning up in my searches. https://councildataproject.org https://councildataproject.org covers Seattle*, Portland, Alameda, Denver, etc. I haven't seen one on the ANCs in DC yet.
mlissner 4y ago
This uses AWS's Textract service, but if you're doing a LOT of extraction, that gets pretty expensive pretty quickly. We do thousands of pages daily on CourtListener.com and created an open source microservice for this purpose. It can take PDFs, DOCX, DOC, TXT, HTML, or a handful of other files and extract the text, doing OCR if necessary: https://free.law/projects/doctor https://free.law/projects/doctor We're always looking for more people to use and improve it.
gvkhna 4y ago
This is awesome! Cities don’t do a great job of organizing data. It’s unclear if there’s enough funding/interest to have better access to data like this. But I certainly wish there was! I sent a message to author directly but wanted to add as well. Has anyone used tsvector index on Postgres for this type of full text search? I’ve had great experience with it (for this type of full text/document data) but there isn’t much information out there about this kind of index and how to best utilize it.

60 points by djha-skin 4y ago

Hrmm, Berkeley resident here. It's clear that the city possesses the original digital copies of the last several decades of these documents, and the fact that their system of record produces them as images is just a weird quirk. Instead of OCRing them, wouldn't it be better to just get the city to fix their system? I'm not the only one who thinks so. Berkeleyside recently wrote about it: https://www.berkeleyside.org/2022/08/12/new-city-website-limits-access-to-vast-archive-of-berkeley-records https://www.berkeleyside.org/2022/08/12/new-city-website-lim...

toomuchtodo 4y ago
FOIA request for the entire corpus on digital media and upload to the Internet Archive as a collection? The archive OCRs PDFs as part of derive operations when items are uploaded. Could even crowdsource the FOIA request using Muckrock.com.
- jeffbee 4y ago
  Ok but the originals were available on the city's website until literally a few weeks ago. I don't think disappearing them was intended, it was just the inevitable result of hiring some consultant clowns to redo the site. It's a very open city, with a full-time staffed archive at the central library who will find whatever you want to read.
  - toomuchtodo 4y ago
    I was not aware. Hopefully these docs can be returned to their previously publicly available glory in short order. Putting them in the Internet Archive ensures access in perpetuity.
- gvkhna 4y ago
  I’m not too clear on the intricacies of FOIA requests but some data cities have, they charge for. Would FOIA be able to get that as well? Otherwise I feel this type of request could be denied for being too large/expensive in nature to fulfill. For instance many counties provide historical tax records for a fee, companies like Zillow pay for this data. Curious if you know more for the purpose of freeing more data.

defrost 4y ago

Digitised full text search is a start. The next level up is full linked cross referencing; bio files on all people that appear in the minutes, the ability to trace their interactions through time, the disclosure of what public ownership records they have for land and companies in the immediate and adjacent regions, what boards they have positions on. If they are there to (say) lobby against local public transportation then who are they affiliated with that is also lobbying against the same in other county areas | states, etc. Disclaimer: we did this some years ago for global mineral exploration and project development, linking public records from companies, stock exchanges, land bureaus, etc. and eventually onsold the data and the process [1]. It can be a grind to set up and get swinging, but ultimately worthwhile and can be funded via a subscription model. [1] https://www.spglobal.com/marketintelligence/en/campaigns/metals-mining https://www.spglobal.com/marketintelligence/en/campaigns/met...

jeffbee 4y ago
You must get accused of “doxxing” a lot. Bad actors seem to believe they have a right to appear at every council meeting completely free from whatever they’ve said or done in the past.
- defrost 4y ago
  I have no knowledge of any dox'ing accusations from the first mid 1970's incarnation through until ~2008 (the last time I had substantial inside access to the mineral intelligence database). Mineral rights leases were all public record, board positions for traded companies are required disclosures to stock exchnages, chaining subsidiaries and linking them to shell ownership of capital | leases etc related to projects is all fair game, it's just peeling back the obfuscation. We could list all mineral and energy projects, companies, board members, leases, etc related to (say) the Koch brothers by degrees of interest (ownership) and produce a chrnology of aquisition .. but at no point did we ever pursue or pass on their home address, social calender, personal photographs, etc beyond what was already released to a stock exchange or in a prospectus. WRT City and Suburban councils we still do a similar thing here in the state where I live, peeling back connections to real estate companies, shopping | parking moguls, pipe laying companies etc. If people are going to have their rates go toward some project put forward in council (road kerbing, for example), it's good to know that the person pushing this is related to a contractor that will directly benefit (and might be paying a kickback for votes).

thadk 4y ago

There are several USA aggregator platforms that help with these kind of public meetings, but they are a bit regional. Original city marked as * https://www.documenters.org https://www.documenters.org / https://www.citybureau.org https://www.citybureau.org covers Chicago*, Atlanta, Cleveland, Detroit, Fresno, Minneapolis, Omaha (backed by MuckRock products) and helps activate volunteers. NYC* has an aggregator around their many Community Boards somewhere but it is not turning up in my searches. https://councildataproject.org https://councildataproject.org covers Seattle*, Portland, Alameda, Denver, etc. I haven't seen one on the ANCs in DC yet.

mlissner 4y ago

This uses AWS's Textract service, but if you're doing a LOT of extraction, that gets pretty expensive pretty quickly. We do thousands of pages daily on CourtListener.com and created an open source microservice for this purpose. It can take PDFs, DOCX, DOC, TXT, HTML, or a handful of other files and extract the text, doing OCR if necessary: https://free.law/projects/doctor https://free.law/projects/doctor We're always looking for more people to use and improve it.

gvkhna 4y ago

This is awesome! Cities don’t do a great job of organizing data. It’s unclear if there’s enough funding/interest to have better access to data like this. But I certainly wish there was! I sent a message to author directly but wanted to add as well. Has anyone used tsvector index on Postgres for this type of full text search? I’ve had great experience with it (for this type of full text/document data) but there isn’t much information out there about this kind of index and how to best utilize it.