3 ms·

It's all about the format: My city's school district makes images of the spreadsheets and then publishes them in a pdf. USELESS

by baldfat 10y ago

closed 10y ago
In their defense, that's still better than most academic publications!
otoburb 10y ago
While deplorable from an accessibility standpoint, at least the data is available and out in the open to the public. One small step in the right direction. EDIT: Tabula[1] is an open source tool to extract tables from PDFs into CSV/Excel format in case one really needs to process data trapped in PDF documents. [1] http://tabula.technology http://tabula.technology
- rcthompson 10y ago
  Does Tabula do OCR? That's what would be required for tables embedded as images.
  - josephfrazier 10y ago
    Unfortunately, it doesn't. From https://github.com/tabulapdf/tabula https://github.com/tabulapdf/tabula: > Caveat: Tabula only works on text-based PDFs, not scanned documents. If you can click-and-drag to select text in your table in a PDF viewer (even if the output is disorganized trash), then your PDF is text-based and Tabula should work.
- markovbling 10y ago
  Tabula is great!
rcthompson 10y ago
Happily, this bill actually does care about the format of the data: "The policy and procedures shall include ... technical requirements with the goal of making electronic data sets available to the greatest number of users and for the greatest number of applications, including, whenever practicable, the use of machine readable, non-proprietary technical standards for web publishing" "The Treasurer shall consider various means by which to develop a set of universal data formatting standards to effectuate the purposes of this act, including working with other State departments and agencies, and contracting, if deemed necessary, with nonprofit organizations, commercial vendors or third party groups for this purpose. If such standards are developed and adopted by the Treasurer, they shall be the format that each State department and agency will use to provide existing and future electronic data sets"
- shakna 10y ago
  > the use of machine readable, non-proprietary technical standards for web publishing Wouldn't that cover PDF, even though it might be deeply inappropriate for it's content? PDF has several ISO standards, and is most definently machine readable. PDF should also meet the goal as well: > available to the greatest number of users and for the greatest number of applications Huge number of PDF clients, such as PDF.js that powers Firefox's reader. > If such standards are developed and adopted by the Treasurer, they shall be the format that each State department and agency will use to provide existing and future electronic data sets Knowing how much government uses PDF... It is probably incoming. But it makes little/no sense for spreadsheet data. Great for graphs and prose, however.
  - baldfat 10y ago
    PDF is really a container and can have just about anything inside of it. https://www.w3.org/2013/04/odw/odw13_submission_52.pdf https://www.w3.org/2013/04/odw/odw13_submission_52.pdf PDF with csv attachments is a good compromise but I rarely see that. Usually it is PDF or Excel and the PDF is a nightmare to munge.
koolba 10y ago
> My city's school district makes images of the spreadsheets and then publishes them in a pdf. USELESS You should be able to sue them for misuse of public funds. Dedicating tax money to convert a useful format (CSV, spreadsheet, etc) into a useless or at least less useful one (ex: PDF) is a clear violation.
- baldfat 10y ago
  It isn't a violation if there isn't any policy or law against it.
- baldfat 10y ago
  It isn't a violation if there isn't any policy or law against it.
rhcom2 10y ago
Had the same problem with NJ. I FOIA'd Chris Christie's schedules and got some of the original PDF reports (which was fine, they were readable with some Python library) and some SCANS of the original PDF reports, even with OCR they were useless.

It's all about the format: My city's school district makes images of the spreadsheets and then publishes them in a pdf. USELESS

by baldfat 10y ago

It's all about the format: My city's school district makes images of the spreadsheets and then publishes them in a pdf. USELESS

closed 10y ago

In their defense, that's still better than most academic publications!

otoburb 10y ago

While deplorable from an accessibility standpoint, at least the data is available and out in the open to the public. One small step in the right direction. EDIT: Tabula[1] is an open source tool to extract tables from PDFs into CSV/Excel format in case one really needs to process data trapped in PDF documents. [1] http://tabula.technology http://tabula.technology

rcthompson 10y ago
Does Tabula do OCR? That's what would be required for tables embedded as images.
- josephfrazier 10y ago
  Unfortunately, it doesn't. From https://github.com/tabulapdf/tabula https://github.com/tabulapdf/tabula: > Caveat: Tabula only works on text-based PDFs, not scanned documents. If you can click-and-drag to select text in your table in a PDF viewer (even if the output is disorganized trash), then your PDF is text-based and Tabula should work.
markovbling 10y ago
Tabula is great!

rcthompson 10y ago

Happily, this bill actually does care about the format of the data: "The policy and procedures shall include ... technical requirements with the goal of making electronic data sets available to the greatest number of users and for the greatest number of applications, including, whenever practicable, the use of machine readable, non-proprietary technical standards for web publishing" "The Treasurer shall consider various means by which to develop a set of universal data formatting standards to effectuate the purposes of this act, including working with other State departments and agencies, and contracting, if deemed necessary, with nonprofit organizations, commercial vendors or third party groups for this purpose. If such standards are developed and adopted by the Treasurer, they shall be the format that each State department and agency will use to provide existing and future electronic data sets"

shakna 10y ago
> the use of machine readable, non-proprietary technical standards for web publishing Wouldn't that cover PDF, even though it might be deeply inappropriate for it's content? PDF has several ISO standards, and is most definently machine readable. PDF should also meet the goal as well: > available to the greatest number of users and for the greatest number of applications Huge number of PDF clients, such as PDF.js that powers Firefox's reader. > If such standards are developed and adopted by the Treasurer, they shall be the format that each State department and agency will use to provide existing and future electronic data sets Knowing how much government uses PDF... It is probably incoming. But it makes little/no sense for spreadsheet data. Great for graphs and prose, however.
- baldfat 10y ago
  PDF is really a container and can have just about anything inside of it. https://www.w3.org/2013/04/odw/odw13_submission_52.pdf https://www.w3.org/2013/04/odw/odw13_submission_52.pdf PDF with csv attachments is a good compromise but I rarely see that. Usually it is PDF or Excel and the PDF is a nightmare to munge.

koolba 10y ago

> My city's school district makes images of the spreadsheets and then publishes them in a pdf. USELESS You should be able to sue them for misuse of public funds. Dedicating tax money to convert a useful format (CSV, spreadsheet, etc) into a useless or at least less useful one (ex: PDF) is a clear violation.

baldfat 10y ago
It isn't a violation if there isn't any policy or law against it.
baldfat 10y ago
It isn't a violation if there isn't any policy or law against it.

rhcom2 10y ago

Had the same problem with NJ. I FOIA'd Chris Christie's schedules and got some of the original PDF reports (which was fine, they were readable with some Python library) and some SCANS of the original PDF reports, even with OCR they were useless.