7 ms·

Public records are public. The fact that some government organizations make it hard to retrieve public records is a flaw in the system. I'd be in favor of a na

Public records are public. The fact that some government organizations make it hard to retrieve public records is a flaw in the system. I'd be in favor of a national law requiring all public records to be published in machine-readable form. In the mean time, it is our civic responsibility to conspire to circumvent these misbehaving public services.

huhtenberg 7y ago
Public records are published based on certain demand assumptions. If a real-world demand for, say, some GIS data is hundreds of requests per day, then a crawler that comes in with hundreds requests PER MINUTE will obviously stress the infrastructure. Adjusting infrastructure to cope is not an instant process, nor is it a sure thing to begin with given all the budgeting formalities. So your "civic duty" will ultimately result in destruction of these services, because they simply don't have the means to deal with such thoughtless activism.
- ijpoijpoihpiuoh 7y ago
  You've made an unfounded assumption -- that is, that the person you're responding to is scraping irresponsibly. If they are, as they say, simply replacing human researchers with the equivalent bots, then the net load change from automation is zero, or possibly even negative.
floatingatoll 7y ago
If such a national law were passed with funding guaranteed for open publication of records, I would endorse your point of view. No such funding exists, and municipalities are regularly denied tax increases by their voters for any reason — much less public records publication that would often embarrass and humiliate those same voters. So in essence you're asking them to cut public services and staffing in order to give hundreds of dollars of IT costs a month to for-profit businesses who can't be bothered to pay some small fraction of their revenue for the costs of delivering those records. It is our civic responsibility to republish those records for free as citizens. Doing so for profit at the expense of citizens is unethical. If OP republishes all records received in a freely-downloadable, unrestricted form, then I would happily help them fix their scrapers. They, of course, do not.
- ijpoijpoihpiuoh 7y ago
  Often what the municipalities are doing for public records is harder and more expensive than just publishing an API. So The funding excuse doesn't really cut muster with me.
  - floatingatoll 7y ago
    Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access? The public records are public. Charging for them is, by the above arguments, immoral. Therefore, not only the municipalities but also the businesses profiting from those public records owe us their scraped data, for free, without regard for profit concerns. Not one for-profit business does so. Why is their immoral action acceptable, when the same action by a municipality is not?
    - devxvda 7y ago
      Loveland (now apparently called Landgrid). https://landgrid.com/ https://landgrid.com/
      - zaphod4prez 7y ago
        Loveland is such a cool organization
    - stickfigure 7y ago
      There's nothing immoral about charging for content that you've aggregated. People sell dictionaries. The problem here is that instead of building APIs (or just posting to FTP sites), governments are building offices and funding staff to answer snail mail requests. Or building sophisticated web forms and search engines. It's obvious how we got to this point (before the internet, you obtained public records by walking into an office) but it's long past time to change. We don't need fancy web forms to search and find data; cut all that out and just provide data in machine readable form to anyone who wants it. Someone will build a pretty commercial interface to public records data. Chances are, they can do it for less than the 8-figure sum required for UI development in the public sector. Win-win.

huhtenberg 7y ago

Public records are published based on certain demand assumptions. If a real-world demand for, say, some GIS data is hundreds of requests per day, then a crawler that comes in with hundreds requests PER MINUTE will obviously stress the infrastructure. Adjusting infrastructure to cope is not an instant process, nor is it a sure thing to begin with given all the budgeting formalities. So your "civic duty" will ultimately result in destruction of these services, because they simply don't have the means to deal with such thoughtless activism.

ijpoijpoihpiuoh 7y ago
You've made an unfounded assumption -- that is, that the person you're responding to is scraping irresponsibly. If they are, as they say, simply replacing human researchers with the equivalent bots, then the net load change from automation is zero, or possibly even negative.

floatingatoll 7y ago

If such a national law were passed with funding guaranteed for open publication of records, I would endorse your point of view. No such funding exists, and municipalities are regularly denied tax increases by their voters for any reason — much less public records publication that would often embarrass and humiliate those same voters. So in essence you're asking them to cut public services and staffing in order to give hundreds of dollars of IT costs a month to for-profit businesses who can't be bothered to pay some small fraction of their revenue for the costs of delivering those records. It is our civic responsibility to republish those records for free as citizens. Doing so for profit at the expense of citizens is unethical. If OP republishes all records received in a freely-downloadable, unrestricted form, then I would happily help them fix their scrapers. They, of course, do not.

ijpoijpoihpiuoh 7y ago
Often what the municipalities are doing for public records is harder and more expensive than just publishing an API. So The funding excuse doesn't really cut muster with me.
- floatingatoll 7y ago
  Can you name a single for-profit public records scraper who republishes the parsed data scraped without charging for data access? The public records are public. Charging for them is, by the above arguments, immoral. Therefore, not only the municipalities but also the businesses profiting from those public records owe us their scraped data, for free, without regard for profit concerns. Not one for-profit business does so. Why is their immoral action acceptable, when the same action by a municipality is not?
  - devxvda 7y ago
    Loveland (now apparently called Landgrid). https://landgrid.com/ https://landgrid.com/
    - zaphod4prez 7y ago
      Loveland is such a cool organization
  - stickfigure 7y ago
    There's nothing immoral about charging for content that you've aggregated. People sell dictionaries. The problem here is that instead of building APIs (or just posting to FTP sites), governments are building offices and funding staff to answer snail mail requests. Or building sophisticated web forms and search engines. It's obvious how we got to this point (before the internet, you obtained public records by walking into an office) but it's long past time to change. We don't need fancy web forms to search and find data; cut all that out and just provide data in machine readable form to anyone who wants it. Someone will build a pretty commercial interface to public records data. Chances are, they can do it for less than the 8-figure sum required for UI development in the public sector. Win-win.