6 ms·

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically

• prepend 8y ago
  This is a cool project. I was trying to find the source for the project on the page. There’s an oss page [0] but that’s about the software used. Is this an open source project? Or what’s the way for licensing to use with US data? [0] https://data-in.place/open-source https://data-in.place/open-source
• kerkeslager 8y ago
  As long as those spreadsheets/database files are accessible to someone with technical skill, people can pull in the data and use tools to make it more accessible and useful. Ideally, yes, the data is useful to begin with, but as long as it's available, there's nothing stopping individuals with the skills from making it useful. Of course, there are exceptions: the PDFs that are often provided by the prosecution as part of the discovery process are prohibitively difficult to deal with, and should be considered a violation of Brady vs. Maryland, IMO.
  • ethbro 8y ago
    Exactly. Accessible and machine readable are necessary but not sufficient. Thankfully, civil society can reasonably pick up the slack. In regards to modern day transparency requirements, it seems like laws should include a reasonableness clause. Making records available to the public but requiring them to be hand photocopied vs. making them available in electronic form in a custom format. Both open. But two very different magnitudes of effort.
  • _bohm 8y ago
    I've spent a great deal of time parsing data out of government PDFs that isn't attainable by any other means as a part of my job. In the process I've learned how difficult this information can be to access even for people who don't require it to be in a machine readable format. It certainly has been an interesting exercise in how far simple web scraping tools can be pushed, though.
    • ams6110 8y ago
      Amazon Textract was recently announced, sounds like it might be good for that. Haven't tried it myself. https://aws.amazon.com/about-aws/whats-new/2018/11/introducing-amazon-textract-now-in-preview-easily-extract-text-and-data-from-virtually-any-document/ https://aws.amazon.com/about-aws/whats-new/2018/11/introduci...
      • _bohm 8y ago
        I applied to the beta, but they never got back to me :\
        • bpchaps 8y ago
          Have you tried Apache's tika? It's pretty decent.
          • _bohm 8y ago
            Nope, I'll have to give it a spin. Thanks for the recommendation!
    • TheAceOfHearts 8y ago
      Do you have any tool suggestions or general advice for someone trying to do this? A while back I was trying to extract text from some government PDFs in order to make the information more accessible for others, but I became a bit overwhelmed when I started reading up on PDFs.
      • 8y ago
• newleaf 8y ago
  In case you haven't seen it, I think https://www.data.gov/ https://www.data.gov/ is an attempt to answer your point about making it "accessible and useful to people." There's room for improvement, but it's a start.
  • org3432 8y ago
    I thought there would be a flood of projects analysing the data when it came out, but it seems like the idea everyone applauded but not much came out of it. Steve Ballmer's http://usafacts.org http://usafacts.org seems like the first real attempt though.
    • FPGAhacker 8y ago
      That's pretty interesting, thanks for mentioning it.
    • gizmodo59 8y ago
      +1 for this. This site is beautifully designed and loads very fast in mobile.
    • pfd1986 8y ago
      Is there a HN thread discussing these reports? I'd love to hear folks' opinions about them... Interesting facts to me in the latest report [1]: - Page 44: "Our economy has grown at a steady rate despite changes in economic policy". I expected to see a lot more fluctuation on this data. - Page 30: "There have been more suicide gun deaths than homicide gun deaths every year since 1981". This is crazy to me given how much we hear about gun homicide being a problem in this country. - Page 29: Crime rate has declined but "The number of incarcerated persons has increased by 330% since 1980". [1] https://static.usafacts.org/public/resources/USAFactsReport2018.pdf https://static.usafacts.org/public/resources/USAFactsReport2...
      • hueving 8y ago
        >Page 44: "Our economy has grown at a steady rate despite changes in economic policy". I expected to see a lot more fluctuation on this data. Barring some calamity, this is pretty much as expected. Tariffs impact a very small percentage of the economy with large size, and the rest of it with a small overhead, much like a fed interest rate hike. Even with this hawkish fed, there hasn't been anything overly harmful to the economy from a policy perspective. > This is crazy to me given how much we hear about gun homicide being a problem in this country. Anything that's politicized gets this special treatment. Kid kills brother with car, blurb in local newspaper. Kid kills brother with gun, national news and Tweets from Presidential candidates. Texting and driving is as bad as drinking and driving when it comes to number of deaths (dwarfing gun deaths as well), yet people do it like it's no big deal. Most cities still only levy small fines (in comparisons to DUIs) for doing it. Not all accidental death is created equal. >Page 29: Crime rate has declined but "The number of incarcerated persons has increased by 330% since 1980" Welcome to the US where the war on drugs gives us authoritarian level incarceration rates.
• bproven 8y ago
  >However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people True, I guess HGTG applies well here: “But the plans were on display…” “On display? I eventually had to go down to the cellar to find them.” “That’s the display department.” “With a flashlight.” “Ah, well, the lights had probably gone.” “So had the stairs.” “But look, you found the notice, didn’t you?” “Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”
• grawprog 8y ago
  >Making public data open by default can arguable be an imporant step towards fostering societal equity I think this was one of my biggest shocks doing work for the government, collecting public data, payed by tax funded grants. Public data isn't for the public. We went into this project with all these starry eyed dreams of making a public online database and freely posting everything we collected, with maps and interactive tools, status reports. It was part of our grant proposal. Then reality came and we found out public data meant a government password protected database with access fees where our data would be available to people willing to pay for it or we'd lose our funding. The data were for companies or individuals willing to pay the government not for the public. This still doesn't sit well with me nearly 6 years later. That was never what we wanted out of that project and it wasn't what was planned or accepted when we wrote our proposal.
  • crankylinuxuser 8y ago
    Simply put, this government isn't ours. It belongs to the corporate heads and the monied elite and the lobbyists who write the laws that are uusually summarily passed by congress.
  • Wowfunhappy 8y ago
    How much did access cost? Ideally, I'd prefer the data be free, but if the fee was (mostly) nominal, I'd consider that almost as good...
• bobwaycott 8y ago
  > Making public data open by default can arguable be an imporant step towards fostering societal equity. Out of curiosity, what’s the argument here for how public data being open by default is an important step toward fostering societal equity?
  • alpyne 8y ago
    Here's a paper by the Sunlight Foundation that goes into this a bit: https://www.scribd.com/doc/263776138/The-Social-Impact-of-Open-Data https://www.scribd.com/doc/263776138/The-Social-Impact-of-Op...
    • bobwaycott 8y ago
      Thanks!
    • p_b_r 8y ago
      I disagree - that's not a link to a paper by the Sunlight Foundation. It's a link to a site that wants you to agree to try a monthly subscription before you can download anything at all. Especially in the context of a discussion about public data, that's an important distinction. If you can, please provide an URL to the actual content?
      • spondyl 8y ago
        The article linked above, hosted on Scribd, is actually one of the official distributions provided by the Sunlight Foundation. The Scribd user who uploaded it was one of the paper's authors. Having said that, here's a link to the version hosted on the Sunlight Foundation's domain: http://assets.sunlightfoundation.com.s3.amazonaws.com/policy/SocialImpactofOpenData.pdf http://assets.sunlightfoundation.com.s3.amazonaws.com/policy... You can also verify what I said about the Scribd version by checking out the original press release/announcement here: https://sunlightfoundation.com/2015/05/05/a-new-approach-to-measuring-the-impact-of-open-data/ https://sunlightfoundation.com/2015/05/05/a-new-approach-to-...
• hartator 8y ago
  I actually rather have the raw data accessible instead of doctored version.
  • amyjess 8y ago
    Why not both? Make the raw data available for those of use who want to write machine-parsing algorithms, and also make it available in human-readable and easily digestible form for the broader public.
• Vinnl 8y ago
  The FAIR principles make a lot of sense to me: Findable, Accessible, Interoperable and Reproducible. https://en.wikipedia.org/wiki/FAIR_data https://en.wikipedia.org/wiki/FAIR_data
• gibsonf1 8y ago
  It would be extremely helpful if the government data used the same Solid protocol (Linked Data Platform) which will likely be getting widespread use starting next year. (See Tim Berner Lee's Inrupt.com Solid startup)
  • trystero 8y ago
    What are the signs that make you think Solid will see widespread use, and by whom? I'm genuinely curious.
• scarejunba 8y ago
  I prefer when it is available as a spreadsheet or database file. The crucial element is my having access to the data. After that, any differential amount of information I have over everyone else is an improvement. For instance, consider the statistics on accidents between cars and bikes in California. You can get the numbers yourself from the government. When someone says that more accidents are adjudged to be because the bicyclist is at fault, you can reference the truth and ruin the credibility of the person making the assertion, thereby allowing political advancement of your own cause. No one can use the technique against you because you are capable of acquiring the knowledge and won't make wrong assertions. Only other people will make them. Having verifiable true information over someone is power. It's better non-democratized so long as I fall within the circle of power.
  • jtloong 8y ago
    I think it needs to be both. Accessible in it's raw format as well as approachable for non-technical users to interact with public information.
• cantthinkofone 8y ago
  I agree that the government will need to work on a transparent front-end to make this data universally accessible. Nonprofits without the budgets for advanced tech workers and without volunteers will need clearly organized links to download. They may not know how to do shell scripting. The more data everyone can use, rather than data that can be owned and commoditized or utilized only by specialists is a good thing.
• vertexFarm 8y ago
  I was kind of concerned right off the bat with that numbered list: 1. public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security I'm sure literally everything that they wish to keep opaque will declared to be covered under one or both of these incredibly vague categories and nothing will significantly change. Is there any elaboration in the bill that defines what they can call a matter of privacy or security? Even if there is, it wouldn't matter much because how are people going to tell if they keep it locked down in the first place? And they would not risk any sort of real blowback for abusing this and getting caught. Tell me there have not been far, far worse scandals that resulted in no consequences for the perps and cowed silence from the public. I don't think they're hiding the X-Files in there or anything, but this won't magically cause a more transparent, just, or equitable government unless it has serious teeth and tight language. And 2. federal agencies should use evidence when they make public policy Somehow I wonder if the data from the Kansas experiment will be taken into consideration and turned into public policy by this current administration, or if they will cherry-pick evidence selectively to justify only wildly unpopular legislation because someone (possibly an industry with a conflict of interest?) contrives some p-hacked research to back it up. Just because something is scientific doesn't necessarily mean it's good government. It is often so, but I'm always very wary when they trot out a bill with lots of bold language touting justice and democracy, truth, stuff like that. If the US legislature passes a bill called "protect innocent puppies from being kicked in the name of god and freedom" you can be 95% sure that this bill will enable a great wave of puppy-kicking despite its holy name.
• mancerayder 8y ago
  Making public data open by default can arguable be an imporant step towards fostering societal equity. Define equity here?
  • danso 8y ago
    Not the parent commenter, but open data policies could contribute to levelingn the playing field for society. The vast majority of records requests are currently made by corporations and other business interests.
    • hyperdunc 8y ago
      Having accessible data isn't enough. The average person also needs the tools to process it in a meaningful way. This is where corporations are ahead - with relatively unlimited resources to turn that government data into insights that influence business decisions.
• pestrov 8y ago
  I would also mention Data USA by MIT Media Labs Collective Learning group, with an effort to combine multiple sources into a single geographic profiles rich with visualization and data sources https://datausa.io https://datausa.io
• Dudemann 8y ago
  What do you mean by "democratize data-science"?