13 ms·

Congress votes to make open government data the default in the United States

1130 points by danso 8y ago

cptskippy 8y ago
> there was a carve out “for data that does not concern monetary policy,” which relates to the Federal Reserve, among others. Does that mean they won't open data that affects monetary policy or that they will only open data that affects monetary policy? Either way that seems huge.
- deleted 8y ago
  [deleted]
jcriddle4 8y ago
Interesting that the Federal Reserve was exempted from this legislation.
- lykr0n 8y ago
  The Federal Reserve [board] bases its data on data from the Reserve Banks (think Federal Reserve of New York)- which are private and get data from banks chartered in the region (so the New York Fed is partially owned by JP Morgan). Requiring the Federal Reserve [board] to release underlying data could get sticky and be a legal minefield.
  - zrail 8y ago
    Also the St Louis Fed provides provides FRED, which is pretty open for most purposes. https://fred.stlouisfed.org/ https://fred.stlouisfed.org/
- indigochill 8y ago
  Is it? My understanding is that although the board is appointed by the President and Congress, once appointed they don't answer to the government, so their data isn't really "government data". But I'll be the first to admit I know very little about how the Federal Reserve works.
  - IncRnd 8y ago
    That's a real shame, because we wouldn't have a federal debt if we didn't have the Federal Reserve. More people should actually know how it works.
- n-exploit 8y ago
  Or course, some of the most important economic data sets won't be shared. It's almost like they want to cover up the systemic disempowerment of the American electorate.
- dannyw 8y ago
  The federal reserve is a curious semi-company-agency with apppontment by president and Congress, but profits going to shareholders (large banks).
- IncRnd 8y ago
  You can access the Reserve's information online as part of the Federal Reserve System's Open Government webpage implementing the Open Government Directive from 2009: https://www.federalreserve.gov/open/open.htm https://www.federalreserve.gov/open/open.htm Despite all that, the Federal Reserve is not a government institution, so such legislation would be similar to singling out your business to post all information in machine readable format online.
mizzao 8y ago
Can someone who is more familiar with the actual policy shed some light on this?
- danso 8y ago
  FWIW, the author, Alex Howard (who I'm friends-via-Twitter with), is as familiar with this as anyone. He was previously a senior analyst at the Sunlight Foundation, which is a prominent open-government organization: https://sunlightfoundation.com/author/ahoward/ https://sunlightfoundation.com/author/ahoward/ The submitted post includes a link [0] to an article he wrote a couple days back, which provides more context on "How did open government data get into the US Code?", including the nitty-gritty of how the original bill was proposed in the last session but ultimately left out of legislation. Howard writes that the legislation was "one of the primary legislative priorities for me during my years as a senior analyst and then deputy director at the Sunlight Foundation" [0] https://e-pluribusunum.org/2018/12/20/senate-passes-evidence-based-policymaking-bill-setting-up-historic-win-for-open-government-data/ https://e-pluribusunum.org/2018/12/20/senate-passes-evidence...
  - cptskippy 8y ago
    > (who I'm friends-via-Twitter with) Full disclosure? Odd brag?
    - danso 8y ago
      It's someone who I mostly learned/knew through social media and am friendly with, even though we've never worked together and maybe have met in person a couple times at conferences.
      - cptskippy 8y ago
        I wasn't sure if you were disclosing the relationship because of the nature of your comment or if you were just bragging about knowing them. It is/was odd.
pratheekrebala 8y ago
FOIA officers will still find a way to send me scanned PDFs of spreadsheets.
- ekianjo 8y ago
  You obviously did not read the article. > public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security
  - clessg 8y ago
    https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html > Please don't insinuate that someone hasn't read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."
  - danso 8y ago
    This policy exists in many states. But plenty of local reporters have tales about how officials respond with PDFs because they either don't realize that the PDFs are generated from a (legacy) system that could output XLS/CSV, or because they fear that Excel sheets/text files are alterable. The benefit of having the law, though, makes it easier to convince officials to tell their IT people to get it right, under threat of lawsuit.
  - mikestew 8y ago
    Not only are you being rude by implying the article went unread, you’re also showing an astounding lack of imagination (perhaps stifled by a need to point out the errors of others). Parent is saying that despite whatever parts of the law you care to snarkily quote, government employees will find a way to screw it up.
  - Rexxar 8y ago
    Scanned pdf of a spreadsheet is clearly machine-readable, as you can read this document with your computer.
todipa 8y ago
Open data is the first step. I wish I could do more business with my gov't (State and Local) through the internet. "Copies of documents cannot be ordered through this website, by email or over the telephone." Only fax and snail mail... https://www.dos.ny.gov/corps/faq_copies.page.asp https://www.dos.ny.gov/corps/faq_copies.page.asp
- weiming 8y ago
  Is this because they are required to make this info accessible, but are not required to make it easy?
  - jaggederest 8y ago
    More importantly, there's no specific funding set aside to support the effort to make it available. Government at all levels lives and dies by funding.
- ehsankia 8y ago
  It's worth noting that easy to access isn't always the best. A great example of this is mugshots, which are open data, but there are now websites that automatically scrape and index people's mugshots and use SEO to rank highly on searches. They then blackmail the person to have it removed. As long as the data is accessible and reasonable easy to get, where journalists and data scientists who really care about the data can get it, then I think we're in a good place.
- deleted 8y ago
  [deleted]
- soared 8y ago
  There are fairly cheap services that turn your emails into faxes. I've used them previously and they worked great.
yohann305 8y ago
I wonder if this open data can be (will be?) used by a foreign entity and be turned against the american people? It should be open but not free-for-all
- icebraining 8y ago
  "open but not free-for-all" What would this mean in practice?
  - yohann305 8y ago
    Available to us citizens only
    - matthewmacleod 8y ago
      That is–not to put too fine a point on it–a dumb idea. It’s essentially impossible to enforce, and doesn’t stop the data from being accessed by anybody else.
    - quickthrower2 8y ago
      Waste of time, easy to get around by asking an American friend to forward the data.
- astazangasta 8y ago
  I'm not convinced there are really any important national security secrets other than passcodes and so on. The rest seems to be excuses for hiding corruption of various kinds. Secrecy in government is incompatible with democracy by definition.
  - rjf72 8y ago
    You have to think about how people respond to things. Social media in its current state has basically turned into a system that takes an event, removes all context, spins it in the most negative way imaginable, and then dogpiles on it to no end -- virtue signaling for imaginary points. There are often lots of things in government, and in life in general, where you must choose between two very negative choices. And the decision there does not necessarily imply that you're in any way satisfied with it, but you see it as the decision least likely to produce an awful outcome. Social media and this sort of logic are wholly and completely incompatible. Take Khashoggi as a contemporary example. Undoubtedly there have been countless heavily classified conversations weighting the pros and cons of any action against Saudi Arabia or MBS as a result of this. How we are extremely dependent on Saudi Arabia for reasons outside the scope of this post, and they know this. And this is all happening at a time when Saudi:US relations souring would greatly stand to strengthen the geopolitical position and power of nations such as China and Russia. In my opinion it's extremely likely we will do nothing, but that's because it's better than doing something. If this was not classified, social media would throw a nonstop hissy fit. Not because they actually care or think we're making the wrong decision, but because it's an incredibly easy way score those imaginary points and followers which are the hottest commodity since sliced bread. Another issue here is that knowing the mechanisms of our decision making here, and how close we came to 'breaking', would be incredibly valuable information to nations such as Saudi Arabia which they could then use to exploit the US. So while I do agree with you that classification is very often abused, I also think there are indeed vast amounts of information that must be classified. People are not mature enough to impartially process our decision making processes, and the details of such processes would provide invaluable information to other nations which could then be used to exploit the US.
compute_me 8y ago
Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aiming to make open data more accessible and to democratize data-science: https://data-in.place/ https://data-in.place/ ...
- prepend 8y ago
  This is a cool project. I was trying to find the source for the project on the page. There’s an oss page [0] but that’s about the software used. Is this an open source project? Or what’s the way for licensing to use with US data? [0] https://data-in.place/open-source https://data-in.place/open-source
- kerkeslager 8y ago
  As long as those spreadsheets/database files are accessible to someone with technical skill, people can pull in the data and use tools to make it more accessible and useful. Ideally, yes, the data is useful to begin with, but as long as it's available, there's nothing stopping individuals with the skills from making it useful. Of course, there are exceptions: the PDFs that are often provided by the prosecution as part of the discovery process are prohibitively difficult to deal with, and should be considered a violation of Brady vs. Maryland, IMO.
  - ethbro 8y ago
    Exactly. Accessible and machine readable are necessary but not sufficient. Thankfully, civil society can reasonably pick up the slack. In regards to modern day transparency requirements, it seems like laws should include a reasonableness clause. Making records available to the public but requiring them to be hand photocopied vs. making them available in electronic form in a custom format. Both open. But two very different magnitudes of effort.
  - _bohm 8y ago
xivzgrev 8y ago
Finally a bipartisan moment, looks like most reps were onboard.
- friedman23 8y ago
  They have to be for the law to pass?
  - srtjstjsj 8y ago
    Not just most reps were onboard; nearly all reps voted in favor
yostrovs 8y ago
I'm glad they were able to do it before the new Congress came on board.
- srtjstjsj 8y ago
  What has that to do with anything? The bill had large majority bipartisan support (which makes me suspect it's toothless...)
  - yostrovs 8y ago
    There's a reason it was passed during the lame duck session of the Senate, just before change of control. That's when urgent matters that are guaranteed to be ignored by the next Congress are handled. I'm surprised this needs explaining.
crankylinuxuser 8y ago
The first problem here is the data formats state and federal governments use. You'll see a hodgepodge, but primarily MS Office $version . The biggest problem with this terrible binary format is that metadata can leak a great deal that should have not been released. So this leads to PDF output of word/excel. The next area is that especially local government offices have no way of setting up a data portal. I'm working on this right now, where the only way to get data out of Bloomington,IN is to do FOIA requests every week/month over the data you want. This absolutely should be available via a portal, and not locked behind "in person, mail, fax at cost of .10$ a page".
tehlike 8y ago
With all the problems its government has, united states is still at it fostering innovation. Open data will bring innovation and accountability. Great move
- perfmode 8y ago
  > federal agencies should use evidence when they make public policy the cynic in me just figures that this moves the goal post such that special interest groups will adapt to produce the right evidence for the desired outcomes
  - CptFribble 8y ago
    You're probably right, but at least in that case there will still be a paper-trail of "data" that motivated entities can point to and make a case against. I would rather have bogus evidence in the official record, that I can analyze and challenge, than no evidence at all.
  - tehlike 8y ago
    Probably, but there'll be others punching through the data for other useful things.
- yxhuvud 8y ago
  Innovation? Other nations (like Sweden) have had its data public for many decades now.
  - akudha 8y ago
    UK is another great example. There is no "innovation" going on here. It is the data owned by the citizens, doesn't take that much effort to release it - they are collecting the data anyway, all they have to do is make it available to the general public. Still, a step in the right direction
    - tehlike 8y ago
      The innovation will come from vast amount of data coming from government agencies. That's what i meant. US government has much more room for improvement (because of its complex structure (federal vs state agencies, etc), large population and so on) - This is very different from pretty much the rest of the world.
HillaryBriss 8y ago
when I first saw this story, I thought it was a joke ala The Onion. so I looked for it on NYTimes, WaPo, CNN but didn't find it. i still can't find it there.
- danso 8y ago
  Open data is a relatively niche topic/policy in politics. It wouldn’t be front page news on a normal day, and so it’s not a surprise if it doesn’t make the weekend headlines, on a week in which the federal govt went into shutdown over the border wall, nevermind the resignation of the defense secretary.
- rdiddly 8y ago
  The part about using evidence to make decisions is a bit Onionesque, I concur.
- logicallee 8y ago
  What struck you as funny or The Onion-like about it?
ghaff 8y ago
For those who don't know, as the article mentions in passing, there's already quite a bit of data online available from data.gov (which was started by the Obama administration).
rpedela 8y ago
Does anyone know if this applies to PACER and its fees?
- jayess 8y ago
  Very unlikely.
- digiphile 8y ago
  It does not. Only CFO Act agencies in the executive branch. But there is a bill to open up PACER.
miguelrochefort 8y ago
Hopefully they release it in RDF.
dgellow 8y ago
As a next step I would love to see an exploration of a legal system where a change of a law or regulation is backed by both data (data and methodology directly referenced by the law) and a description of the expected impact. Something such as: > By changing law A related to B, we expect the increase of C to be at least D in the next E months. If not achieved, the change is reversed/reduced. Hopefully that would allow experimentation without taking the risk of creating a system too bad in case the implementation or policy isn’t good enough. Just dreaming here :)
- justin_oaks 8y ago
  I've dreamed the same thing. Since every law is a trade-off, having both positive and negative effects, I wish each law would enumerate the expected/possible positive and negative effects. In other words, I want the trade-off to be explicit and for the lawmakers to express why they believe the positive effects outweigh the negatives ones.
- deleted 8y ago
  [deleted]
rubyfan 8y ago
What type of policy implications does this actually have?
cantthinkofone 8y ago
It's good to see the US govt making an effort to step up its technical level. A vast if hidden problem in the political sphere is that most politicians do not have a technical education. This creates a serious misconfiguration of the govt alongside other centers of soft power like large cap tech companies. There is no way around it that these companies have to work with the government to secure public interests from 21st century threats. Just today I read an article about how black hat hackers are targeting outdated industrial control systems more vigorously than ever before. The government on its own without technical upgrades cannot face down this problem in its current condition. Which is why opening up data is a beneficial thing. Openness of data is a double edged sword. It will make malicious agents' job easier to have as much data as possible in a consistently machine readable format, but it will also help those on the other side. If tech is one of the things that can bolster and improve government, tech needs to work in the optimal environment. Which is one with open data.
dandare 8y ago
All I want for this Christmass is an authoritative list of all US federal agencies. I kid you not, there is no such list and the number of federal agencies is uncertain. From Wikipedia: >Legislative definitions of a federal agency are varied, and even contradictory, and the official United States Government Manual offers no definition. While the Administrative Procedure Act definition of "agency" applies to most executive branch agencies, Congress may define an agency however it chooses in enabling legislation, and subsequent litigation, often involving the Freedom of Information Act and the Government in the Sunshine Act. These further cloud attempts to enumerate a list of agencies.
- mehrdadn 8y ago
  What do you mean by federal agency (given that the definition itself is unclear)?
- acjohnson55 8y ago
  That seems like the issue is that the answer varies with chosen semantics, not unavailability of information.
- killjoywashere 8y ago
  Check the budget. Bills.
- showerst 8y ago
  It’s not quite an official list due to the issue you mentioned but anything that realistically rounds up to an “agency” will publish notices or regulations in the federal register, and the fedreg API has an endpoint to get all agencies.
  - Breza 8y ago
    That's a really clever idea!
- PerryCox 8y ago
  It seems like the solution here would be to have a government agency that counts how many government agencies there are. I believe that could be done with an executive order but correct me if I'm wrong.
jjt-yn_t 8y ago
I read that as s/data/secrets/g and thought "hmmm..." and then realized I had too many tabs open. Also, that it is the 1970's already.
erentz 8y ago
"The Open, Public, Electronic, and Necessary Government Data Act (AKA the OPEN Government Data Act)" Please, could we end this obsession with backronyms in congress. Perhaps some congressperson could create a suitably titled backronym act to rid us of backronym acts. This should just be the one singular "Government Data Act" and it should be amended as needed to cover any law changes around "Government Data".
- amyboyd 8y ago
  BACKRONYM Act: Banning Acronym Cleverness Keeping Representatives Occupied Nutting Your Mom Act (struggled with the NYM so gave up and went with something childish...)
  - missblit 8y ago
    Naming Your Measures / Mandates.
burtonator 8y ago
We should go the path of Sweden and make all IRS records public too...
pavon 8y ago
Has anyone read any good analysis about Title I of HR4174 requiring evidence when making public policy? Skimming the bill, I don't have a clue what the implications are other than requiring agencies to make some reports. It sounds like a good thing on the surface, but it has been a talking point of the Trump administration, and put into practice in a rather Orwellian fashion. Basically, the implied subtext is that regulators should not put any restrictions on industry unless the evidence is completely unambiguously in favor of the regulation. However, science is never 100% certain even at its best. And you can always drag up some study that contradicts the strong consensus of the field, whether by fluke or intentional design of the study. In other words "evidence based policymaking" has been euphemism for "must give alternative facts equal weight".
qwerty456127 8y ago
Cool! And they should also adopt UK-like government web design principles.
- philipashlock 8y ago
  See https://designsystem.digital.gov https://designsystem.digital.gov
75dvtwin 8y ago
Does anybody know a public source of net worth + annual earnings+ board positions of US Senator/congressman spouses+children ?
pteredactyl 8y ago
Tangent, but related shameless plug. Especially to San Franciscans: https://theconstituent.net https://theconstituent.net. It makes legislation easier to access. And eventually easier to engage with.
havermeyer 8y ago
I'm biased (I work on the BigQuery team) but I'm always excited to see more public datasets made available in BigQuery: https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/. It would be great to have government data available through a variety of cloud services with free exports. Some personal favorites among BigQuery public datasets include NOAA GHCN[0], the Census Bureau's Zip Code Tabulation Area [1], and FEC Campaign Finance [2]. [0] https://console.cloud.google.com/marketplace/details/noaa-public/ghcn-d?filter=solution-type:dataset&filter=category:climate&id=9d500d1d-fda4-4413-a789-d8786fd6592e https://console.cloud.google.com/marketplace/details/noaa-pu... [1] https://console.cloud.google.com/marketplace/details/bigquery-public-data/zipcode_area?filter=solution-type:dataset&filter=category:maps&id=b0de5205-0a3f-4f1f-8cd4-47cfa001de63 https://console.cloud.google.com/marketplace/details/bigquer... [2] https://console.cloud.google.com/marketplace/details/bigquery-public-data/fec-campaign-finance?filter=solution-type:dataset&filter=category:science-research&id=af2dc7e0-f9bf-4a10-9047-fbc5e43eecb7 https://console.cloud.google.com/marketplace/details/bigquer...
torgian 8y ago
Well that’s cool, but the wording worries me: public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security federal agencies should use evidence when they make public policy The word “should” is used in both one and two. If my time in the government taught me anything, it’s that “should” is only slightly stronger than “may”. If an instruction says “shall”, then it is required to be done.
systematical 8y ago
I've been working with the GPOs api. The engineer is quite responsive on github and the api is pretty snappy. Constantly asking for feedback and releasing new features. I think we're headed in the right direction. A shame my project isn't further a long for a shameless plug. If you're interested in informing the public on legislation, have experience on the hill or UI/UX experience hit me up. I'm just some dude with an idea who lives in a terminal. Money is secondary.
peterwwillis 8y ago
Too bad thing related to "security" won't be included. All I want for Christmas: accountability for the DoD budget, which for 2019 will be $717 Billion, the majority of the USG's discretionary spending. In 2015, an audit of the budget revealed $125 Billion in wasteful spending, and this was covered up. In 2016, the Office of Inspector General for DoD said that the Army made 6.5 Trillion in wrongful adjustments to its 2015 accounting. https://en.wikipedia.org/wiki/United_States_Department_of_Defense#Criticism https://en.wikipedia.org/wiki/United_States_Department_of_De... We still do not know how many contractors the DOD employs, or how much money they are paid, because the numbers are just not recorded. We do know, though, that the numbers and budgets we do have are often inaccurately reported, according to the DOD OIG. People whine a lot about paying taxes, but the politicians that always complain about taxes are extracting record amounts of tax money for a military that is mismanaged, doesn't do its accounting properly, can't build modern fighting vehicles, and doesn't record basic information like how many people they employ.
tabtab 8y ago
One has to be careful here. Lots of internal details may give trolls and conspiracy theorists fodder to generate controversy and fake news, often by taking things out of context. It may slow work-flow because workers will be hesitant to write anything without expensive pre-vetting.

1130 points by danso 8y ago

> there was a carve out “for data that does not concern monetary policy,” which relates to the Federal Reserve, among others. Does that mean they won't open data that affects monetary policy or that they will only open data that affects monetary policy? Either way that seems huge.

deleted 8y ago
[deleted]

jcriddle4 8y ago

Interesting that the Federal Reserve was exempted from this legislation.

lykr0n 8y ago
The Federal Reserve [board] bases its data on data from the Reserve Banks (think Federal Reserve of New York)- which are private and get data from banks chartered in the region (so the New York Fed is partially owned by JP Morgan). Requiring the Federal Reserve [board] to release underlying data could get sticky and be a legal minefield.
- zrail 8y ago
  Also the St Louis Fed provides provides FRED, which is pretty open for most purposes. https://fred.stlouisfed.org/ https://fred.stlouisfed.org/
indigochill 8y ago
Is it? My understanding is that although the board is appointed by the President and Congress, once appointed they don't answer to the government, so their data isn't really "government data". But I'll be the first to admit I know very little about how the Federal Reserve works.
- IncRnd 8y ago
  That's a real shame, because we wouldn't have a federal debt if we didn't have the Federal Reserve. More people should actually know how it works.
n-exploit 8y ago
Or course, some of the most important economic data sets won't be shared. It's almost like they want to cover up the systemic disempowerment of the American electorate.
dannyw 8y ago
The federal reserve is a curious semi-company-agency with apppontment by president and Congress, but profits going to shareholders (large banks).
IncRnd 8y ago
You can access the Reserve's information online as part of the Federal Reserve System's Open Government webpage implementing the Open Government Directive from 2009: https://www.federalreserve.gov/open/open.htm https://www.federalreserve.gov/open/open.htm Despite all that, the Federal Reserve is not a government institution, so such legislation would be similar to singling out your business to post all information in machine readable format online.

mizzao 8y ago

Can someone who is more familiar with the actual policy shed some light on this?

danso 8y ago
FWIW, the author, Alex Howard (who I'm friends-via-Twitter with), is as familiar with this as anyone. He was previously a senior analyst at the Sunlight Foundation, which is a prominent open-government organization: https://sunlightfoundation.com/author/ahoward/ https://sunlightfoundation.com/author/ahoward/ The submitted post includes a link [0] to an article he wrote a couple days back, which provides more context on "How did open government data get into the US Code?", including the nitty-gritty of how the original bill was proposed in the last session but ultimately left out of legislation. Howard writes that the legislation was "one of the primary legislative priorities for me during my years as a senior analyst and then deputy director at the Sunlight Foundation" [0] https://e-pluribusunum.org/2018/12/20/senate-passes-evidence-based-policymaking-bill-setting-up-historic-win-for-open-government-data/ https://e-pluribusunum.org/2018/12/20/senate-passes-evidence...
- cptskippy 8y ago
  > (who I'm friends-via-Twitter with) Full disclosure? Odd brag?
  - danso 8y ago
    It's someone who I mostly learned/knew through social media and am friendly with, even though we've never worked together and maybe have met in person a couple times at conferences.
    - cptskippy 8y ago
      I wasn't sure if you were disclosing the relationship because of the nature of your comment or if you were just bragging about knowing them. It is/was odd.

pratheekrebala 8y ago

FOIA officers will still find a way to send me scanned PDFs of spreadsheets.

ekianjo 8y ago
You obviously did not read the article. > public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security
- clessg 8y ago
  https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html > Please don't insinuate that someone hasn't read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."
- danso 8y ago
  This policy exists in many states. But plenty of local reporters have tales about how officials respond with PDFs because they either don't realize that the PDFs are generated from a (legacy) system that could output XLS/CSV, or because they fear that Excel sheets/text files are alterable. The benefit of having the law, though, makes it easier to convince officials to tell their IT people to get it right, under threat of lawsuit.
- mikestew 8y ago
  Not only are you being rude by implying the article went unread, you’re also showing an astounding lack of imagination (perhaps stifled by a need to point out the errors of others). Parent is saying that despite whatever parts of the law you care to snarkily quote, government employees will find a way to screw it up.
- Rexxar 8y ago
  Scanned pdf of a spreadsheet is clearly machine-readable, as you can read this document with your computer.

todipa 8y ago

Open data is the first step. I wish I could do more business with my gov't (State and Local) through the internet. "Copies of documents cannot be ordered through this website, by email or over the telephone." Only fax and snail mail... https://www.dos.ny.gov/corps/faq_copies.page.asp https://www.dos.ny.gov/corps/faq_copies.page.asp

weiming 8y ago
Is this because they are required to make this info accessible, but are not required to make it easy?
- jaggederest 8y ago
  More importantly, there's no specific funding set aside to support the effort to make it available. Government at all levels lives and dies by funding.
ehsankia 8y ago
It's worth noting that easy to access isn't always the best. A great example of this is mugshots, which are open data, but there are now websites that automatically scrape and index people's mugshots and use SEO to rank highly on searches. They then blackmail the person to have it removed. As long as the data is accessible and reasonable easy to get, where journalists and data scientists who really care about the data can get it, then I think we're in a good place.
deleted 8y ago
[deleted]
soared 8y ago
There are fairly cheap services that turn your emails into faxes. I've used them previously and they worked great.

yohann305 8y ago

I wonder if this open data can be (will be?) used by a foreign entity and be turned against the american people? It should be open but not free-for-all

icebraining 8y ago
"open but not free-for-all" What would this mean in practice?
- yohann305 8y ago
  Available to us citizens only
  - matthewmacleod 8y ago
    That is–not to put too fine a point on it–a dumb idea. It’s essentially impossible to enforce, and doesn’t stop the data from being accessed by anybody else.
  - quickthrower2 8y ago
    Waste of time, easy to get around by asking an American friend to forward the data.
astazangasta 8y ago
I'm not convinced there are really any important national security secrets other than passcodes and so on. The rest seems to be excuses for hiding corruption of various kinds. Secrecy in government is incompatible with democracy by definition.
- rjf72 8y ago
  You have to think about how people respond to things. Social media in its current state has basically turned into a system that takes an event, removes all context, spins it in the most negative way imaginable, and then dogpiles on it to no end -- virtue signaling for imaginary points. There are often lots of things in government, and in life in general, where you must choose between two very negative choices. And the decision there does not necessarily imply that you're in any way satisfied with it, but you see it as the decision least likely to produce an awful outcome. Social media and this sort of logic are wholly and completely incompatible. Take Khashoggi as a contemporary example. Undoubtedly there have been countless heavily classified conversations weighting the pros and cons of any action against Saudi Arabia or MBS as a result of this. How we are extremely dependent on Saudi Arabia for reasons outside the scope of this post, and they know this. And this is all happening at a time when Saudi:US relations souring would greatly stand to strengthen the geopolitical position and power of nations such as China and Russia. In my opinion it's extremely likely we will do nothing, but that's because it's better than doing something. If this was not classified, social media would throw a nonstop hissy fit. Not because they actually care or think we're making the wrong decision, but because it's an incredibly easy way score those imaginary points and followers which are the hottest commodity since sliced bread. Another issue here is that knowing the mechanisms of our decision making here, and how close we came to 'breaking', would be incredibly valuable information to nations such as Saudi Arabia which they could then use to exploit the US. So while I do agree with you that classification is very often abused, I also think there are indeed vast amounts of information that must be classified. People are not mature enough to impartially process our decision making processes, and the details of such processes would provide invaluable information to other nations which could then be used to exploit the US.

compute_me 8y ago

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aiming to make open data more accessible and to democratize data-science: https://data-in.place/ https://data-in.place/ ...

prepend 8y ago
This is a cool project. I was trying to find the source for the project on the page. There’s an oss page [0] but that’s about the software used. Is this an open source project? Or what’s the way for licensing to use with US data? [0] https://data-in.place/open-source https://data-in.place/open-source
kerkeslager 8y ago
As long as those spreadsheets/database files are accessible to someone with technical skill, people can pull in the data and use tools to make it more accessible and useful. Ideally, yes, the data is useful to begin with, but as long as it's available, there's nothing stopping individuals with the skills from making it useful. Of course, there are exceptions: the PDFs that are often provided by the prosecution as part of the discovery process are prohibitively difficult to deal with, and should be considered a violation of Brady vs. Maryland, IMO.
- ethbro 8y ago
  Exactly. Accessible and machine readable are necessary but not sufficient. Thankfully, civil society can reasonably pick up the slack. In regards to modern day transparency requirements, it seems like laws should include a reasonableness clause. Making records available to the public but requiring them to be hand photocopied vs. making them available in electronic form in a custom format. Both open. But two very different magnitudes of effort.
- _bohm 8y ago

xivzgrev 8y ago

Finally a bipartisan moment, looks like most reps were onboard.

friedman23 8y ago
They have to be for the law to pass?
- srtjstjsj 8y ago
  Not just most reps were onboard; nearly all reps voted in favor

yostrovs 8y ago

I'm glad they were able to do it before the new Congress came on board.

srtjstjsj 8y ago
What has that to do with anything? The bill had large majority bipartisan support (which makes me suspect it's toothless...)
- yostrovs 8y ago
  There's a reason it was passed during the lame duck session of the Senate, just before change of control. That's when urgent matters that are guaranteed to be ignored by the next Congress are handled. I'm surprised this needs explaining.

crankylinuxuser 8y ago

The first problem here is the data formats state and federal governments use. You'll see a hodgepodge, but primarily MS Office $version . The biggest problem with this terrible binary format is that metadata can leak a great deal that should have not been released. So this leads to PDF output of word/excel. The next area is that especially local government offices have no way of setting up a data portal. I'm working on this right now, where the only way to get data out of Bloomington,IN is to do FOIA requests every week/month over the data you want. This absolutely should be available via a portal, and not locked behind "in person, mail, fax at cost of .10$ a page".

tehlike 8y ago

With all the problems its government has, united states is still at it fostering innovation. Open data will bring innovation and accountability. Great move

perfmode 8y ago
> federal agencies should use evidence when they make public policy the cynic in me just figures that this moves the goal post such that special interest groups will adapt to produce the right evidence for the desired outcomes
- CptFribble 8y ago
  You're probably right, but at least in that case there will still be a paper-trail of "data" that motivated entities can point to and make a case against. I would rather have bogus evidence in the official record, that I can analyze and challenge, than no evidence at all.
- tehlike 8y ago
  Probably, but there'll be others punching through the data for other useful things.
yxhuvud 8y ago
Innovation? Other nations (like Sweden) have had its data public for many decades now.
- akudha 8y ago
  UK is another great example. There is no "innovation" going on here. It is the data owned by the citizens, doesn't take that much effort to release it - they are collecting the data anyway, all they have to do is make it available to the general public. Still, a step in the right direction
  - tehlike 8y ago
    The innovation will come from vast amount of data coming from government agencies. That's what i meant. US government has much more room for improvement (because of its complex structure (federal vs state agencies, etc), large population and so on) - This is very different from pretty much the rest of the world.

HillaryBriss 8y ago

when I first saw this story, I thought it was a joke ala The Onion. so I looked for it on NYTimes, WaPo, CNN but didn't find it. i still can't find it there.

danso 8y ago
Open data is a relatively niche topic/policy in politics. It wouldn’t be front page news on a normal day, and so it’s not a surprise if it doesn’t make the weekend headlines, on a week in which the federal govt went into shutdown over the border wall, nevermind the resignation of the defense secretary.
rdiddly 8y ago
The part about using evidence to make decisions is a bit Onionesque, I concur.
logicallee 8y ago
What struck you as funny or The Onion-like about it?

ghaff 8y ago

For those who don't know, as the article mentions in passing, there's already quite a bit of data online available from data.gov (which was started by the Obama administration).

rpedela 8y ago

Does anyone know if this applies to PACER and its fees?

jayess 8y ago
Very unlikely.
digiphile 8y ago
It does not. Only CFO Act agencies in the executive branch. But there is a bill to open up PACER.

miguelrochefort 8y ago

Hopefully they release it in RDF.

dgellow 8y ago

As a next step I would love to see an exploration of a legal system where a change of a law or regulation is backed by both data (data and methodology directly referenced by the law) and a description of the expected impact. Something such as: > By changing law A related to B, we expect the increase of C to be at least D in the next E months. If not achieved, the change is reversed/reduced. Hopefully that would allow experimentation without taking the risk of creating a system too bad in case the implementation or policy isn’t good enough. Just dreaming here :)

justin_oaks 8y ago
I've dreamed the same thing. Since every law is a trade-off, having both positive and negative effects, I wish each law would enumerate the expected/possible positive and negative effects. In other words, I want the trade-off to be explicit and for the lawmakers to express why they believe the positive effects outweigh the negatives ones.
deleted 8y ago
[deleted]

rubyfan 8y ago

What type of policy implications does this actually have?

cantthinkofone 8y ago

It's good to see the US govt making an effort to step up its technical level. A vast if hidden problem in the political sphere is that most politicians do not have a technical education. This creates a serious misconfiguration of the govt alongside other centers of soft power like large cap tech companies. There is no way around it that these companies have to work with the government to secure public interests from 21st century threats. Just today I read an article about how black hat hackers are targeting outdated industrial control systems more vigorously than ever before. The government on its own without technical upgrades cannot face down this problem in its current condition. Which is why opening up data is a beneficial thing. Openness of data is a double edged sword. It will make malicious agents' job easier to have as much data as possible in a consistently machine readable format, but it will also help those on the other side. If tech is one of the things that can bolster and improve government, tech needs to work in the optimal environment. Which is one with open data.

dandare 8y ago

All I want for this Christmass is an authoritative list of all US federal agencies. I kid you not, there is no such list and the number of federal agencies is uncertain. From Wikipedia: >Legislative definitions of a federal agency are varied, and even contradictory, and the official United States Government Manual offers no definition. While the Administrative Procedure Act definition of "agency" applies to most executive branch agencies, Congress may define an agency however it chooses in enabling legislation, and subsequent litigation, often involving the Freedom of Information Act and the Government in the Sunshine Act. These further cloud attempts to enumerate a list of agencies.

mehrdadn 8y ago
What do you mean by federal agency (given that the definition itself is unclear)?
acjohnson55 8y ago
That seems like the issue is that the answer varies with chosen semantics, not unavailability of information.
killjoywashere 8y ago
Check the budget. Bills.
showerst 8y ago
It’s not quite an official list due to the issue you mentioned but anything that realistically rounds up to an “agency” will publish notices or regulations in the federal register, and the fedreg API has an endpoint to get all agencies.
- Breza 8y ago
  That's a really clever idea!
PerryCox 8y ago
It seems like the solution here would be to have a government agency that counts how many government agencies there are. I believe that could be done with an executive order but correct me if I'm wrong.

jjt-yn_t 8y ago

I read that as s/data/secrets/g and thought "hmmm..." and then realized I had too many tabs open. Also, that it is the 1970's already.

erentz 8y ago

"The Open, Public, Electronic, and Necessary Government Data Act (AKA the OPEN Government Data Act)" Please, could we end this obsession with backronyms in congress. Perhaps some congressperson could create a suitably titled backronym act to rid us of backronym acts. This should just be the one singular "Government Data Act" and it should be amended as needed to cover any law changes around "Government Data".

amyboyd 8y ago
BACKRONYM Act: Banning Acronym Cleverness Keeping Representatives Occupied Nutting Your Mom Act (struggled with the NYM so gave up and went with something childish...)
- missblit 8y ago
  Naming Your Measures / Mandates.

burtonator 8y ago

We should go the path of Sweden and make all IRS records public too...

pavon 8y ago

Has anyone read any good analysis about Title I of HR4174 requiring evidence when making public policy? Skimming the bill, I don't have a clue what the implications are other than requiring agencies to make some reports. It sounds like a good thing on the surface, but it has been a talking point of the Trump administration, and put into practice in a rather Orwellian fashion. Basically, the implied subtext is that regulators should not put any restrictions on industry unless the evidence is completely unambiguously in favor of the regulation. However, science is never 100% certain even at its best. And you can always drag up some study that contradicts the strong consensus of the field, whether by fluke or intentional design of the study. In other words "evidence based policymaking" has been euphemism for "must give alternative facts equal weight".

qwerty456127 8y ago

Cool! And they should also adopt UK-like government web design principles.

philipashlock 8y ago
See https://designsystem.digital.gov https://designsystem.digital.gov

75dvtwin 8y ago

Does anybody know a public source of net worth + annual earnings+ board positions of US Senator/congressman spouses+children ?

pteredactyl 8y ago

Tangent, but related shameless plug. Especially to San Franciscans: https://theconstituent.net https://theconstituent.net. It makes legislation easier to access. And eventually easier to engage with.

havermeyer 8y ago

I'm biased (I work on the BigQuery team) but I'm always excited to see more public datasets made available in BigQuery: https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/. It would be great to have government data available through a variety of cloud services with free exports. Some personal favorites among BigQuery public datasets include NOAA GHCN[0], the Census Bureau's Zip Code Tabulation Area [1], and FEC Campaign Finance [2]. [0] https://console.cloud.google.com/marketplace/details/noaa-public/ghcn-d?filter=solution-type:dataset&filter=category:climate&id=9d500d1d-fda4-4413-a789-d8786fd6592e https://console.cloud.google.com/marketplace/details/noaa-pu... [1] https://console.cloud.google.com/marketplace/details/bigquery-public-data/zipcode_area?filter=solution-type:dataset&filter=category:maps&id=b0de5205-0a3f-4f1f-8cd4-47cfa001de63 https://console.cloud.google.com/marketplace/details/bigquer... [2] https://console.cloud.google.com/marketplace/details/bigquery-public-data/fec-campaign-finance?filter=solution-type:dataset&filter=category:science-research&id=af2dc7e0-f9bf-4a10-9047-fbc5e43eecb7 https://console.cloud.google.com/marketplace/details/bigquer...

torgian 8y ago

Well that’s cool, but the wording worries me: public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security federal agencies should use evidence when they make public policy The word “should” is used in both one and two. If my time in the government taught me anything, it’s that “should” is only slightly stronger than “may”. If an instruction says “shall”, then it is required to be done.

systematical 8y ago

I've been working with the GPOs api. The engineer is quite responsive on github and the api is pretty snappy. Constantly asking for feedback and releasing new features. I think we're headed in the right direction. A shame my project isn't further a long for a shameless plug. If you're interested in informing the public on legislation, have experience on the hill or UI/UX experience hit me up. I'm just some dude with an idea who lives in a terminal. Money is secondary.

peterwwillis 8y ago

Too bad thing related to "security" won't be included. All I want for Christmas: accountability for the DoD budget, which for 2019 will be $717 Billion, the majority of the USG's discretionary spending. In 2015, an audit of the budget revealed $125 Billion in wasteful spending, and this was covered up. In 2016, the Office of Inspector General for DoD said that the Army made 6.5 Trillion in wrongful adjustments to its 2015 accounting. https://en.wikipedia.org/wiki/United_States_Department_of_Defense#Criticism https://en.wikipedia.org/wiki/United_States_Department_of_De... We still do not know how many contractors the DOD employs, or how much money they are paid, because the numbers are just not recorded. We do know, though, that the numbers and budgets we do have are often inaccurately reported, according to the DOD OIG. People whine a lot about paying taxes, but the politicians that always complain about taxes are extracting record amounts of tax money for a military that is mismanaged, doesn't do its accounting properly, can't build modern fighting vehicles, and doesn't record basic information like how many people they employ.

tabtab 8y ago

One has to be careful here. Lots of internal details may give trolls and conspiracy theorists fodder to generate controversy and fake news, often by taking things out of context. It may slow work-flow because workers will be hesitant to write anything without expensive pre-vetting.