9 ms·
President Signs Government-Wide Open Data Bill
- mLuby 8y ago>The OPEN Government Data Act requires all non-sensitive government data to be made available in open and machine-readable formats by default. That sounds pretty awesome (and expensive)!
- dak1 8y agoNot if you expand what's considered sensitive!
- mooman219 8y agoI'd always want to make sure no PII is accidentally leaked. Example: in 1997, researchers at MIT showed that using only gender, date of birth and ZIP code, it is possible to identify the majority of US residents! They proved their point by identifying the Massachusetts governor's medical records in a publicly-available dataset that was presumed anonymous. In 2010, Netflix published an “anonymized” dataset of movie ratings by users. After it was released to the public, researchers were able to identify many Netflix users, even though the dataset only contained user ID, movie, rating, and rating date.
- AckSyn 8y agoOf course they used identifying information to actually identify people. This is 101 level of forensics and research.
- aaomidi 8y agoMeh. Voter records are all basically public and most have full name, address, phone number and even sometimes party affiliations. I'd argue that keeping those public is also a net good for society.
- skookumchuck 8y agoThere are two issues in tension here. One is a right to privacy, but the other is a right to audit who is actually voting. If the voter rolls were secret, there's nothing to stop a massive fraud by those in power to stay in power.
- mattferderer 8y agoParty affiliation should not be public record. I would like to understand why it is.
- hndamien 8y agoKind of defeats the purpose of a public ballot.
- anth_anm 8y agoNo, you can have the lists of voters without having any information about who they voted for (or in this case, party affiliation which gives who they probably voted for).
- hndamien 8y agoWow! Did I write "public" - I meant "secret" - it's been a long day. Of all the comments I've made that should have been down voted, the one I made above was it. HN gods are having mercy on me it seems.
- skookumchuck 8y agoIn the US we have a secret ballot. Some effort is gone to to not collect information about what you voted for, only that you voted. Countries with sham elections tend to be open ballot, as the voters need to show they support the current ruler. This is why the selected candidate wins with 98% of the vote or some such. You can't even get 70% of people to agree on which direction the sun rises. A secret ballot is necessary for free elections.
- anth_anm 8y agoIt's not just PII, it's anything the government just doesn't want people to know. For whatever reason. I'm just pointing this out because I'd like to see a movement towards actually more open government.
- tvanantwerp 8y agoWhere I work, a tax policy think tank, we purchase from the IRS an anonymized data set of about 100k sample tax returns that we use for modeling the effects of changes to tax policy. We've got an agreement with the IRS that we can never share that dataset, and I'm pretty sure this is why. While I'd love to be able to make our tax model results more transparent, the risk of de-anonymization is too high. I think another tax policy group is trying to create a synthetic dataset that is close to the sample in terms of outputs but is entirely made up, so that it could be used for verification of results by third parties--I hope they succeed.
- mmmmmmmmm 8y agoYou've been doing some bathroom reading I see.
- brokenmachine 8y ago> researchers at MIT showed that using only gender, date of birth and ZIP code, it is possible to identify the majority of US residents! Source? How were they able to do that?
- maxerickson 8y agoThere are large databases that have names, zipcodes and dates of birth in them. If you include year, birth dates carry an awful lot of information.
- Someone 8y ago2 genders times ~25,000 date of births (70-ish years) times 40,000 zip codes equals 2 billion. The population won’t be uniformly distributed over date of birth or zip code, but that gives a factor of six or so leeway. The 87% paper is https://dataprivacylab.org/projects/identifiability/paper1.pdf https://dataprivacylab.org/projects/identifiability/paper1.p.... It says: ”It was found that 87% (216 million of 248 million) of the population in the United States had reported characteristics that likely made them unique based only on {5-digit ZIP, gender, date of birth}. About half of the U.S. population (132 million of 248 million or 53%) are likely to be uniquely identified by only {place, gender, date of birth}, where place is basically the city, town, or municipality in which the person resides. And even at the county level, {county, gender, date of birth} are likely to uniquely identify 18% of the U.S. population. In general, few characteristics are needed to uniquely identify a person.” More info likely available via http://latanyasweeney.org/work/identifiability.html http://latanyasweeney.org/work/identifiability.html (discussed in https://news.ycombinator.com/item?id=2942967 https://news.ycombinator.com/item?id=2942967)
- wodenokoto 8y agoWhat did they cross reference the IDs with for the Netflix data?
- bilbo0s 8y agoTrue. Kind of depends on what they say is "sensitive". Probably not much of a facility for appealing that designation either. I sorta look at it like the "First Step" prison reform act, it's only the start. Baby steps.
- skookumchuck 8y agoI suspect it's more expensive to make it not machine-readable, considering that it likely starts out in a computer, and if it is not stored that way, it become essentially inaccessible even to the people who stored it.
- Buttons840 8y agoDoes it apply to state governments?
- rmason 8y agoNo, but I believe it can't help but be an indicator of the trend toward open government. Right now there are around six states with open data laws. I have been working for two years with a group trying to add Michigan to the list. Tomorrow I am publishing an open letter to our governor asking her to support our efforts. I will post a link on HN.
- macawfish 8y ago*This will make it much easier for him to pass the data off to Russian spies
- maxxxxx 8y agoI am surprised that in the current political climate any reasonable laws can pass. My only explanation is that there were no lobbyists fighting it.
- HumanDrivenDev 8y agoLobbyists are probably fighting laws you don't like as well.
- maxxxxx 8y agoNo question.
- TomMckenny 8y agoEven a broken clock is right twice a day.
- craftyguy 8y agoUnless it is a digital clock, then it is never right.
- maxsavin 8y agoTime isn't real
- deleted 8y ago[deleted]
- yellowapple 8y agoDepends on how broken. If the hour and minute hands are swapped it could be right many times more than merely twice.
- vilda 8y agoAnother example is the First Step Act that has been signed Dec 21 last year. It is being considered especially important for poor and black people.
- kgwxd 8y agoDoes he get paid to work today? Edit: It was an honest question I wanted to know the answer to, but I've got my answer.
- zbyte64 8y agoDepends, is the secret service still able to pay trump hotel for their stay?
- iratewizard 8y agoThe whole dollar
- ajr0 8y ago> The Chief Data Officer (CDO) will "(1) be responsible for lifecycle data management" I am very interested in what this type of lifecycle might look like considering most data I feel should be kept forever. I wonder how a lifecycle might collide with the challenges that bit rot[0] are dealing with. [0] https://www.theguardian.com/technology/2015/feb/13/what-is-bit-rot-and-is-vint-cerf-right-to-be-worried https://www.theguardian.com/technology/2015/feb/13/what-is-b...
- ocdtrekkie 8y agoThis is already getting really exciting. Some government entities (including lower levels like local, county, and state governments) have moved to digitizing their old paper and microfilm records. But if they're expected to maintain many types of records essentially forever, it places a constant burden to continue to update and migrate data in perpetuity, whereas paper or microfilm can sit in a box in a closet for decades. For the most part, common file formats like PDFs, JPGs, and TIFs are likely to be understood for a very, very long time, but you don't just have file storage, you have systems to manage, index, and find those files, and those systems are likely to need constant maintenance.
- talmand 8y agoIt's not like paper in boxes don't have their own maintenance costs.
- fixermark 8y agoBut different and arguably on-average cheaper costs. Paper can rot if it's not kept climate-controlled, but not nearly as fast as bits stored to non-volatile media if the power goes out and there isn't a hot backup.
- pbhjpbhj 8y agoYou can keep a whole warehouse worth of paper files on one drive though. 1TB is like 500M pages of text (1000 pallets).
- mrcactu5 8y agohow did we convince Trump to do that?
- anonytrary 8y agoThe media would make Trump to be a 5 year old toddler, but the guy isn't stupid. I'm sure there are millions of people that actually think Trump is a complete idiot. The truth is often less colorful than people would hope. Trump is like anyone else -- he has made good choices and bad choices.
- bhhaskin 8y agoI think people lose sight of this. You don't make (or keep) that kind of money by being stupid. Ethics is a different story.
- welcome_dragon 8y agoWtf guys? You can't have a reasonable discussion like this!
- anonytrary 8y agoHonestly, I'm a bit surprised that people on HN downvote literally anything that is neutral on Trump. "Orange man bad" is an epidemic that affects even the smart people.
- bhhaskin 8y agoIt's a pretty sad state of affairs. Makes it nearly impossible to have any sort of honest conversation about politics, and definitely has a chilling effect. I debated even making the comment because I knew it would draw down votes.
- dmix 8y agoI think the entire point of viciously attacking anyone who shows any sign of being part of the "opponents" ideology is to chill that persons speech. They just don't realize that type of tactic not only fails to silence people but also (quietly) pushes tons of people who normally don't have a horse in the game to the other side. There are plenty of politically ambivalent people who don't want a world ruled by an ideology opposed to open debate/free thought and which actively seeks to destroy context/intention in language in the name of "progress".
- digiphile 8y agoI wrote up this news here: https://e-pluribusunum.org/2019/01/15/president-trump-signs-historic-open-government-data-bill-into-law/ https://e-pluribusunum.org/2019/01/15/president-trump-signs-... This was the subject of a previous thread, here: https://news.ycombinator.com/item?id=18746132 https://news.ycombinator.com/item?id=18746132
- skookumchuck 8y ago> I wrote up this news here: "the American president who has done the most to damage democracy in modern history" Your article starts off with an unsupported opinion irrelevant to the issue, it's not very auspicious for the rest of it.
- PaulBGD_ 8y agoI know the author doesn't claim to be unbiased, but it's rather off-putting to start an article like that.
- digiphile 8y agoIt's rather off-putting to have a corrupt, amoral president who uses transparency as a weapon against his political opponents, lies, and doesn't show respect for civil servants.
- deleted 8y ago[deleted]
- digiphile 8y agoI was the deputy director of the Sunlight Foundation and have covered the space for a decade. I tracked Trump's record on open government and documented it daily: https://sunlightfoundation.com/tracking-trumps-attacks-on-transparency/ https://sunlightfoundation.com/tracking-trumps-attacks-on-tr... It is, in other words, just my opinion, but a statement of fact. Moreover, bill in question is the OPEN Government Data Act. This president's record on transparency, accountability, ethics and civic engagement is atrocious. Therefore, these issues are directly related.
- earth2mars 8y agoSomehow I saw "re" before the "Signs" that got me excited!
- En_gr_Student 8y agoThis sounds like an amazing thing! Anyone serious does their work reproducibly. This shouldn't add more than a little bit about storage on devices and paper, in terms of costs.
- torstenvl 8y agoI am deeply afraid of the impact of this law. The amount of meta-work required to consolidate and annotate data we collect, in order to prepare it for public consumption, seems likely to hurt government efficiency. In addition to the administrative burden, it appears to ignore the fact that non-sensitive information, in sufficient quantity and correlation, becomes sensitive information. Perhaps my skepticism is misplaced, but my initial reaction is that this sounds better in the abstract than it will turn out to be in practice.
- da_chicken 8y agoInefficiency and high expense is the primary burden of open governments, representative governments, and democratic governments. If you want a cheap, efficient government, you want an absolute monarchy. This is why corporations tend to be structured into rather strict hierarchies that bear no small resemblance to feudal kingdoms. That's also why they're terrible at meeting worker demands.
- gaius 8y agoTrump can’t do anything right, can he?
- deleted 8y ago[deleted]
- whatshisface 8y ago>I am deeply afraid It's a meta-comment, but this term sounds slightly misplaced when you're talking about bureaucrats having to do more paperwork.
- rising-sky 8y agoMight not be misplaced if the commenter _is_ a bureaucrat
- jointpdf 8y ago
- tracker1 8y agoHaven't dug in... does this include data as part of government funded grant studies? It is a nice start to more open data from the govt.
- jointpdf 8y agoI searched/skimmed through the full text of the bill, and it doesn't seem that it applies to data used/generated by grant-based research (or other projects). The language of the bill is heavily focused on executive agencies. That would be ideal, although a large portion of grant funding goes to medical research (e.g. via NIH), where it would be difficult to anonymize the data. They could require that it be sanitized (differential privacy, etc.), but I don't know how that could be verified effectively/efficiently. The grants process is quite time-consuming for the grantees (and the grantors) without this requirement. *not a lawyer
- count 8y agoData generated by a researcher on a government grant generally includes a provision in the grant that says the govt owns the data, doesn't it? And if the govt owns it, this seems to indicate it should be open...
- jointpdf 8y agoI'm not really knowledgeable enough on policy to know how this might interact with the new law. But based on my reading, it seems that institutions conducting federally-funded research are required to retain any data generated by the research, but not necessarily that the government "owns" it, per se. There was a change in policy in 1999 that required the ability for the public to use FOIA to access grant-funded research data, with some limitations. "To balance the need for public access while protecting the research process, OMB’s revision limits the kinds of data that will be made accessible (it excludes personal and business-related confidential data) and limits applicability to federally funded data relating to published research findings produced under a federal award and used in developing an agency action that has the force and effect of law." https://fas.org/sgp/crs/secrecy/R42983.pdf https://fas.org/sgp/crs/secrecy/R42983.pdf
- ams6110 8y agoIs this the first act with a recursive name acronym?
- crabl 8y agoThe deep irony here is that https://www.data.gov/ https://www.data.gov/ is still down due to the government shutdown.
- e40 8y agoCan we stop with these comments? We get it, and we all know there's a shutdown. It doesn't add anything to the discussion.
- randyrand 8y agoJust curious, do government services really not have any reserve funding? It seems like avoiding shutdowns could be solved pretty easily by having reserve funds (at least for 1-3 months or so). But perhaps that's the point. Shutdowns are supposed to be inconvenient.
- ghostly_s 8y agoAgencies would need legislation authorizing them to retain funds for this purpose.
- anth_anm 8y agoI don't think there is any point, it's just poor government design in general.
- Maxious 8y agoIndeed. "Many agencies, particularly the military, would intentionally run out of money, obligating Congress to provide additional funds to avoid breaching contracts." https://en.wikipedia.org/wiki/Antideficiency_Act https://en.wikipedia.org/wiki/Antideficiency_Act
- mschuster91 8y ago> It seems like avoiding shutdowns could be solved pretty easily by having reserve funds (at least for 1-3 months or so Or by having the government continue working according to last year's budget planning, like many other countries do. I know of no government shutdown in the Western countries except in the US. Even for "third world" countries such things would be disastrous as armies don't like not being fed and paid.
- teddyh 8y agoI am reminded of the "API" decision made by Jeff Bezos at Amazon, as famously described by Steve Yegge: So one day Jeff Bezos issued a mandate. He's doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion -- back around 2002 I think, plus or minus a year -- he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses. His Big Mandate went something along these lines: 1) All teams will henceforth expose their data and functionality through service interfaces. 2) Teams must communicate with each other through these interfaces. 3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network. 4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care. 5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. 6) Anyone who doesn't do this will be fired. https://plus.google.com/+RipRowan/posts/eVeouesvaVX https://plus.google.com/+RipRowan/posts/eVeouesvaVX
- randyrand 8y agoThis sounds "great", but how exactly does your team that works on say, a matrix library for computer graphics, expose its data over the network? Perhaps instead of it being "all teams" it should be "all cpu processes"?
- sephoric 8y agoThis is how Bezos built AWS, so he meant services that are needed to build websites, such as databases, servers, containers, queues, caches, storage...
- anth_anm 8y ago> This is how Bezos built AWS Bezos didn't build AWS, people who worked for him did. They also didn't do it as part of the normal day to day at Amazon, no matter how much Amazon pushes the "it's the tech we use already" narrative.
- RickJWagner 8y agoAt least two things to like in this: 1) Open is better 2) Bipartisan legislation, indicates some progress is possible