9 ms·
Gaffer: Large-scale graph database by GCHQ
- aselzer 11y agoLove the way all the tests use "customer" and "product" nodes.
- swalsh 11y agoHaha i wonder what the product is when you're a spy agency
- cs02rm0 11y agoIt smells very sanitised, the code comments are oddly uniform. Still, tax payers money is used to write this stuff, good on them for open sourcing it.
- paulgayham 11y agoAlmost like contractors were involved, and it was written to specs.
- shifte 11y agoIt certainly gives you a feel for how they do development internally. End of the readme mentions a new version coming soon, instead of say, iterating on this one. Also, the main contributor is stripped of any personally identifiable information.
- executesorder66 11y ago> stripped of any personally identifiable information. The hypocrisy is staggering.
- mark_l_watson 11y agoInteresting to get open source from the British sort-of equivalent to the NSA. I just looked at the code: Java code that sits on top of the Hadoop file system. Supports date-binned data storage so it looks applicable to systems where you want to toss out old data occasionally.
- acveilleux 11y agoYep, and the Accummulo store it's built upon was open-sourced by the NSA.
- josephcooney 11y agoThe NSA released their Nifi project to the Apache foundation not long ago....we live in strange times.
- akerro 11y agoAny ideas how they could use it?
- tomschlick 11y agoGraph databases are perfect for identifying connections between who talks to who, relatives, contacts, etc.
- samstave 11y agoDidn't echelon effectively invent this concept? oracle was founded to serve the CIA for fucks sake.
- acveilleux 11y agoEvery oppressive regime's secret apparatus is basically a large scale database management system. Whether that's with files and folders a la Stasi or with huge data centers in Utah... Given that, it's not surprising they'd be involved in funding and doing R&D on database systems.
- Hermel 11y agoIronically, "Gaffer" is the German word for rubberneck.
- genericpseudo 11y agoBritish English word for "boss", traditionally in blue-collar jobs (building sites, factories, that kind of thing) but used informally everywhere.
- SixSigma 11y agoGaffer is the name of the head electrician in film production, the assistant is called the Best Boy. https://en.wikipedia.org/wiki/Gaffer_%28filmmaking%29 https://en.wikipedia.org/wiki/Gaffer_%28filmmaking%29 Hence "Gaffer tape" which is black, rather than duct tape, which is the same construction but grey.
- osense 11y agohttps://github.com/gchq https://github.com/gchq Cybersquatters...
- finnn 11y agoI've heard rumor that dead accounts like that can be taken over by someone else by emailing GitHub. A friend of mine did it, but I don't remember all the details of the situation
- unfunco 11y agoI did this a couple of weeks ago, and it was extraordinarily efficient: https://help.github.com/articles/name-squatting-policy/ https://help.github.com/articles/name-squatting-policy/ It took about two-hours from clicking the contact a human button to them releasing the name, they simply release it back and ask you to register quickly before someone else gets there first.
- int_handler 11y agoIt is curious to me why GCHQ didn't just contact GitHub to acquire github.com/gchq but instead decided to go with the long and cumbersome github.com/GovernmentCommunicationsHeadquarters. Perhaps it is a British thing [1]. [1] https://www.addedbytes.com/blog/if-php-were-british/ https://www.addedbytes.com/blog/if-php-were-british/
- fishnchips 11y agoMaybe they tried but from my experience GitHub does not care one bit about cybersquatting. On the other hand I'm not James Bond so who knows ;)
- int_handler 11y agoGitHub explicitly has a Name Squatting Policy [1] that states: > Account names may not be inactively held for future use. GitHub account name squatting is prohibited. Inactive accounts may be renamed or removed by GitHub staff at their discretion. I have been able to acquire a couple of GitHub names that have been inactive for several years by contacting GitHub support, and they usually reply within a day or so. [1] https://help.github.com/articles/name-squatting-policy/ https://help.github.com/articles/name-squatting-policy/
- Ianvdl 11y agoFrom [1]: > Gaffer stores data in Accumulo, but inserting data and retrieving it again requires the user to have no knowledge of Accumulo. As Gaffer stores data in Accumulo, it is horizontally scalable so that very large data sets can be dealt with. It has an API that allows users to retrieve the data they care about, filtered according to their requirements and aggregated over the time window of interest. It supports bulk update and continuous update. Seems like a very useful tool, especially if you already have accumulo infrastructure running. Docs need a bit more work I feel, but it's not terrible for a single page. [1]: https://github.com/GovernmentCommunicationsHeadquarters/Gaffer/blob/master/UserGuide.md https://github.com/GovernmentCommunicationsHeadquarters/Gaff...
- nshstjgsbt 11y agoThis is just a PR move to get the tech community to hate them less, even if only by a little bit. They want to muddy the waters and insert the idea into people's minds that "we're not all bad."
- unfunco 11y agoI don't think that's the case, the software is taxpayer funded so it makes sense to release it. I think it's more likely that it will help with recruitment, pull requests of high-quality might provide a potential interview or job offer.
- voltagex_ 11y agoLook at the Issues. Trolls are going to ruin it as the path of least resistance will be to disable them.
- EvanPlaice 11y agoIf it's 'taxpayer funded' as you say, it should be licensed under the public domain. No 'if', 'ands', or 'buts'. If it's paid for by the people it's the property of the people.
- spyspy 11y agoAnd if you pay enough in taxes you should get your own tomahawk cruise missile.
- EvanPlaice 11y agoNot sure what you'd do with a tomahawk (besides hump it for fortitude). Anyway... In the US the policy is: A United States government work is prepared by an officer or employee of the United States government as part of that person's official duties. It is not subject to copyright in the United States and there are no copyright restrictions on reproduction, derivative works, distribution, performance, or display of the work. Anyone may, without restriction under U.S. copyright laws: - reproduce the work in print or digital form - create derivative works - perform the work publicly - display the work - distribute copies or digitally transfer the work to the public by sale or other transfer of ownership, or by rental, lease, or lending. Source: https://www.usa.gov/government-works https://www.usa.gov/government-works US government based software developer agencies like 18f and USDS (United States Digital Service) license all of their code to the public domain. The UK is slightly different: https://github.com/GovernmentCommunicationsHeadquarters/Gaffer/issues/3 https://github.com/GovernmentCommunicationsHeadquarters/Gaff... But there is the OGL (Open Government License): http://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/licensing-for-re-use/guidance-for-information-providers/make-information-available-ogl/ http://www.nationalarchives.gov.uk/information-management/re...
- sammorrowdrums 11y agoThis is not long after MI6 were looking for NodeJS Devs https://news.ycombinator.com/item?id=10532855 https://news.ycombinator.com/item?id=10532855
- sammorrowdrums 11y agoNot that they're necessarily talking to each other about PR in the tech community, but looks like maybe the British Gov is trying to attract some talent and get devs engaged. Maybe they just want it improved for free :-p
- branchless 11y agoMake a commit to save some memory which allows them to expand their criteria which leads to your arrest for thought-crime.
- DanBC 11y agoGCHQ / CESG have been putting details of some of the tech they work with on their public webpage for a few years now. https://www.gchq-careers.co.uk/about-gchq.html https://www.gchq-careers.co.uk/about-gchq.html https://www.gchq-careers.co.uk/departments/technology-and-engineering.html https://www.gchq-careers.co.uk/departments/technology-and-en... The "applied research" is interesting: https://www.gchq-careers.co.uk/departments/applied-research.html https://www.gchq-careers.co.uk/departments/applied-research.... I think the internet archive has some older pages with relevant information. If this is GCHQ it's gently worrying - a github profile is pretty low on my list of what I want in the way of transparency and accountability.
- Cakez0r 11y agoIt seems like they are making a recruitment push (http://www.theinquirer.net/inquirer/news/2435685/gchq-is-using-graffiti-to-find-hipster-coders http://www.theinquirer.net/inquirer/news/2435685/gchq-is-usi...) and are trying to repair their reputation a bit.
- ExxKA 11y agoI wonder if this is for real
- DanBC 11y agoIt's not mentioned on the GCHQ / CESG website, as far as I can tell.
- tangled 11y agoI'd be interested in seeing the buy vs. build analysis for this project. Are there any pre-existing projects that have similar features? And assuming that this project is used to process classified data, what impact does this have on the selection process? e.g. is it possible to use closed-source solutions?
- jwr 11y agoThis can't be real. I needed something exactly like this and was about to start looking tomorrow. And, lo and behold, GCHQ delivers. As for my future software needs, I expect the code to be written at MI-6 and delivered just-in-time by James Bond.
- jkxyz 11y agoJust don't mention anything about a new JavaScript framework in any of your private communications, or they might spot it and make a new one of those too.
- emehrkay 11y agoGet a [TinekerPop3](http://tinkerpop.incubator.apache.org http://tinkerpop.incubator.apache.org) interface and I'll update my python libs to support it (one day...in the future...when I have time...and interest).
- dignati 11y agoThe name is ridiculous in german. A "gaffer" is a stalker or someone who looks when he's not supposed to. Quite fitting.
- car 11y ago"gaffer" translates to "rubbernecker".
- sehr 11y agoVery weird! I've always heard it used to refer to English soccer club coaches
- bdisraeli 11y agoAlso the person in charge of lighting on a film set.
- alessioalex 11y agoYou mean English football clubs. Nobody says soccer in Europe mate :p
- Raticide 11y agoIn English English 'gaffer' means 'boss'.
- metasean 11y agoOr the head electrician on a film or theatrical set - https://en.wikipedia.org/wiki/Gaffer_%28filmmaking%29 https://en.wikipedia.org/wiki/Gaffer_%28filmmaking%29
- wildpeaks 11y agoAnd in french a "gaffeur" is someone who makes hilarious mistakes.
- cwp 11y agoReally? The spies release a graph database where edges can have statistics attached, like oh, say, counts? We're being trolled. Edit: Looking for forward to the release of RingTone, a system for processing very high volumes of call detail records in real time.
- deleted 11y ago[deleted]
- typon 11y agoDo ends justify the means when it comes to knowledge being added to the world open-source repository of software? Should we, as a community, reject these people's hard work or just use it while also understanding that they're evil? I'm conflicted.
- unquietcode 11y agoWe can't just be consumers here, even though there is no money changing hands. This goes beyond the issue of trusting their code. We should send a clear message that 'dirty bits' are not welcome in community-built software. In effect, this is the only punishment you can dole out to an open source project--that is, choosing not to adopt it. Our 'ethics' as computer scientists are increasingly under fire and I think it's wise to know when to say 'no', especially when the hand that feeds is also the hand that beats you mercilessly.
- throwaway94874 11y agoIf you think what they do is evil then explain why. Don't pretend that everyone thinks that way so much so that it doesn't even need an explanation.
- typon 11y agoIt has been discussed to death here why what they do is evil. If you want a security expert's opinion, read Bruce Schneier's blog, and if you are inclined to learn more about the ethics, this page is great: http://cs.stanford.edu/people/eroberts/cs201/projects/ethics-of-surveillance/index3.html http://cs.stanford.edu/people/eroberts/cs201/projects/ethics...
- conceit 11y agoIt's simple, they are evil, so what they do has to be evil. They are evil because ... we are good ... and they obviously separate themselves by observing others, while hiding in secrecy. It's more complex than that, but here we go.
- adrtessier 11y agoI feel as you do. As much as I am loathe to accept anything coming out of one of these organizations as an open-source project, I think it may actually do us more harm than good by attacking this project out of protest for the way they use their tools. We can be as negative as we wish towards these agencies, but rejecting any and all attempts at communicating with the open source community is a strong way to reinforce their already insular culture. Embracing these projects in some way or another can possibly work as positive feedback toward greater organizational transparency, if not by the brass, than by the developers and engineers that work in these organizations.
- infinity0 11y agoWhy is the British government doing research into something that terrorists could use to further their extremist agendas?
- vskarine 11y agoI like how it is on top of Accumulo which was created by NSA: https://en.wikipedia.org/wiki/Apache_Accumulo https://en.wikipedia.org/wiki/Apache_Accumulo Great to see spy agencies cooperating in open source :P