11 ms·
Graph Mining Library
- tomrod 3y agoThis is a big deal, I think. I'm guessing it's not widely used internally anymore if they are open sourcing it. What is used instead?
- simonw 3y agoI don't think "not widely used internally anymore" is a common rationale for open sourcing something. Generally I'd expect companies to open source things when it's proven itself internally and they want to reap the benefits of open source: - Make internal engineers happy - engineers like having their code released outside the bounds of their company - Prestige, which can help with hiring - External contributions (not even code necessarily, just feedback from people who are using it can be amazingly useful for improving the software) - Ability to hire people in the future who already know important parts of your technical stack, and don't need internal training on it - Externally produced resources that help people learn how to use the software (tutorials, community discussion forums etc) If the software is no longer used internally, open sourcing it is MORE expensive - first you have to get through the whole process of ensuring the code you are releasing can be released (lots of auditing, legal reviews etc), and then you'll have to deal with a flow of support requests which you can either deal with (expensive, especially if your internal knowledge of the tool is atrophying) or ignore (expensive to your reputation).
- mlinhares 3y agoGoogle has done this before so it's not surprising people would think that.
- tomrod 3y agoTo be fair, I can often be uninformed with regards to some of the smaller movements of techcos.
- palata 3y agoIMO, you forget one important point: control. If your open source project/protocol is the most popular, and you have the governance over it, then you decide where it goes. Chromium is open source, but Google controls it, and everyone who depends on it has to follow. If Chromium was not open source, maybe Firefox would be more popular, and Google would not have control over that. > or ignore (expensive to your reputation). I don't think that anything is expensive for Google. They can do whatever they want.
- nivekney 3y agoIt's based on ParlayLib, which is for shared-memory multicore machines. Highly suspect that they moved the algorithms on to distributed systems.
- wilsynet 3y agoAs merely two examples, both gRPC and Kubernetes are important to Google, and yet Google opened sourced them. "No longer used" is not the criteria Google uses to make their software OSS. FYI, I work at Google.
- jefftk 3y agoGoogle Wave is the only counterexample I can think of, where it was "we're deprecating this project, but releasing it as open source".
- bsimpson 3y agoI don't think Google generally opensources _products_ - either it always is open source (Android) or never is (web apps). I can't think of an example where a product was closed source, released as open source, and continually maintained. Open source at Google generally takes the form of libraries rather than products. Often, that's something that an individual engineer is working on, and it's easier to open source than get the copyright reassigned (since Google by default owns any code you write). There are also libraries that are open sourced for business reasons - e.g. SDKs. You can tell the difference, because most individually-driven libraries contain the copy "Not an official Google product" in the README.
- progval 3y ago> I can't think of an example where a product was closed source, released as open source, and continually maintained. I found one after some searching: Nomulus. https://opensource.googleblog.com/2016/10/introducing-nomulus-open-source.html https://opensource.googleblog.com/2016/10/introducing-nomulu...
- tomrod 3y agoThanks for clarifying
- PaulHoule 3y agoI'd say both of those are actively harmful products (like PFOS or cigarettes) that hurt Google's competition by being open sourced. Google wrecked their own productivity, the least they could do was wreck everybody else's.
- spankalee 3y agoMost projects try not to open something that's not going to be maintained. If they do it's usually rather loudly called out in the readme.
- meneer_oke 3y agoWould it be possible to explain why it's big deal.
- xxpor 3y agoI was hoping this would mine literal stats graphs for anomaly detection
- blitzar 3y agoI think they use the word "graph" to mean a different thing to what I use the word for.
- lanstin 3y agohttps://en.wikipedia.org/wiki/Graph_theory https://en.wikipedia.org/wiki/Graph_theory It's interesting and deceptively simple at first.
- supriyo-biswas 3y agoThat’s relatively easy, see https://en.m.wikipedia.org/wiki/Interquartile_range https://en.m.wikipedia.org/wiki/Interquartile_range
- deleted 3y ago[deleted]
- charcircuit 3y agoMost of these files have a double license header.
- ldhulipala 3y agoThanks for pointing this out (fixed now).
- specproc 3y agoIf you're working on this repo, can we plz haz docs?
- ldhulipala 3y agoThanks, yes, this is on the list of TODOs! (also, to open-source the tests)
- specproc 3y agoThank you kindly!
- xw3098 3y agoIt would be nice to have a bit of documentation on what makes this library special as well. It’s a significant time investment to learn a library like this one well. So some information on why one should choose this over, say, http://snap.stanford.edu/ http://snap.stanford.edu/ Would be very helpful.
- choppaface 3y agoDid you release it without docs so that you could add it to your Perf packet?
- whitten 3y agoGithub says it is C, C++, and Starland. What is Starland ?
- ashout33 3y agoif I had to guess, that is a typo and should be starlark, which is the language used for bazel build files. bazel is the build system they use
- jefftk 3y agoGithub says "Starlark 6.2%", so it looks like whitten's typo, not GitHub's.
- nolok 3y agoOn which keyboard layout is rk into nd a typo ...
- mcpeepants 3y agoon any layout operated by a human, who may at times type the wrong word entirely
- macintux 3y ago…and/or fall victim to autocorrect.
- bsimpson 3y ago"STARLAND VOCAL BAND? THEY SUCK!"
- Laremere 3y agoIt's Starlark, the language for configuring the build system Bazel. Bazel is the open source port of Google's internal build system, Blaze. Starlark is a subset of Python.
- deleted 3y ago
- zekenie 3y agosome examples would be super helpful!
- corentin88 3y agoInteresting fact: the first commit is 2 years old and is entitled "Boilerplate for new Google open source project". Either they rewrite git history or it took about 2 years to get approval on making this repo public.
- thfuran 3y agoNow that's bureaucracy.
- numpad0 3y agoI'd agree if last commit was 2 years ago.
- j2kun 3y agoThe code has an internal analogue, and the tooling lets you choose whether to export the entire git history or squash it. They may have chosen the former, in which case it could just be 2 years to migrate and rework the code to be ready for open sourcing. In that time I imagine there were four reorgs and countless priority shifts :)
- spankalee 3y agoIf you know you want to open source a project eventually, it's easier if you start it in the open source part of the internal repo with all the licensing and headers in place. Open sourcing existing code is harder because you need to review that it hasn't used something that can't be opened. So probably they just started the project two years ago, had aspiration to open source, and finally just did now. Some teams might publish earlier, some like to wait until it's had enough internal usage to prove it out.
- hiddencost 3y agoThat could be the template it was cloned from
- deleted 3y ago[deleted]
- 3y ago
- sbrother 3y agoI might be (very) far behind the times, but does this have any relationship with Pregel?
- deleted 3y ago[deleted]
- cmckn 3y agoPregel is a distributed graph processing system, this (AFAICT) is a library for working with graphs in-memory on a single computer.
- dllthomas 3y agoDoes it accept graphviz?
- ponyous 3y agoNo idea where is the hype coming from, who is actually upvoting this? 0 Docs, 0 examples, 0 explanation of how is it useful. Is "Graph Mining" so ubiquitous that people know what this is all about?
- ldhulipala 3y agoWe are updating the README to be more descriptive; in the meantime, please see https://gm-neurips-2020.github.io/ https://gm-neurips-2020.github.io/ or https://research.google/teams/graph-mining/ https://research.google/teams/graph-mining/
- ldhulipala 3y agoThere are now more documents linked to in the README.md and an example you can try to run: https://github.com/google/graph-mining https://github.com/google/graph-mining
- deleted 3y ago[deleted]
- bafe 3y agoIt was hyped some years ago. There are plenty of legitimate applications of graphs, perhaps the library offers well optimized implementation of important algorithms. But the past hype around all things "graph" was not rational. As always, you can't solve all problems with a graph as you can't with a neural network or with any other structure/algorithm
- afandian 3y agoCan someone with familiarity with Bazel give any clues how to build? `bazel build` does something, but I end up with `bazel-build` and `bazel-build` with no obvious build artefacts.
- elteto 3y agoIn bazel //... is the equivalent of the 'all' target in make: bazel build //... bazel test //... bazel query //... The last one should list all targets (from what I remember).
- afandian 3y agoThanks! That last one lists 84 results. None looks obviously like 'main'. Trying a random one: bazel run //in_memory/clustering:graph ERROR: Cannot run target //in_memory/clustering:graph I'm going to wait until someone updates the readme I think!
- tfsh 3y agoconsidering the repo doesn't contain a cc_binary build rule, I'm inclined to believe there's no demo, the easiest way to get started (if you want to play around from scratch) would be to add a cc_binary, point that to a main.cpp file which depends on the library targets you want, e.g "//in_memory/clustering:graph" and ensure there's sufficient visibility from the targets.
- elteto 3y agoThen most likely this is meant to be used primarily as a library. You should wait until they open source the tests (soon, per another commenter). Those will be runnable targets.
- hashar 3y ago`bazel run` is for a rule that has been marked `executable = True` and there is no such rule in the repository. If you `bazel build //...`, you should get the compiled libs under `bazel-out/*fastbuild/bin/`.
- MarkMarine 3y agoWhew. Lots of complaints from people who probably will never need to use this code. If you need docs just read the .h files, they have extensive comments. I’m sure they’ll add them or maybe, just maybe, you could write some to contribute. This would have made some of my previous work much easier, it’s really nice to see google open source this.
- deleted 3y ago[deleted]
- ls612 3y agoI think it’s that it’s not at all obvious how to even build the damn thing so at least a little bit of readme would have been nice. I agree with the sentiment this looks like a super cool tool.
- PaulHoule 3y agoIt says you're supposed to leave a ticket if you have questions or comments... A README file isn't much to ask for.
- MarkMarine 3y agoI’m not saying it’s too much to ask for, but also, when you’re doing distributed in memory graph mining (which means you’ve got an application with a big enough graph that you need to do this, and the technical expertise to need the algorithms in this open source package) maybe it’s expected that you can read the bazel files and bazel docs yourself and figure it out. Or just write a make file and cut all the bazel build optimization out. They don’t put instructions on how to start a F1 car inside the cockpit, you don’t hop into a fighter jet and look for the push to start easy button, it’s expected that when you’re at that level you bring expertise.
- PaulHoule 3y agoYeah, and somebody who is that smart can probably pack their data structures efficiently and find an approximation to do the job on a macbook pro that people with too many resources need a 1000 machine cluster to do. And get the coding and the computation done in the time that the C++ compiler is still chewing on the headers of the bloated library. (At times I’ve been that guy.) But seriously, there is such a thing as industrialization. Google is notorious though for hiring 180 IQ people and getting them to perform at a 35 IQ level because there the documentation makes no sense, a procedure which should be done in one step really takes 300, fighting with C++, etc. They can afford to do it because it is just so profitable to help that guy who shows up on every. single. video. roll. who says “you can’t lose weight by exercising”, “you can’t lose weight by cutting carbs” who links to some video that drones on hours and hours and signs you up for some subscription you can never cancel to sell scam supplements. Shame on them. BTW, with high-complexity software there is enough question that you got it working right that you expect a significant process of testing that it works for your application. For instance if you got a hydrocode for simulating the explosion of a nuclear weapon you would not take for granted that you had built it properly and were using it properly until you'd done a lot of validation work. A system like that isn't a product unless it comes with that validation suite. The same is true for automated trading software (gonna hook it up straight to the market without validation? hope you have $100M to burn!) ... now there was that time a really famous CS professor e-mailed me a short C program that was said to do something remarkable that crashed before it even got into main() which did teach me a thing about C but that's not what a professional programmer does.
- esafak 3y agoGraph mining was "so hot right now" ten years ago. Remember GraphX (https://spark.apache.org/graphx/ https://spark.apache.org/graphx/) and GraphLab (https://en.wikipedia.org/wiki/GraphLab https://en.wikipedia.org/wiki/GraphLab) ? Or graph databases? I guess it coincided with the social network phenomenon. Much more recently geometric learning (ML on graphs and other structures) shone, until LLMs stole their thunder. I still think geometric learning has a lot of life left in it, and I would like to see it gain popularity.
- PaulHoule 3y agoThere are "graph databases" which see graphs as a universal approach to data, see RDF and SPARQL and numerous pretenders. For that matter, think of a C program where the master data structure is a graph of pointers. In a graph like that there is usually a huge number of different edge types such as "is married to", "has yearly average temperature", ... Then there are "graph algorithms" such as PageRank, graph centrality, and such. In a lot of those cases there is one edge type or a small number of edge cases. There are some generic algorithms you can apply to graphs with many typerd edges edges such as the magic SPARQL pattern ?s1 ?p ?o . ?s2 ?p ?o . which finds ?s1 and ?s2 that share a relationship ?p with some ?o and is the basis for a similarity metric between ?s1 and ?s2. Then there are the cases that you pick out nodes with some specific ?p and apply some graph algorithm to those. The thing about graphs is, in general, they are amorphous and could have any structure (or lack of structure) at all which can be a disaster from a memory latency perspective. Specific graphs usually do have some structure with some locality. There was a time I was using that magic SPARQL pattern and wrote a program that would have taken 100 years to run and then repacked the data structures and discovered an approximation that let me run the calculation in 20 minutes. Thus practitioners tend to be skeptical about general purpose graph processing libraries as you may very have a problem that I could code up a special-purpose answer to in less time than you'll spend fighting with the build system for that thing that runs 1000x faster. ---- If you really want to be fashionable though, arXiv today is just crammed with papers about "graph neural networks" that never seem to get hyped elsewhere. YOShInOn has made me a long queue of GNN papers to look at but I've only skimmed a few. A lot of articles say they can be applied to the text analysis problems I do but they don’t seem to really perform better than the system YOShInOn and I use so I haven’t been in a hurry to get into them.
- PaulHoule 3y agoTowards the end of my relationship with a business partner, he was really impressed with a graph processing library released by Intel (because it was Intel), while my thoughts were "ho hum, this looks like it was done by a student" (like a student who got a C-, not a A student) and thought about how much I liked my really crude graph processing scripts in Pig that were crazy fast because they used compressed data structures and well-chosen algorithms.
- Daan21 3y ago[dead]
- pharmakom 3y agoCan someone explain what this library might be useful for?
- oddthink 3y agoClustering. I used the correlation clusterer from here for a problem that I could represent as a graph of nodes with similarity measures (this data looks like this other data) and strong repelling features (this data is known to be different from this other, so never merge them).
- nologic01 3y agoGraph algorithms cry out for some standardization. Think blas and lapack.
- bigbillheck 3y agoConsider: https://graphblas.org https://graphblas.org
- nologic01 3y agoI wonder how much overlap this new project with graphblas and older graph libraries like boost::graph https://www.boost.org/doc/libs/1_83_0/libs/graph/doc/ https://www.boost.org/doc/libs/1_83_0/libs/graph/doc/
- 0x6461188A 3y agoHow is this usable. I see no documentation. There is a docs folder but all it contains is a code of conduct.
- mathisfun123 3y agoI'm not trying to be snarky but have you considered reading the code? Like I'll be honest I can't remember the last time I looked at docs at all instead of reading the code itself.
- n3150n 3y agoAre you for real? I'm also not trying to be snarky but... $ cat $(find . -type f | grep -vE LICENSE\|README\|BUILD\|bazel\|git\|docs) | sort -u | wc -l 8360 unique lines scattered across more than 100 files. Good luck deciphering that in a single day! By the way, the first issue in the repo is a "Request for a more verbose README", which I agree with.
- mathisfun123 3y agomy guy what exactly are you expecting here? this is free as in beer code (apache license). no one is forcing you to use this and no one is asking anything of you for using it. i fully support people releasing their code (that took enormous amounts of blood sweat tears to get working) absolutely however they want to. if i'm interestd enough i'll figure it out and thank them. so as i see it you have like three options if you are unhappy with that: 1. close the tab 2. dig into the impl and learn as you go 3. do 2 but also write docs i just really believe i've covered literally all the cases that any reasonable (not whiney, not entitled) person would concede. > the first issue in the repo is a "Request for a more verbose README", which I agree with. posted today - do you think it might have something to do with this post we find ourselves convening on? i.e. no one was so bothered about a lack of docs until now? edit: i forgot actually something else you could do: email the author and ask nicely for some tips.
- jshobrook 3y agoYou could try using something like Adrenaline https://www.useadrenaline.com/ https://www.useadrenaline.com/ I built it exactly for this use case :)
- itissid 3y agoNoob Q: Would this library be a (good?) candidate to be integrated with a wrappers/extension libraries to have all the graph based clustering algorithms in one place(assuming they are not already)? Or do(better?) frameworks for the same function as this code already exist(maybe networkx?)?
- emmanueloga_ 3y agoFor those wanting to play with graphs and ML I was browsing the arangodb docs recently and I saw that it includes integrations to various graph libraries and machine learning frameworks [1]. I also saw a few jupyter notebooks dealing with machine learning from graphs [2]. Integrations include: * NetworkX -- https://networkx.org/ https://networkx.org/ * DeepGraphLibrary -- https://www.dgl.ai/ https://www.dgl.ai/ * cuGraph (Rapids.ai Graph) -- https://docs.rapids.ai/api/cugraph/stable/ https://docs.rapids.ai/api/cugraph/stable/ * PyG (PyTorch Geometric) -- https://pytorch-geometric.readthedocs.io/en/latest/ https://pytorch-geometric.readthedocs.io/en/latest/ -- 1: https://docs.arangodb.com/3.11/data-science/adapters/ https://docs.arangodb.com/3.11/data-science/adapters/ 2: https://github.com/arangodb/interactive_tutorials#machine-learning https://github.com/arangodb/interactive_tutorials#machine-le...