8 ms·
Why I'm Lukewarm on Graph Neural Networks
- GradientAssent 6y agoAnyone have a mirror?
- VHRanger 6y agoAuthor here. Seems like another post of mine (my take on BTC) was randomly posted and became popular at the same time I posted this one. I'm trying to get the wordpress host's cache to handle the load.
- whymauri 6y agoAlso posted on Reddit: https://www.reddit.com/r/MachineLearning/comments/kqazpd/d_why_im_lukewarm_on_graph_neural_networks/ https://www.reddit.com/r/MachineLearning/comments/kqazpd/d_w...
- osipov 6y agoYou can use https://web.archive.org/web/20210104154304/https://www.singlelunch.com/2020/12/28/why-im-lukewarm-on-graph-neural-networks/ https://web.archive.org/web/20210104154304/https://www.singl... since the original website is experiencing HN bear hug.
- lostmsu 6y agoBy no means I am an expert in deep learning, but lately I've been considering graph NNs snake oil of neural networks. There are no impressive results on any common tasks and that picture is supported by a recent paper claiming transformers to "contain a graph network inside".
- timr 6y agoGraph NNs are widely used in the chemical domain -- molecules are easily modeled as graphs, so it's a natural fit.
- VHRanger 6y agoSo are graph embedding methods, which the post discusses
- dumb1224 6y agoThe post can't be accessed with a Internal Server Error. Or is it just my ISP? I often confuse myself with other representation learning methods such as graph kernels https://en.wikipedia.org/wiki/Graph_kernel https://en.wikipedia.org/wiki/Graph_kernel Are there any relations between them at all?
- VHRanger 6y agoNo, my blog went down from hitting the front page of HN
- enriquto 6y agoAnd for large molecules (e.g., proteins) graph methods really start to shine.
- astrophysician 6y agoIn my opinion there aren't many deep learning applications that are fundamentally snake-oil, it's more that people who don't quite understand the limitations and advantages of particular methods that end up being (witting or unwitting) snake-oil salespeople. Deep graph NN's are very useful for lots of data that are inherently graph-structured (another commenter above mentioned chemistry applications, and there are lots of other examples). Whether or not for the time-being they give SOTA results on common datasets, being able to work directly with graph-structured data is quite appealing in many cases.
- mrfox321 6y agoGoogle maps improved their worse-case residuals of traffic estimation using GNNs (messaging passing Nets, not graph embeddings).
- vladTheInhaler 6y agoA bunch of fully connected layers also "contains a CNN inside", but empirically CNNs lead to better performance on image classification tasks. When you can cut away an enormous, and mostly irrelevant part of the parameter space, that has a lot of value.
- Der_Einzige 6y agoSo wrong oh my God! This beauty got an outstanding paper award at COLING 2020. GNNs are not applied to "common tasks" because they are primarily suited for environments where graph structure is present. In this case, predicting how the Indian congress will vote. https://www.aclweb.org/anthology/2020.coling-main.426/ https://www.aclweb.org/anthology/2020.coling-main.426/
- leecarraher 6y agoI suspect we will not see a revolution in deep learning from GNN without a corresponding hardware(FPGA?) or optimization method(HSIC? HTMs?) advance. GNNs trained by backprop are making many of the same mistakes that LSTMs did: solve one important problem(exploding/vanishing gradients), but introduce a bunch of hyperparameters that bring along their own set of problems. GRUs are successful in my opinion, because they remove some of those tunable parameters. Of course as the post suggested, not being able to tune something, often loses out against some more tuned and curated solution. GNNs being the newest version, having tons of parameters to tweek. In the end do we get a better, ultimately more generalizable solution? Or do we just get more hyperparameters to tune, and spend more time and money for a modest, unrepeatable gain.
- sooheon 6y agoThat hardware advance may be closer than expected: https://www.graphcore.ai https://www.graphcore.ai
- orange3xchicken 6y agoA lot of valid criticism of this post on the reddit ml subreddit: https://www.reddit.com/r/MachineLearning/comments/kqazpd/d_why_im_lukewarm_on_graph_neural_networks/ https://www.reddit.com/r/MachineLearning/comments/kqazpd/d_w...
- sischoel 6y agoI don't know much about graph neural networks although it is a topic that I want to study in the next few months. But what bothered me in your article is what you wrote about graph data structures. NetworkX is indeed very slow, this is due to two facts: - NetworkX is a pure Python implementation and does not relay on some methods written in a faster language like C. - They use dictionaries to represent the graphs, which may have some advantages when mutating graphs, but of course have much worse locality than an adjacency list or a some sparse matrix format. But even for python there are much faster libraries such as igraph. The data structure in igraph is an edge list. A lot of single core graph libraries use an adjacency list internally, and while it is true that the index list for each vertex can be somewhere arbitrary in memory, they usually do not behave like a linked list, unless you graph is really sparse. One of the most used operations in graph algorithms is to iterate over the neighbors of a vertex, and for this, adjacency lists are very good. They also have a small advantage over CSR matrices for adding or removing edges, and they might use slightly less memory, as their index type only needs to be able to index all vertices and not all edges, so they need half of the space, which is better for caches.
- tehjoker 6y agoThere's graph-tool but it is annoying to compile sometimes: https://graph-tool.skewed.de/ https://graph-tool.skewed.de/
- enriquto 6y agoIndeed, the best support for large graphs in Python is found inside scipy.sparse
- jim22k 6y agoAnyone who is serious about graph analysis in Python using linear algebra should check out GraphBLAS (graphblas.org). The inclusion of semirings makes things like SSSP and BFS extremely elegant on sparse adjacency matrices. The main implementation is SuiteSparse::GraphBLAS, a C library which has two Python bindings (search for grblas or pygraphblas). Disclosure: I'm the author of grblas. Both are available on conda-forge for easy installation. If you want to try it out without installing, here is a binder link: https://mybinder.org/v2/gh/metagraph-dev/grblas/HEAD?filepath=notebooks%2FIntro%20to%20GraphBLAS%20%2B%20SSSP%20example.ipynb https://mybinder.org/v2/gh/metagraph-dev/grblas/HEAD?filepat...
- deleted 6y ago[deleted]
- mrtranscendence 6y agoNetworkX isn't a bad library, nor is it only suitable for "babies". Yeesh. Actual people work on this stuff, you know, and they may have different goals, requirements, etc than you. This whole post reeks of someone who's so in love with being a maverick speaking uncomfortable truths that they've lost sight of this human element. If you're offended by (say) research papers that fail to break enough new ground to satisfy you, sorry about your luck, but again, not everyone's optimizing the same function as you. You'll probably do better bringing people around when you're not implying (or outright saying) their work is shit.
- VHRanger 6y agoI never said NX is bad, but it is for "baby" graphs. NX can't scale past a few hundred thousand nodes. I appreciate NX's place in the ecosystem, but its implementation leaves a large gap to be filled for an intermediate library that is a Pandas analogue for graphs.
- dimatura 6y agoI have to say I agree. NX is totally fine for the applications it was built for. I appreciate the simple API and the flexibility of using pure Python. There's a reason it's so popular. I'm not an expert in graph neural networks, so can't really say much about the novelty of Node2Vec. But I do think it's often misguided to judge scientific work as trivial or incremental in retrospect. Specially in a relatively young field like deep learning, where four years (Node2Vec is from 2016) is a long time.
- beaupletga 6y agoI need to handle (for work) a graph with 40 million nodes and more than 130 million edges. As expected, networkx couldn't handle more than a million nodes so I had to search for python libs which might handle that much data. This is why I've been using your lib (https://github.com/VHRanger/nodevectors https://github.com/VHRanger/nodevectors) for at least 2 weeks now as well as these 2 other libs: https://github.com/louisabraham/fastnode2vec https://github.com/louisabraham/fastnode2vec and https://github.com/sknetwork-team/scikit-network https://github.com/sknetwork-team/scikit-network. What do they have in common? They handle sparse graphs (using CSR representations). Having a graph with several million nodes isn't just some edge case, social graph for instance grow way faster than anyone could expect. So I do totally agree that there is a gap between popular graph libs which handle very small graphs and real life libs which need to handle way bigger graphs. Btw, thanks for the work you've done with 'nodevectors'. PS: I'm not criticizing either networkx which is very handy and quite good when prototyping a solution.
- VHRanger 6y agoCheers. Please raise an issue on my library's github or drop me an email if you run into issues using my lib. I'm always happy to help others on their projects.
- Labo333 6y agoHi, I'm the author of [1] nodevectors implements many more algorithms than fastnode2vec that focuses on node2vec. I'm pretty sure fastnode2vec is the fastest node2vec implementation because it uses CSR format, JIT compilation but also supports multiprocessing. The sampling algorithm has been improved compared to the paper (and to all other implementations) and allows truly linear memory consumption. On the other hand, nodevectors has a lot of very cool methods like ProNE so you should definitely try it on your data. However, the original ProNE code is probably faster as it is written in C++ and uses multiple cores while nodevectors just uses the installed BLAS (you can probably get a massive speedup by installing OpenBLAS). [1]: https://github.com/louisabraham/fastnode2vec https://github.com/louisabraham/fastnode2vec
- FL33TW00D 6y agoInteresting post, I’m working on a whole graph classification problem at the moment. If GNNs aren’t the future, then where else should I be looking?
- sanxiyn 6y agoYou should look at label propagation. Combining Label Propagation and Simple Models Out-performs Graph Neural Networks: https://arxiv.org/abs/2010.13993 https://arxiv.org/abs/2010.13993
- sooheon 6y agoLinked paper is focused on node classification, parent is asking for whole graph prediction.
- VHRanger 6y agoFor whole graph classification, I think GNNs are still the future.
- fedd 6y agoI tried to implement a graph neural network back in 2006... https://github.com/fedd/weighscore https://github.com/fedd/weighscore
- SeeDave 6y agoIf you want to feel old: 2020 - 2006 = 14. 2006 - 14 = 1992. When the 'World Wide Web' first launched....
- deleted 6y ago[deleted]
- Der_Einzige 6y agoThis is like someone going "I installed NLTK and it's slow and doesn't support transformers or word2vec!". In fact, I think the author of the article may have actually done exactly that.
- eachro 6y agoHas the author looked at how GNNs are used in say predicting molecular properties[1] or molecule generation[2]? GNNs are the state of the art on those tasks and those problems are certainly much harder than the Cora/Citeseer/etc semisupervised node label prediction tasks. [1] https://arxiv.org/abs/1704.01212 https://arxiv.org/abs/1704.01212 [2] http://proceedings.mlr.press/v80/jin18a/jin18a.pdf http://proceedings.mlr.press/v80/jin18a/jin18a.pdf
- mdawg 6y agoNice article