9 ms·
NetworkX 3.0 - create, manipulate, and study complex networks in Python
- jim22k 4y agoI presented the new dispatching feature at PyData Global back in December. The first 30 minutes of the follow video: https://youtu.be/pd-Sbm8lHTc?t=46 https://youtu.be/pd-Sbm8lHTc?t=46 It shows how to use the feature, some benchmarks, and how other graph libraries can become plugins to the NetworkX dispatching mechanism.
- tiger_eye 4y agoGreat talk! This slide shows some speedups when using the GraphBLAS [1] backend [2]: https://i.postimg.cc/mr3mkKtx/100x-Faster-Network-X.png https://i.postimg.cc/mr3mkKtx/100x-Faster-Network-X.png [1] https://github.com/python-graphblas/python-graphblas https://github.com/python-graphblas/python-graphblas [2] https://github.com/python-graphblas/graphblas-algorithms https://github.com/python-graphblas/graphblas-algorithms
- motohagiography 4y agoI've used networkx as an intermediate graph layer in web apps before talking to a db, which seemed redundant, except being able to reason about my data as a graph was everything I needed.
- BerislavLopac 4y agoNetworkX documentation is the source of one of my favourite surreal sentences: A lobster is a tree that reduces to a caterpillar when pruning all leaves. [0, with a tiny alteration] [0] https://networkx.org/documentation/stable/reference/generated/networkx.generators.random_graphs.random_lobster.html https://networkx.org/documentation/stable/reference/generate...
- xrayarx 4y agoNetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. At some point it might really start to be more efficient to have tags for the posts
- anigbrowl 4y agoI've updated it based on your suggestion. I originally left off a description because NetworkX is the library for doing graph things in Python and I kind of assumed people would know. While it's not the most performant or cutting edge, it's so well-established that everyone else aims for interface compatibility with it, much like pandas and tabular data. It's pretty neat that they've decided to leverage this by opening up the backend, allowing more performant/application specific offerings like SNAP or CUgraph to do the heavy lifting.
- LoganDark 4y ago> I kind of assumed people would know Not everyone writes Python, and not all Python programmers have heard of this particular library.
- decremental 4y agoNot all people are programmers and not all people have heard of programming. This post is catered to an audience that probably knows what NetworkX is.
- LoganDark 4y agoThis is not the same thing. Hacker News as a website is catered to an audience that is mostly familiar with programming. I would say it's safe to assume for any given visitor that they are probably aware of basic computing and programming terminology, but not that they would know what "NetworkX" means. The new title is fitting (it specifies what it's a Python library now).
- Sosh101 4y agoI've never heard of it.
- jimnotgym 4y agoI have written Python for 10 years on and off and have never come across it, or if I did I had forgotten about it.
- hbogert 4y agoGood memories of this library when the professor for the course complex networks said you were allowed to use any existing libraries. That labwork, needless to say, was a walk in the park.
- anigbrowl 4y agoBy the way if you want to explore network science but don't really know where to start, consider Gephi (currently being refactored, and just updated a few days ago) or Cytoscape (if you're more drawn to bioinformatics). Both make it easy to load/generate standard datasets, import tabular data, and have a good selection of plugins. It's easy to kick stuff out to either from NetworkX using /gefx or graphml, and you'll be able to experiment with a wide variety of layout algorithms and metrics. If you don't find the toy/benchmark networks intuitive, considering hitting the HN API for your source material; it's a lot easier to grasp the topic by studying relationships that are already familiar.
- nurbl 4y agoI've had a lot of luck with Gephi for fairly complex graphs (~10000 nodes and edges), in particular it has some very powerful layout algorithms. It can be very helpful to have immediate visual feedback when untangling something like that. Nice to see that development is ongoing, the GUI used to feel a bit clunky but 0.10 seems a lot better!
- anigbrowl 4y agoSeconding this. Get Gephi, install as many plugins as possible, and dig into the plugin pages to find the underlying papers. Also true for Cytoscape, but that tool focuses a little more on connectivity to bio databases and extremely specific stuff like proteomics.
- GraphLover9000 4y agoDo you have any advice/ideas about ways to learn about graph layout algorithms themselves, including dynamic/real-time algorithms (which allow for user interaction)? I have been skimming through various papers and the first book on this page [0], in particular the chapter on force directed algorithms, because they seem to be the earliest and most general graph drawing methods. [0] http://graphdrawing.org/books.html http://graphdrawing.org/books.html
- anigbrowl 4y ago
- college_physics 4y agowow, great to see that networkx keeps evolving. the graphblas backend is particularly intriguing. does networkx expose the graphblas API at the python level or is it working behind the scenes for computation?
- dallathee 4y agoGraphBLAS is wrapped by https://github.com/python-graphblas/python-graphblas/ https://github.com/python-graphblas/python-graphblas/ and the algorithms are available at https://github.com/python-graphblas/graphblas-algorithms https://github.com/python-graphblas/graphblas-algorithms. NetworkX only dispatches the computation for a subset of algorithms to graphblas-algorithms right now. If you want the graphblas API in python, https://github.com/python-graphblas/python-graphblas/ https://github.com/python-graphblas/python-graphblas/ is the right place :)
- robertlagrant 4y agoThis library is in my head also known as "how to solve Advent of Code[0] problems in even fewer lines of Python". [0] https://adventofcode.com https://adventofcode.com
- deleted 4y ago[deleted]
- AndrewOMartin 4y agoSo, is it pronounced "Network Ex" or simply "Networks"?
- mcyc 4y agoNetworkX is my go-to library for helping people get started with open source (like mentoring new devs that want git practice, practice working in a large codebase, etc.). I feel it is laid out in an easy to understand way and all the main parts do one clear thing that can be understood even with only a basic knowledge of graph theory. It also hits the sweet spot of being an obviously useful and widely used package, but without the army of devs working on it, so there is still a lot of low-hanging fruit for tangible contributions. I compare it to something like Django or scikit-learn, which seem insanely dense and difficult to understand even as an experienced dev, and there are so many people working on them that it's hard to find easy but non-trivial contributions for a beginner to make.
- veducha 4y agoDo you have any recommendations on how to get started? I have been using open source for around 10 years and always wanted to contribute. I am a confident python scripter and have used NetworkX in the past for various academic projects, however I have no idea how to get started as a beginner dev and contribute to open issues and bugs. Heck, I don't even know how large codebases are structured! Any hints or recommendation are welcome! :D
- dallathee 4y agoA good place to start specifically for NetworkX would be to go through the new contributor documentation: https://networkx.org/documentation/latest/developer/new_contributor_faq.html https://networkx.org/documentation/latest/developer/new_cont... We also have some structured projects https://networkx.org/documentation/latest/developer/projects.html https://networkx.org/documentation/latest/developer/projects... but they are usually for programs like GSoC/Outreachy. Feel free to start a discussion https://github.com/networkx/networkx/discussions https://github.com/networkx/networkx/discussions if you are looking for something specific :) [I am one of the NetworkX devs]
- mcyc 4y agoIf projects have a "good first issue" tag, they can sometimes be a good place to start. The problem is they are mostly "add tests" or "expand docs", which are important but not very exciting. Imo the best thing is to have a mentor who hopefully can find a reasonable thing to improve/add/fix and can help calm your nerves (submitting a pull request can be kinda scary haha). With NetworkX, in the past there were a lot of minor inefficiencies (things like materializing a list comprehension when you don't need to, etc) that are 1) easy to understand 2) don't require in depth knowledge of graph theory 3) are generally applicable to all python code and 4) easy to measure the results. They aren't the flashiest contributions, but the people I've done it with (6 or 7 at this point) have all seemed to enjoy it. To be clear, I'm _not_ affiliated with NetworkX, but if you are interested in trying it out, my contact info is in my profile. Feel free to reach out.
- antman 4y agoA key feature of this release is pluggable backends incl gpu based ones which should greatly affect performance https://github.com/networkx/networkx/pull/6000 https://github.com/networkx/networkx/pull/6000
- samsquire 4y agoI use NetworkX rather unusually to parse Graphviz dot specification files in my devops pipeline tool, mazzle. They allow me to define infrastructure build instructions as a dot file that looks similar to this. It's similar to an advanced makefile. This segment of the infrastructure specification sets a dependency on the bastion box, the vault AMI (Amazon Machine Image) and web server provisioned by terraform on the source AMI. Then it sets a dependency of uploading nodejs to the repository server and a dependency on dpkg scan packages. It also tells machines to join themselves to a Kubernetes cluster. digraph G { label="pipeline"; rankdir=TB; "packer/source-ami*" -> { "terraform/bastion"; "packer/vault-ami*"; "terraform/web" } "terraform/bastion" -> "@repository-upload/nodejs_12.13.1_amd64" "terraform/repository" -> "@repository-upload/nodejs_12.13.1_amd64" "@repository-upload/nodejs_12.13.1_amd64" -> "@shell/dpkg-scanpackages" "terraform/web" -> "@ansible/kubernetes-join"; "terraform/bastion" -> "@ansible/kubernetes-join"; "terraform/services" -> "@ansible/kubernetes-join"; "@ansible/kubernetes" -> "@ansible/kubernetes-join"; ... } Graph formats are really flexible and compact and I would like to see them used for more things. A topological search solves many problems. https://devops-pipeline.com/ https://devops-pipeline.com/
- samsquire 4y agoI should add, I also use the resulting dependency graph to parallelise the build. So if there's two packer builds that do not depend on eachother, they can run in parallel since they are slow. https://github.com/samsquire/devops-schedule https://github.com/samsquire/devops-schedule https://github.com/samsquire/parallel-workers https://github.com/samsquire/parallel-workers
- VectorLock 4y agoNice I like the concept of turning around Graphviz dot files and turning them from an intermediate representation of visualization into the actual source of truth for the system. Cool idea.
- liotier 4y agoFor my telco job, I whipped up a shortest path routing proof of concept using NetworkX in a couple hours, right after stumbling on it, with my rudimentary Python scripting skills. The sort of extremely productive library that makes one feel heroic !
- martinsmit 4y agoCan someone comment on the user experience of NetworkX vs comparable packages like igraph? I've used Graphs.jl (formerly LightGraphs.jl) and I was unimpressed as it felt quite cumbersome and unintuitive.
- Xcelerate 4y agoI just started using NetworkX a few days ago and was trying to figure out a way to find all paths in a DAG from a source node to a destination node, with the requirement that no path is a superpath of any other path. I ended up writing my own approach that iterates through all pairs of paths but have been bothered since then that I couldn’t find a function in NetworkX to do this; I’m sure it probably exists but I don’t know the right terminology to search for it. Anyone know what this function might be called?
- math_dandy 4y agoWill nx.all_simple_paths do the job? (A simple path is a path with no repeated nodes.)
- ironSkillet 4y agoI am trying to imagine how it is even possible for this to happen in a DAG. One path from A to B cannot be a strict subset of another (because this implies a cycle) - I unless you mean something else by "superpath".
- Xcelerate 4y agoIf you have paths [A-B-C-Z, A-B-Z, A-D-Z], path A-B-C-Z would be eliminated from that set since it is a superpath of A-B-Z. Is my terminology incorrect?
- ironSkillet 4y agoAh, I see what you mean - I was not thinking about the problem correctly.
- itomato 4y agoI have had some luck using NetworkX to render GraphML generated from Jira issue history in a Jupyter notebook, similar to this: https://towardsdatascience.com/from-dataframe-to-network-graph-bbb35c8ab675 https://towardsdatascience.com/from-dataframe-to-network-gra... If there is another method I'd like to hear about it.
- tekkk 4y agoIs it now finally faster than GraphLab? Couple years ago when I last used them, there were some operations like PageRank that was magnitudes faster with GraphLab. Hope they have caught up now. GraphLab has been deprecated for a long while now. Curious did or will they ever make it open-source.
- dallathee 4y ago> Is it now finally faster than GraphLab? NetworkX(pure python with dict as the data structure) itself will never be faster than any C++ based library (I think GraphLab doesn't even exist now?) but with the new experimental backend plugins, you should be able to use NetworkX API and dispatch the computation to more efficient backends like GraphBLAS, and hopefully CuGraph in the future (if you have GPUs). A bit more info: https://twitter.com/networkx_team/status/1612478129649459202 https://twitter.com/networkx_team/status/1612478129649459202
- dilawar 4y agonetworkx and pypy is a great combination. pypy really shines on pure python libraries like networkx and myhdl.
- graphviz 4y agoTaking a broad view of the area, it's interesting that tables and networks are both fundamental ways of representing data in many situation, and often complementary, but the technology for working with tables is far more mature. This spans the low end of representation (CSV files, data frames in pandas) to database systems that provide essential capabilities like query languages and consistency checking, to entire application platforms like Oracle. The equivalent systems for networks aren't as mature. There's still a big opportunity to meet this need, and you can see a lot of progress in technology like NetworkX and Neo4J. A great service that network programming systems provide, as highlighted in some of the comments here, is simply to provide a framework for integrating useful functions for operating or computing on networks so they can be delivered to a broad audience. To illustrate, in the land of tables, if I need to join two tables, and count rows or sort and show the top-N, I know what software on my computer does that. If I have a network and want to find a shortest path or do some other simple pattern matching on paths, and maybe show the result graphically, what am I supposed to use? It's really not clear. And, in 2023, why not both? Applications need both tabular and network views of data, and in some ways they are not that different. That the popular relational data platforms have not already rushed in to fill this need give some reason to pause and consider the situation further or maybe is an argument that tables and networks should have similar importance. From real life, maybe one issue is that the technology for computing on tables is pretty well understood and manageable, but there are a lot of sharp edges around combinatorial structures like networks. From a recent real-life example, say I just want to make an attributed network representing employees in a company, and match them for an internal project. Is this as easy as sorting rows in a table? Hardly. Can you even find a textbook pseudocode algorithm and code it in a graph query language that is recognizable as that algorithm? These look like tough problems. At least NetworkX and systems like it provide a framework to integrate those implementation for a developer audience.