4 ms·

Pylint is a great tool, but it could definitely be faster. One school of thought says: of course it's too slow, it's written in Python, if you want it faster yo

Pylint is a great tool, but it could definitely be faster. One school of thought says: of course it's too slow, it's written in Python, if you want it faster you should rewrite it in C. But I don't think such a drastic and destabilizing change is necessary. There are critical sites in the code where a slightly wrong move is made and it gets magnified into something larger. This post points out one such site. The 80/20 rule says that fixing those errors will give most of the benefit of a full rewrite with way less effort. A few years ago I got fed up with how slow Pylint was, so I made some changes to improve its performance. The main issue was that the tree traversal code was written generically. It was nice and elegant in terms of readability, but it meant that there was a lot of unnecessary runtime type-checking. It also meant that a lot of work was getting done for no reason. For example, if you are trying to apply lint rules for assign statements, you only want to check places where assign statement can legally occur. But Pylint was checking every single node for assign statements, including places where they cannot occur, such as inside function calls. Breaking up that generic logic into specialized instances had an enormous impact on performance. Here are the PRs that implemented these changes: https://github.com/PyCQA/astroid/pull/497 https://github.com/PyCQA/astroid/pull/519 https://github.com/PyCQA/astroid/pull/552

user5994461 6y ago
> if you want it faster you should rewrite it in C. But I don't think such a drastic and destabilizing change is necessary. I've worked around the python linter and it is written in C. The bits in python (especially our custom rules) were horribly slow. Lint is building a tree of the code and going through it recursively. Going through a large tree in pure python is horribly slow, it has to interpret every single line of code again and resolve every variable again, because they could change anytime, the overhead is insane. It's the one thing that python cannot do, parsing tree. If python had a JIT it might be doable but currently it isn't.
- xapata 6y ago
  Try PyPy.
  - PaulHoule 6y ago
    Now that PyPy is up to Python 3.6 and has the bugs from the 3.5 series fixed I am using it for real work. The contextvars polyfill brings in the one thing I really need from Py 3.7. I have a toolkit that puffs up XML, JSON, whatever files into an RDF graph that has extra blank nodes that let you annotate anything. A year ago I was complaining about how slow it was, with PyPy it is 5 times faster and I am not complaining. I hear people get similar speed-ups for branchy monte carlo simulations too.
- Recursing 6y ago
  black (the python auto formatter) uses mypyc https://github.com/python/mypy/tree/master/mypyc https://github.com/python/mypy/tree/master/mypyc
  - rattray 6y ago
    That looks exciting! Compiles mypy-annotated Python to a Python C extension. Claims a ~4x speedup, but still quite unstable/buggy, so who knows how much performance they'll have to give up for working software. Fingers crossed!
hinkley 6y ago
Smaller memory footprints can make code faster, but making code faster often creates a bigger memory footprint. For instance, pipelining an operation to allow multiple CPUs to work on a problem at once can reduce the pressure on the tall tent pole, be it IO bottlenecks or one phase. Time is still dictated by the tallest tent poles, but not also by the fourth through tenth tallest poles. But now everything is happening at once, which means all of the temporary data structures exist in memory in parallel, instead of sequentially. Heavily paraphrasing someone else: The simple things get taken care of early. If we're standing around talking about problems in a successful tool, they're complicated. Although I disagree with that person on one point: we can also be talking about problems that were complicated to diagnose (but straightforward to fix).

user5994461 6y ago

> if you want it faster you should rewrite it in C. But I don't think such a drastic and destabilizing change is necessary. I've worked around the python linter and it is written in C. The bits in python (especially our custom rules) were horribly slow. Lint is building a tree of the code and going through it recursively. Going through a large tree in pure python is horribly slow, it has to interpret every single line of code again and resolve every variable again, because they could change anytime, the overhead is insane. It's the one thing that python cannot do, parsing tree. If python had a JIT it might be doable but currently it isn't.

xapata 6y ago
Try PyPy.
- PaulHoule 6y ago
  Now that PyPy is up to Python 3.6 and has the bugs from the 3.5 series fixed I am using it for real work. The contextvars polyfill brings in the one thing I really need from Py 3.7. I have a toolkit that puffs up XML, JSON, whatever files into an RDF graph that has extra blank nodes that let you annotate anything. A year ago I was complaining about how slow it was, with PyPy it is 5 times faster and I am not complaining. I hear people get similar speed-ups for branchy monte carlo simulations too.
Recursing 6y ago
black (the python auto formatter) uses mypyc https://github.com/python/mypy/tree/master/mypyc https://github.com/python/mypy/tree/master/mypyc
- rattray 6y ago
  That looks exciting! Compiles mypy-annotated Python to a Python C extension. Claims a ~4x speedup, but still quite unstable/buggy, so who knows how much performance they'll have to give up for working software. Fingers crossed!

hinkley 6y ago

Smaller memory footprints can make code faster, but making code faster often creates a bigger memory footprint. For instance, pipelining an operation to allow multiple CPUs to work on a problem at once can reduce the pressure on the tall tent pole, be it IO bottlenecks or one phase. Time is still dictated by the tallest tent poles, but not also by the fourth through tenth tallest poles. But now everything is happening at once, which means all of the temporary data structures exist in memory in parallel, instead of sequentially. Heavily paraphrasing someone else: The simple things get taken care of early. If we're standing around talking about problems in a successful tool, they're complicated. Although I disagree with that person on one point: we can also be talking about problems that were complicated to diagnose (but straightforward to fix).