7 ms·

Spoiler alert, the leak is in libxml2, not Python code.

bbernard 10y ago
In libxml2? That's a reasonable guess, but I would be open to other possibilities I were you, hehe :) Stay tuned!
g-clef 10y ago
That was my first thought when I saw the headline "I bet it's lxml". I have lost count of the number of times I've had to do special nonsense to handle lxml's memory leaks. (I've even had to go so far as to launch parsing code in a subprocess just so the part that created an etree object would quit & return all its RAM.)
nerdponx 10y ago
No kidding. I actually once had a freaky memory leak of my own that I was never able to fix. I was in fact using libxml2 there, I wonder if that was the cause.
- bbernard 10y ago
  There's more than one leak actually. It's quite a party :)
- rudolf0 10y ago
  Exact same thing. Odd unsolved memory leak, using lxml for HTML parsing. I think I'll stay away from it in the future. Unfortunate, since it's the fastest HTML parser in Python that I know of.
  - maxerickson 10y ago
    Have you compared it to the C implementation of ElementTree? With stanzas like for event, n in ElementTree.iterparse(xmlfile, events=("end",)): if n.tag == "blar": # do stuff n.clear() it doesn't even take much memory to parse files of arbitrary size.
deleted 10y ago
[deleted]
mwcampbell 10y ago
The post's tags, displayed at the top, spoiled it for me.
Animats 10y ago
This sort of thing is why calling C from Python is a marginal idea. Debugging such C is tough, especially if it manipulates Python objects in complex ways. There are many invariants of the Python data structures which must be carefully maintained by hand in C code. Getting one wrong will result in obscure bugs like the parent. "Pickle", Python's old seralizer, had a similar leak problem. It has a cache. You can reuse a Pickle stream, which is done in interprocess communication. But the cache of previously-send objects wasn't being cleared at the end of each Pickle block. I found that, did a workaround, and submitted a bug report. Not sure it was ever fixed properly. I still have a bug report in on CPickle, a "faster" implementation of Pickle written in C.[1] In a complicated situation with multiple threads using CPickle, memory becomes corrupted and the program will crash. It doesn't happen using Python Pickle, so I just quit using CPickle. The bug report got the usual "reproduce in a simpler situation" reply to make it go away, and the bug report remains open. It may be the same bug as this one [2] from 2012, although I doubt it. For parsing HTML, I use html4lib. It's slower, but it's all in Python. [1] http://bugs.python.org/issue23655 http://bugs.python.org/issue23655 [2] http://bugs.python.org/issue12680 http://bugs.python.org/issue12680
- bbernard 10y ago
  I've had quite a few issues with cPickle myself, so I see what you mean. Indeed, Python packages built over C extensions can be quite hard to debug, as seen with lxml. But what makes it even harder is the fact that lxml is partly built using Cython... so you deal with Python code, C code generated with Cython, and pure C code (libxml2).
  - dom0 10y ago
    While Cython - in my experience - doesn't have many bugs, it still has some, and can, at times, generate blaringly wrong C code (iirc one simple example: slice assigmnent to a uint8_t* tries to call a Python runtime function on the uint8_t* as a PyObject*).
    - bbernard 10y ago
      Good to know! My only experience so far with Cython comes from lxml. It's a weird language that seems to have a lot of corner cases, though. Just like C++/CLI.

In libxml2? That's a reasonable guess, but I would be open to other possibilities I were you, hehe :) Stay tuned!

That was my first thought when I saw the headline "I bet it's lxml". I have lost count of the number of times I've had to do special nonsense to handle lxml's memory leaks. (I've even had to go so far as to launch parsing code in a subprocess just so the part that created an etree object would quit & return all its RAM.)

nerdponx 10y ago

No kidding. I actually once had a freaky memory leak of my own that I was never able to fix. I was in fact using libxml2 there, I wonder if that was the cause.

bbernard 10y ago
There's more than one leak actually. It's quite a party :)
rudolf0 10y ago
Exact same thing. Odd unsolved memory leak, using lxml for HTML parsing. I think I'll stay away from it in the future. Unfortunate, since it's the fastest HTML parser in Python that I know of.
- maxerickson 10y ago
  Have you compared it to the C implementation of ElementTree? With stanzas like for event, n in ElementTree.iterparse(xmlfile, events=("end",)): if n.tag == "blar": # do stuff n.clear() it doesn't even take much memory to parse files of arbitrary size.

deleted 10y ago

[deleted]

mwcampbell 10y ago

The post's tags, displayed at the top, spoiled it for me.

Animats 10y ago

This sort of thing is why calling C from Python is a marginal idea. Debugging such C is tough, especially if it manipulates Python objects in complex ways. There are many invariants of the Python data structures which must be carefully maintained by hand in C code. Getting one wrong will result in obscure bugs like the parent. "Pickle", Python's old seralizer, had a similar leak problem. It has a cache. You can reuse a Pickle stream, which is done in interprocess communication. But the cache of previously-send objects wasn't being cleared at the end of each Pickle block. I found that, did a workaround, and submitted a bug report. Not sure it was ever fixed properly. I still have a bug report in on CPickle, a "faster" implementation of Pickle written in C.[1] In a complicated situation with multiple threads using CPickle, memory becomes corrupted and the program will crash. It doesn't happen using Python Pickle, so I just quit using CPickle. The bug report got the usual "reproduce in a simpler situation" reply to make it go away, and the bug report remains open. It may be the same bug as this one [2] from 2012, although I doubt it. For parsing HTML, I use html4lib. It's slower, but it's all in Python. [1] http://bugs.python.org/issue23655 http://bugs.python.org/issue23655 [2] http://bugs.python.org/issue12680 http://bugs.python.org/issue12680

bbernard 10y ago
I've had quite a few issues with cPickle myself, so I see what you mean. Indeed, Python packages built over C extensions can be quite hard to debug, as seen with lxml. But what makes it even harder is the fact that lxml is partly built using Cython... so you deal with Python code, C code generated with Cython, and pure C code (libxml2).
- dom0 10y ago
  While Cython - in my experience - doesn't have many bugs, it still has some, and can, at times, generate blaringly wrong C code (iirc one simple example: slice assigmnent to a uint8_t* tries to call a Python runtime function on the uint8_t* as a PyObject*).
  - bbernard 10y ago
    Good to know! My only experience so far with Cython comes from lxml. It's a weird language that seems to have a lot of corner cases, though. Just like C++/CLI.