4 ms·

I understand how the example can leak a resource from the OS standpoint, but I don’t understand how this ends up being a memory leak for the Python process. It

by Ecco 5y ago

I understand how the example can leak a resource from the OS standpoint, but I don’t understand how this ends up being a memory leak for the Python process. It’s not the first time I am hearing about memleaks for a GC-ed language, and I never really understood how it was possible. Would anyone have a super simple example?

deleted 5y ago
[deleted]
persedes 5y ago
Not sure if that classifies as a memory leak,but it can happen with mutable types (lists for example) attached to some global variable or as a default in a class definition / function input. For example: class A a = [] def __init__(self): self.a.append(1) Or def foo(a=[]): # code that adds to a...
js2 5y ago
Python uses reference counting. The exception was preventing the code from reaching the point where it sets the reference to the socket to None. This isn't clear in the minimized example in the post, but if you follow the link to the PR[1], the original code looked like this: def close(self): if self.sock is not None: self._shutdown_transport() # Call shutdown first to make sure that pending messages # reach the AMQP broker if the program exits after # calling this method. self.sock.shutdown(socket.SHUT_RDWR) self.sock.close() self.sock = None self.connected = False If either shutdown (or close) raises an exception, self.sock isn't set to None. Thus Python retains the reference. Aside, but setting self.sock to None will eventually close the socket once Python GC's the reference[2]. It's still best to explicitly call close though. From [2]: > Python takes the automatic shutdown a step further, and says that when a socket is garbage collected, it will automatically do a close if it’s needed. But relying on this is a very bad habit. If your socket just disappears without doing a close, the socket at the other end may hang indefinitely, thinking you’re just being slow. Please close your sockets when you’re done. Another way you can have leaks with a reference counting GC is through circular references. Python provides WeakRef for such situations, but it's up to the programmer to use it correctly. 1. https://github.com/celery/celery/issues/4843#issuecomment-991394349 https://github.com/celery/celery/issues/4843#issuecomment-99... 2. https://docs.python.org/3/howto/sockets.html#disconnecting https://docs.python.org/3/howto/sockets.html#disconnecting
- tmarice 5y ago
  Python uses a generational reference counting GC, it should be able to handle circular references.
  - js2 5y ago
    Sheesh, I've somehow gone two decades of Python programming without realizing that. Thank you for the correction.
    - shpx 5y ago
      More info on CPython's GC: https://devguide.python.org/garbage_collector/ https://devguide.python.org/garbage_collector/

by Ecco 5y ago

deleted 5y ago

[deleted]

persedes 5y ago

Not sure if that classifies as a memory leak,but it can happen with mutable types (lists for example) attached to some global variable or as a default in a class definition / function input. For example: class A a = [] def __init__(self): self.a.append(1) Or def foo(a=[]): # code that adds to a...

js2 5y ago

Python uses reference counting. The exception was preventing the code from reaching the point where it sets the reference to the socket to None. This isn't clear in the minimized example in the post, but if you follow the link to the PR[1], the original code looked like this: def close(self): if self.sock is not None: self._shutdown_transport() # Call shutdown first to make sure that pending messages # reach the AMQP broker if the program exits after # calling this method. self.sock.shutdown(socket.SHUT_RDWR) self.sock.close() self.sock = None self.connected = False If either shutdown (or close) raises an exception, self.sock isn't set to None. Thus Python retains the reference. Aside, but setting self.sock to None will eventually close the socket once Python GC's the reference[2]. It's still best to explicitly call close though. From [2]: > Python takes the automatic shutdown a step further, and says that when a socket is garbage collected, it will automatically do a close if it’s needed. But relying on this is a very bad habit. If your socket just disappears without doing a close, the socket at the other end may hang indefinitely, thinking you’re just being slow. Please close your sockets when you’re done. Another way you can have leaks with a reference counting GC is through circular references. Python provides WeakRef for such situations, but it's up to the programmer to use it correctly. 1. https://github.com/celery/celery/issues/4843#issuecomment-991394349 https://github.com/celery/celery/issues/4843#issuecomment-99... 2. https://docs.python.org/3/howto/sockets.html#disconnecting https://docs.python.org/3/howto/sockets.html#disconnecting

tmarice 5y ago
Python uses a generational reference counting GC, it should be able to handle circular references.
- js2 5y ago
  Sheesh, I've somehow gone two decades of Python programming without realizing that. Thank you for the correction.
  - shpx 5y ago
    More info on CPython's GC: https://devguide.python.org/garbage_collector/ https://devguide.python.org/garbage_collector/