5 ms·

> why is it so typical for a Python we worker to have its memory grow infinitely with each request processed? It's not. The cases where it happened to me durin

> why is it so typical for a Python we worker to have its memory grow infinitely with each request processed? It's not. The cases where it happened to me during the last 10 years and dozens of web projects spanning on multiple frameworks were generally me doing something stupid: - using a mutable object as default value or in a class attribute - doing something in __del__ - keeping DEBUG=True for Django (which is very well known for causing memory leaks) > Is this just life now? Nope. I have big Django apps running (like 500k users/day streaming video sites) and it doesn't happen. > Are there any good tools out there for figuring out what part of the application keeps requesting objects that don’t get destroyed? Yes for course: http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-python-memory-leaks/ http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-pyt...

ubernostrum 9y ago
It's hard to call DEBUG in Django a "memory leak" when the "leak" is documented behavior -- when DEBUG=True, Django keeps a running per-connection log in memory of database queries issued. For, y'know, debugging :) This does cause memory use to increase with the number of queries issued, but a debug log is not really what I'd call a leak (and genuinely leaking memory in Python takes some effort).
- ComputerGuru 9y ago
  Who keeps a running, unbounded log of all connections in memory? I’m not a pythonista, so please forgive my ignorance here, but: is this just the default configuration for Django? Does it ship out-of-the-box with adapters to log that to MySQL or similar instead? Why does it not log the last n connections for some sane (and configurable) default value of n instead?
  - bjt 9y ago
    > Who keeps a running, unbounded log of all connections in memory? It does that only when DEBUG=True, which is not the recommended production configuration.
  - traverseda 9y ago
    With django, you don't need to set up a webserver or mysql. You can just run `./manage.py runserver` and it will run in it's own webserver and use sqlite. That is what the "debug=True" variable does. Sets things up for running in a minimal development environment. Another feature? It restarts the server when any source files change. Suffice to say, it's really not meant to be run in anything like production. https://docs.djangoproject.com/en/dev/ref/settings/#debug https://docs.djangoproject.com/en/dev/ref/settings/#debug >Never deploy a site into production with DEBUG turned on. >Did you catch that? NEVER deploy a site into production with DEBUG turned on. >Still, note that there are always going to be sections of your debug output that are inappropriate for public consumption. File paths, configuration options and the like all give attackers extra information about your server. >It is also important to remember that when running with DEBUG turned on, Django will remember every SQL query it executes. This is useful when you’re debugging, but it’ll rapidly consume memory on a production server.
    - ComputerGuru 9y ago
      I don’t know. If I were designing that system, I’d note that most queries would be extremely similar and would benefit insanely from being compressed in situ or would (given were talking DB here) be a perfect fit for normalization. You mentioned SQLite - a second “querylog.db” with a table of queries and a table of instances (query time, parameters, whatever) would be an obvious option (at the expense of slightly exaggerating your db response times, but I’m sure debug mode is already slower as is).
      - traverseda 9y ago
        When running in debug mode, generally the server is restarted every few minutes at worst. As the app is under active development and the server restarts whenever the app changes. The reason it logs those queries is just for debugging. There are a lot of different ways they could do it, but they don't matter because you can just turn that feature off, or set it up to log in a different way, or whatever else. Logging to a DB would be worse, because then it stores a bunch of useless queries that all look very similar. That logging is used for debugging, and wouldn't be something you'd want to use in production. Why hasn't anyone already done this? Because it's trivia, not an actual problem that needs fixing.
  - jsmeaton 9y ago
    Since no one has mentioned yet, it IS bounded to the last 9000 queries by default.
IgorPartola 9y ago
I found pdb to be useful when I wrote my own code from scratch. I found it useless when using a large app on top of Django. Oh really? I have a lot of dicts and tuples? Who knew. I would rather see something like a count of object allocated but not released at the end of each request cycle and their reference graph. Because when I have found memory leaks it has had to do with some kind of cyclic dependencies or creating objects that some kind of long lived object like a database adapter wanted to hold onto.
- sametmax 9y ago
  You got muppy for that: https://pythonhosted.org/Pympler/muppy.html https://pythonhosted.org/Pympler/muppy.html Also objgraph: https://www.darkcoding.net/software/finding-memory-leaks-in-python-with-objgraph/ https://www.darkcoding.net/software/finding-memory-leaks-in-... If you are lucky to use Python 3.6+, you got tracemalloc: https://docs.python.org/3/library/tracemalloc.html https://docs.python.org/3/library/tracemalloc.html
  - IgorPartola 9y ago
    Thanks. Those do look excellent.
pwaai 9y ago
> Nope. I have big Django apps running (like 500k users/day streaming video sites) and it doesn't happen. Mind sharing your stack? I'm currently torn between Laravel and Django. It won't be the first time I've used an MVC framework (CodeIgniter, some Rails) but I like working with Python. However, Laravel also seems to be heavily used. I don't have anything against PHP but wanting to compound my familiarity with Python is one big reason for leaning towards Django...
- sametmax 9y ago
  Varnish, nginx, gunicorn, Django, redis, celery, postgres most of the time. Sometime a bit of crossbar.io. Laravel is a nice framework. I prefer it to symfony, espacially since they love Vue.js which is now my fav front end lib. But the definitive advantage of using Python is that you get a much more versatile language than PHP. Python is not just good for the Web. It's heavily used for scripting, automation, sysadmin, data analysis, GUI, pentesting, etc. People don't just use Python with HTTP in mind. You find it at Apple, NASA, at Sony, Fedora, in the state of Geneva, in french schools, in 3D (Maya, Blender) or geography (most GIS)... Basically, you get a tool that you can use for a lot more stuff, because the ecosystem has been developed for a lot more types of tasks.
  - mycelium 9y ago
    My default stack is almost exactly yours, except > gunicorn Any particular reason you use it over uwsgi? I last did real research in 2013 and have not revisited since then.
    - alexnewman 9y ago
      I use uwsgi when I wanna use the whole box. Now with k8s use gunicorn
    - sametmax 9y ago
      Ease of use. Uwsgi is faster, but it's not the place in my stack where i need speed. The cost of transforming nginx packets in wsgi object is very small compared to all the rest.
misterbowfinger 9y ago
>- using a mutable object as default value or in a class attribute Python noob here. Can you expand on that a little? Or perhaps link a post? I can't think of an example or how it'd impact memory
- hamandcheese 9y ago
  It would only lead to a memory leak depending on your usage, but it’s pretty common gotcha that can be annoying with or without memory issues. The following is some pretty good reading on the matter, and the example given is also one that would lead to a memory leak: https://stackoverflow.com/questions/1132941/least-astonishment-and-the-mutable-default-argument https://stackoverflow.com/questions/1132941/least-astonishme...
- nitely 9y ago
  Here[0]. Mutable objects as default values create a single instance of the object. For iterables (list, set, etc) the behaviour is particularly surprising coz it accumulates/memoize every item ever added (i.e a list that grows/leaks with every call to the function). If you use a good IDE (like pycharm), it will warn you about this stuff. [0] http://docs.python-guide.org/en/latest/writing/gotchas/ http://docs.python-guide.org/en/latest/writing/gotchas/

ubernostrum 9y ago

It's hard to call DEBUG in Django a "memory leak" when the "leak" is documented behavior -- when DEBUG=True, Django keeps a running per-connection log in memory of database queries issued. For, y'know, debugging :) This does cause memory use to increase with the number of queries issued, but a debug log is not really what I'd call a leak (and genuinely leaking memory in Python takes some effort).

ComputerGuru 9y ago
Who keeps a running, unbounded log of all connections in memory? I’m not a pythonista, so please forgive my ignorance here, but: is this just the default configuration for Django? Does it ship out-of-the-box with adapters to log that to MySQL or similar instead? Why does it not log the last n connections for some sane (and configurable) default value of n instead?
- bjt 9y ago
  > Who keeps a running, unbounded log of all connections in memory? It does that only when DEBUG=True, which is not the recommended production configuration.
- traverseda 9y ago
  With django, you don't need to set up a webserver or mysql. You can just run `./manage.py runserver` and it will run in it's own webserver and use sqlite. That is what the "debug=True" variable does. Sets things up for running in a minimal development environment. Another feature? It restarts the server when any source files change. Suffice to say, it's really not meant to be run in anything like production. https://docs.djangoproject.com/en/dev/ref/settings/#debug https://docs.djangoproject.com/en/dev/ref/settings/#debug >Never deploy a site into production with DEBUG turned on. >Did you catch that? NEVER deploy a site into production with DEBUG turned on. >Still, note that there are always going to be sections of your debug output that are inappropriate for public consumption. File paths, configuration options and the like all give attackers extra information about your server. >It is also important to remember that when running with DEBUG turned on, Django will remember every SQL query it executes. This is useful when you’re debugging, but it’ll rapidly consume memory on a production server.
  - ComputerGuru 9y ago
    I don’t know. If I were designing that system, I’d note that most queries would be extremely similar and would benefit insanely from being compressed in situ or would (given were talking DB here) be a perfect fit for normalization. You mentioned SQLite - a second “querylog.db” with a table of queries and a table of instances (query time, parameters, whatever) would be an obvious option (at the expense of slightly exaggerating your db response times, but I’m sure debug mode is already slower as is).
    - traverseda 9y ago
      When running in debug mode, generally the server is restarted every few minutes at worst. As the app is under active development and the server restarts whenever the app changes. The reason it logs those queries is just for debugging. There are a lot of different ways they could do it, but they don't matter because you can just turn that feature off, or set it up to log in a different way, or whatever else. Logging to a DB would be worse, because then it stores a bunch of useless queries that all look very similar. That logging is used for debugging, and wouldn't be something you'd want to use in production. Why hasn't anyone already done this? Because it's trivia, not an actual problem that needs fixing.
- jsmeaton 9y ago
  Since no one has mentioned yet, it IS bounded to the last 9000 queries by default.

IgorPartola 9y ago

I found pdb to be useful when I wrote my own code from scratch. I found it useless when using a large app on top of Django. Oh really? I have a lot of dicts and tuples? Who knew. I would rather see something like a count of object allocated but not released at the end of each request cycle and their reference graph. Because when I have found memory leaks it has had to do with some kind of cyclic dependencies or creating objects that some kind of long lived object like a database adapter wanted to hold onto.

sametmax 9y ago
You got muppy for that: https://pythonhosted.org/Pympler/muppy.html https://pythonhosted.org/Pympler/muppy.html Also objgraph: https://www.darkcoding.net/software/finding-memory-leaks-in-python-with-objgraph/ https://www.darkcoding.net/software/finding-memory-leaks-in-... If you are lucky to use Python 3.6+, you got tracemalloc: https://docs.python.org/3/library/tracemalloc.html https://docs.python.org/3/library/tracemalloc.html
- IgorPartola 9y ago
  Thanks. Those do look excellent.

pwaai 9y ago

> Nope. I have big Django apps running (like 500k users/day streaming video sites) and it doesn't happen. Mind sharing your stack? I'm currently torn between Laravel and Django. It won't be the first time I've used an MVC framework (CodeIgniter, some Rails) but I like working with Python. However, Laravel also seems to be heavily used. I don't have anything against PHP but wanting to compound my familiarity with Python is one big reason for leaning towards Django...

sametmax 9y ago
Varnish, nginx, gunicorn, Django, redis, celery, postgres most of the time. Sometime a bit of crossbar.io. Laravel is a nice framework. I prefer it to symfony, espacially since they love Vue.js which is now my fav front end lib. But the definitive advantage of using Python is that you get a much more versatile language than PHP. Python is not just good for the Web. It's heavily used for scripting, automation, sysadmin, data analysis, GUI, pentesting, etc. People don't just use Python with HTTP in mind. You find it at Apple, NASA, at Sony, Fedora, in the state of Geneva, in french schools, in 3D (Maya, Blender) or geography (most GIS)... Basically, you get a tool that you can use for a lot more stuff, because the ecosystem has been developed for a lot more types of tasks.
- mycelium 9y ago
  My default stack is almost exactly yours, except > gunicorn Any particular reason you use it over uwsgi? I last did real research in 2013 and have not revisited since then.
  - alexnewman 9y ago
    I use uwsgi when I wanna use the whole box. Now with k8s use gunicorn
  - sametmax 9y ago
    Ease of use. Uwsgi is faster, but it's not the place in my stack where i need speed. The cost of transforming nginx packets in wsgi object is very small compared to all the rest.

misterbowfinger 9y ago

>- using a mutable object as default value or in a class attribute Python noob here. Can you expand on that a little? Or perhaps link a post? I can't think of an example or how it'd impact memory

hamandcheese 9y ago
It would only lead to a memory leak depending on your usage, but it’s pretty common gotcha that can be annoying with or without memory issues. The following is some pretty good reading on the matter, and the example given is also one that would lead to a memory leak: https://stackoverflow.com/questions/1132941/least-astonishment-and-the-mutable-default-argument https://stackoverflow.com/questions/1132941/least-astonishme...
nitely 9y ago
Here[0]. Mutable objects as default values create a single instance of the object. For iterables (list, set, etc) the behaviour is particularly surprising coz it accumulates/memoize every item ever added (i.e a list that grows/leaks with every call to the function). If you use a good IDE (like pycharm), it will warn you about this stuff. [0] http://docs.python-guide.org/en/latest/writing/gotchas/ http://docs.python-guide.org/en/latest/writing/gotchas/