5 ms·

It's hard to call DEBUG in Django a "memory leak" when the "leak" is documented behavior -- when DEBUG=True, Django keeps a running per-connection log in memory

It's hard to call DEBUG in Django a "memory leak" when the "leak" is documented behavior -- when DEBUG=True, Django keeps a running per-connection log in memory of database queries issued. For, y'know, debugging :) This does cause memory use to increase with the number of queries issued, but a debug log is not really what I'd call a leak (and genuinely leaking memory in Python takes some effort).

ComputerGuru 9y ago
Who keeps a running, unbounded log of all connections in memory? I’m not a pythonista, so please forgive my ignorance here, but: is this just the default configuration for Django? Does it ship out-of-the-box with adapters to log that to MySQL or similar instead? Why does it not log the last n connections for some sane (and configurable) default value of n instead?
- bjt 9y ago
  > Who keeps a running, unbounded log of all connections in memory? It does that only when DEBUG=True, which is not the recommended production configuration.
- traverseda 9y ago
  With django, you don't need to set up a webserver or mysql. You can just run `./manage.py runserver` and it will run in it's own webserver and use sqlite. That is what the "debug=True" variable does. Sets things up for running in a minimal development environment. Another feature? It restarts the server when any source files change. Suffice to say, it's really not meant to be run in anything like production. https://docs.djangoproject.com/en/dev/ref/settings/#debug https://docs.djangoproject.com/en/dev/ref/settings/#debug >Never deploy a site into production with DEBUG turned on. >Did you catch that? NEVER deploy a site into production with DEBUG turned on. >Still, note that there are always going to be sections of your debug output that are inappropriate for public consumption. File paths, configuration options and the like all give attackers extra information about your server. >It is also important to remember that when running with DEBUG turned on, Django will remember every SQL query it executes. This is useful when you’re debugging, but it’ll rapidly consume memory on a production server.
  - ComputerGuru 9y ago
    I don’t know. If I were designing that system, I’d note that most queries would be extremely similar and would benefit insanely from being compressed in situ or would (given were talking DB here) be a perfect fit for normalization. You mentioned SQLite - a second “querylog.db” with a table of queries and a table of instances (query time, parameters, whatever) would be an obvious option (at the expense of slightly exaggerating your db response times, but I’m sure debug mode is already slower as is).
    - traverseda 9y ago
      When running in debug mode, generally the server is restarted every few minutes at worst. As the app is under active development and the server restarts whenever the app changes. The reason it logs those queries is just for debugging. There are a lot of different ways they could do it, but they don't matter because you can just turn that feature off, or set it up to log in a different way, or whatever else. Logging to a DB would be worse, because then it stores a bunch of useless queries that all look very similar. That logging is used for debugging, and wouldn't be something you'd want to use in production. Why hasn't anyone already done this? Because it's trivia, not an actual problem that needs fixing.
    - ubernostrum 9y ago
      As others have noted, the behavior is there for use in lighter-weight debugging/dev situations when you want to interact with the code and also have the ability to inspect what it's doing in detail (either by directly poking things from inside a Python interpreter, or through plugins like the debug toolbar). Trying to design a hyper-efficient compressed query storage mechanism would be massive overkill for such a thing. The simplest thing that could possibly work for this is an in-memory list of the queries as strings, so that's what it does.
- jsmeaton 9y ago
  Since no one has mentioned yet, it IS bounded to the last 9000 queries by default.

ComputerGuru 9y ago

Who keeps a running, unbounded log of all connections in memory? I’m not a pythonista, so please forgive my ignorance here, but: is this just the default configuration for Django? Does it ship out-of-the-box with adapters to log that to MySQL or similar instead? Why does it not log the last n connections for some sane (and configurable) default value of n instead?

bjt 9y ago
> Who keeps a running, unbounded log of all connections in memory? It does that only when DEBUG=True, which is not the recommended production configuration.
traverseda 9y ago
With django, you don't need to set up a webserver or mysql. You can just run `./manage.py runserver` and it will run in it's own webserver and use sqlite. That is what the "debug=True" variable does. Sets things up for running in a minimal development environment. Another feature? It restarts the server when any source files change. Suffice to say, it's really not meant to be run in anything like production. https://docs.djangoproject.com/en/dev/ref/settings/#debug https://docs.djangoproject.com/en/dev/ref/settings/#debug >Never deploy a site into production with DEBUG turned on. >Did you catch that? NEVER deploy a site into production with DEBUG turned on. >Still, note that there are always going to be sections of your debug output that are inappropriate for public consumption. File paths, configuration options and the like all give attackers extra information about your server. >It is also important to remember that when running with DEBUG turned on, Django will remember every SQL query it executes. This is useful when you’re debugging, but it’ll rapidly consume memory on a production server.
- ComputerGuru 9y ago
  I don’t know. If I were designing that system, I’d note that most queries would be extremely similar and would benefit insanely from being compressed in situ or would (given were talking DB here) be a perfect fit for normalization. You mentioned SQLite - a second “querylog.db” with a table of queries and a table of instances (query time, parameters, whatever) would be an obvious option (at the expense of slightly exaggerating your db response times, but I’m sure debug mode is already slower as is).
  - traverseda 9y ago
    When running in debug mode, generally the server is restarted every few minutes at worst. As the app is under active development and the server restarts whenever the app changes. The reason it logs those queries is just for debugging. There are a lot of different ways they could do it, but they don't matter because you can just turn that feature off, or set it up to log in a different way, or whatever else. Logging to a DB would be worse, because then it stores a bunch of useless queries that all look very similar. That logging is used for debugging, and wouldn't be something you'd want to use in production. Why hasn't anyone already done this? Because it's trivia, not an actual problem that needs fixing.
  - ubernostrum 9y ago
    As others have noted, the behavior is there for use in lighter-weight debugging/dev situations when you want to interact with the code and also have the ability to inspect what it's doing in detail (either by directly poking things from inside a Python interpreter, or through plugins like the debug toolbar). Trying to design a hyper-efficient compressed query storage mechanism would be massive overkill for such a thing. The simplest thing that could possibly work for this is an in-memory list of the queries as strings, so that's what it does.
jsmeaton 9y ago
Since no one has mentioned yet, it IS bounded to the last 9000 queries by default.