5 ms·

A guide to organizing settings in Django

98 points by senko 5y ago

nettleseyeball 5y ago
Interesting read! Just wanted to say though I don't think the critique of the "settings/__init__.py" approach is accurate. You can easily access environment variables inside these files using os.environ, and it doesn't require a file locally that isn't present in the repo. As you say you just specify which settings file to use with the DJANGO_SETTINGS_MODULE env var. Am I missing something?
- senko 5y ago
  Thanks! Once you switch to using environment variables, having additional specialized settings files is a complexity you don't need, in my experience. So I prefer to simplify without losing much in terms of convenience.
silvester23 5y ago
So, they figured out they can can use os.environ within the settings.py? Am I missing something?
- senko 5y ago
  OP here. You can use whatever you want in settings.py. It's "why" that matters. Yeah, you can boil it down to "use env", but do people know why it's usually the best approach? In my experience, many don't, hence the blog post. I mean, if it were obvious and accepted truth, Django would be doing this by default already.
  - leetrout 5y ago
    The community writ large is already using django-environ to solve the problems you faced. https://github.com/joke2k/django-environ https://github.com/joke2k/django-environ
leetrout 5y ago
This isn't wrong advice but their tone of authority seems a bit misplaced. 1. Make settings.py a package 2. Please do not put anything more than imports in __init__.py since you are adding logic downstream. 3. Use django environ https://github.com/joke2k/django-environ https://github.com/joke2k/django-environ which gives you a defined schema of your environment needs with casting and default values when needed. Now you keep your .env files, keep your setting overrides (settings/base.py, settings/local.py, etc) when needed AND you have defined a schema with non-string types for your environment variables. From the readme you can see how you make a schema with a casted type and default value: env = environ.Env( DEBUG=(bool, False) )
- senko 5y ago
  OP here. There are different styles you could go about, all of them having in common that you wan't to pull things from .env nicely. I personally don't prefer django-environ because it's a bit too verbose/magical for my tastes, so use python-dotenv. I would recommend against using `local.py`. It's easy to overcomplicate things here and one of the things I worry about is how easy it is to have a new person set the project up and start working on it. So, in general, absolutely agree with you. Details (which package to use and how exactly to set up the layout) are always going to be preference-based. Or at least until the nice folks from Black alleviate us from that pain as well :)
  - leetrout 5y ago
    Sure. Claiming "too verbose/magical" for a library that adds structure to generally unstructured, smattered config data seems a bit misplaced. You are entitled to your opinion but I don't think trying to defend your stance with homemade env parsing functions while writing an authoritative sounding blog post helps the community as a whole when there are sound libraries and patterns that solve this exact problem.
- whalesalad 5y ago
  That’s how I do it. .env files are terrible if you’re using them with a language specific loader. The entire point of env is that it’s an abstraction you can manipulate externally. You can setup the env, then launch the program. It allows for runtime changes. https://whalesalad.com/blog/doing-python-configuration-right https://whalesalad.com/blog/doing-python-configuration-right You can do it right without packages or external dependencies.
danpalmer 5y ago
The next step from this is the django-configurations package. While it's not perfect, it does help when settings get very complex. We've got a Django site that we run in production for different partners. We've got 560 settings, and then dev/test/prod configs for each partner, plus we vary the list of INSTALLED_APPS per partner. django-configurations is pretty nice for making this all manageable, and mypy helps to make sure everything is consistent.
- mands 5y ago
  +1 for django-configurations, it's been invaluable for us in helping to keep settings managable as we build / test / deploy across multiple environments and modes
rmnclmnt 5y ago
Dynamic configuration is the first step, second step is the validation of configuration. To that aim, one can use Pydantic Settings management with extensive validation to avoid footguns down the road. If validation is configured properly, if the application starts without crashing, it is already on good tracks.
VWWHFSfQ 5y ago
We just have a single settings.py and use django-environ to populate it. # settings.py from environ import Env ENV = Env() DEBUG = ENV.bool("DEBUG", True) SECRET_KEY = ENV.str("SECRET_KEY", "debug-secret-key") ALLOWED_HOSTS = ENV.list("ALLOWED_HOSTS", default=["*"]) ... # Running with local .env $ foreman run -- python3 manage.py runserver # Running with prod.env $ foreman run -e prod.env -- python3 manage.py runserver It's easy and scales well with a lot of settings/environments/developers.
- danpalmer 5y ago
  I realise that this is probably just an example for HN so this isn't criticism of your comment! But in case anyone reading doesn't know already, `runserver` is not appropriate for running in production. It doesn't have any security hardening, is single threaded and blocking, will reload code as it changes, and is generally very slow. It's generally best to use something like gunicorn, uwsgi, uvicorn, etc, and also to then host behind something like nginx which is designed to be exposed to potentially malicious users.
  - justusw 5y ago
    Furthermore I would like to add that having DEBUG set to True by default could lead you to accidentally enabling Django debug mode in production which could lead to accidentally leaking secrets.
    - VWWHFSfQ 5y ago
      We've gone back and forth on this a number of times internally and decided for the most part to just keep Django's defaults. There are a number of different ways that having "secure by default" can break your local development experience. For instance, if you accidentally run your local Django server with DEBUG=False, SECURE_SSL_REDIRECT=True, SESSION_COOKIE_SECURE=True, and any of the SECURE_HSTS_* settings your browser will basically remember that for all of eternity and you will no longer be able to just use http://localhost:8000/ http://localhost:8000/ without having to go figure out how to get it to "forget" all that stuff. So for local development we follow the principle of least surprise, and we have a fairly rigorous policy/procedure for configuring and auditing production environment settings.
buixuanquy 5y ago
For newbie, I think the best way to learn how to organize a Django project is using cookiecutter-django. They follow most best practices, and you have already worked docker-compose config to quickly up and running without problems.
mattupstate 5y ago
While this might be decent advice around the internal details of a Django application's settings, what's missing is a mention of who this approach is designed for. It appears geared for developers who work on the project because the internal details (module names like settings.local, settings.production) leak to the command line interface. This might be fine in some cases, but this advice is not universal. All apps have a configuration API and you might consider how that looks depending on who/what is running the application, even if it's people who work on the code.
abishekg 5y ago
I use python-decouple with django and it’s solved this pretty elegantly. A dotenv file for local and env variables for production. Isn’t that the 12 factor recommendation? https://github.com/henriquebastos/python-decouple https://github.com/henriquebastos/python-decouple
- wombatpm 5y ago
  Seconded. I switched from django-environ to python-decouple. Now that python-decouple defers to environment variables over .env files I find that I have our DevOps people spin up containers with secrets configured using their tool de jure while I just have a local .env file. We just make sure our env.sample is up to date with respect to our .env. DevOps folk use that to define environment variables for the container.
deleted 5y ago
[deleted]
Alir3z4 5y ago
Don't do this. Keep everything in settings.py and use environment variables (django-environ) to configure it. In a project with tens and tens of apps and so many modules requiring configurations and easily reaching more than thousand settings key, keeping 1 settings.py has not failed me. With the approach mentioned in the article, you'll get to a point where you'll be having so different setting module for each env. 01. local 02. testing 03. CI/CI 04. E2E 05. Docker 06. Kubernetes 07. staging 08. release 09. production 10. k8s_win.py 11. k8s_linux.py 12. k8s_mac.py 13. heroku.py 14. pie.py 15. home.py 16. vacation.py 17. ... Soon you'll get to a place where you need to kind of merge 2 of them (settings/docker.py and settings/kubernetes.py) or worse, several of them and then hell is unleashed. You get confused, your team gets confused, the new team member is confused, no one has any idea where to look at. Just keep it in 1 place, let your brain stay healthier and prevent early cancer.
- senko 5y ago
  OP here. I agree with you. Quoting from the article: > We can now go back to only having settings.py that imports dotenv, fetches the configuration, and initializes Django settings as needed.
- jmath 5y ago
  anything to prevent early cancer
- marcosdumay 5y ago
  Well, now you have the exactly equivalent problem of managing your environment variables. What is better depends on your tooling.
- mattupstate 5y ago
  I'd go so far as to say that configuration details for specific runtime environments should never be the concern of the application/service/program itself. It's almost always a smell to me when I see "dev" "prod" "stage" "local" in code unless it's an abstraction over the configuration for some sort of logical runtime profile.
- scwoodal 5y ago
amirkdv 5y ago
Having experimented with a bunch of different ways to organize complex django settings, I think what OP suggests is a local optimum that's hard to scale and doesn't solve a bunch of important problems (eg ok, credentials are now in local.py which is not committed to git. Now how do I get it to people's dev machines / CI?) My key observation is this: If you're actually deploying and maintaining a django app you can't escape having quite a few environment variables that control different pieces of config. Once I accept that I can't shake these off, I'd rather minimize all other complexity, including having to reason about multiple settings modules that import/override each other in creative ways. What I've settled on is: 1. Every thing controlled by env vars, not choice of settings module 2. the one and only settings.py toggles config based on env vars, say something like this: STORAGE_MODE = os.environ['STORAGE_MODE'] STATICFILES_STORAGE = { 'local': '...StaticFileStorage', 's3': '...S3Boto3Storage' }[STORAGE_MODE] EDIT: wording and typos
ensignavenger 5y ago
This seems overly complex to me. They have their settings in a .env file, they have a separate program/script that loads those settings into the environment variables, then they have another program that reads them out of the environment and parses them into python objects in settings.py... so why not just put them in settings.py to begin with? I have also never agreed with the common suggestion of putting secrets into env vars- I believe secrets belong in something like Hashicorp's Vault and stored on the running system using proper OS level access control.

98 points by senko 5y ago

Interesting read! Just wanted to say though I don't think the critique of the "settings/__init__.py" approach is accurate. You can easily access environment variables inside these files using os.environ, and it doesn't require a file locally that isn't present in the repo. As you say you just specify which settings file to use with the DJANGO_SETTINGS_MODULE env var. Am I missing something?

senko 5y ago
Thanks! Once you switch to using environment variables, having additional specialized settings files is a complexity you don't need, in my experience. So I prefer to simplify without losing much in terms of convenience.

silvester23 5y ago

So, they figured out they can can use os.environ within the settings.py? Am I missing something?

senko 5y ago
OP here. You can use whatever you want in settings.py. It's "why" that matters. Yeah, you can boil it down to "use env", but do people know why it's usually the best approach? In my experience, many don't, hence the blog post. I mean, if it were obvious and accepted truth, Django would be doing this by default already.
- leetrout 5y ago
  The community writ large is already using django-environ to solve the problems you faced. https://github.com/joke2k/django-environ https://github.com/joke2k/django-environ

leetrout 5y ago

This isn't wrong advice but their tone of authority seems a bit misplaced. 1. Make settings.py a package 2. Please do not put anything more than imports in __init__.py since you are adding logic downstream. 3. Use django environ https://github.com/joke2k/django-environ https://github.com/joke2k/django-environ which gives you a defined schema of your environment needs with casting and default values when needed. Now you keep your .env files, keep your setting overrides (settings/base.py, settings/local.py, etc) when needed AND you have defined a schema with non-string types for your environment variables. From the readme you can see how you make a schema with a casted type and default value: env = environ.Env( DEBUG=(bool, False) )

senko 5y ago
OP here. There are different styles you could go about, all of them having in common that you wan't to pull things from .env nicely. I personally don't prefer django-environ because it's a bit too verbose/magical for my tastes, so use python-dotenv. I would recommend against using `local.py`. It's easy to overcomplicate things here and one of the things I worry about is how easy it is to have a new person set the project up and start working on it. So, in general, absolutely agree with you. Details (which package to use and how exactly to set up the layout) are always going to be preference-based. Or at least until the nice folks from Black alleviate us from that pain as well :)
- leetrout 5y ago
  Sure. Claiming "too verbose/magical" for a library that adds structure to generally unstructured, smattered config data seems a bit misplaced. You are entitled to your opinion but I don't think trying to defend your stance with homemade env parsing functions while writing an authoritative sounding blog post helps the community as a whole when there are sound libraries and patterns that solve this exact problem.
whalesalad 5y ago
That’s how I do it. .env files are terrible if you’re using them with a language specific loader. The entire point of env is that it’s an abstraction you can manipulate externally. You can setup the env, then launch the program. It allows for runtime changes. https://whalesalad.com/blog/doing-python-configuration-right https://whalesalad.com/blog/doing-python-configuration-right You can do it right without packages or external dependencies.

danpalmer 5y ago

The next step from this is the django-configurations package. While it's not perfect, it does help when settings get very complex. We've got a Django site that we run in production for different partners. We've got 560 settings, and then dev/test/prod configs for each partner, plus we vary the list of INSTALLED_APPS per partner. django-configurations is pretty nice for making this all manageable, and mypy helps to make sure everything is consistent.

mands 5y ago
+1 for django-configurations, it's been invaluable for us in helping to keep settings managable as we build / test / deploy across multiple environments and modes

rmnclmnt 5y ago

Dynamic configuration is the first step, second step is the validation of configuration. To that aim, one can use Pydantic Settings management with extensive validation to avoid footguns down the road. If validation is configured properly, if the application starts without crashing, it is already on good tracks.

VWWHFSfQ 5y ago

We just have a single settings.py and use django-environ to populate it. # settings.py from environ import Env ENV = Env() DEBUG = ENV.bool("DEBUG", True) SECRET_KEY = ENV.str("SECRET_KEY", "debug-secret-key") ALLOWED_HOSTS = ENV.list("ALLOWED_HOSTS", default=["*"]) ... # Running with local .env $ foreman run -- python3 manage.py runserver # Running with prod.env $ foreman run -e prod.env -- python3 manage.py runserver It's easy and scales well with a lot of settings/environments/developers.

danpalmer 5y ago
I realise that this is probably just an example for HN so this isn't criticism of your comment! But in case anyone reading doesn't know already, `runserver` is not appropriate for running in production. It doesn't have any security hardening, is single threaded and blocking, will reload code as it changes, and is generally very slow. It's generally best to use something like gunicorn, uwsgi, uvicorn, etc, and also to then host behind something like nginx which is designed to be exposed to potentially malicious users.
- justusw 5y ago
  Furthermore I would like to add that having DEBUG set to True by default could lead you to accidentally enabling Django debug mode in production which could lead to accidentally leaking secrets.
  - VWWHFSfQ 5y ago
    We've gone back and forth on this a number of times internally and decided for the most part to just keep Django's defaults. There are a number of different ways that having "secure by default" can break your local development experience. For instance, if you accidentally run your local Django server with DEBUG=False, SECURE_SSL_REDIRECT=True, SESSION_COOKIE_SECURE=True, and any of the SECURE_HSTS_* settings your browser will basically remember that for all of eternity and you will no longer be able to just use http://localhost:8000/ http://localhost:8000/ without having to go figure out how to get it to "forget" all that stuff. So for local development we follow the principle of least surprise, and we have a fairly rigorous policy/procedure for configuring and auditing production environment settings.

buixuanquy 5y ago

For newbie, I think the best way to learn how to organize a Django project is using cookiecutter-django. They follow most best practices, and you have already worked docker-compose config to quickly up and running without problems.

mattupstate 5y ago

While this might be decent advice around the internal details of a Django application's settings, what's missing is a mention of who this approach is designed for. It appears geared for developers who work on the project because the internal details (module names like settings.local, settings.production) leak to the command line interface. This might be fine in some cases, but this advice is not universal. All apps have a configuration API and you might consider how that looks depending on who/what is running the application, even if it's people who work on the code.

abishekg 5y ago

I use python-decouple with django and it’s solved this pretty elegantly. A dotenv file for local and env variables for production. Isn’t that the 12 factor recommendation? https://github.com/henriquebastos/python-decouple https://github.com/henriquebastos/python-decouple

wombatpm 5y ago
Seconded. I switched from django-environ to python-decouple. Now that python-decouple defers to environment variables over .env files I find that I have our DevOps people spin up containers with secrets configured using their tool de jure while I just have a local .env file. We just make sure our env.sample is up to date with respect to our .env. DevOps folk use that to define environment variables for the container.

deleted 5y ago

[deleted]

Alir3z4 5y ago

Don't do this. Keep everything in settings.py and use environment variables (django-environ) to configure it. In a project with tens and tens of apps and so many modules requiring configurations and easily reaching more than thousand settings key, keeping 1 settings.py has not failed me. With the approach mentioned in the article, you'll get to a point where you'll be having so different setting module for each env. 01. local 02. testing 03. CI/CI 04. E2E 05. Docker 06. Kubernetes 07. staging 08. release 09. production 10. k8s_win.py 11. k8s_linux.py 12. k8s_mac.py 13. heroku.py 14. pie.py 15. home.py 16. vacation.py 17. ... Soon you'll get to a place where you need to kind of merge 2 of them (settings/docker.py and settings/kubernetes.py) or worse, several of them and then hell is unleashed. You get confused, your team gets confused, the new team member is confused, no one has any idea where to look at. Just keep it in 1 place, let your brain stay healthier and prevent early cancer.

senko 5y ago
OP here. I agree with you. Quoting from the article: > We can now go back to only having settings.py that imports dotenv, fetches the configuration, and initializes Django settings as needed.
jmath 5y ago
anything to prevent early cancer
marcosdumay 5y ago
Well, now you have the exactly equivalent problem of managing your environment variables. What is better depends on your tooling.
mattupstate 5y ago
I'd go so far as to say that configuration details for specific runtime environments should never be the concern of the application/service/program itself. It's almost always a smell to me when I see "dev" "prod" "stage" "local" in code unless it's an abstraction over the configuration for some sort of logical runtime profile.
scwoodal 5y ago

amirkdv 5y ago

Having experimented with a bunch of different ways to organize complex django settings, I think what OP suggests is a local optimum that's hard to scale and doesn't solve a bunch of important problems (eg ok, credentials are now in local.py which is not committed to git. Now how do I get it to people's dev machines / CI?) My key observation is this: If you're actually deploying and maintaining a django app you can't escape having quite a few environment variables that control different pieces of config. Once I accept that I can't shake these off, I'd rather minimize all other complexity, including having to reason about multiple settings modules that import/override each other in creative ways. What I've settled on is: 1. Every thing controlled by env vars, not choice of settings module 2. the one and only settings.py toggles config based on env vars, say something like this: STORAGE_MODE = os.environ['STORAGE_MODE'] STATICFILES_STORAGE = { 'local': '...StaticFileStorage', 's3': '...S3Boto3Storage' }[STORAGE_MODE] EDIT: wording and typos

ensignavenger 5y ago

This seems overly complex to me. They have their settings in a .env file, they have a separate program/script that loads those settings into the environment variables, then they have another program that reads them out of the environment and parses them into python objects in settings.py... so why not just put them in settings.py to begin with? I have also never agreed with the common suggestion of putting secrets into env vars- I believe secrets belong in something like Hashicorp's Vault and stored on the running system using proper OS level access control.