6 ms·

Docker at Spotify

65 points by pini42 13y ago

tinco 13y ago
I like how no one is paying heed to the "Don't use Docker in production" warning on the Docker site. Every once in a while they release a version that's buggy (like 0.7.3) and we fix to the previous version and wait for the next version.
quaunaut 13y ago
I've been watching Docker ever since Flynn was announced, and started using it seriously this last December. Primarily, I'm trying to use it as a faster alternative to Chef/Puppet, running it over a Vagrant precise64 box, to potentially be something I can shove onto a foreign machine without much worry. Overall, this has actually worked fairly well- getting certain containers up and running, and having them work together, hasn't been very difficult. But, there's one place that's consistently been a thorn in my side: PostgreSQL. Currently, there is this article in Docker.io's docs[1] that claims to help you set up a Postgres container, but so far I've not seen it work for me or anyone else I've spoken to about it. It's primarily come from how Postgres is traditionally installed/ran: 1. Service is started. This can either be an 'active' Postgres server, or in the background. This creates a Postmaster.pid file, which ensures no other Postgres servers are running on the machine(I'm not entirely sure why, beyond ease-of-use scenarios). 2. Service is stopped. This deletes the Postmaster.pid file. In the case of Docker containers, this presents a bit of a problem. If you try to start the Postgres container that way, it will run, but the Postmaster.pid seems to sit outside of it. In the process, once you shut that container down, you actually can't start another without error- the Postmaster.pid stops you. The other alternative, is having a container running with Postgres as a background process to something else. This can work, but you start having to use services that can keep a process running while letting you access /bin/bash to still get at Postgres. Neither of these potential solutions allow you to retain data, either. I've done a lot of poking around, but I haven't had a lot of success in figuring out how to get past this. Those who are using Docker as a Puppet/Chef replacement, especially with Vagrant- what are you usually doing for a database solution? 1. http://docs.docker.io/en/latest/examples/postgresql_service/ http://docs.docker.io/en/latest/examples/postgresql_service/
- FooBarWidget 13y ago
  Maybe you are not shutting Postgres down correctly? 'docker kill' is kill `kill -9`, it doesn't let the container shutdown gracefully. `docker stop` on the hand sends SIGTERM to the container and gives it a chance to shut down correctly. As for running Postgres as a background process of something else: I actually recommend this approach. I think running a single process inside a Docker container is a bad idea. By doing that, it does not start important system services such as syslog, and does not allow the container to really work correctly as a proper Unix system should. That is why we have created https://github.com/phusion/baseimage-docker https://github.com/phusion/baseimage-docker, and we base all our Docker containers on that. The Github page that I linked to explains the problem in detail, explains why a lot of Docker containers are actually not working properly, explains how it solves that problem, and explains how you can use the solution.
- jhgg 13y ago
  Have you looked at https://github.com/jeffutter/dokku-postgresql-plugin https://github.com/jeffutter/dokku-postgresql-plugin ? It's built to run with dokku, but the setup steps in the install and command file show how to setup postgres successfully.
- boothead 13y ago
  Here's my docker file for running a postgres 9.3 with plv8 FROM boothead/saucy ENV DEBIAN_FRONTEND noninteractive # Add repository and install PostgreSQL 9.3 RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main 9.3" >> /etc/apt/sources.list RUN apt-get update RUN apt-get -y --force-yes -f install postgresql-9.3 postgresql-client-9.3 libicu48 wget RUN wget http://www.mirrorservice.org/sites/archive.ubuntu.com/ubuntu//pool/universe/libv/libv8/libv8-3.7.12.22_3.7.12.22-3_amd64.deb RUN dpkg -i libv8-3.7.12.22_3.7.12.22-3_amd64.deb RUN apt-get -y --force-yes -f install postgresql-9.3-plv8 # Postgres is started now, shut it down and replace config RUN service postgresql stop ADD pg_hba.conf /etc/postgresql/9.3/main/ ADD pg_ident.conf /etc/postgresql/9.3/main/ ADD postgresql.conf /etc/postgresql/9.3/main/ # Create superuser, tempo db and plv8 language ADD create_user.sh / RUN /bin/bash /create_user.sh RUN rm /create_user.sh # Configure for running in container EXPOSE 5432 # VOLUME ["/var/lib/postgresql/9.3/main"] CMD ["/bin/su", "postgres", "-c", "/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf"] create_user.sh is basically service postgresql start; su echo <SQL> | psql; service postgresql stop; This is fine for me to spin up a postgres instance at some known state for development, but it's a bit dissatisfying for managing the data separately from the container. Anyone got any tips for integrating volumes with this?
afandian 13y ago
If there are any Spotify people reading, please please please fix the search/filter regression. You broke it a year-ish ago. It's nice to hear about innovations but a bit galling to paying customers when you break an essential feature and don't fix it. To reproduce: be on an artist page. Now try and find a track with a given name on that page. I dare you. Without reading every single track with your eyes. You used to be able to filter that list. http://community.spotify.com/t5/Spotify-Ideas/0-8-8-Bring-back-the-filter-option-on-album-artist-pages/idi-p/261856 http://community.spotify.com/t5/Spotify-Ideas/0-8-8-Bring-ba...
casca 13y ago
"* Please note Docker is currently under heavy developement. It should not be used in production (yet)." [1] [1] http://www.docker.io/learn_more/ http://www.docker.io/learn_more/
- allbutlost 13y ago
  This is addressed in the talk. It's a short talk, so may be worth listening to it to see how it was used.
  - casca 13y ago
    Agreed, but I'd have thought that it was worth raising for those who are not familiar with where Docker is in the development cycle.
ermintrude 13y ago
They let devs ssh onto production boxes and do what they want?!?! "Things diverge pretty quickly" - I'm sure they do. That sounds like a recipe for disaster... Docker looks promising but until there's a way to allocate maximum resources to a container I wouldn't use it in production. VMs are much slower to start but at least a runaway process on one won't affect other VMs on the same hardware. Docker only lets you hint that only a certain number of cores should be used, so a bad process might monopolise the physical machine. Also until other tools (ansible, salt, chef, etc) help with provisioning you need to edit the dockerfile to change parameters (eg for settings that depend on your environment like smtp end points test vs prod, etc). If these points are addressed I think it'll be awesome though.
- danielbln 13y ago
  Docker lets you restrict memory and cpu consumption on a per-container basis. Configuration like endpoints are usually fed into the container via environment variables, not by changing stuff around in the Dockerfile. This way you can use the same container in multiple environments.
- ldng 13y ago
  Aren't cgroups be more then a hitting ? Or do you mean that by default does not setup strict enough cgroups ?
  - ermintrude 13y ago
    It seems cpu.shares allow a proportional amount of CPU to be allocated, but LXC may use up to 100% of all idle CPUs. You can't specify a hard-limit of "only use 1GHz when I have a 2GHz CPU" apparently (according to here: http://comments.gmane.org/gmane.linux.kernel.containers.lxc.general/2475 http://comments.gmane.org/gmane.linux.kernel.containers.lxc....). So I guess that could mean that if the containers are using 100% of all CPUs, there wouldn't be resources left e.g. for monitoring on the host? Anyone got any experience with this?
    - deleted 13y ago
      [deleted]
deleted 13y ago
[deleted]
ksec 13y ago
Why Docker? Why not Packer.io? Why No Love for OSv?

65 points by pini42 13y ago

tinco 13y ago

I like how no one is paying heed to the "Don't use Docker in production" warning on the Docker site. Every once in a while they release a version that's buggy (like 0.7.3) and we fix to the previous version and wait for the next version.

quaunaut 13y ago

I've been watching Docker ever since Flynn was announced, and started using it seriously this last December. Primarily, I'm trying to use it as a faster alternative to Chef/Puppet, running it over a Vagrant precise64 box, to potentially be something I can shove onto a foreign machine without much worry. Overall, this has actually worked fairly well- getting certain containers up and running, and having them work together, hasn't been very difficult. But, there's one place that's consistently been a thorn in my side: PostgreSQL. Currently, there is this article in Docker.io's docs[1] that claims to help you set up a Postgres container, but so far I've not seen it work for me or anyone else I've spoken to about it. It's primarily come from how Postgres is traditionally installed/ran: 1. Service is started. This can either be an 'active' Postgres server, or in the background. This creates a Postmaster.pid file, which ensures no other Postgres servers are running on the machine(I'm not entirely sure why, beyond ease-of-use scenarios). 2. Service is stopped. This deletes the Postmaster.pid file. In the case of Docker containers, this presents a bit of a problem. If you try to start the Postgres container that way, it will run, but the Postmaster.pid seems to sit outside of it. In the process, once you shut that container down, you actually can't start another without error- the Postmaster.pid stops you. The other alternative, is having a container running with Postgres as a background process to something else. This can work, but you start having to use services that can keep a process running while letting you access /bin/bash to still get at Postgres. Neither of these potential solutions allow you to retain data, either. I've done a lot of poking around, but I haven't had a lot of success in figuring out how to get past this. Those who are using Docker as a Puppet/Chef replacement, especially with Vagrant- what are you usually doing for a database solution? 1. http://docs.docker.io/en/latest/examples/postgresql_service/ http://docs.docker.io/en/latest/examples/postgresql_service/

FooBarWidget 13y ago
Maybe you are not shutting Postgres down correctly? 'docker kill' is kill `kill -9`, it doesn't let the container shutdown gracefully. `docker stop` on the hand sends SIGTERM to the container and gives it a chance to shut down correctly. As for running Postgres as a background process of something else: I actually recommend this approach. I think running a single process inside a Docker container is a bad idea. By doing that, it does not start important system services such as syslog, and does not allow the container to really work correctly as a proper Unix system should. That is why we have created https://github.com/phusion/baseimage-docker https://github.com/phusion/baseimage-docker, and we base all our Docker containers on that. The Github page that I linked to explains the problem in detail, explains why a lot of Docker containers are actually not working properly, explains how it solves that problem, and explains how you can use the solution.
jhgg 13y ago
Have you looked at https://github.com/jeffutter/dokku-postgresql-plugin https://github.com/jeffutter/dokku-postgresql-plugin ? It's built to run with dokku, but the setup steps in the install and command file show how to setup postgres successfully.
boothead 13y ago
Here's my docker file for running a postgres 9.3 with plv8 FROM boothead/saucy ENV DEBIAN_FRONTEND noninteractive # Add repository and install PostgreSQL 9.3 RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main 9.3" >> /etc/apt/sources.list RUN apt-get update RUN apt-get -y --force-yes -f install postgresql-9.3 postgresql-client-9.3 libicu48 wget RUN wget http://www.mirrorservice.org/sites/archive.ubuntu.com/ubuntu//pool/universe/libv/libv8/libv8-3.7.12.22_3.7.12.22-3_amd64.deb RUN dpkg -i libv8-3.7.12.22_3.7.12.22-3_amd64.deb RUN apt-get -y --force-yes -f install postgresql-9.3-plv8 # Postgres is started now, shut it down and replace config RUN service postgresql stop ADD pg_hba.conf /etc/postgresql/9.3/main/ ADD pg_ident.conf /etc/postgresql/9.3/main/ ADD postgresql.conf /etc/postgresql/9.3/main/ # Create superuser, tempo db and plv8 language ADD create_user.sh / RUN /bin/bash /create_user.sh RUN rm /create_user.sh # Configure for running in container EXPOSE 5432 # VOLUME ["/var/lib/postgresql/9.3/main"] CMD ["/bin/su", "postgres", "-c", "/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf"] create_user.sh is basically service postgresql start; su echo <SQL> | psql; service postgresql stop; This is fine for me to spin up a postgres instance at some known state for development, but it's a bit dissatisfying for managing the data separately from the container. Anyone got any tips for integrating volumes with this?

afandian 13y ago

If there are any Spotify people reading, please please please fix the search/filter regression. You broke it a year-ish ago. It's nice to hear about innovations but a bit galling to paying customers when you break an essential feature and don't fix it. To reproduce: be on an artist page. Now try and find a track with a given name on that page. I dare you. Without reading every single track with your eyes. You used to be able to filter that list. http://community.spotify.com/t5/Spotify-Ideas/0-8-8-Bring-back-the-filter-option-on-album-artist-pages/idi-p/261856 http://community.spotify.com/t5/Spotify-Ideas/0-8-8-Bring-ba...

casca 13y ago

"* Please note Docker is currently under heavy developement. It should not be used in production (yet)." [1] [1] http://www.docker.io/learn_more/ http://www.docker.io/learn_more/

allbutlost 13y ago
This is addressed in the talk. It's a short talk, so may be worth listening to it to see how it was used.
- casca 13y ago
  Agreed, but I'd have thought that it was worth raising for those who are not familiar with where Docker is in the development cycle.

ermintrude 13y ago

They let devs ssh onto production boxes and do what they want?!?! "Things diverge pretty quickly" - I'm sure they do. That sounds like a recipe for disaster... Docker looks promising but until there's a way to allocate maximum resources to a container I wouldn't use it in production. VMs are much slower to start but at least a runaway process on one won't affect other VMs on the same hardware. Docker only lets you hint that only a certain number of cores should be used, so a bad process might monopolise the physical machine. Also until other tools (ansible, salt, chef, etc) help with provisioning you need to edit the dockerfile to change parameters (eg for settings that depend on your environment like smtp end points test vs prod, etc). If these points are addressed I think it'll be awesome though.

danielbln 13y ago
Docker lets you restrict memory and cpu consumption on a per-container basis. Configuration like endpoints are usually fed into the container via environment variables, not by changing stuff around in the Dockerfile. This way you can use the same container in multiple environments.
ldng 13y ago
Aren't cgroups be more then a hitting ? Or do you mean that by default does not setup strict enough cgroups ?
- ermintrude 13y ago
  It seems cpu.shares allow a proportional amount of CPU to be allocated, but LXC may use up to 100% of all idle CPUs. You can't specify a hard-limit of "only use 1GHz when I have a 2GHz CPU" apparently (according to here: http://comments.gmane.org/gmane.linux.kernel.containers.lxc.general/2475 http://comments.gmane.org/gmane.linux.kernel.containers.lxc....). So I guess that could mean that if the containers are using 100% of all CPUs, there wouldn't be resources left e.g. for monitoring on the host? Anyone got any experience with this?
  - deleted 13y ago
    [deleted]

deleted 13y ago

[deleted]

ksec 13y ago

Why Docker? Why not Packer.io? Why No Love for OSv?