9 ms·

PostgreSQL Performance on Docker

157 points by porker 12y ago

felixgallo 12y ago
I pray to god nobody uses docker and postgres or any other database until this question is resolved. The most rational explanation for why postgres would be 'faster' under docker virtualization than it is raw-on-the-operating-system would be that libcontainer/docker are skipping over some fundamental atomicity guarantee.
- feld 12y ago
  I can't wait for their data to disappear.
  - msane 12y ago
    You sound like a real stand-up guy.
    - steffan 12y ago
      Impatient, too
- deleted 12y ago
  [deleted]
- jfoutz 12y ago
  I had a somewhat snarky answer earlier. I'd say it's much more likely the pre-configured docker digital ocean makes available is on hardware that isn't as heavily used as a regular old postgress configuration. That is, there's more load in general on the postgress machine rather than the docker machine, you don't get perfect isolation in those environments.
  - kerr23 12y ago
    I ran all of the benchmarks on the same machine. I would see it possible that the pre-configured Digital Ocean machine favors docker processes over non-docker somehow.
- cyphar 12y ago
  Docker is just glue code. That's all it is. It sets up your bind mounts, cgroups and namespaces. It then execs the process and gtfos. The way the kernel is dealing with the namespace'd postgres is the difference.
casca 12y ago
This kind of benchmarking should really be run on physical hardware where you know that the underlying resources are not being switched out. The repeated running somewhat mitigates this, but when you don't control the hardware, decisions are being made that are beyond your visibility. The almost 50% more context switches for normal Postgres is very telling. If those had any disk implication which is quite possible with the 512Mb of RAM, it could easily explain the discrepancy.
- antocv 12y ago
  Indeed, doing benchmarking on an environment you dont control - which probably runs other stuff by the side. This benchmark is useless.
deleted 12y ago
[deleted]
themgt 12y ago
If this is the Dockerfile [1] and he's not mounting a volume in, then he's running the Postgres data dir on a btrfs snapshot vs. probably ext4 w/o Docker [1]: https://bitbucket.org/davidkerr/docker/src/c23b2040321b65d38665ede901b1be4edb1318c3/postgresql/9.3/ubuntu/Dockerfile?at=master https://bitbucket.org/davidkerr/docker/src/c23b2040321b65d38...
- kerr23 12y ago
  Mentioned above that i started the container with --volume. The 3 docker conditions where No flags, --volume /.../, --volume /.../ --net=host
shykes 12y ago
EDIT: incredibly, I'm seeing people use this benchmark to argue both 1) that Docker is bad for performance and 2) that Docker is magically faster (and probably does something stupid to be faster). Talk about an illustration of what's wrong with benchmarks! Neither of these statements are correct. It's possible to configure Docker for negligible performance overhead. Docker definitely does not do magic tricks to make things faster. Mostly it sets up the storage backend of your choice and then gets out of the way. Whatever you're seeing is probably a result of your particular host configuration combined with your particular storage backend configuration. My initial comment below is a detailed response to the "Docker is terrible for performance" camp. TLDR: "read the Docker manual before trying to benchmark it". It looks like the guy didn't bother to mark /var/lib/postgres as a volume, which is the recommended way of running a database in Docker. It can be fixed with the following line in his Dockerfile: VOLUME /var/lib/postgresql This will make sure his persistent (and IO-critical) data is bind-mounted from the underlying host filesystem, instead of depending on Docker's copy-on-write backend (which could be btrfs, aufs or lvm/devmapper depending on his configuration). Read the docs, fix your configuration, and try again. Hint: the IO overhead will be negligible. Also: it will end up being a benchmark of your underlying host storage, not of Docker.
- hosay123 12y ago
  So basically, you need to short circuit one of Docker's core features to get decent performance. I like it.
  - wmf 12y ago
    Even ignoring performance, putting persistent data (like databases) on volumes is a best practice since the normal filesystem inside the container is ephemeral.
    - dsissitka 12y ago
      "...since the normal filesystem inside the container is ephemeral." Not by default: # docker run -i -t ubuntu bash root@48dbef418f4c:/# echo foo > /bar root@48dbef418f4c:/# exit # docker restart 48dbef418f4c 48dbef418f4c # docker attach 48dbef418f4c root@48dbef418f4c:/# cat /bar foo root@48dbef418f4c:/# Docker won't delete a container unless you tell it to.
ForHackernews 12y ago
If this is running in a DigitalOcean droplet, isn't it already running inside a virtualization layer? How valid are benchmarks in this circumstance?
- chris_mahan 12y ago
  Could it be that the virtualization layer does something with docker?
  - jpgvm 12y ago
    No. First of all. Docker is -not- virtualisation. It's -containerization-. Effectively namespacing and little else. In Linux namespaces are hierarchal, I can arrange them in a tree. Where at the top namespace A can have 2 sub-namespaces B and C. A can see the process table of B and C, but B and C can only see their respective process tables. This is called pid name spacing. Full containerisation on Linux actually requires a bunch of different namespacing, firstly the pid namespaces above but also uids, network (yes, in Docker your container has a fully namespaced network stack in including it's own (multiple!) route tables and adapters) and also device/filesystem etc. The key take away here is NOTHING is being virtualised, we are simply only showing a section of the system to each process depending on it's position in the namespace hierarchy. What this means is Docker (well really Linux.. Docker is just glue that calls this functionality up) is just.. Linux. Nothing fancy and definitely nothing KVM/Xen/VMWare would be able to influence.
    - ticklemyelmo 12y ago
      > The key take away here is NOTHING is being virtualised The droplet itself is virtualized.
      - jpgvm 12y ago
        Yes I realize this, if you read my comment I was addressing that KVM couldn't have interfered with Docker if it tried.
    - 12y ago
tbrock 12y ago
You are running a benchmark in a virtualized environment with no guarantees about access to the underlying hardware from one moment to the next. It's also the smallest instance type. At least if you had chosen the largest you'd possibly have the entire box. With the smallest instance type however anyone with a larger instance than you is going to steal cycles and I/O away the second they have any load. What makes people think benchmarking on virtualized hardware is at all worthwhile? (Serious question) It's like writing a blog post about which way the wind was blowing for the last five minutes. This is all nonsense.
- kerr23 12y ago
  I ran the benchmark multiple times and I got very consistent results. I know that doesn't eliminate the VM factor, but it certainly minimizes it. Also, this test wasn't meant to be scientific. I'm not selling docker, postgres or digital ocean, just satisfying a personal curiosity. Benchmarks are touchy things in general, I understand that. I published the results and I'd love to see someone publish conflicting results, or verify them. I've tried to be as transparent as possible.
- tszming 12y ago
  It is not nonsense. It is not nonsense because people really run their applications (e.g. PostgreSQL) in virtualized environment these days (i.e. AWS, DO, Linode, Google Cloud etc.) People still need to estimate the performance they will have under these situations - and these benchmarks are relevant to them.
- _pmf_ 12y ago
  > What makes people think benchmarking on virtualized hardware is at all worthwhile? Benchmarking is worthwile. Blogging about any perceived objective results is indeed questionable.
scoopr 12y ago
Just as an uninformed guess, wouldn't docker effectively be just moving the postgres process to another cgroup task group, where it might be scheduled more fairly when the benchmark process is hogging the system?
keypusher 12y ago
Until shown otherwise, I am partial to the idea that this is just a difference between PG on Ubuntu vs PG on CentOS. It's possible docker is simply a red herring, and that there are too many confounding variables for any of these benchmarks to be meaningful. I would be interested to see more rigorous benchmarks of virtualized environments though.
- kerr23 12y ago
  I did a control benchmark between PG Ubuntu vs PG on CentOS, the results are in the post.

157 points by porker 12y ago

I pray to god nobody uses docker and postgres or any other database until this question is resolved. The most rational explanation for why postgres would be 'faster' under docker virtualization than it is raw-on-the-operating-system would be that libcontainer/docker are skipping over some fundamental atomicity guarantee.

feld 12y ago
I can't wait for their data to disappear.
- msane 12y ago
  You sound like a real stand-up guy.
  - steffan 12y ago
    Impatient, too
deleted 12y ago
[deleted]
jfoutz 12y ago
I had a somewhat snarky answer earlier. I'd say it's much more likely the pre-configured docker digital ocean makes available is on hardware that isn't as heavily used as a regular old postgress configuration. That is, there's more load in general on the postgress machine rather than the docker machine, you don't get perfect isolation in those environments.
- kerr23 12y ago
  I ran all of the benchmarks on the same machine. I would see it possible that the pre-configured Digital Ocean machine favors docker processes over non-docker somehow.
cyphar 12y ago
Docker is just glue code. That's all it is. It sets up your bind mounts, cgroups and namespaces. It then execs the process and gtfos. The way the kernel is dealing with the namespace'd postgres is the difference.

casca 12y ago

This kind of benchmarking should really be run on physical hardware where you know that the underlying resources are not being switched out. The repeated running somewhat mitigates this, but when you don't control the hardware, decisions are being made that are beyond your visibility. The almost 50% more context switches for normal Postgres is very telling. If those had any disk implication which is quite possible with the 512Mb of RAM, it could easily explain the discrepancy.

antocv 12y ago
Indeed, doing benchmarking on an environment you dont control - which probably runs other stuff by the side. This benchmark is useless.

deleted 12y ago

[deleted]

themgt 12y ago

If this is the Dockerfile [1] and he's not mounting a volume in, then he's running the Postgres data dir on a btrfs snapshot vs. probably ext4 w/o Docker [1]: https://bitbucket.org/davidkerr/docker/src/c23b2040321b65d38665ede901b1be4edb1318c3/postgresql/9.3/ubuntu/Dockerfile?at=master https://bitbucket.org/davidkerr/docker/src/c23b2040321b65d38...

kerr23 12y ago
Mentioned above that i started the container with --volume. The 3 docker conditions where No flags, --volume /.../, --volume /.../ --net=host

shykes 12y ago

EDIT: incredibly, I'm seeing people use this benchmark to argue both 1) that Docker is bad for performance and 2) that Docker is magically faster (and probably does something stupid to be faster). Talk about an illustration of what's wrong with benchmarks! Neither of these statements are correct. It's possible to configure Docker for negligible performance overhead. Docker definitely does not do magic tricks to make things faster. Mostly it sets up the storage backend of your choice and then gets out of the way. Whatever you're seeing is probably a result of your particular host configuration combined with your particular storage backend configuration. My initial comment below is a detailed response to the "Docker is terrible for performance" camp. TLDR: "read the Docker manual before trying to benchmark it". It looks like the guy didn't bother to mark /var/lib/postgres as a volume, which is the recommended way of running a database in Docker. It can be fixed with the following line in his Dockerfile: VOLUME /var/lib/postgresql This will make sure his persistent (and IO-critical) data is bind-mounted from the underlying host filesystem, instead of depending on Docker's copy-on-write backend (which could be btrfs, aufs or lvm/devmapper depending on his configuration). Read the docs, fix your configuration, and try again. Hint: the IO overhead will be negligible. Also: it will end up being a benchmark of your underlying host storage, not of Docker.

hosay123 12y ago
So basically, you need to short circuit one of Docker's core features to get decent performance. I like it.
- wmf 12y ago
  Even ignoring performance, putting persistent data (like databases) on volumes is a best practice since the normal filesystem inside the container is ephemeral.
  - dsissitka 12y ago
    "...since the normal filesystem inside the container is ephemeral." Not by default: # docker run -i -t ubuntu bash root@48dbef418f4c:/# echo foo > /bar root@48dbef418f4c:/# exit # docker restart 48dbef418f4c 48dbef418f4c # docker attach 48dbef418f4c root@48dbef418f4c:/# cat /bar foo root@48dbef418f4c:/# Docker won't delete a container unless you tell it to.

ForHackernews 12y ago

If this is running in a DigitalOcean droplet, isn't it already running inside a virtualization layer? How valid are benchmarks in this circumstance?

chris_mahan 12y ago
Could it be that the virtualization layer does something with docker?
- jpgvm 12y ago
  No. First of all. Docker is -not- virtualisation. It's -containerization-. Effectively namespacing and little else. In Linux namespaces are hierarchal, I can arrange them in a tree. Where at the top namespace A can have 2 sub-namespaces B and C. A can see the process table of B and C, but B and C can only see their respective process tables. This is called pid name spacing. Full containerisation on Linux actually requires a bunch of different namespacing, firstly the pid namespaces above but also uids, network (yes, in Docker your container has a fully namespaced network stack in including it's own (multiple!) route tables and adapters) and also device/filesystem etc. The key take away here is NOTHING is being virtualised, we are simply only showing a section of the system to each process depending on it's position in the namespace hierarchy. What this means is Docker (well really Linux.. Docker is just glue that calls this functionality up) is just.. Linux. Nothing fancy and definitely nothing KVM/Xen/VMWare would be able to influence.
  - ticklemyelmo 12y ago
    > The key take away here is NOTHING is being virtualised The droplet itself is virtualized.
    - jpgvm 12y ago
      Yes I realize this, if you read my comment I was addressing that KVM couldn't have interfered with Docker if it tried.
  - 12y ago

tbrock 12y ago

You are running a benchmark in a virtualized environment with no guarantees about access to the underlying hardware from one moment to the next. It's also the smallest instance type. At least if you had chosen the largest you'd possibly have the entire box. With the smallest instance type however anyone with a larger instance than you is going to steal cycles and I/O away the second they have any load. What makes people think benchmarking on virtualized hardware is at all worthwhile? (Serious question) It's like writing a blog post about which way the wind was blowing for the last five minutes. This is all nonsense.

kerr23 12y ago
I ran the benchmark multiple times and I got very consistent results. I know that doesn't eliminate the VM factor, but it certainly minimizes it. Also, this test wasn't meant to be scientific. I'm not selling docker, postgres or digital ocean, just satisfying a personal curiosity. Benchmarks are touchy things in general, I understand that. I published the results and I'd love to see someone publish conflicting results, or verify them. I've tried to be as transparent as possible.
tszming 12y ago
It is not nonsense. It is not nonsense because people really run their applications (e.g. PostgreSQL) in virtualized environment these days (i.e. AWS, DO, Linode, Google Cloud etc.) People still need to estimate the performance they will have under these situations - and these benchmarks are relevant to them.
_pmf_ 12y ago
> What makes people think benchmarking on virtualized hardware is at all worthwhile? Benchmarking is worthwile. Blogging about any perceived objective results is indeed questionable.

scoopr 12y ago

Just as an uninformed guess, wouldn't docker effectively be just moving the postgres process to another cgroup task group, where it might be scheduled more fairly when the benchmark process is hogging the system?

keypusher 12y ago

Until shown otherwise, I am partial to the idea that this is just a difference between PG on Ubuntu vs PG on CentOS. It's possible docker is simply a red herring, and that there are too many confounding variables for any of these benchmarks to be meaningful. I would be interested to see more rigorous benchmarks of virtualized environments though.

kerr23 12y ago
I did a control benchmark between PG Ubuntu vs PG on CentOS, the results are in the post.