4 ms·

The < 2MB figure is with huge pages (2MB ones in this case) enabled. I used 16GB of shared_buffers. The measurement at the end shows how much of a difference in

The < 2MB figure is with huge pages (2MB ones in this case) enabled. I used 16GB of shared_buffers. The measurement at the end shows how much of a difference in memory usage that makes: The page tables use 6.5MB per process without huge pages, 132KB with. So sure, you can have shared buffers set so large that it's more than 2MB per process - but there is a fair bit of headroom to 2MB. So I don't think it's particularly misleading. I think threads are the right call, and that we should move to threads at some point. Partially due to the overhead you mention, but more importantly because there's a lot of features that are much harder with separate processes. We have a fair bit of redundant code due to it: one version working with plain pointers, one with pointers differing between processes, do deal with dynamically allocated shared memory for things like parallel query.

eloff 6y ago
It's a decent ballpark estimate, but it's definitely not a constant factor. People with really huge database deployments (like machines with 4TB of ram) must be crazy if they're not aggressively using pgbouncer to keep the number of connections down. I'm actually quite heartened to hear that the attitude among Postgres developers is changing. Back in the day whenever people would bring up threads vs processes they got shouted down because processes are clearly superior and threads were "just implemented as processes on linux anyway". That's going to be one hell of a job to change now though. To Tom Lane: I told you so :)
- anarazel 6y ago
  > must be crazy if they're not aggressively using pgbouncer to keep the number of connections down. With the limitations around PG poolers that's not always that easy :( I've worked on a number of very large postgres instances - the backend memory usage wasn't usually a major issue for the very large ones (i.e. not schema sharded ones, where it's a large issue). The TLB miss ratio however was a major bottleneck on them, even with huge pages (perhaps even because of huge pages, because the TLB for huge pages used to be so small). > I'm actually quite heartened to hear that the attitude among Postgres developers is changing. Back in the day whenever people would bring up threads vs processes they got shouted down because processes are clearly superior and threads were "just implemented as processes on linux anyway". [...] > > To Tom Lane: I told you so :) The project is bigger than Tom Lane ;) I think it's been pretty clear that threads have more advantages than disadvantages (which are substantial - hello mmap_sem) for quite a while. But that doesn't necessarily mean that we should have changed it a couple years back - as you say, it's not a small change, and it'll cause some disruption. There arguably were (and perhaps are) more crucial issues. > That's going to be one hell of a job to change now though. Yea. But it's doable. Personally I think the PG internal changes not the hardest parts - that's having to deal with all the extension out there. Both to ensure that they are adapted, but also managing the pain of having to deal with all the API evolution.

eloff 6y ago

It's a decent ballpark estimate, but it's definitely not a constant factor. People with really huge database deployments (like machines with 4TB of ram) must be crazy if they're not aggressively using pgbouncer to keep the number of connections down. I'm actually quite heartened to hear that the attitude among Postgres developers is changing. Back in the day whenever people would bring up threads vs processes they got shouted down because processes are clearly superior and threads were "just implemented as processes on linux anyway". That's going to be one hell of a job to change now though. To Tom Lane: I told you so :)

anarazel 6y ago
> must be crazy if they're not aggressively using pgbouncer to keep the number of connections down. With the limitations around PG poolers that's not always that easy :( I've worked on a number of very large postgres instances - the backend memory usage wasn't usually a major issue for the very large ones (i.e. not schema sharded ones, where it's a large issue). The TLB miss ratio however was a major bottleneck on them, even with huge pages (perhaps even because of huge pages, because the TLB for huge pages used to be so small). > I'm actually quite heartened to hear that the attitude among Postgres developers is changing. Back in the day whenever people would bring up threads vs processes they got shouted down because processes are clearly superior and threads were "just implemented as processes on linux anyway". [...] > > To Tom Lane: I told you so :) The project is bigger than Tom Lane ;) I think it's been pretty clear that threads have more advantages than disadvantages (which are substantial - hello mmap_sem) for quite a while. But that doesn't necessarily mean that we should have changed it a couple years back - as you say, it's not a small change, and it'll cause some disruption. There arguably were (and perhaps are) more crucial issues. > That's going to be one hell of a job to change now though. Yea. But it's doable. Personally I think the PG internal changes not the hardest parts - that's having to deal with all the extension out there. Both to ensure that they are adapted, but also managing the pain of having to deal with all the API evolution.