11 ms·

> I'm honestly surprised it took them so long to reach this conclusion. I'm not. You can get a long way with conventional IPC, and OS processes provide a lot

> I'm honestly surprised it took them so long to reach this conclusion. I'm not. You can get a long way with conventional IPC, and OS processes provide a lot of value. For most PostgreSQL instances the TLB flush penalty is at least 3rd or 4th on the list of performance concerns, far below prevailing storage and network bottlenecks. I share the concerns cited in this LWN story. Reworking this massive code base around multithreading carries a large amount of risk. PostgreSQL developers will have to level up substantially to pull it off. A PostgreSQL endorsed "second-system" with the (likely impossible, but close enough that it wouldn't matter) goal of 100% client compatibility could be a better approach. Adopting a memory safe language would make this both tractable and attractive (to both developers and users.) The home truth is that any "new process model" effort would actually play out exactly this way, so why not be deliberate about it?

atonse 3y ago
Would this basically be a new front end? Like the part that handles sockets and input? Or more if a rewrite of subsystems? Like the query planner or storage engine etc?
- topspin 3y ago
  Both, I'd imagine. With regard to client compatibility there are related precedents for this already; the PostgreSQL wire protocol has emerged as a de facto standard. Cockroachdb and ClickHouse are two examples that come to mind.
nextaccountic 3y ago
From what I gather postgres isn't doing conventional IPC but instead it uses shared memory, which means the same mechanism threads use but with way higher complexity
- topspin 3y ago
  As does Oracle, and others. I'm aware. IPC, to me, includes the conventional shared memory resources (memory segments, locks, semaphores, condition variable, etc.) used by these systems: resources acquired by processes for the purpose of communication with other processes. I get it though. The most general concept of shared memory is not coupled to an OS "process." You made me question whether my concept of term IPC was valid, however. So what does one do when a question appears? Stop thinking immediately and consult a language model! Q: Is shared memory considered a form of interprocess communication? GPT-4: Yes, shared memory is indeed considered a form of interprocess communication (IPC). It's one of the several mechanisms provided by an operating system to allow processes to share and exchange data. ... Why does citing ChatGPT make me feel so ugly inside?
  - TeMPOraL 3y ago
    I always understood IPC, "interprocess communication", in general sense, as anything and everything that can be used by processes to communicate with each other - of course with a narrowing provision that common use of the term refers to those means that are typically used for that purpose, are relatively efficient, and the process in question run on the same machine. In that view, I always saw shared memory as IPC, in that it is a tool commonly used to exchange data between processes, but of course it is not strictly tied to any process in particular. This is similar to files, which if you squint are a form of IPC too, and are also not tied to any specific process. > Why does citing ChatGPT make me feel so ugly inside? That's probably because, in cases like this, it's not much different to stating it yourself, but is more noisy.
  - faangsticle 3y ago
    > Why does citing ChatGPT make me feel so ugly inside? Its the modern let me Google that for you. Just like people don't care what the #1 result on Google is, they also don't care what ChatGPT has to say about it. If they did, they'd ask it themselves.
  - pritambarhate 3y ago
    Without a credible source to reconfirm what ChatGPT said, one can’t really assume what ChatGPT says is correct.
- mgaunard 3y ago
  What do you think IPC is?
- wbl 3y ago
  Not necessarily. Man 3 shmem if you want a journey back to some bad ideas.
gmokki 3y ago
Would something like opt-in sharing of pages between processes that oracle has been trying to get into kernel be the correct option: https://lwn.net/ml/linux-kernel/cover.1682453344.git.khalid.aziz@oracle.com/ https://lwn.net/ml/linux-kernel/cover.1682453344.git.khalid.... Postmaster would just share the already shared memory between processes (containing also the locks). That explicit part of memory would opt-in to thread -like sharing and thus get faster/less tlb switching and lower memory usage. While all the rest of the state would still be per-process and safe. tl;dr super share the existing shared memory area with kernel patch All operating systems not supporting it would keep working as is.
- topspin 3y ago
  Yes, it would mitigate the TLB problem. Interesting that Oracle is also looking to solve this problem, but not by multithreading the Oracle RDBMS.

atonse 3y ago

Would this basically be a new front end? Like the part that handles sockets and input? Or more if a rewrite of subsystems? Like the query planner or storage engine etc?

topspin 3y ago
Both, I'd imagine. With regard to client compatibility there are related precedents for this already; the PostgreSQL wire protocol has emerged as a de facto standard. Cockroachdb and ClickHouse are two examples that come to mind.

nextaccountic 3y ago

From what I gather postgres isn't doing conventional IPC but instead it uses shared memory, which means the same mechanism threads use but with way higher complexity

topspin 3y ago
As does Oracle, and others. I'm aware. IPC, to me, includes the conventional shared memory resources (memory segments, locks, semaphores, condition variable, etc.) used by these systems: resources acquired by processes for the purpose of communication with other processes. I get it though. The most general concept of shared memory is not coupled to an OS "process." You made me question whether my concept of term IPC was valid, however. So what does one do when a question appears? Stop thinking immediately and consult a language model! Q: Is shared memory considered a form of interprocess communication? GPT-4: Yes, shared memory is indeed considered a form of interprocess communication (IPC). It's one of the several mechanisms provided by an operating system to allow processes to share and exchange data. ... Why does citing ChatGPT make me feel so ugly inside?
- TeMPOraL 3y ago
  I always understood IPC, "interprocess communication", in general sense, as anything and everything that can be used by processes to communicate with each other - of course with a narrowing provision that common use of the term refers to those means that are typically used for that purpose, are relatively efficient, and the process in question run on the same machine. In that view, I always saw shared memory as IPC, in that it is a tool commonly used to exchange data between processes, but of course it is not strictly tied to any process in particular. This is similar to files, which if you squint are a form of IPC too, and are also not tied to any specific process. > Why does citing ChatGPT make me feel so ugly inside? That's probably because, in cases like this, it's not much different to stating it yourself, but is more noisy.
- faangsticle 3y ago
  > Why does citing ChatGPT make me feel so ugly inside? Its the modern let me Google that for you. Just like people don't care what the #1 result on Google is, they also don't care what ChatGPT has to say about it. If they did, they'd ask it themselves.
- pritambarhate 3y ago
  Without a credible source to reconfirm what ChatGPT said, one can’t really assume what ChatGPT says is correct.
mgaunard 3y ago
What do you think IPC is?
wbl 3y ago
Not necessarily. Man 3 shmem if you want a journey back to some bad ideas.

gmokki 3y ago

Would something like opt-in sharing of pages between processes that oracle has been trying to get into kernel be the correct option: https://lwn.net/ml/linux-kernel/cover.1682453344.git.khalid.aziz@oracle.com/ https://lwn.net/ml/linux-kernel/cover.1682453344.git.khalid.... Postmaster would just share the already shared memory between processes (containing also the locks). That explicit part of memory would opt-in to thread -like sharing and thus get faster/less tlb switching and lower memory usage. While all the rest of the state would still be per-process and safe. tl;dr super share the existing shared memory area with kernel patch All operating systems not supporting it would keep working as is.

topspin 3y ago
Yes, it would mitigate the TLB problem. Interesting that Oracle is also looking to solve this problem, but not by multithreading the Oracle RDBMS.