4 ms·

It would be interesting to have something between threads and processes. I'll call them heavy-threads for sake of discussion. Like light-threads, heavy-threads

by mastax 3y ago

It would be interesting to have something between threads and processes. I'll call them heavy-threads for sake of discussion. Like light-threads, heavy-threads would share the same process-security-boundary and therefore switching between them would be cheap. No need to flush TLB, I$, D$. Like processes, heavy-threads would have mostly-separate address spaces by default. Similar to forking a process, they could share read-only mappings for shared libraries, code, COW global variables, and explicitly defined shared writable memory regions. Like processes, heavy-threads would isolate failure states. A C++ exception, UNIX signal, segfault, etc. would kill only the heavy-thread responsible.

dasyatidprime 3y ago
So what would be different between those and forked processes?
ShroudedNight 3y ago
I've been pondering / ruminating with this too; I've been somewhat surprised that few operating systems have played with reserving per-thread address space as thread-local storage, or requiring something akin to a 'far' pointer to access commonly-addressed shared memory.
wbl 3y ago
You cannot COW and share the TLB state. The caches aren't flushed in process changes either: it's that the data is different so evictions happen.
mattashii 3y ago
> No need to flush TLB TLB isn't "flushed" so much as it is useless across different memory address spaces. Switching processes means switching address spaces, which means you have to switch the contents of the TLB to the new process' TLB entries, which eventually indeed flushes the TLB, but that is only over time, not necessarily the moment you switch processes. > Like processes, heavy-threads would have mostly-separate address spaces by default. This thus conflicts with the need to not flush TLBs. You can't not change TLB contents across address spaces.
mike_hearn 3y ago
There are some problems. 1. Mostly separate address spaces requires changing the TLB on context switch (modern hw lets it be partial). You could use MPKs to share a single address space with fast protection switches. 2. Threads share the global heap, but your heavy threads would require explicitly defined shared writeable memory regions, so presumably each one has its own heap. That's a fair bit of overhead. 3. Failure isolation is more complicated than deciding what to kill. The expand on the last point, Postgres doesn't isolate failures to a single process because they do share memory and might corrupt those shared memory regions. But even if you don't have shared memory failure recovery isn't always easy. Software has to be written specifically to plan for it. You can kill processes because everything in the OS is written around allowing for that possibility, for example, shells know what to do if a sub-process is killed unexpectedly. Killing a heavy thread (=process) is no good if the parent process is going to wait for a reply from it forever because it wasn't written to handle the process going away.
anarazel 3y ago
> Like light-threads, heavy-threads would share the same process-security-boundary and therefore switching between them would be cheap. No need to flush TLB, I$, D$. > Like processes, heavy-threads would have mostly-separate address spaces by default. Similar to forking a process, they could share read-only mappings for shared libraries, code, COW global variables, and explicitly defined shared writable memory regions. I don't think you realistically can have separate address spaces and not have TLB etc impact. If they're separate address spaces, you need separate TLB entries => lower TLB hit ratio.

by mastax 3y ago

dasyatidprime 3y ago

So what would be different between those and forked processes?

ShroudedNight 3y ago

I've been pondering / ruminating with this too; I've been somewhat surprised that few operating systems have played with reserving per-thread address space as thread-local storage, or requiring something akin to a 'far' pointer to access commonly-addressed shared memory.

wbl 3y ago

You cannot COW and share the TLB state. The caches aren't flushed in process changes either: it's that the data is different so evictions happen.

mattashii 3y ago

> No need to flush TLB TLB isn't "flushed" so much as it is useless across different memory address spaces. Switching processes means switching address spaces, which means you have to switch the contents of the TLB to the new process' TLB entries, which eventually indeed flushes the TLB, but that is only over time, not necessarily the moment you switch processes. > Like processes, heavy-threads would have mostly-separate address spaces by default. This thus conflicts with the need to not flush TLBs. You can't not change TLB contents across address spaces.

mike_hearn 3y ago

There are some problems. 1. Mostly separate address spaces requires changing the TLB on context switch (modern hw lets it be partial). You could use MPKs to share a single address space with fast protection switches. 2. Threads share the global heap, but your heavy threads would require explicitly defined shared writeable memory regions, so presumably each one has its own heap. That's a fair bit of overhead. 3. Failure isolation is more complicated than deciding what to kill. The expand on the last point, Postgres doesn't isolate failures to a single process because they do share memory and might corrupt those shared memory regions. But even if you don't have shared memory failure recovery isn't always easy. Software has to be written specifically to plan for it. You can kill processes because everything in the OS is written around allowing for that possibility, for example, shells know what to do if a sub-process is killed unexpectedly. Killing a heavy thread (=process) is no good if the parent process is going to wait for a reply from it forever because it wasn't written to handle the process going away.

anarazel 3y ago

> Like light-threads, heavy-threads would share the same process-security-boundary and therefore switching between them would be cheap. No need to flush TLB, I$, D$. > Like processes, heavy-threads would have mostly-separate address spaces by default. Similar to forking a process, they could share read-only mappings for shared libraries, code, COW global variables, and explicitly defined shared writable memory regions. I don't think you realistically can have separate address spaces and not have TLB etc impact. If they're separate address spaces, you need separate TLB entries => lower TLB hit ratio.