3 ms·
So, last time this was posted I commented that I was interested, but couldn't use this at work because of the GPL license. Since then I implemented my own M:N
by loeg 15y ago
So, last time this was posted I commented that I was interested, but couldn't use this at work because of the GPL license.
Since then I implemented my own M:N userspace:kernel threading library (which is what lthread boils down to) based on Russ Cox's BSD-licensed libtask. You know what I found?
1) Many libc routines require a surprisingly large amount of stack. 64kiB was the smallest power of 2 I could find where I wouldn't overflow the stack somewhere in a libc call. (This isn't a problem with stack-copying, but it is a problem if you have dedicated per-lthread stacks).
2) It was slow! Just dropping the M:N library and using blocking network I/O with 200+ kernel pthreads was vastly faster.
3) Many pthread-functions get very, very confused if they are used from different pthreads (same logical "lthread"). (This happens when a userspace scheduler swap happens between matching operations.) For example, pthread mutexes don't like to be locked in one pthread and unlocked in another pthread (same "lthread"). I ran into other things that match this category, but can't remember them off of the top of my head.
I ended up scrapping it and just using a pool of kernel pthreads. It works, the code is pretty clear (blocking IO! whatever, man), and it's fast enough.
Edit: By "fast enough", I mean "can fill 2x 1Gbit pipes from disk" (without any perf-focused work thus far).
- ralph 15y agoOut of interest, what implementation of libc was that?
- loeg 15y agoA hacked up FreeBSD libc (we make a FreeBSD-derived operating system). Some of the authentication/credentials code, for example, creates a bunch of large stack frames.
- loeg 15y agoSince I can't edit this post anymore, here's the relevant code: https://github.com/cemeyer/taskmn https://github.com/cemeyer/taskmn (Buyer beware: this is basically a snapshot of it at the point we decided to drop it entirely; I poked it enough to get it to compile under GCC / linux, but haven't verified functionality at all.) More edits: 4) Re: Stack-copying; I found that without any attempt at optimizing memory use, my "light" threads were consuming on the order of ~600 bytes of stack (when de-scheduled); with a stack copying approach and a run-time stack of 128kiB, calling large-stack-using libc functions was fine. 5) Re: per-task dedicated stack memory; the default amount of memory allocated per-pthread on our platform is 128kiB, so 64kiB isn't great savings. I detected stack corruption at 32kiB and below by setting a canary value (at the top of the allocated stack) in the scheduler before running a ready "lthread."
- beagle3 15y agoThanks for sharing your code and experience. 600 bytes per de-scheduled thread sounds excellent; It makes a million mostly-dormant threads feasible in less than a gig of memory -- I think that's the use case lthread-style threads shine in. You wouldn't be able to do that with blocking kernel threads (you'd need 64gig of ram just for stack, and that assumes the kernel scales well enough to handle that). I think overall lower efficiency per thread, even if it's 50% slower, is acceptable in such a use case, given the memory requirements reduction.
- loeg 15y agoNo problem :-). Re: "lower efficiency… is acceptable in such a use case": Maybe — it really depends on your use case. In the case you describe, sure. In my particular (specialized) case: 1) My clients are connected by 1Gb or 10Gb switched ethernet, and they are typically on firewalled local networks. I don't worry about trickle-style DoS attacks. (In other words: my threads are mostly-active, not mostly-dormant.) 2) My clients really only care about sustained streaming throughput. So, if they can max out the underlying disk with 20-30 simultaneous connections, they won't bother throwing 1000s of connections at my server. pthread-per-connection works great for this circumstance (with a pre-allocated thread pool). Use the right tool for the job ;-).
- tomp 15y agoSince you can consider libc routines as blocking code, you could switch back to the C stack (of the original pthread), so you would need only one 64kB stack per pthread.