5 ms·

High-process-count support added to master

84 points by tiffanyh 9y ago

lallysingh 9y ago
6 digit PIDs? These aren't actually stored in decimal, right?
- jrbancel 9y ago
  No, but they are constrained to be less than or equal to 999,999. See http://gitweb.dragonflybsd.org/dragonfly.git/blob/586c43085fc900273732f99de6c9ef43f73ded76:/sys/sys/proc.h#l445 http://gitweb.dragonflybsd.org/dragonfly.git/blob/586c43085f...
- jlgaddis 9y ago
  signed int, IIRC (cf. pid_t)
- fanf2 9y ago
  When FreeBSD moved from 16 bit PIDs to 5 digit PIDs back in about 1999, I got the impression that one reason for not using the full 32 bit space was compatibility with tabular formatting in lots of userspace tools.
theandrewbailey 9y ago
Just make sure that your process destruction doesn't involve a lock in kernel space. 900,000 threads waiting for a lock... yikes! https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/ https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and...
tiffanyh 9y ago
xeon126# uptime 1:42PM up 9 mins, 3 users, load averages: 890407.00, 549381.40, 254199.55 Seeing load averages of ~900,000 blows my mind.
- mritun 9y ago
  > Seeing load averages of ~900,000 blows my mind. More impressive is that you can run "uptime" and it actually responds with an output in reasonable amount of time with said 6-digit load-average.
le-mark 9y ago
My question is: why am I paying cloud providers for virtual machines with some imaginary virtual cpu count (or dynos or whatever) when I could be paying for M process running my server executable (capped at N threads per process)? Why can't I just write a server, bundle up the exe and assets, and run it somewhere? Why do I have to futz about with admining, patching, and hardening the OS when that's not what I care about? Edit, to clarify, it just seems like an os with the capability to host a large number of user processes as here would really allow an order of magnitude reduction in hosting cost. Ie if a machine can host 1,000,000 paying accounts vs 10 vps/containered apps.
- tiffanyh 9y ago
  Not to sound cheeky but it sounds like you want to pay for timeshare on a mainframe. (and I agree, it'd actually be nice if cloud providers framed cost based on processes count not vCPUs. Heroku kind of does that with their concept of "dynos")
  - le-mark 9y ago
    Yes, that's exactly what I'd like, with something more modern than cobol of course :)
  - digi_owl 9y ago
    Given the incessant push towards treating the web as a front end for "cloud apps", timeshare is back baby. Only that now the terminal is replaced with the web browser.
- majke 9y ago
  Welcome to the Docker revolution. Historically it was hard to distribute software. You can't just copy over .elf files since these need dynamic libraries. Dependency hell is real, people invented .deb to solve many problems, but debs were always intended to be installed in global scope, making it hard to package user software. Roll forward and nowadays with namespaces you can "containerize" also disk, which means shared libs. Docker images are a better delivery mechanism than raw elf files, or even debs. Hosting docker images is inherently cheaper than virtual machines. I think Heroku were first large service to realize that.
justin66 9y ago
> With the commits made today, master can support at least 900,000 processes with just a kern.maxproc setting in /boot/loader.conf, assuming the machine has the memory to handle it. They are just four bits away from hitting a really big number.

84 points by tiffanyh 9y ago

lallysingh 9y ago

6 digit PIDs? These aren't actually stored in decimal, right?

jrbancel 9y ago
No, but they are constrained to be less than or equal to 999,999. See http://gitweb.dragonflybsd.org/dragonfly.git/blob/586c43085fc900273732f99de6c9ef43f73ded76:/sys/sys/proc.h#l445 http://gitweb.dragonflybsd.org/dragonfly.git/blob/586c43085f...
jlgaddis 9y ago
signed int, IIRC (cf. pid_t)
fanf2 9y ago
When FreeBSD moved from 16 bit PIDs to 5 digit PIDs back in about 1999, I got the impression that one reason for not using the full 32 bit space was compatibility with tabular formatting in lots of userspace tools.

theandrewbailey 9y ago

Just make sure that your process destruction doesn't involve a lock in kernel space. 900,000 threads waiting for a lock... yikes! https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/ https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and...

tiffanyh 9y ago

xeon126# uptime 1:42PM up 9 mins, 3 users, load averages: 890407.00, 549381.40, 254199.55 Seeing load averages of ~900,000 blows my mind.

mritun 9y ago
> Seeing load averages of ~900,000 blows my mind. More impressive is that you can run "uptime" and it actually responds with an output in reasonable amount of time with said 6-digit load-average.

le-mark 9y ago

My question is: why am I paying cloud providers for virtual machines with some imaginary virtual cpu count (or dynos or whatever) when I could be paying for M process running my server executable (capped at N threads per process)? Why can't I just write a server, bundle up the exe and assets, and run it somewhere? Why do I have to futz about with admining, patching, and hardening the OS when that's not what I care about? Edit, to clarify, it just seems like an os with the capability to host a large number of user processes as here would really allow an order of magnitude reduction in hosting cost. Ie if a machine can host 1,000,000 paying accounts vs 10 vps/containered apps.

tiffanyh 9y ago
Not to sound cheeky but it sounds like you want to pay for timeshare on a mainframe. (and I agree, it'd actually be nice if cloud providers framed cost based on processes count not vCPUs. Heroku kind of does that with their concept of "dynos")
- le-mark 9y ago
  Yes, that's exactly what I'd like, with something more modern than cobol of course :)
- digi_owl 9y ago
  Given the incessant push towards treating the web as a front end for "cloud apps", timeshare is back baby. Only that now the terminal is replaced with the web browser.
majke 9y ago
Welcome to the Docker revolution. Historically it was hard to distribute software. You can't just copy over .elf files since these need dynamic libraries. Dependency hell is real, people invented .deb to solve many problems, but debs were always intended to be installed in global scope, making it hard to package user software. Roll forward and nowadays with namespaces you can "containerize" also disk, which means shared libs. Docker images are a better delivery mechanism than raw elf files, or even debs. Hosting docker images is inherently cheaper than virtual machines. I think Heroku were first large service to realize that.

justin66 9y ago

> With the commits made today, master can support at least 900,000 processes with just a kern.maxproc setting in /boot/loader.conf, assuming the machine has the memory to handle it. They are just four bits away from hitting a really big number.