4 ms·

> up to 1024 Curious where this (rather large, yet still seemingly arbitrary) limit comes from.

throw0101a 3y ago
> Curious where this (rather large, yet still seemingly arbitrary) limit comes from. It is Good Enough for now, while keeping various pre-allocated, statically created structures with-in reasonable size limits: > Global and allocated arrays sized by MAXCPU result in excessive bloat on systems with lower core counts. In addition, some code used u_char (8 bits) to hold a CPU index, which is not valid if MAXCPU is greater than 256. > A number of recent commits addressed these sorts of issues, including at least: […] * https://cgit.freebsd.org/src/commit/?id=9051987e40c5 https://cgit.freebsd.org/src/commit/?id=9051987e40c5 See: > The SMP system now supports up to 1024 cores on amd64 and arm64. Many kernel CPU sets are now dynamically allocated to avoid consuming excessive memory. The kernel cpuset ABI has been updated to support the higher limit. 76887e84be97[1] d1639e43c589[2] 9051987e40c5[3] e0c6e8910898[4] (Sponsored by The FreeBSD Foundation) * https://www.freebsd.org/releases/14.0R/relnotes/#kernel-general https://www.freebsd.org/releases/14.0R/relnotes/#kernel-gene...
toast0 3y ago
Gotta have some limit, 4x the current limit of 256 seems reasonableish. Dual socket Epyc 9654 is 96 cores * 2 threads / core * 2 sockets = 384 threads. Intel says their Xeon Platinum 8490H can live on an 8 socket board[1], if you can find one (SuperMicro has one, no price listed [2]; not sure if this is really an 8 socket system, or if it's 4x dual-socket nodes in one chassis?); 60 * 2 * 8 = 960, so that's within the limit, and 8 socket boards are pretty difficult to find. [1] https://ark.intel.com/content/www/us/en/ark/products/231747/intel-xeon-platinum-8490h-processor-112-5m-cache-1-90-ghz.html https://ark.intel.com/content/www/us/en/ark/products/231747/... [2] https://www.supermicro.com/en/products/system/mp/6u/sys-681e-tr https://www.supermicro.com/en/products/system/mp/6u/sys-681e...
- vluft 3y ago
  9754 is 128/256 now, so 256/512 for that. That supermicro system is 8-way; it's 4 dual-socket motherboards but they're one system, hooked together by backplane boards. You can price supermicro's complete-system-only stuff (all of it now, alas) out on thinkmate or similar sites, but a minimal config (and you'd never buy that for a minimal config) hits around $60k.
crest 3y ago
Bitmaps for logical CPU cores and certain lock-free algorithms don't scale well to arbitrary high CPU counts e.g. reclaiming resources once in a while is O(n^2) or worse or the size of the lock structure is linear to the maximum number of cores etc. The relevant parts of the ABI have been future proofed to allow raising the kernel CPU core count limit without breaking the syscall interface for systems with less cores than the existing limit.

> Curious where this (rather large, yet still seemingly arbitrary) limit comes from. It is Good Enough for now, while keeping various pre-allocated, statically created structures with-in reasonable size limits: > Global and allocated arrays sized by MAXCPU result in excessive bloat on systems with lower core counts. In addition, some code used u_char (8 bits) to hold a CPU index, which is not valid if MAXCPU is greater than 256. > A number of recent commits addressed these sorts of issues, including at least: […] * https://cgit.freebsd.org/src/commit/?id=9051987e40c5 https://cgit.freebsd.org/src/commit/?id=9051987e40c5 See: > The SMP system now supports up to 1024 cores on amd64 and arm64. Many kernel CPU sets are now dynamically allocated to avoid consuming excessive memory. The kernel cpuset ABI has been updated to support the higher limit. 76887e84be97[1] d1639e43c589[2] 9051987e40c5[3] e0c6e8910898[4] (Sponsored by The FreeBSD Foundation) * https://www.freebsd.org/releases/14.0R/relnotes/#kernel-general https://www.freebsd.org/releases/14.0R/relnotes/#kernel-gene...

toast0 3y ago

Gotta have some limit, 4x the current limit of 256 seems reasonableish. Dual socket Epyc 9654 is 96 cores * 2 threads / core * 2 sockets = 384 threads. Intel says their Xeon Platinum 8490H can live on an 8 socket board[1], if you can find one (SuperMicro has one, no price listed [2]; not sure if this is really an 8 socket system, or if it's 4x dual-socket nodes in one chassis?); 60 * 2 * 8 = 960, so that's within the limit, and 8 socket boards are pretty difficult to find. [1] https://ark.intel.com/content/www/us/en/ark/products/231747/intel-xeon-platinum-8490h-processor-112-5m-cache-1-90-ghz.html https://ark.intel.com/content/www/us/en/ark/products/231747/... [2] https://www.supermicro.com/en/products/system/mp/6u/sys-681e-tr https://www.supermicro.com/en/products/system/mp/6u/sys-681e...

vluft 3y ago
9754 is 128/256 now, so 256/512 for that. That supermicro system is 8-way; it's 4 dual-socket motherboards but they're one system, hooked together by backplane boards. You can price supermicro's complete-system-only stuff (all of it now, alas) out on thinkmate or similar sites, but a minimal config (and you'd never buy that for a minimal config) hits around $60k.

crest 3y ago

Bitmaps for logical CPU cores and certain lock-free algorithms don't scale well to arbitrary high CPU counts e.g. reclaiming resources once in a while is O(n^2) or worse or the size of the lock structure is linear to the maximum number of cores etc. The relevant parts of the ABI have been future proofed to allow raising the kernel CPU core count limit without breaking the syscall interface for systems with less cores than the existing limit.