[sparc64] number of processors in a LDOM

[Jan Engelhardt]

On Wednesday 2018-04-04 15:30, Anatoly Pugachev wrote:
>
>Can someone tell me or suggest why does getconf returns total available to a physical machine
>cpu count, and not LDOM allocated processor/vcpu count ?
>
>ttip$ getconf -a | grep PROCESSORS
>_NPROCESSORS_CONF                  256
>_NPROCESSORS_ONLN                  16

It's how the hypervisor populates the MDESC info passed to domains.
It even does that for control domains. Almost makes one want to believe
the 6-core T1s could be functional 8-core ones. (Who knows.)


jengelh at a1:~$ lscpu
Architecture:        sparc64 (T1000)
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Big Endian
CPU(s):              24
On-line CPU(s) list: 0-23

Anyhow, by declaring nr_cpus_present=256 but only nr_cpus_online=16, domains can
get more vCPUs assigned at a later time without rebooting the OS, if one
assumes that the MDESC is meant to be static during runtime and/or that the OS
cannot handle a change of nr_cpus_present (there are possibly some allocations
of exactly this size all over the place). Similar to NR_CPUS.



>I'm raising this issue, because some userspace tools use nproc to run parallel make for
>example. And starting from 4.15+ (but not on 4.14) kernel overcommited CPU usage (for example,
>using make -j256 on a LDOM with 16 vcpus allocated) gets me to the following (reproducible):

I consider defaulting to nr_cpus_present totally wrong for any software.
CPUs can be offlined by the administrator at will.

Second, I would also consider defaulting to on nr_cpus_online to be a subpar
default. Some institutions (or the load balancing system in use there) use
sched_setaffinity as a primitive measure to statically ensure that login shells
never run on the same core as a number cruncher task.
`make -j$(getconf _NPROCESSORS_ONLN)` would be dumb there because only e.g. 2 cores
are available for shells/gcc - the other 14 are assigned to HPC tasks.