/sys/devices/system/cpu/possible is immutable?

[Greg KH]

On Thu, Mar 18, 2021 at 11:10:18PM +0900, Junyeong Jeong wrote:
> Hello everyone :)
> 
> I hope that kernelnewbies mailing list is a suitable place for asking my
> question.
> 
> I wonder that possible-CPU-mask(/sys/devices/system/cpu/possible) can be
> changed after boot in some way or other. I read that it is fixed at boot
> time (https://elixir.bootlin.com/linux/v5.8/source/include/linux/cpumask.h#L50).
> But I am not convinced that it is really immutable even if some cgroup
> or virtualization magic is used.
> 
> Let me account for why I am curious about it.
> Nowadays I am developing BPF library written in rust language.
> In order to call `bpf_lookup_elem()` to get values from
> BPF_MAP_TYPE_PERCPU_ARRAY in userspace, we need to know the correct
> number of per-cpu areas before calling it. Because an out-buffer for
> multiple per-cpu values should be allocated and passed to the
> `bpf_lookup_elem()`. But this process is strongly based on the
> assumption that the number of per-cpu area is always immutable.
> 
> I am referring to /sys/devices/system/cpu/possible file to get to know
> the number of per-cpu areas. I don't know the better way for figuring
> out the number. What I am anxious about is that the number of per-cpu
> areas varies from time to time under some circumstances with cgroup or
> virtualization magic.
> 
> So I checked some cgroup and virtualization ordinary use-cases which did
> not affect the possible-CPU-mask.
> 
> --
> 1.
> $ docker run --cpuset-cpus=0-3 -it ubuntu:20.10 bash
> 
> This does not affect /sys/devices/system/cpu/possible at all. The value
> it contains is the same with the value of the host machine.
> 
> 2.
> $ virsh setvcpus --current ubuntu20.10 5
> 
> Before starting guest OS, the number of maximum vCPU was set to 8 and
> current vCPU was set to 4. While guest OS is running, I changed the
> number of vCPU to 5. And _inside guest OS_, I enabled the new CPU by
> setting /sys/devices/system/cpu/cpu4/online to 1. But
> /sys/devices/system/cpu/possible of guest OS did not change as expected.
> --
> 
> While I was conducting some tests, I realized that it's not possible to
> prove the immutability of possible-CPU-mask using inductive
> method. Because there must be some corner cases that I can never
> imagine.
> 
> 
> Can anyone explain that possible-CPU-mask and the number of per-cpu
> areas never change after boot-time even by cgroup magic or some tricks
> from outside of hypervisors?

That sysfs value itself will not change for the single system while the
kernel is running, but your program could be moved from a system with
one value for that file, to another system with a different value while
it is running without knowing that you were migrated.

But that being said, that file does not show you how many cpus you
actually have access to at that moment in time, it's just a max-value
for that specific kernel build.  CPUs can come and go while your program
is running, so be aware of that.  Your program can also be forbidden to
run on specific cpus if the admin decides to do so.

So be careful here, and use the "normal" api to deal with cpu values and
numbers, never assume that once you start, that the number of cpus your
program has access to will not change.

good luck!

greg k-h