On Mon, 22 Oct 2012, Srivatsa Bhat wrote:

Hi,

You might want to take a look at Documentation/cpu-hotplug.txt for some details regarding online/possible CPUs...

ah, quite so, although that file is definitely out of date. it doesn't mention the command-line parameters "nosmp" or "nr_cpus=" which are defined in kernel/smp.c. more in a bit ... rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ========================================================================

On Tue, 23 Oct 2012, Srivatsa Bhat wrote:

Hi,

On Tue, Oct 23, 2012 at 12:43 AM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:

...

On Mon, 22 Oct 2012, Srivatsa Bhat wrote:

...

Hi,

You might want to take a look at Documentation/cpu-hotplug.txt for some details regarding online/possible CPUs...

ah, quite so, although that file is definitely out of date. it doesn't mention the command-line parameters "nosmp" or "nr_cpus=" which are defined in kernel/smp.c. more in a bit ...

I believe you'll find them in Documentation/kernel-parameters.txt. Of course, it would be good to have them documented or at least mentioned in the above file (cpu-hotplug.txt) as well..

while it's useful, the kernel-parameters.txt file is notoriously behind the times. in this case, it does mention "nosmp" and "nr_cpus" but doesn't mention the X86-only "possible_cpus" parameter. oh, well ... rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ========================================================================

On Mon, 22 Oct 2012, Srivatsa Bhat wrote:

Hi,

You might want to take a look at Documentation/cpu-hotplug.txt for some details regarding online/possible CPUs...

i'll do one more post on this, even though i never imagined it was going to get this complicated. to recap, what i'm trying to understand is this output from dmesg: [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x08] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x09] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0a] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0b] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0c] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0d] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0e] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x0f] disabled) ... snip ... [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 16 CPUs, 8 hotplug CPUs ... snip ... [ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:16 nr_node_ids:1 where my quad-core laptop (which should technically have eight processors) apparently has 16 "possible" CPUs, 8 of which are apparently available for hotplug. first, it seems clear that why the above is happening is that a lot of kernel code that allocates per-CPU variables needs to do that *statically* and can't cope with hotplugged CPUs, so the kernel boot code will automatically add some float to the alleged number of CPUs, just to play it safe. as i read it, then, the kernel code will use the value of "nr_cpu_ids" above equal to 16 on my system, even though only the first 8 correspond to physical processors. so far, so good. from "Documentation/cpu-hotplug.txt": "cpu_possible_mask: Bitmap of possible CPUs that can ever be available in the system. This is used to allocate some boot time memory for per_cpu variables that aren't designed to grow/shrink as CPUs are made available or removed. Once set during boot time discovery phase, the map is static, i.e no bits are added or removed anytime. Trimming it accurately for your system needs upfront can save some boot time memory. See below for how we use heuristics in x86_64 case to keep this under check... "Q: How do we determine how many CPUs are available for hotplug. "A: There is no clear spec defined way from ACPI that can give us that information today. Based on some input from Natalie of Unisys, that the ACPI MADT (Multiple APIC Description Tables) marks those possible CPUs in a system with disabled status. "Andi implemented some simple heuristics that count the number of disabled CPUs in MADT as hotpluggable CPUS. In the case there are no disabled CPUS we assume 1/2 the number of CPUs currently present can be hotplugged." so the above suggests that the number of "possible" CPUs on my system is based on the ACPI values, which will already incorporate some extras just to play it safe. in the file "arch/x86/kernel/smpboot.c", we further read: /* * cpu_possible_mask should be static, it cannot change as cpu's * are onlined, or offlined. The reason is per-cpu data-structures * are allocated by some modules at init time, and dont expect to * do this dynamically on cpu arrival/departure. * cpu_present_mask on the other hand can change dynamically. * In case when cpu_hotplug is not compiled, then we resort to current * behaviour, which is cpu_possible == cpu_present. * - Ashok Raj * * Three ways to find out the number of additional hotplug CPUs: * - If the BIOS specified disabled CPUs in ACPI/mptables use that. * - The user can overwrite it with possible_cpus=NUM * - Otherwise don't reserve additional CPUs. ... snip .... so it looks like the value of 16 comes from the ACPI/mptables. finally, from "drivers/acpi/tables.c", this looks like the code that's called for each possible CPU, that matches the output from "dmesg": void acpi_table_print_madt_entry(struct acpi_subtable_header *header) { if (!header) return; switch (header->type) { case ACPI_MADT_TYPE_LOCAL_APIC: { struct acpi_madt_local_apic *p = (struct acpi_madt_local_apic *)header; printk(KERN_INFO PREFIX "LAPIC (acpi_id[0x%02x] lapic_id[0x%02x] %s)\n", p->processor_id, p->id, (p->lapic_flags & ACPI_MADT_ENABLED) ? "enabled" : "disabled"); } break; and at that point, i think i'm just going to let it go. but that still brings me back to my original question regarding the code for /proc/softirqs in fs/proc/softirqs.c -- why iterate through all *possible* CPUs when some of them will clearly be disabled and not represent actual CPUs? static int show_softirqs(struct seq_file *p, void *v) { int i, j; seq_puts(p, " "); for_each_possible_cpu(i) seq_printf(p, "CPU%-8d", i); seq_putc(p, '\n'); for (i = 0; i < NR_SOFTIRQS; i++) { seq_printf(p, "%12s:", softirq_to_name[i]); for_each_possible_cpu(j) seq_printf(p, " %10u", kstat_softirqs_cpu(i, j)); seq_putc(p, '\n'); } return 0; } anyway, i think i've flogged this enough. rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ========================================================================

On Tue, 23 Oct 2012, Srivatsa Bhat wrote:

I don't think we should worry too much about this.. I can think of a simple reason why one would want to iterate through all possible CPUs.. : IIUC, this function prints, for each softirq, the number of times it ran on a particular CPU. So if a CPU was online for a while and then was taken offline, in order to print the softirq stats properly (including how many times it ran on that CPU when it was online), we will have to iterate through all possible CPUs.. That is a simple and valid reason, IMHO.

that's the same reason i came up with as well, but it still seems somewhat lazy -- how hard would it be to have a bitmap that shows whether a possible CPU was *ever* active/online, and print accordingly? but this inspired another thought. a lot of this complexity is introduced if you configure your kernel with CONFIG_HOTPLUG_CPU, and there are a lot of references to that in the kernel source tree: $ grep -rw CONFIG_HOTPLUG_CPU * | wc -l 332 $ but how many people really need to have a CPU-hotpluggable kernel? it would seem that you could simplify things by just turning off that option. i'm puzzled that the default x86 config files have that option turned on. seriously, how many people out there are hotplugging CPUs? i'm just curious. and is there any obvious drawback to just unconfiguring that option? rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ========================================================================

On Tue, 23 Oct 2012, Srivatsa Bhat wrote:

Ah, that's the thing.. CPU hotplug does not always refer to "physically pull out or plug in CPUs into the system". As you might have gathered from reading the help text, it also refers to "soft" offline/online of CPUs. As a result, CPU hotplug is used in several occasions.. (guess what, you are also using it I bet ;)

of course ... it's early, i wasn't thinking clearly. rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ========================================================================