CPU Load Balancer / Scheduler / CFS

[Robert W]

I'm seeing some bizarre differences between two servers (one running
2.6.32 and the other 2.6.18) with how processes are spread across
CPUs.

I created a simple test client: https://gist.github.com/2306761
which forks 4 times to create processes which just consume cpu.

On 2.6.18, the processes are distributed across CPUs as expected (all
4 processes are consuming 100%).
But on 2.6.32, the distribution varies. Sometimes it's (100%, 33%,
33%, 33%), other times it's (100%, 100%, 50%, 50%), etc.. while all
the other CPUs sit completely idle..

Both servers have many cores (more than 4). Both servers each have 2
physical processors, and are hyper-threaded.

I've done many other similar tests (tried running the "stress" program
with different parameters and I also got bizarre results. It basically
looks like the CPU Load Balancer never moves a process to a different
CPU under any circumstances). I haven't set anything regarding cpusets
/ etc. so it should all be default behavior. Also, if I manually move
the processes to other cpus (using taskset) then I can get them to all
run at 100%.

What could be going on here?