Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

[Andrei Hurynovich]

Hi.

We are trying to build realtime(-ish) system based on rhel6(kernel 
2.6.32-642.1.1.el6.x86_64).

We used isolcpus to remove some cpus from process 
scheduling(isolcpus=2-19 nohz_full=2-19 rcu_nocbs=2-19).

We spin off a program thread that set's its cpu affinity to one of those 
isolated cpus, sets its scheduling chass to SCHED_FIFO, spins in a loop 
and never sched_yield()-s to the kernel.

We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are 
NEVER interrupted.

We are observing that the program thread is indeed realtime and is never 
interrupted.

After some time working like this, the system becomes irresponsive - ssh 
connections start failing with timeout, existing connections hang when 
trying to read/write to physical disks(reading procfs or writing to 
tmpfs is unaffected).

***
According to /proc/sched_debug, it seems that kernel still schedules 
some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for 
example cpu 18 get tasks events/18 and kblockd/18 that are stuck in 
runnable(but not running state), so those kernel processes never got a 
single time slice because our realtime process hogs 100% of cpu.

And kernel/18 and kblockd/18 never migrate to other cpus because these 
tasks are pinned to cpu 18.
***

Check please these /proc/sched_debug snapshots, you can see that 
events/18 and kblockd/18 sum-exec counters are not increasing:
https://gist.github.com/altmind/5cf4aad87a4a082441c1ca9378a06154




The question is: Is it possible to never schedule kernel tasks on 
selected cpus?

-- 
Thanks,
Andrei Hurynovich