Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running
Hi. We are trying to build realtime(-ish) system based on rhel6(kernel 2.6.32-642.1.1.el6.x86_64). We used isolcpus to remove some cpus from process scheduling(isolcpus=2-19 nohz_full=2-19 rcu_nocbs=2-19). We spin off a program thread that set's its cpu affinity to one of those isolated cpus, sets its scheduling chass to SCHED_FIFO, spins in a loop and never sched_yield()-s to the kernel. We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are NEVER interrupted. We are observing that the program thread is indeed realtime and is never interrupted. After some time working like this, the system becomes irresponsive - ssh connections start failing with timeout, existing connections hang when trying to read/write to physical disks(reading procfs or writing to tmpfs is unaffected). *** According to /proc/sched_debug, it seems that kernel still schedules some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for example cpu 18 get tasks events/18 and kblockd/18 that are stuck in runnable(but not running state), so those kernel processes never got a single time slice because our realtime process hogs 100% of cpu. And kernel/18 and kblockd/18 never migrate to other cpus because these tasks are pinned to cpu 18. *** Check please these /proc/sched_debug snapshots, you can see that events/18 and kblockd/18 sum-exec counters are not increasing: https://gist.github.com/altmind/5cf4aad87a4a082441c1ca9378a06154 The question is: Is it possible to never schedule kernel tasks on selected cpus? -- Thanks, Andrei Hurynovich
On Wed, 28 Jun 2017 08:39:07 -0500, Andrei Hurynovich said:
We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are NEVER interrupted.
According to /proc/sched_debug, it seems that kernel still schedules some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for example cpu 18 get tasks events/18 and kblockd/18 that are stuck in runnable(but not running state), so those kernel processes never got a single time slice because our realtime process hogs 100% of cpu.
This is what happens when you have a priority inversion - when you tell the system to give 100% to a process, you shouldn't be surprised when other tasks don't get any service.
The question is: Is it possible to never schedule kernel tasks on selected cpus?
Only if the userspace process on that CPU never makes system calls - which is very unlikely if the process has actual real-time requirements. Also, if your "real-time" process is taking 100% of the CPU, you have a disaster waiting to happen. You have zero headroom for dealing with unexpected events. Thought experiment: What happens if your real-world part of the system has an unexpected error, that requires 1% of a CPU for error recovery? You are forced to either ignore the error or miss a real-time deadline. You might want to think about dividing up your process into 2 parts - one that handles the *actual* real-time work and only uses (for example) 20-30% of a CPU, and the parts that don't have actual real-time constraints that can then run with the rest of the available CPU, but allow other threads (such as kernel) to execute as well.
Thank you Valdis. Yes, I'm basically getting what I want - the RT proc never ever gives up to the system. There are a plenty of cores left to run non-rt tasks on the machine. The question is why this old 2.6 kernel decide that it needs per-cpu events and kblockd tasks. Maybe someone can give a hint in what subsystem's documentation can I find anything about workqueue tasks. /Documentation/kernel-per-CPU-kthreads.txt is great, but described controls appeared only in 3.10.x :) On 06/28/2017 01:04 PM, valdis.kletnieks@vt.edu wrote:
On Wed, 28 Jun 2017 08:39:07 -0500, Andrei Hurynovich said:
We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are NEVER interrupted. According to /proc/sched_debug, it seems that kernel still schedules some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for example cpu 18 get tasks events/18 and kblockd/18 that are stuck in runnable(but not running state), so those kernel processes never got a single time slice because our realtime process hogs 100% of cpu. This is what happens when you have a priority inversion - when you tell the system to give 100% to a process, you shouldn't be surprised when other tasks don't get any service.
The question is: Is it possible to never schedule kernel tasks on selected cpus? Only if the userspace process on that CPU never makes system calls - which is very unlikely if the process has actual real-time requirements.
Also, if your "real-time" process is taking 100% of the CPU, you have a disaster waiting to happen. You have zero headroom for dealing with unexpected events. Thought experiment: What happens if your real-world part of the system has an unexpected error, that requires 1% of a CPU for error recovery? You are forced to either ignore the error or miss a real-time deadline.
You might want to think about dividing up your process into 2 parts - one that handles the *actual* real-time work and only uses (for example) 20-30% of a CPU, and the parts that don't have actual real-time constraints that can then run with the rest of the available CPU, but allow other threads (such as kernel) to execute as well.
-- Thanks, Andrei Hurynovich Charlesworth Research LLC. http://www.charlesworthresearch.com/
On Wed, 28 Jun 2017 14:02:37 -0500, Andrei Hurynovich said:
The question is why this old 2.6 kernel decide that it needs per-cpu events and kblockd tasks.
You have per-cpu events ecause your real-time process issues syscalls, and syscalls do things inside the kernel that require per-CPU infrastructure. You have kblockd and other per-cpu threads because you're using an old ancient kernel that doesn't have Frederic Weisbecker's CONFIG_NO_HZ_FULL support, or the support for a single system-wide kblockd, or a mess of other stuff. So it isn't that it "decides" to do it per-cpu, it's because 2.6.32 had it done that way for simplicity, and in the *8 years* since 2.6.32 was released, people cleaned much of that stuff up. [/usr/src/linux-next] git diff --shortstat v2.6.32 next-20170627 64638 files changed, 17302278 insertions(+), 5102365 deletions(-) Yes, you're 17 *million* lines of code behind the times. Much of what you are complaining about was solved half a decade or more ago. Or as Greg suggested, use a modern kernel. :)
On Wed, Jun 28, 2017 at 08:39:07AM -0500, Andrei Hurynovich wrote:
Hi.
We are trying to build realtime(-ish) system based on rhel6(kernel 2.6.32-642.1.1.el6.x86_64).
Wow, you do realize that is a _very_ old and obsolete kernel, supported by no one except Red Hat. If you stick with it, you are going to have to get your support from them, not the community, as you are already paying for it :) Good luck! greg k-h
On Wed, 2017-06-28 at 08:39 -0500, Andrei Hurynovich wrote:
Hi.
We are trying to build realtime(-ish) system based on rhel6(kernel 2.6.32-642.1.1.el6.x86_64).
We used isolcpus to remove some cpus from process scheduling(isolcpus=2-19 nohz_full=2-19 rcu_nocbs=2-19).
We spin off a program thread that set's its cpu affinity to one of those isolated cpus, sets its scheduling chass to SCHED_FIFO, spins in a loop and never sched_yield()-s to the kernel.
We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are NEVER interrupted.
You want an actual realtime kernel for that to work right. The real time kernel currently supported by Red Hat is 3.10 based, not 2.6.32 based. -- All rights reversed
participants (4)
-
Andrei Hurynovich -
Greg KH -
Rik van Riel -
valdis.kletnieks＠vt.edu