Userspace starving kernel threads for CPU cycle.
Hi, First time poster here. I have a machine with 80 cores and I had a burst of requesting more cpu period than the available number of CPUs, my guess was that this resulted in starvation for the kernel threads, is my guess plausible or do kernel threads get preference for scheduling. Also pardon my ignorance if my question sounds dumb. Thanks, Abejide Ayodele It always seems impossible until it's done. --Nelson Mandela
On Fri, 08 Jun 2018 16:49:48 -0500, Àbéjídé Àyodélé said:
I have a machine with 80 cores and I had a burst of requesting more cpu period than the available number of CPUs, my guess was that this resulted in starvation for the kernel threads, is my guess plausible or do kernel threads get preference for scheduling.
The short answer: 'man 7 sched' The long answer: The first question is, of course: "Did you see any actual evidence of kernel threads being starved?" In general, CPU-bound execution threads get lower priority than threads that are I/O bound or otherwise bursty in behavior, so kernel threads (which almost never go CPU bound) end up getting scheduled before cpu-bound threads. However, it *is* possible to shoot yourself in the foot by setting artificially high priorities on CPU-bound processes and/or setting processor affinity or fully tickless kernels to end up accidentally starving some kernel threads.
The short answer: 'man 7 sched' Thanks I read this and I think I might still be confused. I am using cgroups and have cpu.cfs_quota_us configured as 2300000 and cpu.cfs_period_us configured as 100000 for 3 different cgroups of, all of these I assume equates 69 cpus along with a couple of other cgroups with cpu.cfs_quota_us configured on a 80 cpu machine which is why I made my original guess. The first question is, of course: "Did you see any actual evidence of kernel threads being starved?" I have a couple of very similar machines with similar workloads and observed the below type of messages in dmesg on several of them: rcu_sched detected stalls on CPUs Sending NMI from CPU 43 to CPUs 14 watchdog: BUG: soft lockup - CPU#26 stuck for 22s [migration/54:335] ixgbe 0000:19:00.1 eno2: initiating reset due to tx timeout Which is why I have this hypothesis. I am still unclear if the cgroup group controller makes guarantees such that tasks in the cgroup cannot be preempted even if a kernel thread requires cpu time. Thanks for your time! Abejide Ayodele It always seems impossible until it's done. --Nelson Mandela
On Fri, 08 Jun 2018 23:05:33 -0500, Àbéjídé Àyodélé said:
rcu_sched detected stalls on CPUs Sending NMI from CPU 43 to CPUs 14 watchdog: BUG: soft lockup - CPU#26 stuck for 22s [migration/54:335] ixgbe 0000:19:00.1 eno2: initiating reset due to tx timeout
The BUG should have caused a stack traceback which would have told us where the CPU was stuck for 22 seconds. That's usually not a starvation issue, it's more often a deadlock (or livelock) condition where the thread on CPU#26 was waiting for something to happen, and it never happened.
participants (2)
-
valdis.kletnieks＠vt.edu -
Àbéjídé Àyodélé