mount command stuck at synchronize_rcu on a RT patched kernel.
Hello All, We are facing an issue with 3.10.65 kernel + RT69 patch on a 12core xeon. We are running real time applications (higher priority than kernel threads, requiring the resolution of 1ms or less) on ioslated CPU's which are configured through kernel boot arguments as indicated below. Also, we have offloaded the RCU call backs to other book keeping CPU cores. But when we try to perform some general commands like mount some directory over NFS, the command hangs and never completes till we stop our real time applications. The moment real time applications are stopped, command finishes immediately. When we see the backtrace of the mount command, it waiting for "synchronize_rcu". Since we have already offloaded RCU call backs, we expect nothing to be done on isolcpus except rcuc process on isolcpus waking the offloaded threads on other cores. Can someone help us understand what kernel parameters/config needs to be modified/tuned to achieve the proper behavior of the system ? Thanks in advance. Backtrace: ---------- [root@localhost ~]# ps -eaf | grep mount root 17915 4532 0 19:13 pts/0 00:00:00 umount /bng4 root 17916 17915 0 19:13 pts/0 00:00:00 /sbin/umount.nfs /bng4 root 17962 17945 0 19:14 pts/2 00:00:00 grep --color=auto mount [root@localhost ~]# cat /proc/17916/stack [<ffffffff810612b4>] wait_rcu_gp+0x44/0x50 [<ffffffff810c6ec9>] synchronize_rcu+0x29/0x40 [<ffffffff8126446a>] nfs_free_server+0xea/0x180 [<ffffffff8126e49e>] nfs_kill_super+0x1e/0x30 [<ffffffff8115a4ed>] deactivate_locked_super+0x3d/0x60 [<ffffffff8115ab16>] deactivate_super+0x46/0x60 [<ffffffff811770ef>] mntput_no_expire+0xef/0x150 [<ffffffff81178376>] SyS_umount+0x96/0x3d0 [<ffffffff818cbb46>] system_call_fastpath+0x1a/0x1f [<ffffffffffffffff>] 0xffffffffffffffff [root@localhost ~]# Boot arguments: ---------------- [root@localhost ~]# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.10.65-rt69 raid=noautodetect console=tty0,115200 ip=none root=/dev/ram0 ramdisk_size=12582912 rw clock=pit rcu_nocbs=1-8 rcu_nocb_poll isolcpus=1-8 irqaffinity=0 tsc=perfect selinux=0 enforcing=0 intel_iommu=on iommu=pt default_hugepagesz=2M hugepagesz=2M hugepages=4096 nohz_full=1-8 idle=poll noswap pci=pcie_bus_perf pcie_aspm=off [root@localhost ~]# Below is our RCU config while compiling the kernel + rt patch. # RCU Subsystem CONFIG_TREE_PREEMPT_RCU=y CONFIG_PREEMPT_RCU=y CONFIG_RCU_STALL_COMMON=y # CONFIG_RCU_USER_QS is not set CONFIG_RCU_FANOUT=64 CONFIG_RCU_FANOUT_LEAF=16 # CONFIG_RCU_FANOUT_EXACT is not set # CONFIG_TREE_RCU_TRACE is not set CONFIG_RCU_BOOST=y CONFIG_RCU_BOOST_PRIO=85 CONFIG_RCU_BOOST_DELAY=500 CONFIG_RCU_NOCB_CPU=y CONFIG_RCU_NOCB_CPU_NONE=y # CONFIG_RCU_NOCB_CPU_ZERO is not set # CONFIG_RCU_NOCB_CPU_ALL is not set # RCU Debugging # CONFIG_PROVE_RCU_DELAY is not set # CONFIG_SPARSE_RCU_POINTER is not set # CONFIG_RCU_TORTURE_TEST is not set CONFIG_RCU_CPU_STALL_TIMEOUT=21 CONFIG_RCU_CPU_STALL_VERBOSE=y CONFIG_RCU_CPU_STALL_INFO=y # CONFIG_RCU_TRACE is not set Thanks & Regards Mahesh
On Mon, Jun 04, 2018 at 03:55:34PM +0530, Mahesh Sivapuram wrote:
Hello All,
We are facing an issue with 3.10.65 kernel + RT69 patch on a 12core xeon.
First off, that is a very old and "obsolete" kernel version. As you are probably stuck at that version due to a support contract, I strongly suggest you get support from the company you are paying that money to, as they are the only ones that really know what is in that kernel. There's almost nothing the community can do to help you out, sorry.
We are running real time applications (higher priority than kernel threads, requiring the resolution of 1ms or less) on ioslated CPU's which are configured through kernel boot arguments as indicated below. Also, we have offloaded the RCU call backs to other book keeping CPU cores. But when we try to perform some general commands like mount some directory over NFS, the command hangs and never completes till we stop our real time applications. The moment real time applications are stopped, command finishes immediately.
When we see the backtrace of the mount command, it waiting for "synchronize_rcu". Since we have already offloaded RCU call backs, we expect nothing to be done on isolcpus except rcuc process on isolcpus waking the offloaded threads on other cores.
Messing with where RCU callbacks run from, is a _very_ risky thing to do. There are built-in assumptions that RCU uses to be able to do its work properly, and some of those are that it can touch specific CPUs. Also, I don't think the 3.10.y kernel had all of the "isolate cpus" logic properly implemented in it, I would strongly suggest moving to a newer kernel version (4.17?) that way if things still do not work, the community has a chance to provide some help. good luck! greg k-h
participants (2)
-
Greg KH -
Mahesh Sivapuram