mount command stuck at synchronize_rcu on a RT patched kernel.

[Mahesh Sivapuram]

Hello All,

We are facing an issue with 3.10.65 kernel + RT69 patch on a 12core xeon.
We are running real time applications (higher priority than kernel threads,
requiring the resolution of 1ms or less) on ioslated CPU's which are
configured through kernel boot arguments as indicated below. Also, we have
offloaded the RCU call backs to other book keeping CPU cores. But when we
try to perform some general commands like mount some directory over NFS,
the command hangs and never completes till we stop our real time
applications. The moment real time applications are stopped, command
finishes immediately.

When we see the backtrace of the mount command, it waiting for
"synchronize_rcu". Since we have already offloaded RCU call backs, we
expect nothing to be done on isolcpus except rcuc process on isolcpus
waking the offloaded threads on other cores.

Can someone help us understand what kernel parameters/config needs to be
modified/tuned to achieve the proper behavior of the system ?

Thanks in advance.

Backtrace:
----------
[root at localhost ~]# ps -eaf | grep mount
root     17915  4532  0 19:13 pts/0    00:00:00 umount /bng4
root     17916 17915  0 19:13 pts/0    00:00:00 /sbin/umount.nfs /bng4
root     17962 17945  0 19:14 pts/2    00:00:00 grep --color=auto mount
[root at localhost ~]# cat /proc/17916/stack
[<ffffffff810612b4>] wait_rcu_gp+0x44/0x50
[<ffffffff810c6ec9>] synchronize_rcu+0x29/0x40
[<ffffffff8126446a>] nfs_free_server+0xea/0x180
[<ffffffff8126e49e>] nfs_kill_super+0x1e/0x30
[<ffffffff8115a4ed>] deactivate_locked_super+0x3d/0x60
[<ffffffff8115ab16>] deactivate_super+0x46/0x60
[<ffffffff811770ef>] mntput_no_expire+0xef/0x150
[<ffffffff81178376>] SyS_umount+0x96/0x3d0
[<ffffffff818cbb46>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
[root at localhost ~]#

Boot arguments:
----------------
[root at localhost ~]# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.65-rt69 raid=noautodetect console=tty0,115200
ip=none root=/dev/ram0 ramdisk_size=12582912 rw clock=pit rcu_nocbs=1-8
rcu_nocb_poll isolcpus=1-8 irqaffinity=0 tsc=perfect selinux=0 enforcing=0
intel_iommu=on iommu=pt default_hugepagesz=2M hugepagesz=2M hugepages=4096
nohz_full=1-8 idle=poll noswap pci=pcie_bus_perf pcie_aspm=off
[root at localhost ~]#

Below is our RCU config while compiling the kernel + rt patch.

# RCU Subsystem
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_STALL_COMMON=y
# CONFIG_RCU_USER_QS is not set
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_PRIO=85
CONFIG_RCU_BOOST_DELAY=500
CONFIG_RCU_NOCB_CPU=y
CONFIG_RCU_NOCB_CPU_NONE=y
# CONFIG_RCU_NOCB_CPU_ZERO is not set
# CONFIG_RCU_NOCB_CPU_ALL is not set
# RCU Debugging
# CONFIG_PROVE_RCU_DELAY is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=21
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_RCU_CPU_STALL_INFO=y
# CONFIG_RCU_TRACE is not set

Thanks & Regards
Mahesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180604/490ec418/attachment-0001.html>