mount command stuck at synchronize_rcu on a RT patched kernel.

[Greg KH]

On Mon, Jun 04, 2018 at 03:55:34PM +0530, Mahesh Sivapuram wrote:
> Hello All,
> 
> We are facing an issue with 3.10.65 kernel + RT69 patch on a 12core xeon.

First off, that is a very old and "obsolete" kernel version.  As you are
probably stuck at that version due to a support contract, I strongly
suggest you get support from the company you are paying that money to,
as they are the only ones that really know what is in that kernel.
There's almost nothing the community can do to help you out, sorry.

> We are running real time applications (higher priority than kernel threads,
> requiring the resolution of 1ms or less) on ioslated CPU's which are
> configured through kernel boot arguments as indicated below. Also, we have
> offloaded the RCU call backs to other book keeping CPU cores. But when we
> try to perform some general commands like mount some directory over NFS,
> the command hangs and never completes till we stop our real time
> applications. The moment real time applications are stopped, command
> finishes immediately.
> 
> When we see the backtrace of the mount command, it waiting for
> "synchronize_rcu". Since we have already offloaded RCU call backs, we
> expect nothing to be done on isolcpus except rcuc process on isolcpus
> waking the offloaded threads on other cores.

Messing with where RCU callbacks run from, is a _very_ risky thing to
do.  There are built-in assumptions that RCU uses to be able to do its
work properly, and some of those are that it can touch specific CPUs.

Also, I don't think the 3.10.y kernel had all of the "isolate cpus"
logic properly implemented in it, I would strongly suggest moving to a
newer kernel version (4.17?) that way if things still do not work, the
community has a chance to provide some help.

good luck!

greg k-h