Debugging a kernel freeze

[phil]

On 01/12/17 14:05, Victor Ascroft wrote:
> I have a iMX6 running a 4.9 kernel with a custom kernel driver communicating
> with a FPGA over PCIe. The driver is not built in to the kernel but loaded as
> a module after complete boot up. During the running of the system, after a few
> hours the kernel completely freezes. No kernel panics or stack traces, nothing.
> I have access to the serial console.

I've done a lot of work with the imx6 and an Altera Cyclone IV FPGA 
connected via PCIe bus and I've not experienced any major issues with 
this setup.

> In such a scenario what are the ways to debug and try locating the source of
> the problem? I am not looking for a solution for my problem but things or
> approaches one can go about trying while trying to fix such a scenario?

This is a difficult situation and it will take a lot of time to debug 
but you really just need to spend time picking apart the driver. You 
should try disabling various parts and adding dynamic debug messages or 
tracing.

My first suspicion in these cases however is always with interrupts. 
There have been a few times when our FPGA code has a fault and the 
interrupts fail, so my first port of call is to usually disable 
interrupts in my driver and replace them with highres timers. Also you 
might want to look at load balancing the interrupts, ARM processors keep 
interrupts to one core (or they did in the kernels I've been using) and 
you can either manually assign the interrupts to other cores or use 
irqbalance to do so automatically. I prefered the manual solution as 
irqbalance didn't seem to assign my workload efficiently across the 
cores. At any rate you should probably be monitoring the interrupts.

Good Luck!

Regards,

Philip Downer