Trying to debug interrupt flood after unbind
I am trying to load a driver for an Exar serial chip, but that chip is gobbled up by the 8250 driver on boot. So, I use the "unbind" command in /sys/bus/pci/drivers/serial to remove the device from the clutches of 8250. Based on cobbled together google searches, I use the following to unbind it (assuming the address in /sys/bus/pci/drivers/serial is 0000:04:00.0. sudo echo -n "0000:04:00.0" | tee ./unbind The address disappears from that dir when I do this command, so I'm assuming it works. I then go and install the Exar-provided driver. Within about 3 seconds, I then get a system notification that the IRQ used by the Exar driver has been disabled. I can also go look at /proc/interrupts and see a huge amount of interrupts happening on that IRQ. There's a crash message in the log: [ 167.938861] irq 17: nobody cared (try booting with the "irqpoll" option) [ 167.938868] CPU: 0 PID: 801 Comm: Xorg Tainted: G O 3.16.6-2-desktop #1 [ 167.938871] Hardware name: RTD Embedded Technologies, Inc CMA34CR/CMA34CR, BIOS v3.72.51.0009-1.1.85582 02/09/2015 09:39:58 [ 167.938873] ffff880148672cc4 ffffffff8161ab03 ffff880148672c00 ffffffff810b8acd [ 167.938877] ffff880148672c00 0000000000000011 0000000000000000 ffffffff810b9011 [ 167.938880] 0000000000000000 0000000000000000 0000000000000011 0000000000000000 [ 167.938883] Call Trace: [ 167.938899] [<ffffffff8100519e>] dump_trace+0x8e/0x350 [ 167.938905] [<ffffffff81005506>] show_stack_log_lvl+0xa6/0x190 [ 167.938909] [<ffffffff81006c01>] show_stack+0x21/0x50 [ 167.938914] [<ffffffff8161ab03>] dump_stack+0x49/0x6a [ 167.938922] [<ffffffff810b8acd>] __report_bad_irq+0x2d/0xc0 [ 167.938928] [<ffffffff810b9011>] note_interrupt+0x241/0x290 [ 167.938935] [<ffffffff810b67f1>] handle_irq_event_percpu+0xa1/0x1d0 [ 167.938940] [<ffffffff810b695e>] handle_irq_event+0x3e/0x60 [ 167.938945] [<ffffffff810b9b58>] handle_fasteoi_irq+0x88/0x160 [ 167.938949] [<ffffffff810050fd>] handle_irq+0x1d/0x30 [ 167.938955] [<ffffffff81624549>] do_IRQ+0x49/0xe0 [ 167.938959] [<ffffffff816224ad>] common_interrupt+0x6d/0x6d [ 167.938967] [<ffffffff81620dce>] _raw_spin_unlock_irqrestore+0xe/0x30 [ 167.938974] [<ffffffff815c3f05>] unix_poll+0x25/0xb0 [ 167.938980] [<ffffffff81513fa9>] sock_poll+0x49/0x110 [ 167.938986] [<ffffffff811caf40>] do_select+0x390/0x7a0 [ 167.938991] [<ffffffff811cb4e4>] core_sys_select+0x194/0x2b0 [ 167.938995] [<ffffffff811cb6aa>] SyS_select+0xaa/0xf0 [ 167.938999] [<ffffffff8162182d>] system_call_fastpath+0x1a/0x1f [ 167.939015] [<00007f7563abda43>] 0x7f7563abda42 [ 167.939016] handlers: [ 167.939020] [<ffffffffa0549110>] serialxr_interrupt [xr17v35x] [ 167.939023] Disabling IRQ #17 So, as near as I can tell, when the Exar driver is inserted, an interrupt flood occurs, and the Exar driver (the only interrupt handler on that IRQ) does not respond to any of them. I put in some debug code and verified that the Exar interrupt handler is called... but the handler just returns with an IRQ_NONE value. I've tried: 1) Multiple CPUs from different families (Core i7, Core 2 Duo, AMD G-Series) and it occurs with all of them. 2) Kernels 3.16 and 4.2 and it occurs with both of them. 3) Disabling ModemManager and it still happens. 4) Contacting Exar about this. They could not reproduce the problem. 5) openSUSE 13.2 and Fedora 20. It happened with openSUSE, but NOT with Fedora 20. 6) The reference Exar implementation vs. our implementation, and it occurs with both of them. Do you have any suggestions on how I can discover what is sending all of those interrupts? Are there kernel tools specifically for that? Thank you. Rob Groner
On Tue, May 31, 2016 at 06:41:24PM +0000, Rob Groner wrote:
I am trying to load a driver for an Exar serial chip, but that chip is gobbled up by the 8250 driver on boot.
Why does the in-kernel driver not support this chip? Do you have a pointer to the Exar source anywhere?
So, I use the “unbind” command in /sys/bus/pci/ drivers/serial to remove the device from the clutches of 8250. Based on cobbled together google searches, I use the following to unbind it (assuming the address in /sys/bus/pci/drivers/serial is 0000:04:00.0.
That's a huge hammer, are you sure that's the best way to do this? Why not just add your device to a blacklist in the 8250 driver?
So, as near as I can tell, when the Exar driver is inserted, an interrupt flood occurs, and the Exar driver (the only interrupt handler on that IRQ) does not respond to any of them. I put in some debug code and verified that the Exar interrupt handler is called… but the handler just returns with an IRQ_NONE value.
Sounds like a bug in the Exar driver, why isn't it handling the interrupt properly? thanks, greg k-h
On Tue, 2016-05-31 at 13:14 -0700, Greg KH wrote:
On Tue, May 31, 2016 at 06:41:24PM +0000, Rob Groner wrote:
I am trying to load a driver for an Exar serial chip, but that chip is gobbled up by the 8250 driver on boot.
Why does the in-kernel driver not support this chip? Do you have a pointer to the Exar source anywhere?
This is kind of an ongoing topic since around December The in-kernel driver doesn't support the GPIO on the Exar chip, which frustrates our customers because our board requires use of the GPIO. I've been working with Sudip Mukherjee to add GPIO to the in-kernel driver, but that's stalled for reasons (he's waiting for someone else to submit something similar). In the meantime, I'm just trying to find any workaround for our customers that doesn't involve a kernel build.
So, I use the “unbind” command in /sys/bus/pci/ drivers/serial to remove the device from the clutches of 8250. Based on cobbled together google searches, I use the following to unbind it (assuming the address in /sys/bus/pci/drivers/serial is 0000:04:00.0.
That's a huge hammer, are you sure that's the best way to do this? Why not just add your device to a blacklist in the 8250 driver?
I know about blacklisting a module, but I've not heard of blacklisting a device for a particular module. Is this without having to rebuild the kernel? That would be perfectly acceptable. I'm happy to use smaller hammers.
So, as near as I can tell, when the Exar driver is inserted, an interrupt flood occurs, and the Exar driver (the only interrupt handler on that IRQ) does not respond to any of them. I put in some debug code and verified that the Exar interrupt handler is called… but the handler just returns with an IRQ_NONE value.
Sounds like a bug in the Exar driver, why isn't it handling the interrupt properly?
Exar is pretty responsive overall, but despite my complete ease at producing this problem, they've been unable to recreate it, so they just shrug. That's why I've put it on myself to try and get whatever info I can about what is happening and either find the bug in their driver or be able to point them in the direction. Rob
On Tue, May 31, 2016 at 04:38:37PM -0400, Rob Groner wrote:
On Tue, 2016-05-31 at 13:14 -0700, Greg KH wrote:
On Tue, May 31, 2016 at 06:41:24PM +0000, Rob Groner wrote:
I am trying to load a driver for an Exar serial chip, but that chip is gobbled up by the 8250 driver on boot.
Why does the in-kernel driver not support this chip? Do you have a pointer to the Exar source anywhere?
This is kind of an ongoing topic since around December.
yes, and new driver was provided to have the gpio as a separate platform driver which was also tested by Rob and he confirmed that everything is working as expected. But the reviews asked for a total rewrite with the exar codes split out from 8250_pci, and i got busy with a change of dayjob. If you want to have a look at the final version that was submitted then it is at: http://www.spinics.net/lists/linux-serial/msg20859.html I thought to do it this weekend but I will be again travelling to UK this weekend so again postponed for atleast one month. Regards Sudip
On Thu, 2016-06-02 at 10:38 +0100, Sudip Mukherjee wrote:
On Tue, May 31, 2016 at 04:38:37PM -0400, Rob Groner wrote:
On Tue, 2016-05-31 at 13:14 -0700, Greg KH wrote:
On Tue, May 31, 2016 at 06:41:24PM +0000, Rob Groner wrote:
I am trying to load a driver for an Exar serial chip, but that chip is gobbled up by the 8250 driver on boot.
Why does the in-kernel driver not support this chip? Do you have a pointer to the Exar source anywhere?
This is kind of an ongoing topic since around December.
yes, and new driver was provided to have the gpio as a separate platform driver which was also tested by Rob and he confirmed that everything is working as expected. But the reviews asked for a total rewrite with the exar codes split out from 8250_pci, and i got busy with a change of dayjob. If you want to have a look at the final version that was submitted then it is at:
I thought to do it this weekend but I will be again travelling to UK this weekend so again postponed for atleast one month.
I have no criticism or complaint for how long it is taking or will take to get the in-kernel driver in good working order...I greatly appreciate all of the help that has been given and offered. What I'm trying to do is come up with a working alternative until the driver fix is in. In my perfect world, there would be a way for them to disentangle the Exar chip from the 8250 driver, and then they could use the provided Exar driver (until such time as the built-in driver does everything needed). Is there any other workaround other than "unbind"? Because unbind doesn't work for some reason. And if there is no other alternative to unbind, then is there some way I can try to identify where the interrupt flood is coming from when I use it? I've looked at the Exar driver, and when it receives the interrupt, it checks all of its registers, and if there is no interrupt bit there, it says "not mine", which is either true or not true. I'll investigate that possibility. Rob
participants (4)
-
Greg KH -
Rob Groner -
Rob Groner
-
Sudip Mukherjee