How to handle Hotplug with UIO userspace driver
Hi All, We're working on a custom PCIe based hardware, which can be hotplugged into the system. We're using the UIO approach of having a minimal kernel driver which basically handles the interrupt and provides a mmap of the device memory to userspace and userspace driver which mmap's the provided memory and manages the device. We're using systemd and udev rules to spawn off a user process whenever the hardware is plugged into the system. Now my question is, how can we gracefully inform the userspace process if someone pulls out the hardware (sitting on a PCI slot). Even with full hotplug support, there can always be a case of "surprise removal", where someone yanks the board without going through the whole "attention button" routine. While the kernel driver has no problem with hot-removal, I see that if I "unregister" our UIO driver in response to the removal, then the mapping for the hardware's physical memory would become "invalid" (done by UIO on unregsiter). This could possibly cause the user-space process to crash due to illegal memory access. Is there a "standard" way to handle this scenario, i.e for hotpluggable hardware using UIO? Basically, we'd want to "unregister" the UIO device only after the userspace process is informed of the removal so that it could unmap/close the device and exit gracefully. We could try doing it using some systemd trick, where we put a dependency on one of the UIO created files (/dev/uioX), so that when the file is destroyed, systemd could kill/shutdown the process. Although, there's no guarantee that the userspace process won't try to access the mmaped space during that interval, so this isn't a reliable solution either. Any hints, thoughts? Thanks., -mandeep
On Wed, Nov 26, 2014 at 11:27:07AM -0800, Mandeep Sandhu wrote:
Hi All,
We're working on a custom PCIe based hardware, which can be hotplugged into the system.
We're using the UIO approach of having a minimal kernel driver which basically handles the interrupt and provides a mmap of the device memory to userspace and userspace driver which mmap's the provided memory and manages the device.
We're using systemd and udev rules to spawn off a user process whenever the hardware is plugged into the system.
Now my question is, how can we gracefully inform the userspace process if someone pulls out the hardware (sitting on a PCI slot). Even with full hotplug support, there can always be a case of "surprise removal", where someone yanks the board without going through the whole "attention button" routine.
While the kernel driver has no problem with hot-removal, I see that if I "unregister" our UIO driver in response to the removal, then the mapping for the hardware's physical memory would become "invalid" (done by UIO on unregsiter). This could possibly cause the user-space process to crash due to illegal memory access.
You should just get 0xff on the memory reads, right? You can catch the signal for invalid memory accesses.
Is there a "standard" way to handle this scenario, i.e for hotpluggable hardware using UIO?
Your kernel driver knows when the device is removed, so have it tell userspace this. Or, just tie into libudev and have your userspace program be notified when the device goes away. Do you have a pointer to the source of your uio kernel driver anywhere? thanks, greg k-h
While the kernel driver has no problem with hot-removal, I see that if I "unregister" our UIO driver in response to the removal, then the mapping for the hardware's physical memory would become "invalid" (done by UIO on unregsiter). This could possibly cause the user-space process to crash due to illegal memory access.
You should just get 0xff on the memory reads, right? You can catch the signal for invalid memory accesses.
Wouldn't I get an illegal memory access, as UIO would've removed the mappings that it setup when I registered the UIO driver? I saw the userspace process segfault after removal.
Is there a "standard" way to handle this scenario, i.e for hotpluggable hardware using UIO?
Your kernel driver knows when the device is removed, so have it tell userspace this. Or, just tie into libudev and have your userspace program be notified when the device goes away.
Right. This is what I'm trying to do using a udev "remove" rule. Although the question still remains, what will happen if the device is being actively accessed _before_ the signal reaches the process. Is it safe to do "uio_unregister_device" while a userspace process is accessing the device? I saw a patch for this very situation (UIO & hotplug) being discussed on LKML almost 4 yrs back, although I don't see it in my kernel version - 3.16.0 (Ubuntu 3.16.0-24-generic). The LKML link: https://lkml.org/lkml/2010/9/20/21 Wouldn't this solve the issue? I wonder why it didn't make it into mainline?
Do you have a pointer to the source of your uio kernel driver anywhere?
Not yet, as this is still under active development (we don't have the h/w to test yet :/ ). Maybe once things stabilize a little, we'll probably open-source it along with the h/w too! :) Thanks, -mandeep
thanks,
greg k-h
On Wed, Nov 26, 2014 at 12:33:41PM -0800, Mandeep Sandhu wrote:
> While the kernel driver has no problem with hot-removal, I see that if I > "unregister" our UIO driver in response to the removal, then the mapping for > the hardware's physical memory would become "invalid" (done by UIO on > unregsiter). This could possibly cause the user-space process to crash due to > illegal memory access.
You should just get 0xff on the memory reads, right? You can catch the signal for invalid memory accesses.
Wouldn't I get an illegal memory access, as UIO would've removed the mappings that it setup when I registered the UIO driver? I saw the userspace process segfault after removal.
I don't know, never tried it. What about catching the signal for this?
> Is there a "standard" way to handle this scenario, i.e for hotpluggable > hardware using UIO?
Your kernel driver knows when the device is removed, so have it tell userspace this. Or, just tie into libudev and have your userspace program be notified when the device goes away.
Right. This is what I'm trying to do using a udev "remove" rule.
No, just have libudev tell you about the problem, don't create a whole new rule for this, that's overkill and messy and not dynamic at all.
Although the question still remains, what will happen if the device is being actively accessed _before_ the signal reaches the process. Is it safe to do "uio_unregister_device" while a userspace process is accessing the device?
I don't know, try it and see :)
I saw a patch for this very situation (UIO & hotplug) being discussed on LKML almost 4 yrs back, although I don't see it in my kernel version - 3.16.0 (Ubuntu 3.16.0-24-generic). The LKML link:
https://lkml.org/lkml/2010/9/20/21
Wouldn't this solve the issue? I wonder why it didn't make it into mainline?
No idea, that UIO maintainer must be really lazy and never apply patches :) Try that series on your kernel and see what happens. If it works, please resend that patch series. thanks, greg k-h
Hi Greg,
I saw a patch for this very situation (UIO & hotplug) being discussed on LKML almost 4 yrs back, although I don't see it in my kernel version - 3.16.0 (Ubuntu 3.16.0-24-generic). The LKML link:
https://lkml.org/lkml/2010/9/20/21
Wouldn't this solve the issue? I wonder why it didn't make it into mainline?
No idea, that UIO maintainer must be really lazy and never apply patches :)
Try that series on your kernel and see what happens. If it works, please resend that patch series.
I tried out a dummy uio driver with a userspace program accessing the mmaped space (attached). On doing an unregister of the uio device, I see a crash/bug being triggered. Sometimes the kernel continues to run, sometimes failing during rmmod of my module, sometimes locks up during reboot (trying this on VirtualBox VM) and other times there's no crash at all (but I'm just getting lucky). Although with the patch applied, the user process gets a SIGBUS and exits when I do an uio_unregister_device() call in my driver. So I guess this patch is needed. A generic question, if I have to apply the patches and do a sanity build, which branch should I be applying the patch to? linux-next? linux-stable? Thanks, -mandeep
thanks,
greg k-h
Sorry, sent it too soon. Here's the attached driver and userspace app for simulating a crash. You can run make to build both. Thanks, -mandeep On Mon, Dec 8, 2014 at 3:24 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com> wrote:
Hi Greg,
I saw a patch for this very situation (UIO & hotplug) being discussed on LKML almost 4 yrs back, although I don't see it in my kernel version - 3.16.0 (Ubuntu 3.16.0-24-generic). The LKML link:
https://lkml.org/lkml/2010/9/20/21
Wouldn't this solve the issue? I wonder why it didn't make it into mainline?
No idea, that UIO maintainer must be really lazy and never apply patches :)
Try that series on your kernel and see what happens. If it works, please resend that patch series.
I tried out a dummy uio driver with a userspace program accessing the mmaped space (attached). On doing an unregister of the uio device, I see a crash/bug being triggered. Sometimes the kernel continues to run, sometimes failing during rmmod of my module, sometimes locks up during reboot (trying this on VirtualBox VM) and other times there's no crash at all (but I'm just getting lucky).
Although with the patch applied, the user process gets a SIGBUS and exits when I do an uio_unregister_device() call in my driver. So I guess this patch is needed.
A generic question, if I have to apply the patches and do a sanity build, which branch should I be applying the patch to? linux-next? linux-stable?
Thanks, -mandeep
thanks,
greg k-h
Hi Mandeep, Recently i encountered similar issue with UIO framework in hotplug scenario. I reviewed the earlier patch from 2010 and your 4 patch series. 4 patch series seems to work. I checked the latest kernel source and i dont think your patches have been merged. Hopefully i am not looking at the wrong place. If they are not merged is there a plan to merge them ? I do think these patches are required. I also found a bug in your patch set which is discussed here https://lkml.org/lkml/2017/6/14/30. I was able to workaround/fix it. I can upload a patch if needed. Please advice if you plan to get the patchset merged. Thanks Divakar
participants (3)
-
Divakar -
Greg KH -
Mandeep Sandhu