watchdog pet in kernel module
Hi, I would like to move the hardware watchdog pet code from user space to kernel space inside the ipmi module and would like to know how to go about pet the hardware watchdog at periodic interval. Any pointers help will be really appreciated. Regards, Vipul.
Use queue_delayed_work Thanks, Arun On Tue, Dec 3, 2013 at 12:04 PM, Vipul Jain <vipulsj@gmail.com> wrote:
Hi,
I would like to move the hardware watchdog pet code from user space to kernel space inside the ipmi module and would like to know how to go about pet the hardware watchdog at periodic interval. Any pointers help will be really appreciated.
Regards, Vipul.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Mon, Dec 2, 2013 at 10:34 PM, Vipul Jain <vipulsj@gmail.com> wrote:
Hi,
I would like to move the hardware watchdog pet code from user space to kernel space inside the ipmi module and would like to know how to go about pet the hardware watchdog at periodic interval. Any pointers help will be a cursory glance at Documentation/ipmi.txt states that IPMI comes with a standard watchdog timer. So what are you looking for? really appreciated.
Regards, Vipul.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Mon, Dec 2, 2013 at 11:32 PM, anish singh <anish198519851985@gmail.com>wrote:
On Mon, Dec 2, 2013 at 10:34 PM, Vipul Jain <vipulsj@gmail.com> wrote:
Hi,
I would like to move the hardware watchdog pet code from user space to kernel space inside the ipmi module and would like to know how to go about pet the hardware watchdog at periodic interval. Any pointers help will be a cursory glance at Documentation/ipmi.txt states that IPMI comes with a standard watchdog timer. So what are you looking for?
Hi Anish, currently we configure/pet the watchdog from user space via /dev/ipmi0 device interface and I would like to do the pet part from kernel module. It looks like we do have ipmi_watchdog.c driver that does pet but I am not able to understand what I need to do in order to make it independent of user space and keep pet by its own. Wondering if you could please help me here. Regards, Vipul.
On Tue, 03 Dec 2013 13:15:32 -0800, Vipul Jain said:
currently we configure/pet the watchdog from user space via /dev/ipmi0 device interface and I would like to do the pet part from kernel module.
That's actually defeating the purpose. If you do it from the kernel, you keep the watchdog from detecting a whole set of hangs that can cause userspace to wedge up.
On Tue, Dec 3, 2013 at 2:31 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Tue, 03 Dec 2013 13:15:32 -0800, Vipul Jain said:
currently we configure/pet the watchdog from user space via /dev/ipmi0 device interface and I would like to do the pet part from kernel module.
That's actually defeating the purpose. If you do it from the kernel, you keep the watchdog from detecting a whole set of hangs that can cause userspace to wedge up.
Well we use different mechanism to detect user space hangs and take corrective actions. Hence we want to separate the user space issues from kernel space issues by using hardware watchdog pet in kernel space.
Hi Vipul, I have seen this in a number of commercial software running on RHEL, and on other realtime OS as well. The watchdog mechanism is always working in pair: userspace "feeding" the dog (in the kernel). (btw, feed the dog is a more usually used term than "pet" the dog. sorry for that. google for that and perhaps you can get more info?). Like Valdis said, this way you will know when userspace hang, which is the key criteria for reboot. Why do u want to detect if the kernel hang (versus busy doing something)? Theoretically that is not possible, especially when all interrupt are disabled. On Wed, Dec 4, 2013 at 6:45 AM, Vipul Jain <vipulsj@gmail.com> wrote:
On Tue, Dec 3, 2013 at 2:31 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Tue, 03 Dec 2013 13:15:32 -0800, Vipul Jain said:
currently we configure/pet the watchdog from user space via /dev/ipmi0 device interface and I would like to do the pet part from kernel module.
That's actually defeating the purpose. If you do it from the kernel, you keep the watchdog from detecting a whole set of hangs that can cause userspace to wedge up.
Well we use different mechanism to detect user space hangs and take corrective actions. Hence we want to separate the user space issues from kernel space issues by using hardware watchdog pet in kernel space.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-- Regards, Peter Teoh
On Tue, Dec 3, 2013 at 10:28 PM, Peter Teoh <htmldeveloper@gmail.com> wrote:
Hi Vipul,
I have seen this in a number of commercial software running on RHEL, and on other realtime OS as well. The watchdog mechanism is always working in pair: userspace "feeding" the dog (in the kernel). (btw, feed the dog is a more usually used term than "pet" the dog. sorry for that. google for that and perhaps you can get more info?).
Like Valdis said, this way you will know when userspace hang, which is the key criteria for reboot. Why do u want to detect if the kernel hang (versus busy doing something)? Theoretically that is not possible, especially when all interrupt are disabled.
Hi Peter,
If you don't mind can you please provide me more insight as what can be false alarm I can encounter to move pet inside kernel module? Regards, Vipul.
On Wed, 04 Dec 2013 16:45:44 -0800, Vipul Jain said:
If you don't mind can you please provide me more insight as what can be false alarm I can encounter to move pet inside kernel module?
The issue isn't false alarms - it's failure to alarm when it should. The problem is that it's possible for a kernel to get wedged in such a way that a kernel thread is still able to feed the watchdog timer on a regular basis, but userspace is effectively hung and unable to proceed. For example, if an OOPS happens while a filesystem lock is held, all future userspace references to that filesystem (and possibly all filesystems of the same type) will hang, eventually strangling the box while the kernel is still perfectly able to keep the watchdog working.
On Wed, Dec 4, 2013 at 4:57 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Wed, 04 Dec 2013 16:45:44 -0800, Vipul Jain said:
If you don't mind can you please provide me more insight as what can be false alarm I can encounter to move pet inside kernel module?
The issue isn't false alarms - it's failure to alarm when it should.
The problem is that it's possible for a kernel to get wedged in such a way that a kernel thread is still able to feed the watchdog timer on a regular basis, but userspace is effectively hung and unable to proceed. For example, if an OOPS happens while a filesystem lock is held, all future userspace references to that filesystem (and possibly all filesystems of the same type) will hang, eventually strangling the box while the kernel is still perfectly able to keep the watchdog working.
Hi Valdis,
I see what you are saying but what if the user process that's feeding the dog gets hung and rest of the system is fine then it will bring the whole system down won't it? I basically want to avoid this? Regards, Vipul.
On Thu, Dec 5, 2013 at 9:06 AM, Vipul Jain <vipulsj@gmail.com> wrote:
On Wed, Dec 4, 2013 at 4:57 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Wed, 04 Dec 2013 16:45:44 -0800, Vipul Jain said:
If you don't mind can you please provide me more insight as what can be false alarm I can encounter to move pet inside kernel module?
The issue isn't false alarms - it's failure to alarm when it should.
The problem is that it's possible for a kernel to get wedged in such a way that a kernel thread is still able to feed the watchdog timer on a regular basis, but userspace is effectively hung and unable to proceed. For example, if an OOPS happens while a filesystem lock is held, all future userspace references to that filesystem (and possibly all filesystems of the same type) will hang, eventually strangling the box while the kernel is still perfectly able to keep the watchdog working.
Hi Valdis,
I see what you are saying but what if the user process that's feeding the dog gets hung and rest of the system is fine then it will bring the whole system down won't it? I basically want to avoid this?
Normally the process that feed the dog, is a simple process that JUST periodically set the watchdog device descriptor. Yes, one main() with a while loop just periodically resetting the descriptor. And so it is is not able to respond in time, by inference, OTHER PROCESS must have hung. In other system i saw there is a mother process that monitor a few (not all) of its key child process .... so perhaps one child will have one variable to signal to the mother that it is running. If not responding in time, the mother will clean up everything and then purposely not setting the watchdog, resulting in reboot.
Regards, Vipul.
-- Regards, Peter Teoh
Although /dev/watchdog is available in usermode, but nothing should stop you to write to it from a kernel thread. Rajat On Wed, Dec 4, 2013 at 5:50 PM, Peter Teoh <htmldeveloper@gmail.com> wrote:
On Thu, Dec 5, 2013 at 9:06 AM, Vipul Jain <vipulsj@gmail.com> wrote:
On Wed, Dec 4, 2013 at 4:57 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Wed, 04 Dec 2013 16:45:44 -0800, Vipul Jain said:
If you don't mind can you please provide me more insight as what can be false alarm I can encounter to move pet inside kernel module?
The issue isn't false alarms - it's failure to alarm when it should.
The problem is that it's possible for a kernel to get wedged in such a way that a kernel thread is still able to feed the watchdog timer on a regular basis, but userspace is effectively hung and unable to proceed. For example, if an OOPS happens while a filesystem lock is held, all future userspace references to that filesystem (and possibly all filesystems of the same type) will hang, eventually strangling the box while the kernel is still perfectly able to keep the watchdog working.
Hi Valdis,
I see what you are saying but what if the user process that's feeding the dog gets hung and rest of the system is fine then it will bring the whole system down won't it? I basically want to avoid this?
Normally the process that feed the dog, is a simple process that JUST periodically set the watchdog device descriptor. Yes, one main() with a while loop just periodically resetting the descriptor.
And so it is is not able to respond in time, by inference, OTHER PROCESS must have hung. In other system i saw there is a mother process that monitor a few (not all) of its key child process .... so perhaps one child will have one variable to signal to the mother that it is running. If not responding in time, the mother will clean up everything and then purposely not setting the watchdog, resulting in reboot.
Regards, Vipul.
-- Regards, Peter Teoh
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Thu, Dec 5, 2013 at 10:19 AM, Rajat Sharma <fs.rajat@gmail.com> wrote:
Although /dev/watchdog is available in usermode, but nothing should stop you to write to it from a kernel thread.
Rajat
I don't think /dev/watchdog (literally, I meant) is available in the kernel. It is accessible in userspace, but translated to a different name in the kernel. and moreover, if u access the variable directly, bypassing all the spinlock (see drivers/watchdog and look for "wdt_lock" spinlock) that is implemented around it, u might be going into a racing condition. BUT.....if u really insist probing from inside the kernel....it is not watchdog, it is "process watch", in your own way. ie, u can always write a loop that periodically probe the status of that specific to make sure it is in RUNNING state (vs BLOCKING when it is waiting for some I/O, or locks to complete), and perhaps check the CPU instruction to make sure that it is not going into a tight loop (ie, a userspace program that literally do "while(true) {do_nothing()}....and many other possible "hung" criteria for a process as well. not easy...but extremely complex.
On Wed, Dec 4, 2013 at 5:50 PM, Peter Teoh <htmldeveloper@gmail.com>wrote:
On Thu, Dec 5, 2013 at 9:06 AM, Vipul Jain <vipulsj@gmail.com> wrote:
On Wed, Dec 4, 2013 at 4:57 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Wed, 04 Dec 2013 16:45:44 -0800, Vipul Jain said:
If you don't mind can you please provide me more insight as what can be false alarm I can encounter to move pet inside kernel module?
The issue isn't false alarms - it's failure to alarm when it should.
The problem is that it's possible for a kernel to get wedged in such a way that a kernel thread is still able to feed the watchdog timer on a regular basis, but userspace is effectively hung and unable to proceed. For example, if an OOPS happens while a filesystem lock is held, all future userspace references to that filesystem (and possibly all filesystems of the same type) will hang, eventually strangling the box while the kernel is still perfectly able to keep the watchdog working.
Hi Valdis,
I see what you are saying but what if the user process that's feeding the dog gets hung and rest of the system is fine then it will bring the whole system down won't it? I basically want to avoid this?
Normally the process that feed the dog, is a simple process that JUST periodically set the watchdog device descriptor. Yes, one main() with a while loop just periodically resetting the descriptor.
And so it is is not able to respond in time, by inference, OTHER PROCESS must have hung. In other system i saw there is a mother process that monitor a few (not all) of its key child process .... so perhaps one child will have one variable to signal to the mother that it is running. If not responding in time, the mother will clean up everything and then purposely not setting the watchdog, resulting in reboot.
Regards, Vipul.
-- Regards, Peter Teoh
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-- Regards, Peter Teoh
On Thu, Dec 5, 2013 at 8:45 AM, Vipul Jain <vipulsj@gmail.com> wrote:
On Tue, Dec 3, 2013 at 10:28 PM, Peter Teoh <htmldeveloper@gmail.com>wrote:
Hi Vipul,
I have seen this in a number of commercial software running on RHEL, and on other realtime OS as well. The watchdog mechanism is always working in pair: userspace "feeding" the dog (in the kernel). (btw, feed the dog is a more usually used term than "pet" the dog. sorry for that. google for that and perhaps you can get more info?).
Like Valdis said, this way you will know when userspace hang, which is the key criteria for reboot. Why do u want to detect if the kernel hang (versus busy doing something)? Theoretically that is not possible, especially when all interrupt are disabled.
Hi Peter,
If you don't mind can you please provide me more insight as what can be false alarm I can encounter to move pet inside kernel module?
"Feeding the dog" is simply a periodic timer that wakes up and set a variable. By the fact that the variable can be set/reset, also means that the periodic timer IS working. In userspace, if you just have one process to "feed the watchdog", then essentially we are monitoring whether system-wide the performance overall is good enough so that the periodic timer can be woken up at the required interval to reset the variable. If some process hung, it MAY or MAY not affect the periodicity of this timer process. But if you have the timer embedded inside a particular high priority process you want to monitor, and if it hung, and "feeding the watchdog" will not execute, and the kernel will reboot you (read below - search "reboot"). http://www.mjmwired.net/kernel/Documentation/watchdog/watchdog-api.txt and more insights: http://stackoverflow.com/questions/2020468/who-is-refreshing-hardware-watchd... (and lots of the "RELATED" questions at the side of the above page as well.)
Regards, Vipul.
-- Regards, Peter Teoh
participants (6)
-
anish singh -
Arun KS -
Peter Teoh -
Rajat Sharma -
Valdis.Kletnieks＠vt.edu -
Vipul Jain