Recovering Linux system from hung state via software
Hi Kernel alias, I am a newbie and I am trying to figure out ways where in I can recover the Linux in below two scenarios: 1. my specific process hangs. 2. kernel gets hung partially or completely. I have done some reading and seems like there is softlockup and hardlockup mechanisms in Linux source base that I can use but not sure, if yes I have below questions: 1. Which kernel version is minimum required for this? 2. How do I know that soft and hard lockup are enabled in my kernel? 3. How can I customize the behavior of default action that been taken? 4. Can I use these two lockup mechanism to find out if my process is hung or not? 5. Any pointers to any docs that can help will be appreciated. I will greatly appreciate any help here. Regards, Vipul.
On Fri, Nov 29, 2013 at 8:28 AM, Vipul Jain <vipulsj@gmail.com> wrote:
Hi Kernel alias,
I am a newbie and I am trying to figure out ways where in I can recover the Linux in below two scenarios: 1. my specific process hangs.
how to recover i cannot tell you, because it is application specific (but best is to design your system to reboot completely. eg temporary stuff or files should be stored in memory - eg, tmpfs, and rebooting will be all "gone", not erased and removed securely, but logically "gone"). And how to detect that is this: assuming one mother process is monitoring 10 child process, so inside each child process, simply just setup a PERIODIC (eg, per 5 sec) mechanism to toggle a binary variables through IPC means. It will be reset when the mother process go around checking all the variable status and, if not reset it therefore implies that the particular process might be hung. it can wait further, or continue checking other process. at the end of checking ALL the process, if everything is OK, it should feed the kernel watchdog timer. if the kernel watchdog timer is not reset, the kernel module will then reboot the system. (ie, reboot is from kernel module).
2. kernel gets hung partially or completely.
I have done some reading and seems like there is softlockup and hardlockup mechanisms in Linux source base that I can use but not sure, if yes I have below questions: 1. Which kernel version is minimum required for this? 2. How do I know that soft and hard lockup are enabled in my kernel? 3. How can I customize the behavior of default action that been taken? 4. Can I use these two lockup mechanism to find out if my process is hung or not? 5. Any pointers to any docs that can help will be appreciated.
I will greatly appreciate any help here.
Regards, Vipul.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-- Regards, Peter Teoh
assuming one mother process is monitoring 10 child process, so inside each child process, simply just setup a PERIODIC (eg, per 5 sec) mechanism to toggle a binary variables through IPC means. It will be reset when the mother process go around checking all the variable status and, if not reset it therefore implies that the particular process might be hung. it can wait further, or continue checking other process. at the end of checking ALL the process, if everything is OK, it should feed the kernel watchdog timer. if the kernel watchdog timer is not reset, the kernel module will then reboot the system. (ie, reboot is from kernel module).
Hold on! Why should we reboot the whole system if only some of these processes are misbehaving?!?! Why should other processes suffer due this? Wouldn't it be better to just kill the erroneous process (like how most OS's anyway do, eg: "Force Quit" in Ubuntu, or chrome tabs). Or are these processes the only ones running on the system? -mandeep
On Wed, Dec 4, 2013 at 4:13 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com>wrote:
assuming one mother process is monitoring 10 child process, so inside each child process, simply just setup a PERIODIC (eg, per 5 sec) mechanism to toggle a binary variables through IPC means. It will be reset when the mother process go around checking all the variable status and, if not reset it therefore implies that the particular process might be hung. it can wait further, or continue checking other process. at the end of checking ALL the process, if everything is OK, it should feed the kernel watchdog timer. if the kernel watchdog timer is not reset, the kernel module will then reboot the system. (ie, reboot is from kernel module).
Hold on! Why should we reboot the whole system if only some of these processes are misbehaving?!?! Why should other processes suffer due this? Wouldn't it be better to just kill the erroneous process (like how most OS's anyway do, eg: "Force Quit" in Ubuntu, or chrome tabs).
In many COTS software, the behavior of every process is highly dependent on one-another, especially some of these will talk to hardware, and other are just processing the intermediate data. When something goes wrong, it is difficult to diagnose the faults (which is why faults logging is important, and always done on flash or harddisk, but not temporary filesystem) in realtime (ie, self-diagnosis mechanism), so it is better to reboot. yes, not all process need to trigger reboot, so design it with care. eg, Apache server can always afford to be kill and restart a new one.
Or are these processes the only ones running on the system?
-mandeep
-- Regards, Peter Teoh
You could also look at using existing s/w for monitoring processes like monit. I don't know your exact usecase but you could make your process as a 'service' and have something like systemd or upstart monitor it. These tools will typically 'restart' the hung process on detection. -mandeep On Fri, Nov 29, 2013 at 5:58 AM, Vipul Jain <vipulsj@gmail.com> wrote:
Hi Kernel alias,
I am a newbie and I am trying to figure out ways where in I can recover the Linux in below two scenarios: 1. my specific process hangs. 2. kernel gets hung partially or completely.
I have done some reading and seems like there is softlockup and hardlockup mechanisms in Linux source base that I can use but not sure, if yes I have below questions: 1. Which kernel version is minimum required for this? 2. How do I know that soft and hard lockup are enabled in my kernel? 3. How can I customize the behavior of default action that been taken? 4. Can I use these two lockup mechanism to find out if my process is hung or not? 5. Any pointers to any docs that can help will be appreciated.
I will greatly appreciate any help here.
Regards, Vipul.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Vipul, On Fri, Nov 29, 2013 at 5:58 AM, Vipul Jain <vipulsj@gmail.com> wrote:
Hi Kernel alias,
I am a newbie and I am trying to figure out ways where in I can recover the Linux in below two scenarios: 1. my specific process hangs. 2. kernel gets hung partially or completely.
I have done some reading and seems like there is softlockup and hardlockup mechanisms in Linux source base that I can use but not sure, if yes I have below questions: 1. Which kernel version is minimum required for this? 2. How do I know that soft and hard lockup are enabled in my kernel? 3. How can I customize the behavior of default action that been taken? 4. Can I use these two lockup mechanism to find out if my process is hung or not? 5. Any pointers to any docs that can help will be appreciated.
I will greatly appreciate any help here.
What about crashing the system using sysrq and analysing the ramdump using crash utility? May be the following link give you some pointers, https://wiki.ubuntu.com/Kernel/CrashdumpRecipe Thanks, Arun
Regards, Vipul.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
participants (4)
-
Arun KS -
Mandeep Sandhu -
Peter Teoh -
Vipul Jain