Re: Any trick to remove a D-state dead process without rebooting?
Hi, Valdis: Thanks for the informative reply! There are actually plenty of processes stuck in D-state, among them are systemd-logind and snmpd.
1) Check your dmesg/syslog for any WARN, BUG, or OOPS messages with tracebacks, which probably indicates the kernel bug that caused the problem.
2) cat /proc/NNN/stack will give you a hint where the syscall is wedged.
http://ix.io/CMN It seems that they all have something to do with autofs or filesystem. As shown in this graph: https://i.imgur.com/C1bPHTX.png , the ultimate cause of the massive process massacre is due to memory excessive memory usage of some processes in the noon of Dec 2. However, I did not see OOM in dmesg. Is there a way to adjust system overall OOM aggressiveness so the kernel will kill the offending processes early, before the kernel become mal-functioning? Many thanks, Yun-Chih
On Tue, 05 Dec 2017 12:03:56 +0800, Yun-Chih Chen said:
2) cat /proc/NNN/stack will give you a hint where the syscall is wedged.
http://ix.io/CMN It seems that they all have something to do with autofs or filesystem.
Fortunately, you've hit an actually fixable problem that you may not need a reboot - you just have a lot of processes waiting for a remote NFS file system to respond. Go find out where the mount is from, and why it's not responding. Fixing that will likely clean up your issue.
On Tuesday, December 5, 2017, <valdis.kletnieks@vt.edu> wrote:
On Tue, 05 Dec 2017 12:03:56 +0800, Yun-Chih Chen said:
2) cat /proc/NNN/stack will give you a hint where the syscall is wedged.
http://ix.io/CMN It seems that they all have something to do with autofs or filesystem.
Fortunately, you've hit an actually fixable problem that you may not need a reboot - you just have a lot of processes waiting for a remote NFS file system to respond. Go find out where the mount is from, and why it's not responding. Fixing that will likely clean up your issue.
If it is indeed coming from nfs mount, check if it is mounted in synchronous mode, and try to switch to async. See if it made any differences Regards, Mulyadi -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
On Wed, 06 Dec 2017 10:07:44 +0700, Mulyadi Santosa said:
If it is indeed coming from nfs mount, check if it is mounted in synchronous mode, and try to switch to async. See if it made any differences
Do note that async mode, due to how it works, *does* open a possibility of data loss - on a write, the remote NFS server can say "done" immediately, but then crash before committing the write to disk. However, being stuck in 'D' state means the remote server didn't reply at all, so that won't help much, especially for already-hung processes. Also: In this case, 'cat /proc/mounts' is your friend - using 'df' to see what's mounted will almost certainly also hang.
participants (3)
-
Mulyadi Santosa -
valdis.kletnieks＠vt.edu -
Yun-Chih Chen