Hello Kernel World, We're facing Kernel Panic on CentOS 7 since upgrading from 3.10.0-957 to 3.10.0-1062. I'm thinking about a java memory leak, but not sure. Do you know what's going on here ? Christophe [cid:image001.png@01D58339.CD24E760]
On Tue, Oct 15, 2019 at 07:21:18AM +0000, Christophe DUMONT wrote:
We're facing Kernel Panic on CentOS 7 since upgrading from 3.10.0-957 to 3.10.0-1062. I'm thinking about a java memory leak, but not sure. Do you know what's going on here ?
Looks similar to this: https://access.redhat.com/solutions/4164501 -- Valentin
On Tue, 15 Oct 2019 07:21:18 -0000, Christophe DUMONT said:
We're facing Kernel Panic on CentOS 7 since upgrading from 3.10.0-957 to 3.10.0-1062. I'm thinking about a java memory leak, but not sure. Do you know what's going on here ?
Well, what made you think "Java memory leak"? Java is userspace. If it's leaking memory so far that the kernel has problems, it would probably: a) Have been leaking memory and causing problems in -957 as well b) Died in the OOM (Out Of Memory) code, rather than in the futex() system call. Yes, poorly written Java code will leak memory like a sieve, but this doesn't smell anything remotely like a memory leak. I agree with Valentin that it's probably the bug report he references.
Hello, What made me think about a memory leak is the message : Java Not Tainted 3.10.0-1062.1.1.el7.x86_64. I can’t access the bug report referenced : an active redhat subscription is needed ! Christophe Dumont Ligne directe : 0476842574 De : Valdis Kletnieks <valdis@vt.edu> De la part de Valdis Kletnieks Envoyé : mardi 15 octobre 2019 22:20 À : Christophe DUMONT <Christophe.DUMONT@yoni.fr> Cc : kernelnewbies@kernelnewbies.org Objet : Re: Kernel Panic On Tue, 15 Oct 2019 07:21:18 -0000, Christophe DUMONT said:
We're facing Kernel Panic on CentOS 7 since upgrading from 3.10.0-957 to 3.10.0-1062. I'm thinking about a java memory leak, but not sure.
Do you know what's going on here ?
Well, what made you think "Java memory leak"? Java is userspace. If it's leaking memory so far that the kernel has problems, it would probably: a) Have been leaking memory and causing problems in -957 as well b) Died in the OOM (Out Of Memory) code, rather than in the futex() system call. Yes, poorly written Java code will leak memory like a sieve, but this doesn't smell anything remotely like a memory leak. I agree with Valentin that it's probably the bug report he references.
On Wed, 16 Oct 2019 07:34:01 -0000, Christophe DUMONT said:
What made me think about a memory leak is the message : Java Not Tainted 3.10.0-1062.1.1.el7.x86_64.=20
That just tells you that the currently executing process was java. It says nothing at all about a memory leak, and as I already mentioned, if Java was leaking memory, it would almost certainly have been leaking memory on a previous kernel. The important part almost always isn't the running process, it's the kernel stack traceback, which in this case has 'futex' scribbled *all* over it. General rule of thumb: If you get more than one crash that has a similar traceback that points at a specific syscall, or file system driver, etc, the bug is almost guaranteed to be in that code. If you get a rash of crashes with *different* tracebacks, you probably have some other code that's overlaying memory.
We can put aside Java Memory Leak. I downgraded to kernel 3.10.0-957.27.2.el7.x86_64, there are no more crashes. The bug comes from ‘futex‘ syscall. What can causes that bug and how can i investigate ? Christophe Dumont Ligne directe : 0476842574 De : Valdis Kletnieks <valdis@vt.edu> De la part de Valdis Kletnieks Envoyé : mercredi 16 octobre 2019 16:17 À : Christophe DUMONT <Christophe.DUMONT@yoni.fr> Cc : kernelnewbies@kernelnewbies.org Objet : Re: Kernel Panic On Wed, 16 Oct 2019 07:34:01 -0000, Christophe DUMONT said:
What made me think about a memory leak is the message : Java Not Tainted 3.10.0-1062.1.1.el7.x86_64.=20
That just tells you that the currently executing process was java. It says nothing at all about a memory leak, and as I already mentioned, if Java was leaking memory, it would almost certainly have been leaking memory on a previous kernel. The important part almost always isn't the running process, it's the kernel stack traceback, which in this case has 'futex' scribbled *all* over it. General rule of thumb: If you get more than one crash that has a similar traceback that points at a specific syscall, or file system driver, etc, the bug is almost guaranteed to be in that code. If you get a rash of crashes with *different* tracebacks, you probably have some other code that's overlaying memory.
On Thu, Oct 17, 2019 at 07:48:49AM +0000, Christophe DUMONT wrote:
We can put aside Java Memory Leak.
I downgraded to kernel 3.10.0-957.27.2.el7.x86_64, there are no more crashes.
The bug comes from ‘futex‘ syscall. What can causes that bug and how can i investigate ?
You can try to get the source for these two versions and compare. If the diff is not too big it might give you some idea what changed... -- Valentin
Hello, We downgraded to older kernel. We still facing the same Kernel Panic with different Kernel. Christophe Dumont Ligne directe : 0476842574 De : Valdis Kletnieks <valdis@vt.edu> De la part de Valdis Kletnieks Envoyé : mercredi 16 octobre 2019 16:17 À : Christophe DUMONT <Christophe.DUMONT@yoni.fr> Cc : kernelnewbies@kernelnewbies.org Objet : Re: Kernel Panic On Wed, 16 Oct 2019 07:34:01 -0000, Christophe DUMONT said:
What made me think about a memory leak is the message : Java Not Tainted 3.10.0-1062.1.1.el7.x86_64.=20
That just tells you that the currently executing process was java. It says nothing at all about a memory leak, and as I already mentioned, if Java was leaking memory, it would almost certainly have been leaking memory on a previous kernel. The important part almost always isn't the running process, it's the kernel stack traceback, which in this case has 'futex' scribbled *all* over it. General rule of thumb: If you get more than one crash that has a similar traceback that points at a specific syscall, or file system driver, etc, the bug is almost guaranteed to be in that code. If you get a rash of crashes with *different* tracebacks, you probably have some other code that's overlaying memory.
On Tue, Oct 22, 2019 at 09:38:59AM +0000, Christophe DUMONT wrote:
Hello,
We downgraded to older kernel. We still facing the same Kernel Panic with different Kernel.
Please contact your vendor for support, they are the only ones that can support you as it's a very odd and old kernel you are using, and you are paying for support for it already. Might as well take advantage of it! good luck! greg k-h
participants (4)
-
Christophe DUMONT -
Greg KH -
Valdis Klētnieks -
Valentin Vidić