Crash when sending a lot of messages through a unix socket
Hi, I am trying to log information from within the kernel with the use of a unix socket (/dev/log). My code is a kprobe handler, so while executing I am atomic and uninterrupted. The crashes begin when I do stress tests against my module, so i.e. more than 50 logs per second. The crash messages I get every time are different (from NULL reference, unable to handle page request etc), so I suppose that something is overwritten and this leads to the crash. I don't want to post the code here because it ~100 LOC but these are my steps: Initialization: call sock_create_kern, with PF_UNIX, SOCK_DGRAM, 0 and the address of my socket, then I memset to 0 my sockaddr_un variable and then I set its sun_family to PF_UNIX and copy the "/dev/log" string to sun_path Logging: I have the message that I want to log in variable called buffer and its length in variable called message_length, I set the msg_name of the msghdr to the address of my sockaddr_un variable, than the msn_namelen to sizeof(sockadrr_un), the msg_iov to an iovec variable which contains the buffer and the message_length values, msg_iovlen to 1 and msg_control, msg_controllen to 0. Then I set the fs to KERNEL_DS, call sock_sendmsg with parameters my socket, the address of my msghdr and the message_length and then I restore the fs that there was before setting it to KERNEL_DS. Please note that my socket, sockadd_un and my buffer are static variables. Also I always use the socket which is initialized once in the Initialization state (function). Do you see any "bad practice" or something that could lead to crashes when stressing with a lot of messages the socket? Thank you in advance! Panos -- http://www.cern.ch/psakkos
Hi :) On Wed, May 2, 2012 at 8:43 PM, Panagiotis Sakkos <panos.sakkos@cern.ch> wrote:
Hi,
I am trying to log information from within the kernel with the use of a unix socket (/dev/log). My code is a kprobe handler, so while executing I am atomic and uninterrupted. The crashes begin when I do stress tests against my module, so i.e. more than 50 logs per second.
Maybe that's the problem......under "rapid" call, kprobe handler must allocate stack for unwinding the call more and more. Not to mention that it began to push reentrancy problem to the surface. Regarding reentrancy, IMO think like this...your handler called once...ok fine, twice, perhaps still fine...but maybe more than 50 per sec, not to mention the possibility that they are executed by more than 1 core at once, perhaps that's the problem. race condition.... -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Hi Mulyadi! I can't see your point, how can I have a race condition while I am atomic? Thanks for your response, Panos ________________________________________ From: Mulyadi Santosa [mulyadi.santosa@gmail.com] Sent: 02 May 2012 19:55 To: Panagiotis Sakkos Cc: kernelnewbies@kernelnewbies.org Subject: Re: Crash when sending a lot of messages through a unix socket Hi :) On Wed, May 2, 2012 at 8:43 PM, Panagiotis Sakkos <panos.sakkos@cern.ch> wrote:
Hi,
I am trying to log information from within the kernel with the use of a unix socket (/dev/log). My code is a kprobe handler, so while executing I am atomic and uninterrupted. The crashes begin when I do stress tests against my module, so i.e. more than 50 logs per second.
Maybe that's the problem......under "rapid" call, kprobe handler must allocate stack for unwinding the call more and more. Not to mention that it began to push reentrancy problem to the surface. Regarding reentrancy, IMO think like this...your handler called once...ok fine, twice, perhaps still fine...but maybe more than 50 per sec, not to mention the possibility that they are executed by more than 1 core at once, perhaps that's the problem. race condition.... -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Hi all, Just FYI, I set the MSG_DONTWAIT flag at my logging socket and now instead of crash I get failure of sending the message (which is much better than crashing...) Thanks, Panos -- http://www.cern.ch/psakkos ________________________________________ From: kernelnewbies-bounces@kernelnewbies.org [kernelnewbies-bounces@kernelnewbies.org] on behalf of Panagiotis Sakkos [panos.sakkos@cern.ch] Sent: 03 May 2012 10:31 To: kernelnewbies@kernelnewbies.org Subject: RE: Crash when sending a lot of messages through a unix socket Hi Mulyadi! I can't see your point, how can I have a race condition while I am atomic? Thanks for your response, Panos ________________________________________ From: Mulyadi Santosa [mulyadi.santosa@gmail.com] Sent: 02 May 2012 19:55 To: Panagiotis Sakkos Cc: kernelnewbies@kernelnewbies.org Subject: Re: Crash when sending a lot of messages through a unix socket Hi :) On Wed, May 2, 2012 at 8:43 PM, Panagiotis Sakkos <panos.sakkos@cern.ch> wrote:
Hi,
I am trying to log information from within the kernel with the use of a unix socket (/dev/log). My code is a kprobe handler, so while executing I am atomic and uninterrupted. The crashes begin when I do stress tests against my module, so i.e. more than 50 logs per second.
Maybe that's the problem......under "rapid" call, kprobe handler must allocate stack for unwinding the call more and more. Not to mention that it began to push reentrancy problem to the surface. Regarding reentrancy, IMO think like this...your handler called once...ok fine, twice, perhaps still fine...but maybe more than 50 per sec, not to mention the possibility that they are executed by more than 1 core at once, perhaps that's the problem. race condition.... -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
participants (2)
-
Mulyadi Santosa -
Panagiotis Sakkos