Notify special task kill using wait* functions
Hi, I'm working in a LSM whose goal is to detect and mitigate fork brute force attacks against vulnerable userspace applications. The detection and mitigation works as expected by I'm stuck at this point. The mitigation method used is to kill all the offending tasks involved in the attack. To do so, I kill the tasks using: do_send_sig_info(SIGKILL, SEND_SIG_PRIV, p, PIDTYPE_PID); The question is: How can I notify to wait* functions that the task has been killed by the "Brute" LSM. For example, in the function wait_task_zombie, in the "out_info:" label, the code is the following: out_info: infop = wo->wo_info; if (infop) { if ((status & 0x7f) == 0) { infop->cause = CLD_EXITED; infop->status = status >> 8; } else { infop->cause = (status & 0x80) ? CLD_DUMPED : CLD_KILLED; infop->status = status & 0x7f; } infop->pid = pid; infop->uid = uid; } I think I need to modify this code to achieve the commented goal. But I don't know how to proceed. Is it possible to kill a task in a way that this code can detect ? Now, with this code, we know that a task has been killed and with what signal. But is it possible to know that the task has been killed by the "Brute" LSM. Using for example the 8 upper bits in the status that I think are unused when a process is killed or dumped. It is possible to use the do_send_sig_info passing to the struct kernel_siginfo the necessary information to be able to detect this scenario in the wait* functions? If yes, what info I need to pass? The final purpose of all of this is that using the waitid function from userspace (or waitpid function) we can know that a child task has been killed by the "Brute" LSM or not. I try to inform to userspace that a task has been killed due to the "Brute" mitigation. Sorry, but I'm stuck at this point. Any help would be greatly appreciated. Regards, John Wood
On Tue, 30 Mar 2021 19:34:59 +0200, John Wood said:
The question is: How can I notify to wait* functions that the task has been killed by the "Brute" LSM.
What wait* functions even *care* that your LSM was what killed it? If you're caring about somehow notifying userspace that it was your LSM specifically, remember that if your code works properly, only attackers get notified - and they can then determine "Ah, this system has Brute installed, we need to back off and fly under its radar". You're much better off sending a SIGKILL to the entire process group and be done with it. That way the bad guys get less information.
Hi, On Tue, Mar 30, 2021 at 02:40:38PM -0400, Valdis Klētnieks wrote:
On Tue, 30 Mar 2021 19:34:59 +0200, John Wood said:
The question is: How can I notify to wait* functions that the task has been killed by the "Brute" LSM.
What wait* functions even *care* that your LSM was what killed it?
If you're caring about somehow notifying userspace that it was your LSM specifically, remember that if your code works properly, only attackers get notified - and they can then determine "Ah, this system has Brute installed, we need to back off and fly under its radar".
You're much better off sending a SIGKILL to the entire process group and be done with it. That way the bad guys get less information.
Thanks for the suggestion, but I will expose more info to try to clarify why to notify to userspace can be useful. In a discussion with Andi Kleen in the v5 review [1] he explain me some cons with the current mitiggation method. Without entering in more detail, the mitigation kills all the tasks involved in the attack, but a supervisor can respawn the processes killed and the attack can be started again. So, he suggested that notifying to userspace (via wait*() functions) that a child task has been killed by the "Brute" LSM, the supervisor can adopt the correct policy and avoid respawn the killed processes. [1] https://lore.kernel.org/kernel-hardening/20210227153013.6747-8-john.wood@gmx... Thanks, John Wood
On Fri, 02 Apr 2021 14:49:32 +0200, John Wood said:
the attack can be started again. So, he suggested that notifying to userspace (via wait*() functions) that a child task has been killed by the "Brute" LSM, the supervisor can adopt the correct policy and avoid respawn the killed processes.
[1] https://lore.kernel.org/kernel-hardening/20210227153013.6747-8-john.wood@gmx...
That patch contains the biggest problem with your idea: +Moreover, this method is based on the idea that the protection doesn't act if +the parent crashes. So, it would still be possible for an attacker to fork a +process and probe itself. Then, fork the child process and probe itself again. +This way, these steps can be repeated infinite times without any mitigation. In general, "security" that has an obvious and easy way to bypass it isn't providing any real security at all. If all it takes to bypass it is a double fork, everybody who didn't just fall out of the tree will do a double fork. In other words, anybody who's clued enough to write malware that actually works and does the sort of attack you're trying to prevent should be able to fix the malware to bypass your "security" with just a few added lines of code.
Hi, On Fri, Apr 02, 2021 at 11:50:18PM -0400, Valdis Klētnieks wrote:
On Fri, 02 Apr 2021 14:49:32 +0200, John Wood said:
the attack can be started again. So, he suggested that notifying to userspace (via wait*() functions) that a child task has been killed by the "Brute" LSM, the supervisor can adopt the correct policy and avoid respawn the killed processes.
[1] https://lore.kernel.org/kernel-hardening/20210227153013.6747-8-john.wood@gmx...
That patch contains the biggest problem with your idea:
+Moreover, this method is based on the idea that the protection doesn't act if +the parent crashes. So, it would still be possible for an attacker to fork a +process and probe itself. Then, fork the child process and probe itself again. +This way, these steps can be repeated infinite times without any mitigation.
Sorry, but this paragraph is extracted from the "Other implementations" section (grsecurity to be more clear). Concreatly from the "Scenarios not detected (false negatives)". So, it is a problem that has been addressed with the current implementation. In other words, the "Brute LSM" acts on the parent (brute force attack through the execve system call).
In general, "security" that has an obvious and easy way to bypass it isn't providing any real security at all. If all it takes to bypass it is a double fork, everybody who didn't just fall out of the tree will do a double fork. In other words, anybody who's clued enough to write malware that actually works and does the sort of attack you're trying to prevent should be able to fix the malware to bypass your "security" with just a few added lines of code.
Currently, the scenario you propose is fully mitigated :). And notifying to userspace that all the tasks has been killed by "Brute" not decrease the security. It adds the possibility that the supervisor adopts the correct policy. If you can bypass it send a proof of concept and I will try to improve the implementation :) Thanks, John Wood
On Sat, 03 Apr 2021 09:02:26 +0200, John Wood said:
Currently, the scenario you propose is fully mitigated :). And notifying to userspace that all the tasks has been killed by "Brute" not decrease the security. It adds the possibility that the supervisor adopts the correct policy.
So how do you figure out how far up the chain you kill processes? What does a triple or quadruple fork do? How do you ensure that you deliver the notification to the correct supervisor? (Hint - walking backwards until you hit a process running as root isn't necessarily the right answer, especially once you consider containers and systems where gdm runs as non-root and other weird stuff..) Bonus points if you deal correctly with abuse of LD_PRELOAD to front-end a signal handler that catches SIGSEGV, without breaking the semantics of legitimate signal handlers...
Hi, On Sat, Apr 03, 2021 at 05:34:01PM -0400, Valdis Klētnieks wrote:
On Sat, 03 Apr 2021 09:02:26 +0200, John Wood said:
Currently, the scenario you propose is fully mitigated :). And notifying to userspace that all the tasks has been killed by "Brute" not decrease the security. It adds the possibility that the supervisor adopts the correct policy.
So how do you figure out how far up the chain you kill processes?
When a process crashes with a fatal signal their stats are updated. Also, are updated the "exec" stats (the statistics of the parent process: the process that called the execve system call). Then the stats of the process that crashes are checked. If these show a fast crash rate, this fork hierarchy is killed (all the processes that have the same stats). Then the exec stats are checked but if the info are equal to the fork stats nothing is done. Otherwise, if they show an attack, this fork hierarchy is also killed. If you follow the discussion with Andi Kleen there is a full example. [1] [1] https://lore.kernel.org/kernel-hardening/20210309184054.GA3058@ubuntu/ Now, the process that forks and crashes is mitigated. Also, the process that exec and the child crashes is mitigated. The only drawback here, as point Andi, is that a supervisor respawns some process killed. To avoid this situation he suggest to notify to usersapace via wait* that the task has been killed by "brute". Then, this supervisor has the chance to do what they need (respawn or not).
What does a triple or quadruple fork do? How do you ensure that you deliver the notification to the correct supervisor? (Hint - walking backwards until you hit a process running as root isn't necessarily the right answer, especially once you consider containers and systems where gdm runs as non-root and other weird stuff..)
The number of forks does not matter. It's not a problem here. And, if you follow the annotated discussion, you will see that walking backwards not necessary reach init. The notification is delivered to the supervisor that wants it (via wait).
Bonus points if you deal correctly with abuse of LD_PRELOAD to front-end a signal handler that catches SIGSEGV, without breaking the semantics of legitimate signal handlers...
Hmm, interesting... I need to study it and test. Good point. Thanks. Thanks, John Wood
On Sun, 04 Apr 2021 11:48:37 +0200, John Wood said:
that exec and the child crashes is mitigated. The only drawback here, as point Andi, is that a supervisor respawns some process killed. To avoid this situation he suggest to notify to usersapace via wait* that the task has been killed by "brute". Then, this supervisor has the chance to do what they need (respawn or not).
And how does the kernel know that it's notifying a "real" supervisor process, and not a process started by the bad guy, who can receive the notification and decide to respawn?
Hi, On Sun, Apr 04, 2021 at 05:10:25PM -0400, Valdis Klētnieks wrote:
On Sun, 04 Apr 2021 11:48:37 +0200, John Wood said:
that exec and the child crashes is mitigated. The only drawback here, as point Andi, is that a supervisor respawns some process killed. To avoid this situation he suggest to notify to usersapace via wait* that the task has been killed by "brute". Then, this supervisor has the chance to do what they need (respawn or not).
And how does the kernel know that it's notifying a "real" supervisor process, and not a process started by the bad guy, who can receive the notification and decide to respawn?
Well, I think this is not possible to know. Anyway, I believe that the "bad guy" not rely on the wait* notification to decide to respawn or not. He will do the attack without waiting any notification. Anyway thanks a lot for this new point of view. Thanks, John Wood
On Mon, 05 Apr 2021 09:31:47 +0200, John Wood said:
And how does the kernel know that it's notifying a "real" supervisor process, and not a process started by the bad guy, who can receive the notification and decide to respawn?
Well, I think this is not possible to know. Anyway, I believe that the "bad guy" not rely on the wait* notification to decide to respawn or not. He will do the attack without waiting any notification.
You believe wrong. After my 4 decades of interacting with the computer security community, the only thing that remains a constant is that if you say "I believe that...", there will be *somebody* who will say "Challenge accepted" and try to do the opposite just for the lulz. Then there will be a second guy saying "Hmm.. I wonder how much I could sell a 0-day for..." If you provide a way for an attacker to "fly under the radar" (either by having a hardcoded limit of SYSSEGV/minute that they can carefully limit themselves to, or by letting them set up a "supervisor" process they can abuse, or any other method), attackers *will* use it to prevent being detected. That's the thing about computer security - you have to keep asking yourself "how could the attacker abuse feature X to their benefit?" It's probably *not* even safe to go and kill *all* processes running under the same UID - because if you do that, and a code execution bug is found in the web server software (or back-end stuff launched by it), you just provided an attacker a free DoS of the webserver. Remember - your attacker is somebody who can take a 1-byte buffer overflow, and convert it into a complete root compromise of a system If you think I'm kidding, go look at this paper that analyzes how to exploit a bug in ntpd to get yourself a root shell from a remote system (or whatever other code you want to run): https://www.giac.org/paper/gcih/352/linux-ntpd-buffer-overflow/102270 Of course, that bug was in 2002, and the author had to hand-craft a lot of the support framework. These days, the attacker would probably just craft a module for Metasploit from the team at Rapid7 or other attack tool. Yes, there's open-source exploit tools out there... See https://metasploit.com/ - or at least the YouTube demo https://www.youtube.com/watch?v=cYtDxfKdlqs Make note of how many Windows versions they tested against in the video. And if you don't watch, here's the backstory: A crew call Shadowbroker hacked the NSA and stole a huge collection of exploit tools and dumped them into the public. Somebody else took one of the exploit tools, figured out what it was doing, and tossed a module over to the Metasploit crew - and now there's an automated "type 3 lines to pwn the box" that's almost certainly easier to use than the NSA version.... Now be glad that the guys at GIAC and Rapid7 are the good guys - but remember that the black hats are at least as good, and have toolkits at least as good...
Hi Valdis, On Tue, Apr 06, 2021 at 07:55:36PM -0400, Valdis Klētnieks wrote:
On Mon, 05 Apr 2021 09:31:47 +0200, John Wood said:
And how does the kernel know that it's notifying a "real" supervisor process, and not a process started by the bad guy, who can receive the notification and decide to respawn?
Well, I think this is not possible to know. Anyway, I believe that the "bad guy" not rely on the wait* notification to decide to respawn or not. He will do the attack without waiting any notification.
You believe wrong. After my 4 decades of interacting with the computer security community, the only thing that remains a constant is that if you say "I believe that...", there will be *somebody* who will say "Challenge accepted" and try to do the opposite just for the lulz. Then there will be a second guy saying "Hmm.. I wonder how much I could sell a 0-day for..."
Ok, lesson learned. I agree.
[Great explanation and information]
Wow, I'm impressed. Thank you very much for this great explanation and info. Thanks a lot for do that (insist about this subject). During the discussion [1] you made me realize that I'm totally wrong (and you are totally right :) ). The detection of brute force attacks that happen through the execve system call can be easily bypassed -> Well, I bypass it during the tests using a double exec. So, this part needs more work. [1] https://lore.kernel.org/kernelnewbies/20210330173459.GA3163@ubuntu/ A first thought: Scenario: A process [p1] execs. The child [p2] execs again. The child [p3] crashes. Problem: The brute LSM kills p3 if it forks and crashes with a fast crash rate (fork brute force attack). But the p2 process can start again the p3. Then brute kills p2 (exec brute force attack). Now, if p1 starts p2 the attack can follow without mitigation. New proposal: When brute detects a brute force attack through the fork system call (killing p3) it will mark the binary file executed by p3 as "not allowed". From now on, any execve that try to run this binary will fail. This way it is not necessary to notify nothing to userspace and also we avoid an exec brute force attack due to the respawn of processes [2] by a supervisor (abused or not by a bad guy). [2] https://lore.kernel.org/kernel-hardening/878s78dnrm.fsf@linux.intel.com/ This would imply remove the update (walking up in the processes tree) of the exec stats and add a list of not allowed binaries. What do you think? Any ideas are welcome. I'm open minded :) Regards, John Wood
On Wed, 07 Apr 2021 19:51:51 +0200, John Wood said:
When brute detects a brute force attack through the fork system call (killing p3) it will mark the binary file executed by p3 as "not allowed". From now on, any execve that try to run this binary will fail. This way it is not necessary to notify nothing to userspace and also we avoid an exec brute force attack due to the respawn of processes [2] by a supervisor (abused or not by a bad guy).
You're not thinking evil enough. :) I didn't even finish the line that starts "From now on.." before I started wondering "How can I abuse this to hang or crash a system?" And it only took me a few seconds to come up with an attack. All you need to do is find a way to sigsegv /bin/bash... and that's easy to do by forking, excecve /bin/bash, and then use ptrace() to screw the child process's stack and cause a sigsegv. Say goodnight Gracie...
I didn't even finish the line that starts "From now on.." before I started wondering "How can I abuse this to hang or crash a system?" And it only took me a few seconds to come up with an attack. All you need to do is find a way to sigsegv /bin/bash... and that's easy to do by forking, excecve /bin/bash, and then use ptrace() to screw the child process's stack and cause a sigsegv.
Say goodnight Gracie...
Yes there is certainly DoS potential, but that's kind of inevitable for the proposal. It's a trade between allowing attacks and allowing DoS, with the idea that a DoS is more benign. I'm more worried that it doesn't actually prevent the attacks unless we make sure systemd and other supervisor daemons understand it, so that they don't restart. Any caching of state is inherently insecure because any caches of limited size can be always thrashed by a purposeful attacker. I suppose the only thing that would work is to actually write something to the executable itself on disk, but of course that doesn't always work either. -Andi
Hi Valdis and Andi. Thanks for your comments. On Wed, Apr 07, 2021 at 06:51:48PM -0700, Andi Kleen wrote:
I didn't even finish the line that starts "From now on.." before I started wondering "How can I abuse this to hang or crash a system?" And it only took me a few seconds to come up with an attack. All you need to do is find a way to sigsegv /bin/bash... and that's easy to do by forking, excecve /bin/bash, and then use ptrace() to screw the child process's stack and cause a sigsegv.
Say goodnight Gracie...
Yes there is certainly DoS potential, but that's kind of inevitable for the proposal. It's a trade between allowing attacks and allowing DoS, with the idea that a DoS is more benign.
I'm more worried that it doesn't actually prevent the attacks unless we make sure systemd and other supervisor daemons understand it, so that they don't restart.
I'm working on it. I will send a formal proposal in the next version.
Any caching of state is inherently insecure because any caches of limited size can be always thrashed by a purposeful attacker. I suppose the only thing that would work is to actually write something to the executable itself on disk, but of course that doesn't always work either.
I'm also working on this. In the next version I will try to find a way to prevent brute force attacks through the execve system call with more than one level of forking.
-Andi
Again, thank you very much. John Wood
Any caching of state is inherently insecure because any caches of limited size can be always thrashed by a purposeful attacker. I suppose the only thing that would work is to actually write something to the executable itself on disk, but of course that doesn't always work either.
I'm also working on this. In the next version I will try to find a way to prevent brute force attacks through the execve system call with more than one level of forking.
Thanks. Thinking more about it what I wrote above wasn't quite right. The cache would only need to be as big as the number of attackable services/suid binaries. Presumably on many production systems that's rather small, so a cache (which wouldn't actually be a cache, but a complete database) might actually work. -Andi
Hi, On Fri, Apr 09, 2021 at 08:06:21AM -0700, Andi Kleen wrote:
Any caching of state is inherently insecure because any caches of limited size can be always thrashed by a purposeful attacker. I suppose the only thing that would work is to actually write something to the executable itself on disk, but of course that doesn't always work either.
I'm also working on this. In the next version I will try to find a way to prevent brute force attacks through the execve system call with more than one level of forking.
Thanks.
Thinking more about it what I wrote above wasn't quite right. The cache would only need to be as big as the number of attackable services/suid binaries. Presumably on many production systems that's rather small, so a cache (which wouldn't actually be a cache, but a complete database) might actually work.
Thanks. I will keep it in mind.
-Andi
Regards, John Wood
On Fri, 09 Apr 2021 08:06:21 -0700, Andi Kleen said:
Thinking more about it what I wrote above wasn't quite right. The cache would only need to be as big as the number of attackable services/suid binaries. Presumably on many production systems that's rather small, so a cache (which wouldn't actually be a cache, but a complete database) might actually work.
You also need to consider non-suid things called by suid things that don't sanitize input sufficiently before invocation... Thinking about at - is it really a good thing to try to do this in kernelspace? Or is 'echo 1 > /proc/sys/kernel/print-fatal-signals' and a program to watch the dmesg and take action more appropriate? A userspace monitor would have more options (though a slightly higher risk of race conditions).
Hi, On Fri, Apr 09, 2021 at 07:28:20PM -0400, Valdis Klētnieks wrote:
On Fri, 09 Apr 2021 08:06:21 -0700, Andi Kleen said:
Thinking more about it what I wrote above wasn't quite right. The cache would only need to be as big as the number of attackable services/suid binaries. Presumably on many production systems that's rather small, so a cache (which wouldn't actually be a cache, but a complete database) might actually work.
You also need to consider non-suid things called by suid things that don't sanitize input sufficiently before invocation...
Thinking about at - is it really a good thing to try to do this in kernelspace? Or is 'echo 1 > /proc/sys/kernel/print-fatal-signals' and a program to watch the dmesg and take action more appropriate? A userspace monitor would have more options (though a slightly higher risk of race conditions).
Thanks for the ideas. I need some time to send a formal proposal that works properly. I would like to get feedback at that moment. I think it would be better to discuss about the real patch. Again, thanks. John Wood
participants (3)
-
Andi Kleen -
John Wood -
Valdis Klētnieks