[PATCH 0/4] Help to debug spinlocks
John Wood
john.wood at gmx.com
Sat Dec 19 06:46:37 EST 2020
Hi,
I'm working in a new LSM to detect and mitigate any fork brute force
attack against vulnerable userspace processes. I'm testing the detection
method but I have found some problems that I think are related to locking
since the kernel gets stuck but not crashes. This work is a WIP to obtain
the v3 version. The mitigation, documentation and fine tunning detection
are under construction.
My problem is that I don't be able to find the cause of this behaviour and
any help would be greatly appreciated.
To test this feature I use the following userspace program:
#include <stdio.h>
int main(void)
{
int *p = 0;
*p = 0;
return 0;
}
This program triggers a "Segmentation fault" that is what I want. Then I
run the binary multiple times to obtain many faults. The method used are
the following commands wrote directly in the shell:
while :
do
./test
done
But at this moment the kernel gets stuck and any message is shown. On one
occasion I got the following message.
[ 200.447700] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [test:277]
[ 200.450553] Modules linked in:
[ 200.451208] irq event stamp: 0
[ 200.451868] hardirqs last enabled at (0): [<0000000000000000>] 0x0
[ 200.453186] hardirqs last disabled at (0): [<ffffffffb789cd6b>] copy_process+0x6bb/0x1c40
[ 200.455230] softirqs last enabled at (0): [<ffffffffb789cd6b>] copy_process+0x6bb/0x1c40
[ 200.457316] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 200.458853] CPU: 0 PID: 277 Comm: test Not tainted 5.10.0+ #98
[ 200.460320] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1 04/01/2014
[ 200.462735] RIP: 0010:queued_write_lock_slowpath+0x50/0x90
[ 200.464402] Code: 0d ba ff 00 00 00 3e 0f b1 13 85 c0 74 33 3e 81 03 00 01 00 00 b9 ff 00 00 00 be 00 01 00 00 8b 03 3d 00 01 00 00 74 0c 5
[ 200.469109] RSP: 0000:ffffaed4c0003e38 EFLAGS: 00000206
[ 200.470191] RAX: 0000000000000300 RBX: ffffffffb92dc7e0 RCX: 00000000000000ff
[ 200.471658] RDX: 0000000000000300 RSI: 0000000000000100 RDI: ffffffffb92dc7e0
[ 200.473106] RBP: ffffaed4c0003e48 R08: 0000000000000001 R09: 0000000000000000
[ 200.474625] R10: ffffffffb92dc7f8 R11: 0000000000000000 R12: ffffffffb92dc7e4
[ 200.476410] R13: ffffffffb92dc7f8 R14: ffff8d14c04a2380 R15: ffff8d14c0c8c2d0
[ 200.478179] FS: 00007f3384f5a500(0000) GS:ffff8d14c7800000(0000) knlGS:0000000000000000
[ 200.480313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 200.481826] CR2: 0000000000000000 CR3: 00000000043a0000 CR4: 00000000000006f0
[ 200.483698] Call Trace:
[ 200.484481] <IRQ>
[ 200.485141] do_raw_write_lock+0xae/0xb0
[ 200.486265] _raw_write_lock+0x6c/0x70
[ 200.487366] brute_task_free+0x86/0xf0
[ 200.488477] security_task_free+0x27/0x50
[ 200.489657] __put_task_struct+0x6d/0x150
[ 200.490824] delayed_put_task_struct+0x9b/0x110
[ 200.492142] rcu_core+0x412/0x6d0
[ 200.493113] ? rcu_core+0x3de/0x6d0
[ 200.493864] rcu_core_si+0xe/0x10
[ 200.494568] __do_softirq+0xcf/0x428
[ 200.495325] asm_call_irq_on_stack+0x12/0x20
[ 200.496407] </IRQ>
[ 200.496969] do_softirq_own_stack+0x61/0x70
[ 200.498030] irq_exit_rcu+0xc1/0xd0
[ 200.498913] sysvec_apic_timer_interrupt+0x52/0xb0
[ 200.500179] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 200.501773] RIP: 0010:vprintk_emit+0x134/0x3a0
[ 200.502906] Code: 89 f9 4c 89 f2 44 89 ef e8 b9 fc ff ff 48 c7 c7 e0 ca 15 b9 41 89 c4 e8 3a 1e b2 00 e8 e5 0e 00 00 4c 8b 4d c8 4c 89 cf 3
[ 200.507074] RSP: 0000:ffffaed4c0c63c60 EFLAGS: 00000246
[ 200.508400] RAX: ffffaed4c0c63ca0 RBX: ffffaed4c0c63ce8 RCX: 0000000000000a17
[ 200.510198] RDX: 000000000000002e RSI: ffffffffb7934e26 RDI: 0000000000000246
[ 200.511942] RBP: ffffaed4c0c63ca0 R08: 0000000000000000 R09: 0000000000000246
[ 200.513799] R10: 0000000000000001 R11: 0000000000000000 R12: 000000000000002e
[ 200.515593] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffb8dcb218
[ 200.517388] ? vprintk_emit+0x1b6/0x3a0
[ 200.518403] ? lock_acquire+0x1ae/0x3b0
[ 200.519306] vprintk_default+0x1d/0x20
[ 200.520088] vprintk_func+0x68/0x120
[ 200.520845] ? _raw_spin_unlock_irqrestore+0x47/0x50
[ 200.521904] printk+0x58/0x6f
[ 200.522537] brute_task_fatal_signal+0x1ed/0x210
[ 200.523463] security_task_fatal_signal+0x27/0x40
[ 200.524408] get_signal+0x176/0xc70
[ 200.525122] arch_do_signal+0x34/0x8f0
[ 200.525902] ? force_sig_fault+0x63/0x80
[ 200.526710] ? trace_hardirqs_off+0x13/0xd0
[ 200.527549] exit_to_user_mode_prepare+0x155/0x200
[ 200.528517] irqentry_exit_to_user_mode+0x9/0x30
[ 200.529463] irqentry_exit+0x5e/0x80
[ 200.530232] exc_page_fault+0xad/0x2a0
[ 200.530989] ? asm_exc_page_fault+0x8/0x30
[ 200.531816] asm_exc_page_fault+0x1e/0x30
[ 200.532629] RIP: 0033:0x564ce0c6b13d
[ 200.533394] Code: 5d c3 0f 1f 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa e9 77 ff ff ff f3 0f 1e fa 55 48 89 e5 48 c7 45 f8 00 00 00 00 48 8b e
[ 200.538044] RSP: 002b:00007ffc2423f3b0 EFLAGS: 00010246
[ 200.539210] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f3384f53718
[ 200.541326] RDX: 00007ffc2423f4a8 RSI: 00007ffc2423f498 RDI: 0000000000000001
[ 200.543111] RBP: 00007ffc2423f3b0 R08: 00007f3384f54d80 R09: 00007f3384f54d80
[ 200.544538] R10: 0000000000000000 R11: 00007f3384f15188 R12: 0000564ce0c6b040
[ 200.546349] R13: 00007ffc2423f490 R14: 0000000000000000 R15: 0000000000000000
I don't have any experience debugging errors caused by locking and I don't
know how to proceed.
I turn on the following options in my .config file but during the test it
doesn't appear any "DEADLOCK" warning messages.
#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_LOCK_DEBUGGING_SUPPORT=y
CONFIG_PROVE_LOCKING=y
CONFIG_PROVE_RAW_LOCK_NESTING=y
CONFIG_LOCK_STAT=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y
CONFIG_DEBUG_RWSEMS=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_LOCKDEP=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
# CONFIG_WW_MUTEX_SELFTEST is not set
# CONFIG_SCF_TORTURE_TEST is not set
# CONFIG_CSD_LOCK_WAIT_DEBUG is not set
# end of Lock Debugging (spinlocks, mutexes, etc...)
I also send my work to show the code used. Any help that points me to the
right direction would be greatly appreciated.
Thanks a lot.
John Wood (4):
security: Add LSM hook at the point where a task gets a fatal signal
security/brute: Define a LSM and manage statistical data
securtiy/brute: Detect a brute force attack
Documentation: Add documentation for the Brute LSM
Documentation/admin-guide/LSM/Brute.rst | 186 +++++++
Documentation/admin-guide/LSM/index.rst | 1 +
include/linux/lsm_hook_defs.h | 1 +
include/linux/lsm_hooks.h | 4 +
include/linux/security.h | 4 +
kernel/signal.c | 1 +
security/Kconfig | 11 +-
security/Makefile | 4 +
security/brute/Kconfig | 13 +
security/brute/Makefile | 2 +
security/brute/brute.c | 705 ++++++++++++++++++++++++
security/security.c | 5 +
12 files changed, 932 insertions(+), 5 deletions(-)
create mode 100644 Documentation/admin-guide/LSM/Brute.rst
create mode 100644 security/brute/Kconfig
create mode 100644 security/brute/Makefile
create mode 100644 security/brute/brute.c
--
2.25.1
More information about the Kernelnewbies
mailing list