Trying to understand oom-killer (output)
Sasa Vilic
sasavilic at gmail.com
Thu Feb 3 13:37:26 EST 2022
Hi,
this is my first email to LKML, so please forgive me if this is not a
proper mailing list. I have been puzzled by some oom-kill which I
don't understand. I invested a substantial amount of time trying to
understand oom-killer output, but the numbers just don't add up (I am
sure that I am missing something). You can find oom-killer log output
at the end of email below, here is my analysis so far:
[Don Feb 3 10:26:18 2022] kube-proxy invoked oom-killer:
gfp_mask=0x3000d0, order=2, oom_score_adj=-999
[Don Feb 3 10:26:18 2022] kube-proxy cpuset=kube-proxy mems_allowed=0
[Don Feb 3 10:26:18 2022] CPU: 2 PID: 3780 Comm: kube-proxy Kdump:
loaded Not tainted 3.10.0-957.21.3.el7.x86_64 #1
Kube proxy is triggering oom-killer, by requesting 16kb of memory
(order=2 => 2^2 = 4 pages = 16kb), right?
Trying to decode gfp_mask=0x3000d0
0x200000 #define ___GFP_HARDWALL 0x20000u
0x100000 #define ___GFP_NOMEMALLOC 0x10000u
0x80 #define ___GFP_FS 0x80u
0x40 #define ___GFP_IO 0x40u
0x10 #define ___GFP_WAIT 0x10u
tells me nothing so far.
[Don Feb 3 10:26:18 2022] 1042927 total pagecache pages
[Don Feb 3 10:26:18 2022] 0 pages in swap cache
[Don Feb 3 10:26:18 2022] Swap cache stats: add 0, delete 0, find 0/0
[Don Feb 3 10:26:18 2022] Free swap = 0kB
[Don Feb 3 10:26:18 2022] Total swap = 0kB
[Don Feb 3 10:26:18 2022] 20971384 pages RAM
[Don Feb 3 10:26:18 2022] 0 pages HighMem/MovableOnly
[Don Feb 3 10:26:18 2022] 390901 pages reserved
We got around 80 GiB of RAM (20971384 pages) or of which 1.5 GiB is
reserved (390901). (reserved for the kernel itself I suppose, right?)
Swap is disabled.
We have 4 GiB used by page cache (but I suppose that those page cache
could be freed if needed?).
Looking at Normal pool:
[Don Feb 3 10:26:18 2022] Node 0 Normal free:69016kB min:65192kB
low:81488kB high:97788kB active_anon:44389844kB
inactive_anon:1293312kB active_file:20040kB inactive_file:32380kB
unevictable:44436kB isolated(anon):0kB isolated(file):564kB
present:80740352kB managed:79411360kB mlocked:44448kB dirty:0kB
writeback:20kB mapped:44412kB shmem:4010304kB
slab_reclaimable:18049652kB slab_unreclaimable:14413248kB
kernel_stack:51536kB pagetables:110548kB unstable:0kB bounce:0kB
free_pcp:2392kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:37 all_unreclaimable? no
[Don Feb 3 10:26:18 2022] lowmem_reserve[]: 0 0 0 0
[Don Feb 3 10:26:18 2022] Node 0 Normal: 8989*4kB (UEM) 3771*8kB (UE)
118*16kB (EM) 40*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 69292kB
There is around 4 MiB available before minimum is reached. There is
some fragmentation, but we still got 118 blocks of 16 kB. Therefore
the memory allocation request should not fail.
[Don Feb 3 10:26:18 2022] Mem-Info:
[Don Feb 3 10:26:18 2022] active_anon:11334512 inactive_anon:331655
isolated_anon:0
active_file:5010 inactive_file:8095 isolated_file:141
unevictable:11170 dirty:0 writeback:5 unstable:0
slab_reclaimable:4728777 slab_unreclaimable:3779454
mapped:11225 shmem:1027917 pagetables:28308 bounce:0
free:99164 free_pcp:598 free_cma:0
There are 18 GiB (4728777 pages) used by the kernel that could be
apparently reclaimed, but it is not. Why is the kernel not reclaiming
that space?
There are 14 GiB (3779454 pages) used by the kernel that could not be
reclaimed. Why is my kernel taking so much space?
If I sum up all pages listed in the rss column for each process, I get
40 GiB (10651521 pages). I suppose that the remaining memory is used
by the kernel (18 + 14 GiB), right? What could be the reason for the
kernet to consume so much memory? Could too many threads explain this
behavior?
I still don't get why that 16 KiB allocation did not succeed.
I would appreciate any help that I can get. Thank you in advance!
Here is the whole oom-killer output:
[Don Feb 3 10:26:18 2022] kube-proxy invoked oom-killer:
gfp_mask=0x3000d0, order=2, oom_score_adj=-999
[Don Feb 3 10:26:18 2022] kube-proxy cpuset=kube-proxy mems_allowed=0
[Don Feb 3 10:26:18 2022] CPU: 2 PID: 3780 Comm: kube-proxy Kdump:
loaded Not tainted 3.10.0-957.21.3.el7.x86_64 #1
[Don Feb 3 10:26:18 2022] Hardware name: QEMU Standard PC (i440FX +
PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org
04/01/2014
[Don Feb 3 10:26:18 2022] Call Trace:
[Don Feb 3 10:26:18 2022] [<ffffffffa2363107>] dump_stack+0x19/0x1b
[Don Feb 3 10:26:18 2022] [<ffffffffa235db2a>] dump_header+0x90/0x229
[Don Feb 3 10:26:18 2022] [<ffffffffa1d01292>] ? ktime_get_ts64+0x52/0xf0
[Don Feb 3 10:26:18 2022] [<ffffffffa1d584df>] ? delayacct_end+0x8f/0xb0
[Don Feb 3 10:26:18 2022] [<ffffffffa1dba834>] oom_kill_process+0x254/0x3d0
[Don Feb 3 10:26:18 2022] [<ffffffffa1dba2dd>] ?
oom_unkillable_task+0xcd/0x120
[Don Feb 3 10:26:18 2022] [<ffffffffa1dba386>] ? find_lock_task_mm+0x56/0xc0
[Don Feb 3 10:26:18 2022] [<ffffffffa1dbb076>] out_of_memory+0x4b6/0x4f0
[Don Feb 3 10:26:18 2022] [<ffffffffa235e62e>]
__alloc_pages_slowpath+0x5d6/0x724
[Don Feb 3 10:26:18 2022] [<ffffffffa1dc1454>]
__alloc_pages_nodemask+0x404/0x420
[Don Feb 3 10:26:18 2022] [<ffffffffa1c94dfd>] copy_process+0x1dd/0x1a40
[Don Feb 3 10:26:18 2022] [<ffffffffa1c96811>] do_fork+0x91/0x320
[Don Feb 3 10:26:18 2022] [<ffffffffa23709d6>] ?
trace_do_page_fault+0x56/0x150
[Don Feb 3 10:26:18 2022] [<ffffffffa1c96b26>] SyS_clone+0x16/0x20
[Don Feb 3 10:26:18 2022] [<ffffffffa23761b4>] stub_clone+0x44/0x70
[Don Feb 3 10:26:18 2022] [<ffffffffa2375ddb>] ?
system_call_fastpath+0x22/0x27
[Don Feb 3 10:26:18 2022] Mem-Info:
[Don Feb 3 10:26:18 2022] active_anon:11334512 inactive_anon:331655
isolated_anon:0
active_file:5010 inactive_file:8095 isolated_file:141
unevictable:11170 dirty:0 writeback:5 unstable:0
slab_reclaimable:4728777 slab_unreclaimable:3779454
mapped:11225 shmem:1027917 pagetables:28308 bounce:0
free:99164 free_pcp:598 free_cma:0
[Don Feb 3 10:26:18 2022] Node 0 DMA free:15892kB min:12kB low:12kB
high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB
dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB
bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[Don Feb 3 10:26:18 2022] lowmem_reserve[]: 0 2826 80374 80374
[Don Feb 3 10:26:18 2022] Node 0 DMA32 free:311376kB min:2376kB
low:2968kB high:3564kB active_anon:948204kB inactive_anon:33308kB
active_file:0kB inactive_file:0kB unevictable:244kB isolated(anon):0kB
isolated(file):0kB present:3129192kB managed:2894664kB mlocked:244kB
dirty:0kB writeback:0kB mapped:488kB shmem:101364kB
slab_reclaimable:865456kB slab_unreclaimable:704552kB
kernel_stack:3424kB pagetables:2684kB unstable:0kB bounce:0kB
free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[Don Feb 3 10:26:18 2022] lowmem_reserve[]: 0 0 77547 77547
[Don Feb 3 10:26:18 2022] Node 0 Normal free:69016kB min:65192kB
low:81488kB high:97788kB active_anon:44389844kB
inactive_anon:1293312kB active_file:20040kB inactive_file:32380kB
unevictable:44436kB isolated(anon):0kB isolated(file):564kB
present:80740352kB managed:79411360kB mlocked:44448kB dirty:0kB
writeback:20kB mapped:44412kB shmem:4010304kB
slab_reclaimable:18049652kB slab_unreclaimable:14413248kB
kernel_stack:51536kB pagetables:110548kB unstable:0kB bounce:0kB
free_pcp:2392kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:37 all_unreclaimable? no
[Don Feb 3 10:26:18 2022] lowmem_reserve[]: 0 0 0 0
[Don Feb 3 10:26:18 2022] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U)
0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U)
1*2048kB (M) 3*4096kB (M) = 15892kB
[Don Feb 3 10:26:18 2022] Node 0 DMA32: 2516*4kB (UEM) 560*8kB (UEM)
66*16kB (UEM) 3758*32kB (EM) 1244*64kB (EM) 311*128kB (UEM) 75*256kB
(UEM) 36*512kB (UEM) 18*1024kB (UEM) 0*2048kB 0*4096kB = 311344kB
[Don Feb 3 10:26:18 2022] Node 0 Normal: 8989*4kB (UEM) 3771*8kB (UE)
118*16kB (EM) 40*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 69292kB
[Don Feb 3 10:26:18 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[Don Feb 3 10:26:18 2022] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[Don Feb 3 10:26:18 2022] 1042927 total pagecache pages
[Don Feb 3 10:26:18 2022] 0 pages in swap cache
[Don Feb 3 10:26:18 2022] Swap cache stats: add 0, delete 0, find 0/0
[Don Feb 3 10:26:18 2022] Free swap = 0kB
[Don Feb 3 10:26:18 2022] Total swap = 0kB
[Don Feb 3 10:26:18 2022] 20971384 pages RAM
[Don Feb 3 10:26:18 2022] 0 pages HighMem/MovableOnly
[Don Feb 3 10:26:18 2022] 390901 pages reserved
[Don Feb 3 10:26:18 2022] [ pid ] uid tgid total_vm rss
nr_ptes swapents oom_score_adj name
[Don Feb 3 10:26:18 2022] [ 1807] 0 1807 27264 11751
59 0 0 systemd-journal
[Don Feb 3 10:26:18 2022] [ 1825] 0 1825 84456 19426
69 0 0 lvmetad
[Don Feb 3 10:26:18 2022] [ 1841] 0 1841 11129 386
23 0 -1000 systemd-udevd
[Don Feb 3 10:26:18 2022] [ 3280] 0 3280 32020 11163
53 0 -1000 dmeventd
[Don Feb 3 10:26:18 2022] [ 3289] 0 3289 13880 192
27 0 -1000 auditd
[Don Feb 3 10:26:18 2022] [ 3313] 999 3313 156151 2451
63 0 0 polkitd
[Don Feb 3 10:26:18 2022] [ 3314] 81 3314 15184 523
34 0 -900 dbus-daemon
[Don Feb 3 10:26:18 2022] [ 3320] 0 3320 119375 1271
84 0 0 NetworkManager
[Don Feb 3 10:26:18 2022] [ 3321] 0 3321 66966 629
80 0 0 sssd
[Don Feb 3 10:26:18 2022] [ 3322] 0 3322 5397 255
16 0 0 irqbalance
[Don Feb 3 10:26:18 2022] [ 3325] 0 3325 6224 325
17 0 0 qemu-ga
[Don Feb 3 10:26:18 2022] [ 3336] 0 3336 79483 2310
107 0 0 sssd_be
[Don Feb 3 10:26:18 2022] [ 3377] 0 3377 68920 577
88 0 0 sssd_nss
[Don Feb 3 10:26:18 2022] [ 3378] 0 3378 63840 550
78 0 0 sssd_pam
[Don Feb 3 10:26:18 2022] [ 3379] 0 3379 62228 555
77 0 0 sssd_ssh
[Don Feb 3 10:26:18 2022] [ 3435] 0 3435 6865 586
20 0 0 systemd-logind
[Don Feb 3 10:26:18 2022] [ 3448] 0 3448 31571 326
18 0 0 crond
[Don Feb 3 10:26:18 2022] [ 3475] 0 3475 27526 194
9 0 0 agetty
[Don Feb 3 10:26:18 2022] [ 3573] 0 3573 35460 13602
57 0 -999 kube-proxy
[Don Feb 3 10:26:18 2022] [ 3574] 0 3574 28216 500
58 0 -1000 sshd
[Don Feb 3 10:26:18 2022] [ 3575] 0 3575 143480 3611
97 0 0 tuned
[Don Feb 3 10:26:18 2022] [ 3584] 0 3584 203847 5774
65 0 0 flanneld
[Don Feb 3 10:26:18 2022] [ 3586] 997 3586 19819 281
38 0 0 zabbix_agentd
[Don Feb 3 10:26:18 2022] [ 3587] 997 3587 19819 427
38 0 0 zabbix_agentd
[Don Feb 3 10:26:18 2022] [ 3588] 997 3588 19849 444
39 0 0 zabbix_agentd
[Don Feb 3 10:26:18 2022] [ 3589] 997 3589 19849 436
39 0 0 zabbix_agentd
[Don Feb 3 10:26:18 2022] [ 3591] 997 3591 19849 426
39 0 0 zabbix_agentd
[Don Feb 3 10:26:18 2022] [ 3592] 997 3592 19916 473
39 0 0 zabbix_agentd
[Don Feb 3 10:26:18 2022] [ 3604] 0 3604 267029 99685
392 0 0 rsyslogd
[Don Feb 3 10:26:18 2022] [ 3728] 0 3728 30746 312
56 0 0 keepalived
[Don Feb 3 10:26:18 2022] [ 3730] 0 3730 31796 502
60 0 0 keepalived
[Don Feb 3 10:26:18 2022] [ 3731] 0 3731 32570 1228
62 0 0 keepalived
[Don Feb 3 10:26:18 2022] [ 3902] 0 3902 22934 426
43 0 0 master
[Don Feb 3 10:26:18 2022] [ 3935] 89 3935 25545 494
47 0 0 qmgr
[Don Feb 3 10:26:18 2022] [ 4107] 0 4107 853912 46862
275 0 -999 dockerd
[Don Feb 3 10:26:18 2022] [ 4146] 0 4146 1420875 15073
300 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 6423] 0 6423 750261 68814
372 0 -999 kubelet
[Don Feb 3 10:26:18 2022] [16445] 0 16445 1872 262
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [16464] 1001 16464 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 1404] 38 1404 10548 355
22 0 0 ntpd
[Don Feb 3 10:26:18 2022] [13704] 0 13704 300089 9863
136 0 0 filebeat
[Don Feb 3 10:26:18 2022] [29615] 0 29615 2224 411
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [29780] 0 29780 35813 4433
33 0 999 coredns
[Don Feb 3 10:26:18 2022] [ 2062] 0 2062 1936 234
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 2092] 1001 2092 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 2313] 0 2313 1872 215
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 2332] 1001 2332 667 71
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 2479] 0 2479 1872 220
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 2498] 0 2498 5510 2219
14 0 989 java_bootstrap
[Don Feb 3 10:26:18 2022] [ 2600] 0 2600 1340116 88706
278 0 989 java
[Don Feb 3 10:26:18 2022] [ 2660] 0 2660 1872 244
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 2681] 1001 2681 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 2801] 0 2801 2224 239
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 2826] 1001 2826 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 2925] 0 2925 2224 228
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 2952] 1001 2952 667 71
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 3060] 0 3060 1888 192
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 3080] 1001 3080 667 71
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 3362] 0 3362 2224 246
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 3395] 0 3395 2719 267
10 0 999 nginx
[Don Feb 3 10:26:18 2022] [ 3571] 0 3571 1888 196
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 3613] 0 3613 5510 2220
14 0 989 java_bootstrap
[Don Feb 3 10:26:18 2022] [ 3644] 0 3644 2830 404
9 0 999 nginx
[Don Feb 3 10:26:18 2022] [ 3670] 0 3670 1189835 93385
272 0 989 java
[Don Feb 3 10:26:18 2022] [ 3753] 0 3753 1872 251
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 3789] 1001 3789 667 75
7 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 3904] 0 3904 2240 225
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 3923] 1001 3923 667 70
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 4253] 0 4253 1888 228
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 4272] 0 4272 5510 2218
14 0 961 java_bootstrap
[Don Feb 3 10:26:18 2022] [ 4383] 0 4383 1413025 146345
394 0 961 java
[Don Feb 3 10:26:18 2022] [ 4417] 0 4417 2240 262
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 4438] 0 4438 2719 267
9 0 993 nginx
[Don Feb 3 10:26:18 2022] [ 4524] 0 4524 1872 251
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 4547] 0 4547 2716 269
9 0 993 nginx
[Don Feb 3 10:26:18 2022] [ 4579] 0 4579 2830 383
9 0 993 nginx
[Don Feb 3 10:26:18 2022] [ 4649] 0 4649 1872 243
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 4672] 0 4672 2830 383
8 0 993 nginx
[Don Feb 3 10:26:18 2022] [ 4692] 0 4692 2716 270
9 0 999 nginx
[Don Feb 3 10:26:18 2022] [ 4882] 0 4882 1888 249
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 4904] 0 4904 5510 2220
14 0 967 java_bootstrap
[Don Feb 3 10:26:18 2022] [ 4914] 0 4914 2863 417
9 0 999 nginx
[Don Feb 3 10:26:18 2022] [ 4938] 0 4938 1688100 370101
856 0 967 java
[Don Feb 3 10:26:18 2022] [ 5130] 0 5130 1657107 217119
568 0 961 java
[Don Feb 3 10:26:18 2022] [ 5181] 0 5181 1727641 447718
1062 0 967 java
[Don Feb 3 10:26:18 2022] [23299] 0 23299 1888 229
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [23318] 1001 23318 667 76
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [23409] 0 23409 1872 227
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [23429] 0 23429 5508 2211
14 0 989 java_bootstrap
[Don Feb 3 10:26:18 2022] [23469] 0 23469 1156551 87062
263 0 989 java
[Don Feb 3 10:26:18 2022] [23576] 0 23576 1872 229
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [23611] 1001 23611 667 75
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [23717] 0 23717 1872 213
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [23737] 1001 23737 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [23855] 0 23855 1888 250
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [23878] 0 23878 2716 264
9 0 993 nginx
[Don Feb 3 10:26:18 2022] [23947] 0 23947 2826 371
9 0 993 nginx
[Don Feb 3 10:26:18 2022] [23959] 0 23959 1872 221
10 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [23979] 0 23979 5508 2210
14 0 923 java_bootstrap
[Don Feb 3 10:26:18 2022] [24022] 0 24022 1687586 326834
782 0 923 java
[Don Feb 3 10:26:18 2022] [24102] 0 24102 1872 222
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [24122] 1001 24122 667 71
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [24216] 0 24216 1888 226
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [24235] 0 24235 2717 264
9 0 993 nginx
[Don Feb 3 10:26:18 2022] [24264] 0 24264 2826 372
8 0 993 nginx
[Don Feb 3 10:26:18 2022] [24351] 0 24351 2619741 357131
859 0 923 java
[Don Feb 3 10:26:18 2022] [ 8434] 0 8434 1888 235
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 8452] 1001 8452 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [ 8609] 0 8609 1936 234
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [ 8628] 0 8628 5631 2354
16 0 906 java_bootstrap
[Don Feb 3 10:26:18 2022] [ 8670] 0 8670 3406142 1945540
3955 0 906 java
[Don Feb 3 10:26:18 2022] [10200] 0 10200 2224 202
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [10227] 1001 10227 667 71
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [10371] 0 10371 2224 220
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [10391] 1001 10391 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [10601] 0 10601 1936 229
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [10627] 0 10627 5631 2354
14 0 993 java_bootstrap
[Don Feb 3 10:26:18 2022] [10681] 0 10681 1458786 143558
401 0 993 java
[Don Feb 3 10:26:18 2022] [10820] 0 10820 1872 227
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [10850] 1001 10850 667 73
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [10933] 0 10933 1936 221
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [10952] 0 10952 5632 2354
14 0 989 java_bootstrap
[Don Feb 3 10:26:18 2022] [11003] 0 11003 1173227 93422
275 0 989 java
[Don Feb 3 10:26:18 2022] [11159] 0 11159 1888 220
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [11182] 0 11182 5632 2355
15 0 993 java_bootstrap
[Don Feb 3 10:26:18 2022] [11273] 0 11273 1417988 126907
376 0 993 java
[Don Feb 3 10:26:18 2022] [11301] 0 11301 1872 229
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [11330] 1001 11330 667 74
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [11412] 0 11412 1888 227
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [11433] 1001 11433 667 71
7 0 -998 pod
[Don Feb 3 10:26:18 2022] [11698] 0 11698 1872 231
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [11716] 0 11716 5632 2355
14 0 997 java_bootstrap
[Don Feb 3 10:26:18 2022] [11782] 0 11782 1163544 115567
329 0 997 java
[Don Feb 3 10:26:18 2022] [11960] 0 11960 1872 223
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [11983] 0 11983 5632 2354
14 0 996 java_bootstrap
[Don Feb 3 10:26:18 2022] [12055] 0 12055 982558 95682
274 0 996 java
[Don Feb 3 10:26:18 2022] [12070] 0 12070 1888 231
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [12094] 1001 12094 667 70
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [12456] 0 12456 2224 229
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [12479] 0 12479 5631 2353
14 0 987 java_bootstrap
[Don Feb 3 10:26:18 2022] [12535] 0 12535 1095784 98525
282 0 987 java
[Don Feb 3 10:26:18 2022] [13439] 0 13439 1888 245
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [13456] 1001 13456 667 71
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [13726] 0 13726 1872 216
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [13749] 0 13749 5631 2354
15 0 993 java_bootstrap
[Don Feb 3 10:26:18 2022] [13784] 0 13784 1091829 130305
345 0 993 java
[Don Feb 3 10:26:18 2022] [13929] 0 13929 2224 221
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [13950] 1001 13950 667 72
7 0 -998 pod
[Don Feb 3 10:26:18 2022] [14213] 0 14213 1872 237
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [14236] 0 14236 5631 2354
14 0 948 java_bootstrap
[Don Feb 3 10:26:18 2022] [14369] 0 14369 2654883 860559
2129 0 948 java
[Don Feb 3 10:26:18 2022] [14709] 0 14709 1888 222
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [14729] 1001 14729 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [15009] 0 15009 2240 237
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [15028] 0 15028 5632 2355
14 0 923 java_bootstrap
[Don Feb 3 10:26:18 2022] [15058] 0 15058 3174893 1410674
2957 0 923 java
[Don Feb 3 10:26:18 2022] [11209] 0 11209 2304 227
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [11228] 1001 11228 667 71
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [11358] 0 11358 1888 221
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [11379] 0 11379 5510 2218
13 0 987 java_bootstrap
[Don Feb 3 10:26:18 2022] [11447] 0 11447 1489370 162709
439 0 987 java
[Don Feb 3 10:26:18 2022] [19435] 0 19435 2224 233
10 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [19454] 1001 19454 667 71
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [19571] 0 19571 1952 215
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [19589] 0 19589 5513 2227
14 0 993 java_bootstrap
[Don Feb 3 10:26:18 2022] [19677] 0 19677 2240 242
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [19694] 0 19694 1117148 104436
288 0 993 java
[Don Feb 3 10:26:18 2022] [19698] 1001 19698 1083 71
8 0 -998 pod
[Don Feb 3 10:26:18 2022] [19832] 0 19832 1872 223
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [19851] 0 19851 5512 2228
14 0 974 java_bootstrap
[Don Feb 3 10:26:18 2022] [19907] 0 19907 1481942 157796
393 0 974 java
[Don Feb 3 10:26:18 2022] [20122] 0 20122 1872 247
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [20145] 1001 20145 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [20458] 0 20458 2224 216
9 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [20480] 0 20480 5512 2228
14 0 993 java_bootstrap
[Don Feb 3 10:26:18 2022] [20516] 0 20516 1100754 101849
284 0 993 java
[Don Feb 3 10:26:18 2022] [17440] 0 17440 1952 230
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [17459] 1001 17459 667 72
6 0 -998 pod
[Don Feb 3 10:26:18 2022] [17627] 0 17627 1872 240
8 0 -999 docker-containe
[Don Feb 3 10:26:18 2022] [17645] 0 17645 5631 2354
14 0 745 java_bootstrap
[Don Feb 3 10:26:18 2022] [17708] 0 17708 7009676 2566642
5332 0 745 java
[Don Feb 3 10:26:18 2022] [16892] 89 16892 25501 502
48 0 0 pickup
[Don Feb 3 10:26:18 2022] [30881] 0 30881 645 26
5 0 967 getent
[Don Feb 3 10:26:18 2022] [30928] 0 30928 645 27
5 0 923 getent
[Don Feb 3 10:26:18 2022] [31198] 0 31198 43821 642
83 0 0 sshd
[Don Feb 3 10:26:18 2022] [31214] 0 31214 645 24
5 0 961 getent
[Don Feb 3 10:26:18 2022] [31263] 0 31263 43821 626
84 0 0 sshd
[Don Feb 3 10:26:18 2022] [31286] 0 31286 28892 277
13 0 0 bash
[Don Feb 3 10:26:18 2022] [31358] 0 31358 64395 578
76 0 0 sudo
[Don Feb 3 10:26:18 2022] [31398] 0 31398 31431 1216
18 0 0 htop
[Don Feb 3 10:26:18 2022] [31401] 0 31401 27660 321
55 0 0 sshd
[Don Feb 3 10:26:18 2022] [31406] 0 31406 38752 1567
32 0 0 lvm.py
[Don Feb 3 10:26:18 2022] [31418] 0 31418 28825 227
14 0 0 bash
[Don Feb 3 10:26:18 2022] [31428] 0 31428 32570 1049
59 0 0 keepalived
[Don Feb 3 10:26:18 2022] [31430] 0 31430 18729 243
38 0 0 curl
[Don Feb 3 10:26:18 2022] [31431] 0 31431 32570 1049
59 0 0 keepalived
[Don Feb 3 10:26:18 2022] Out of memory: Kill process 11782 (java)
score 1002 or sacrifice child
[Don Feb 3 10:26:18 2022] Killed process 11782 (java)
total-vm:4654176kB, anon-rss:462268kB, file-rss:0kB, shmem-rss:0kB
Kind Regards,
Sasa Vilic
More information about the Kernelnewbies
mailing list