Lost memory

[Andrea Tomassetti]

Hi everyone,
I've been facing this problem for the past few weeks and I'm out of
ideas of how to analyze it, maybe someone here has some thoughts on
how to address it.
Basically, if I compare the output of `free` and the total RSS
reported by `ps` there are 200 GiB of difference:

$ free
              total        used        free      shared  buff/cache   available
Mem:      521933440   371541084    19246028        4100   131146328   117933472
Swap:      20971516    15208556     5762960

$ ps -eo rss | awk '{ rss += $0 } END { print rss}'
176765072

total used: 386749640
total RSS:  176765072
difference: 209984568 (~200.3 GiB)

I have some programs running that make intense use of networking with
thousands of sockets opened, so I checked sockstat but with no luck:

$ cat /proc/net/sockstat
sockets: used 90319
TCP: inuse 40798 orphan 0 tw 56 alloc 89689 mem 558949 # *4 = 2235796
KiB = ~2.13 GiB
UDP: inuse 31 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

Even /proc/meminfo is quite useless [1].

I started to kill one process at a time till I found the culprit: a
Java process with high network usage. But inspecting this process'
memory consumption didn't give me any conclusion because its memory
consumption appears to be low:

$ sudo pmap -x 268169 | head -n2 | tail -n1; sudo pmap -x 268169 | tail -n 1
Address           Kbytes     RSS   Dirty Mode  Mapping
total kB         171403084 23775460 23697900
                                        ~ 22.7 GiB

But this is the process that is helding the resources because, once
killed, the memory difference/leak disappears.

How is it possible that no utility or kernel information is capable of
outlining this memory leak? What can I possibly do to investigate it
further? For sure there's a problem releasing resources in the Java
code, but I would like to see it at a kernel level.

Any help is very appreciated.

Thank you in advance,
Andrea


[1] $ cat /proc/meminfo
MemTotal:       521933440 kB
MemFree:        20600964 kB
MemAvailable:   118570008 kB
Buffers:           18092 kB
Cached:         125086816 kB
SwapCached:      1792868 kB
Active:         115786784 kB
Inactive:       187218964 kB
Active(anon):   79144440 kB
Inactive(anon): 98768932 kB
Active(file):   36642344 kB
Inactive(file): 88450032 kB
Unevictable:       20556 kB
Mlocked:           20556 kB
SwapTotal:      20971516 kB
SwapFree:        5762448 kB
Dirty:            114392 kB
Writeback:             0 kB
AnonPages:      176129992 kB
Mapped:           315908 kB
Shmem:              4100 kB
KReclaimable:    5322088 kB
Slab:           13095480 kB
SReclaimable:    5322088 kB
SUnreclaim:      7773392 kB
KernelStack:      135600 kB
PageTables:       496504 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    542904956 kB
Committed_AS:   210582988 kB
VmallocTotal:   133143592960 kB
VmallocUsed:      256200 kB
VmallocChunk:          0 kB
Percpu:            62976 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:          32768 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB