Estimating total memory usage in any Linux system
Hi, I was just trying to manually calculate the total memory available in a system to match up the MemTotal shown in /proc/meminfo I know "free -m" command will give the vary high level usage, but I wanted to breakdown the MemTotal numbers to see usage in each category. I am looking for a common formula that should match on almost all systems. So, I came up with below fields from meminfo to closely match the total. ~Total = {MemFree + Active + Inactive + Shmem + Mapped + Slab + KernelStack + PageTables + (init freed) + (Boot Diff) + (low watermarks)} But I am not sure if these fields are correct at least for arm32 system. In some boards I could able to match up these figures, but some not. So I have some doubts: * Is "Mapped" (NR_MAPPED_PAGES) already part of Active pages ? * Is Kernel allocation (alloc_pages, etc.) accounted somewhere ? Is it part of Anon pages ? * What about vmalloc / ioremap pages ? Do we also need to consider VmallocUsed ? ==> In my opinion vmalloc internally used kmalloc and page tables, so I guess it should be already part of Slab or PageTables ?? Below are some data taken from a small embedded arm32 device with 512MB RAM: Kernel Boot Logs: Memory: 99244K/254976K available (10720K kernel code, 1336K rwdata, 4792K rodata, 400K init, 1602K bss, 135252K reserved, 20480K cma-reserved) Total = 99244 + 400 (init) + 20480 (cma) = 120124 Boot Diff = 121204 - 120124 => 1080 kB MemTotal: 121204 kB MemFree: 2540 kB MemAvailable: 51924 kB Buffers: 0 kB Cached: 47492 kB SwapCached: 0 kB Active: 42604 kB Inactive: 24308 kB Active(anon): 19476 kB Inactive(anon): 92 kB Active(file): 23128 kB Inactive(file): 24216 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 8 kB Writeback: 0 kB AnonPages: 19452 kB Mapped: 12644 kB Shmem: 148 kB Slab: 27560 kB SReclaimable: 7224 kB SUnreclaim: 20336 kB KernelStack: 3104 kB PageTables: 1896 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 60600 kB Committed_AS: 1576740 kB VmallocTotal: 777216 kB VmallocUsed: 53696 kB VmallocChunk: 689148 kB -------------------- These are my calculations from the above fields. --------------------- MemFree 2540 Active 42604 Inactive 24308 Mapped 12644 Slab 27560 KernelStack 3104 PageTables 1896 Shmem 148 init freed 400 Diff from boot total 1080 min_free_kbytes 1384 user_reserve_kbytes 3537 My Total ==> 121205 (add all the above) As you can see "My Total" closely matches with the MemTotal from /proc/meminfo But on some systems it does not match. So, I wanted to understand if there is something wrong in my calculation, or there is something wrong in the system itself (like leaks etc,) Note: I don't want to use any tools or utilize for these, but I wanted to manually understand it. I am also trying to explore every vm counter that populates these fields. But if you have any other opinion about it please let me know. Thanks, Pintu
On Mon, 18 May 2020 at 23:27, Pintu Agarwal <pintu.ping@gmail.com> wrote:
Hi,
I was just trying to manually calculate the total memory available in a system to match up the MemTotal shown in /proc/meminfo I know "free -m" command will give the vary high level usage, but I wanted to breakdown the MemTotal numbers to see usage in each category.
I am looking for a common formula that should match on almost all systems. So, I came up with below fields from meminfo to closely match the total. ~Total = {MemFree + Active + Inactive + Shmem + Mapped + Slab + KernelStack + PageTables + (init freed) + (Boot Diff) + (low watermarks)}
But I am not sure if these fields are correct at least for arm32 system. In some boards I could able to match up these figures, but some not.
So I have some doubts: * Is "Mapped" (NR_MAPPED_PAGES) already part of Active pages ? * Is Kernel allocation (alloc_pages, etc.) accounted somewhere ? Is it part of Anon pages ? * What about vmalloc / ioremap pages ? Do we also need to consider VmallocUsed ? ==> In my opinion vmalloc internally used kmalloc and page tables, so I guess it should be already part of Slab or PageTables ??
Below are some data taken from a small embedded arm32 device with 512MB RAM:
Sorry, I forgot to mention the kernel version. Here it is 3.18 Kernel
Kernel Boot Logs: Memory: 99244K/254976K available (10720K kernel code, 1336K rwdata, 4792K rodata, 400K init, 1602K bss, 135252K reserved, 20480K cma-reserved) Total = 99244 + 400 (init) + 20480 (cma) = 120124 Boot Diff = 121204 - 120124 => 1080 kB
MemTotal: 121204 kB MemFree: 2540 kB MemAvailable: 51924 kB Buffers: 0 kB Cached: 47492 kB SwapCached: 0 kB Active: 42604 kB Inactive: 24308 kB Active(anon): 19476 kB Inactive(anon): 92 kB Active(file): 23128 kB Inactive(file): 24216 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 8 kB Writeback: 0 kB AnonPages: 19452 kB Mapped: 12644 kB Shmem: 148 kB Slab: 27560 kB SReclaimable: 7224 kB SUnreclaim: 20336 kB KernelStack: 3104 kB PageTables: 1896 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 60600 kB Committed_AS: 1576740 kB VmallocTotal: 777216 kB VmallocUsed: 53696 kB VmallocChunk: 689148 kB --------------------
These are my calculations from the above fields. --------------------- MemFree 2540 Active 42604 Inactive 24308 Mapped 12644 Slab 27560 KernelStack 3104 PageTables 1896 Shmem 148 init freed 400 Diff from boot total 1080 min_free_kbytes 1384 user_reserve_kbytes 3537 My Total ==> 121205 (add all the above)
As you can see "My Total" closely matches with the MemTotal from /proc/meminfo
But on some systems it does not match. So, I wanted to understand if there is something wrong in my calculation, or there is something wrong in the system itself (like leaks etc,)
Note: I don't want to use any tools or utilize for these, but I wanted to manually understand it.
I am also trying to explore every vm counter that populates these fields.
But if you have any other opinion about it please let me know.
Thanks, Pintu
On Tue, 19 May 2020 22:33:15 +0530, Pintu Agarwal said:
Below are some data taken from a small embedded arm32 device with 512MB RAM:
Sorry, I forgot to mention the kernel version. Here it is 3.18 Kernel
That's a kernel from 2014. You don't seriously expect us to remember how all those counters worked back then, do you? That's a very real issue, not sarcasm: [/usr/src/linux-next] git diff --shortstat v3.18..next-20200518 76288 files changed, 15101441 insertions(+), 5417984 deletions(-) You're asking about what the numbers meant 15 million lines of code ago.
On Wed, 20 May 2020 at 09:31, Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
On Tue, 19 May 2020 22:33:15 +0530, Pintu Agarwal said:
Below are some data taken from a small embedded arm32 device with 512MB RAM:
Sorry, I forgot to mention the kernel version. Here it is 3.18 Kernel
That's a kernel from 2014. You don't seriously expect us to remember how all those counters worked back then, do you?
Actually I don't care about the kernel version. This was just for reference. I know it is weird to manually arrive a MemTotal from the values of meminfo. But I am actually looking for general formula. I could get some of the answers by tracing the kernel code for each counters. But still there are some gaps which I want to clarify. As I see there are several ways to total it: i.e; (PSS way), (Buffers, Cached), (Active, Inactive) According to me, MemTotal = MemFree + Active + Inactive + SwapCached + Mapped + Shmem + Slab + KernelStack + PageTables + (Init Freed from boot) + (alloc pages) + (min_free_bytes + user_reserve_bytes) Did I miss anything here ? With this I could arrive to some result, but I have some doubts: * Is Mapped already included in Active/Inactive (file) ? Because mapped is coming from NR_FILE_MAPPED. ==> I see that it is accounted from mm/rmap.c * Do we also need to incude VmallocUsed here ? I see in some cases people include it too. ==> But I think vmalloc internally calls kmalloc and use page tables, so it should be already accounted in Slab and PageTables ?? ==> Do I still need to include these values from /proc/vmallocinfo ?? * Is it valid to include min_free_kbytes and user_reserve_kbytes here ? Its kind of reserve for internal purpose but still it contributes to the total. * Is kernel internal allocation (alloc_pages) statistics accounted somewhere ? ==> I can see that some counters like (pgalloc_normal, etc. and pgfree) are available as part of /proc/vmstat ===> But this is like overall allocation and free stats. * Apart from these, do we also need to consider Graphics/3D allocation separately ? Thanks, Pintu
On Wed, 20 May 2020 10:26:50 +0530, Pintu Agarwal said:
I know it is weird to manually arrive a MemTotal from the values of meminfo. But I am actually looking for general formula.
You missed the point - the "general formula" has probably changed over the years.
With this I could arrive to some result, but I have some doubts: * Is Mapped already included in Active/Inactive (file) ? Because mapped is coming from NR_FILE_MAPPED.
And that's the sort of question that can have different answers for 3.18 and 5.7.
On Wed, 20 May 2020 at 11:02, Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
On Wed, 20 May 2020 10:26:50 +0530, Pintu Agarwal said:
I know it is weird to manually arrive a MemTotal from the values of meminfo. But I am actually looking for general formula.
You missed the point - the "general formula" has probably changed over the years.
With this I could arrive to some result, but I have some doubts: * Is Mapped already included in Active/Inactive (file) ? Because mapped is coming from NR_FILE_MAPPED.
And that's the sort of question that can have different answers for 3.18 and 5.7.
I am okay for even the latest kernel. I am even trying to get this on 5.5 kernel or higher. But my questions remains the same. What will be the equation for even 5.7 kerne ?
participants (2)
-
Pintu Agarwal -
Valdis Klētnieks