Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64

[Amit Agarwal]

Hi All,

We are running a 32 bit application on RHEL6.3-64 bit OS with kernel 
version 2.6.32-279.el6.x86_64.

While running this application we see the following when running under 
strace:
mmap(offset=33230848, len=2068480) failed with errno=12 for the file 
<so file name>

The system has enough RAM, total 16GB and about 12 GB free. We checked 
buddyinfo on the system, when the application is running and see the 
below trend:
Node 0, zone   Normal   5200  21396  21389  21516  16202  12770   9054  
4459   1430    168    313
Node 0, zone   Normal   5231  21395  21389  21516  16202  12770   9054  
4459   1430    168    313
Node 0, zone   Normal   5128  21401  21389  21516  16202  12770   9054  
4459   1430    168    313
Node 0, zone   Normal   5060  21405  21391  21516  16202  12770   9054  
4459   1430    168    313
..............
Node 0, zone   Normal    483  17946  21342  21516  16202  12770   9054  
4459   1430    168    313
Node 0, zone   Normal    315  17937  21342  21516  16202  12770   9054  
4459   1430    168    313
Node 0, zone   Normal    345  17891  21352  21516  16202  12770   9054  
4459   1430    168    313
Node 0, zone   Normal    278  17785  21352  21516  16202  12770   9054  
4459   1430    168    313


At this point the application crashes with mmap error.


Page Type information on the system:
Page block order: 9
Pages per block:  512

Free pages count per migrate type at order       0      1      2      3 
     4      5      6      7      8      9     10
Node    0, zone      DMA, type    Unmovable      1      0      0      1 
     0      0      1      0      0      0      0
Node    0, zone      DMA, type  Reclaimable      0      0      0      0 
     0      0      0      0      0      0      0
Node    0, zone      DMA, type      Movable      0      0      0      0 
     0      0      0      0      0      0      3
Node    0, zone      DMA, type      Reserve      0      0      0      0 
     0      0      0      0      0      1      0
Node    0, zone      DMA, type      Isolate      0      0      0      0 
     0      0      0      0      0      0      0
Node    0, zone    DMA32, type    Unmovable    410    261    133     51 
    24     14      7      2      0      0      0
Node    0, zone    DMA32, type  Reclaimable   2573   1806   1108    889 
   522    175     31      0      0      0      0
Node    0, zone    DMA32, type      Movable    408    352    345    341 
   328    312    310    260    216    180    356
Node    0, zone    DMA32, type      Reserve      0      0      0      0 
     0      0      0      0      0      2      0
Node    0, zone    DMA32, type      Isolate      0      0      0      0 
     0      0      0      0      0      0      0
Node    0, zone   Normal, type    Unmovable    434    788    518    137 
    58     24     13      4      5      0      0
Node    0, zone   Normal, type  Reclaimable      1    110   2136   3212 
   414     25      8      7      8      0      0
Node    0, zone   Normal, type      Movable   8135  20535  17946  18171 
15732  12719   9033   4448   1417    169    312
Node    0, zone   Normal, type      Reserve      0      0      0      0 
     0      0      0      0      0      0      1
Node    0, zone   Normal, type      Isolate      0      0      0      0 
     0      0      0      0      0      0      0

Number of blocks type     Unmovable  Reclaimable      Movable      
Reserve      Isolate
Node 0, zone      DMA            1            0            6            
1            0
Node 0, zone    DMA32           11           79         1436            
2            0
Node 0, zone   Normal          188          337         6129            
2            0


When we are running the same application on other system, it comes up. 
So, it has to do with memory allocation setting of some sort.

So, Is there some setting that we can use to allow the kernel to 
allocate the 313 - 4M  pages that are free for 4k memory allocations? If 
so, what should we change on the system?

Will setting vm.zone_reclaim_mode to 1 help here?

Do I need to provide more information, if yes, what?

-- 
Thanks,
-aka
http://blog.amit-agarwal.co.in