Why I can't map memory with mmap

Kevin O'Gorman kogorman at gmail.com
Mon Aug 18 20:32:05 EDT 2014


Mandeep has solved it for me.

I was able to set /proc/sys/vm/nr_hugepages to 12288, corresponding to
24GiB of memory, and make a corresponding mmap request.  It takes a while
-- a very noticeable couple of seconds -- but it works.

There are a few things still to be worked out, like why that setting is
reduced when my little program runs.  So that it has to be re-established
in order to run it again.  But it gets me started.


On Mon, Aug 18, 2014 at 1:25 PM, Mandeep Sandhu <mandeepsandhu.chd at gmail.com
> wrote:

> I sent the previous mail too early!
>
> I got it to work for root user by setting "/proc/sys/vm/nr_hugepages" to
> 20 as suggested in Documentation/vm/hugetlbpage.txt. Now I'm able to mmap
> it and it reflects the same in /proc/meminfo as well! :)
>
> HTH,
> -mandeep
>
>
>
> On Mon, Aug 18, 2014 at 1:23 PM, Mandeep Sandhu <
> mandeepsandhu.chd at gmail.com> wrote:
>
>> I tried your program and got the same result, i.e "Resource temporarily
>> unavailable" for an ordinary user and "Cannot allocate memory" if run as
>> root.
>>
>> However, reading the Huge TLB pages doc
>> (Documentation/vm/hugetlbpage.txt), I saw the following para which might
>> explain the error:
>> ...
>> Once a number of huge pages have been pre-allocated to the kernel huge
>> page
>> pool, a user with appropriate privilege can use either the mmap system
>> call
>> or shared memory system calls to use the huge pages. See the discussion of
>> Using Huge Pages, below.
>>
>> The administrator can allocate persistent huge pages on the kernel boot
>> command line by specifying the "hugepages=N" parameter, where 'N' = the
>> number of huge pages requested. This is the most reliable method of
>> allocating huge pages as memory has not yet become fragmented.
>> ...
>> ...
>>
>> I don't have my kernel booted with it, so that might explain why mmap fails for MAP_HUGETLB.
>>
>> I can't restart my machine with the new flag right now, but maybe you can try.
>>
>> HTH,
>>
>> -mandeep
>>
>>
>>
>>
>> On Mon, Aug 18, 2014 at 11:30 AM, Kevin O'Gorman <kogorman at gmail.com>
>> wrote:
>>
>>> Now I'm befuddled.  I modified the source so that it was easy to try
>>> different combinations.  If I sudo it, I can map a 1GiB (1<<30) private
>>> anonymous region if I sudo the program.  I cannot find any combination with
>>> MAP_HUGETLB that works at all.
>>>
>>> Here it is in  working form, with an anonymous map and no backing file.
>>> The mmap request is for a single byte, so it works for ordinary users  Try
>>> uncommenting the MAP_HUGETLB line, and I'd love to hear about a version
>>> that works, even if it's only for root.
>>>
>>> Since I'm aiming for a huge memory region, I figured HUGETLB would make
>>> sense to limit the overhead.
>>>
>>> ============================ CUT HERE ================
>>> /**
>>>  * @file
>>>  * <pre>"Find out the limits on locked memory"
>>>  * Last Modified: Mon Aug 18 11:21:23 PDT 2014
>>>  * @author Kevin O'Gorman
>>>  */
>>>
>>> #define _GNU_SOURCE /* enable some of the mmap flags */
>>> #include <unistd.h>
>>> #include <stdlib.h>
>>> #include <stdio.h>
>>> #include <string.h>
>>> #include <sys/mman.h>
>>>
>>> /* for getrlimit(2) */
>>> #include <sys/time.h>
>>> #include <sys/resource.h>
>>>
>>> /* for open(2) */
>>> #include <sys/types.h>
>>> #include <sys/stat.h>
>>> #include <sys/fcntl.h>
>>>
>>> int
>>> main(int argc, char *argv[])
>>> {
>>>   void *where;
>>>   size_t length = 1;
>>>   struct rlimit rlim;
>>>   int flags = 0;
>>>   int fd = -1;
>>>
>>> //  fd = open("solvedtaq", O_RDWR);  if (fd == -1) perror("open");
>>>   printf("fd is %d\n", fd);
>>>
>>> //  flags |= MAP_HUGETLB;
>>>   flags |= MAP_PRIVATE;
>>>   flags |= MAP_LOCKED;
>>>   flags |= MAP_ANONYMOUS;
>>>
>>>   if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) {
>>>     printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long
>>> long)rlim.rlim_max, (long long)rlim.rlim_cur);
>>>   } else {
>>>     perror("getrlimit failed");
>>>   }
>>>
>>>   where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0);
>>>   if (where != MAP_FAILED) {
>>>     printf("Mapped at %p\n", where);
>>>   } else {
>>>     perror("Mapping failed");
>>>   }
>>>
>>>   return EXIT_SUCCESS;
>>> }
>>> ============================ CUT HERE ================
>>>
>>>
>>>
>>> On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman at gmail.com>
>>> wrote:
>>>
>>>>
>>>> On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks at vt.edu> wrote:
>>>>
>>>>> On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
>>>>>
>>>>> > maybe someone here can help me figure out how to map a big (really
>>>>> big)
>>>>> > work area and lock it in memory.
>>>>>
>>>>> What problem are you trying to solve by doing this?  It usually
>>>>> doesn't make
>>>>> sense to lock more than a small chunk in memory - that's mostly for
>>>>> crypto
>>>>> programs like PGP that are paranoid about keys being written to swap
>>>>> spaces,
>>>>> and a *very* few highly performance oriented programs.  99.8% of the
>>>>> time,
>>>>> the kernel will do a reasonable job of making sure the right pages are
>>>>> in
>>>>> memory.
>>>>>
>>>>> Plus, programmers almost always over-estimate what they *really* need
>>>>> to lock
>>>>> down.  The end result is that performance ends up being *worse*,
>>>>> because they
>>>>> lock out more than needed, leaving all the rest of the processes on
>>>>> the system
>>>>> fighting over a too-small pool of pages.  Really sucks when you
>>>>> actually have
>>>>> to read  from disk every I/O because you no longer have an in-core
>>>>> file cache :)
>>>>>
>>>>> I'm solving a huge alpha-beta search problem, and want a huge cache
>>>> table tocheck permutations.  That's why I increased RAM from 8 to 32 GB,
>>>> and would
>>>> be happy to use the extra 24GB all for the cache.  (Cache table is like
>>>> a hash table but with collisions causing replacement rather than chaining).
>>>>
>>>> Even in my small tests, the cache is giving me speedups in the
>>>> neighborhood of
>>>> a million, so I'm going big. Because this is about speed, I do not
>>>> *ever* want it to
>>>> swap.
>>>>
>>>>
>>>>> > I wrote a little test program to try this out, and it fails.  As a
>>>>> regular
>>>>> > user, perror() tells me some "resource is temporarily unavailable".
>>>>> As
>>>>> > root, it says it "cannot allocate memory".  I'm only asking for 1
>>>>> byte.
>>>>>
>>>>> >   where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE |
>>>>> > MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
>>>>>
>>>>> Remember that no matter what "length" you pass, even if it's 1 byte,
>>>>> the
>>>>> kernel has to allocate at least enough integer pages to cover the
>>>>> request.
>>>>> You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't
>>>>> quite big enough.
>>>>>
>>>>> And if you ask for MAP_HUGETLB, you're going to get at least 1 page.
>>>>> Which
>>>>> is probably 2M in size.  And that can be painful if the default max
>>>>> lockable
>>>>> memory is less than 2M (at least on my Fedora Rawhide box, it's all of
>>>>> 64K).  There's also some interaction danger with MAP_POPULATE and
>>>>> memory
>>>>> fragmentation.  But both POPULATE and HUGETLB have a high chance of
>>>>> not playing well with LOCKED.
>>>>>
>>>>
>>>> Good catch.  Indeed, removing MAP_HUGETLB allows the program to
>>>> succeed.  I'll try increasing the resource limit, but may have to have to
>>>> sudo it, do the mmap() and then chusr to my real ID.
>>>>
>>>> The fd of -1 that some folks thought might be a problem is suggested in
>>>> the man page for use with MAP_ANONYMOUS.  That flag means I'm not using a
>>>> backing file, after all.
>>>>
>>>> --
>>>> Kevin O'Gorman
>>>>
>>>> programmer, n. an organism that transmutes caffeine into software.
>>>>  Please consider the environment before printing this email.
>>>>
>>>
>>>
>>>
>>> --
>>> Kevin O'Gorman
>>>
>>> programmer, n. an organism that transmutes caffeine into software.
>>>  Please consider the environment before printing this email.
>>>
>>> _______________________________________________
>>> Kernelnewbies mailing list
>>> Kernelnewbies at kernelnewbies.org
>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>
>>>
>>
>


-- 
Kevin O'Gorman

programmer, n. an organism that transmutes caffeine into software.
Please consider the environment before printing this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140818/adfc44d9/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 441 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140818/adfc44d9/attachment-0001.gif 


More information about the Kernelnewbies mailing list