Why I can't map memory with mmap

Mon Aug 18 16:25:45 EDT 2014

I sent the previous mail too early!

I got it to work for root user by setting "/proc/sys/vm/nr_hugepages" to 20
as suggested in Documentation/vm/hugetlbpage.txt. Now I'm able to mmap it
and it reflects the same in /proc/meminfo as well! :)

HTH,
-mandeep

On Mon, Aug 18, 2014 at 1:23 PM, Mandeep Sandhu <mandeepsandhu.chd at gmail.com
> wrote:

> I tried your program and got the same result, i.e "Resource temporarily
> unavailable" for an ordinary user and "Cannot allocate memory" if run as
> root.
>
> However, reading the Huge TLB pages doc
> (Documentation/vm/hugetlbpage.txt), I saw the following para which might
> explain the error:
> ...
> Once a number of huge pages have been pre-allocated to the kernel huge page
> pool, a user with appropriate privilege can use either the mmap system call
> or shared memory system calls to use the huge pages. See the discussion of
> Using Huge Pages, below.
>
> The administrator can allocate persistent huge pages on the kernel boot
> command line by specifying the "hugepages=N" parameter, where 'N' = the
> number of huge pages requested. This is the most reliable method of
> allocating huge pages as memory has not yet become fragmented.
> ...
> ...
>
> I don't have my kernel booted with it, so that might explain why mmap fails for MAP_HUGETLB.
>
> I can't restart my machine with the new flag right now, but maybe you can try.
>
> HTH,
>
> -mandeep
>
>
>
>
> On Mon, Aug 18, 2014 at 11:30 AM, Kevin O'Gorman <kogorman at gmail.com>
> wrote:
>
>> Now I'm befuddled.  I modified the source so that it was easy to try
>> different combinations.  If I sudo it, I can map a 1GiB (1<<30) private
>> anonymous region if I sudo the program.  I cannot find any combination with
>> MAP_HUGETLB that works at all.
>>
>> Here it is in  working form, with an anonymous map and no backing file.
>> The mmap request is for a single byte, so it works for ordinary users  Try
>> uncommenting the MAP_HUGETLB line, and I'd love to hear about a version
>> that works, even if it's only for root.
>>
>> Since I'm aiming for a huge memory region, I figured HUGETLB would make
>> sense to limit the overhead.
>>
>> ============================ CUT HERE ================
>> /**
>>  * @file
>>  * <pre>"Find out the limits on locked memory"
>>  * Last Modified: Mon Aug 18 11:21:23 PDT 2014
>>  * @author Kevin O'Gorman
>>  */
>>
>> #define _GNU_SOURCE /* enable some of the mmap flags */
>> #include <unistd.h>
>> #include <stdlib.h>
>> #include <stdio.h>
>> #include <string.h>
>> #include <sys/mman.h>
>>
>> /* for getrlimit(2) */
>> #include <sys/time.h>
>> #include <sys/resource.h>
>>
>> /* for open(2) */
>> #include <sys/types.h>
>> #include <sys/stat.h>
>> #include <sys/fcntl.h>
>>
>> int
>> main(int argc, char *argv[])
>> {
>>   void *where;
>>   size_t length = 1;
>>   struct rlimit rlim;
>>   int flags = 0;
>>   int fd = -1;
>>
>> //  fd = open("solvedtaq", O_RDWR);  if (fd == -1) perror("open");
>>   printf("fd is %d\n", fd);
>>
>> //  flags |= MAP_HUGETLB;
>>   flags |= MAP_PRIVATE;
>>   flags |= MAP_LOCKED;
>>   flags |= MAP_ANONYMOUS;
>>
>>   if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) {
>>     printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long
>> long)rlim.rlim_max, (long long)rlim.rlim_cur);
>>   } else {
>>     perror("getrlimit failed");
>>   }
>>
>>   where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0);
>>   if (where != MAP_FAILED) {
>>     printf("Mapped at %p\n", where);
>>   } else {
>>     perror("Mapping failed");
>>   }
>>
>>   return EXIT_SUCCESS;
>> }
>> ============================ CUT HERE ================
>>
>>
>>
>> On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman at gmail.com>
>> wrote:
>>
>>>
>>> On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks at vt.edu> wrote:
>>>
>>>> On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
>>>>
>>>> > maybe someone here can help me figure out how to map a big (really
>>>> big)
>>>> > work area and lock it in memory.
>>>>
>>>> What problem are you trying to solve by doing this?  It usually doesn't
>>>> make
>>>> sense to lock more than a small chunk in memory - that's mostly for
>>>> crypto
>>>> programs like PGP that are paranoid about keys being written to swap
>>>> spaces,
>>>> and a *very* few highly performance oriented programs.  99.8% of the
>>>> time,
>>>> the kernel will do a reasonable job of making sure the right pages are
>>>> in
>>>> memory.
>>>>
>>>> Plus, programmers almost always over-estimate what they *really* need
>>>> to lock
>>>> down.  The end result is that performance ends up being *worse*,
>>>> because they
>>>> lock out more than needed, leaving all the rest of the processes on the
>>>> system
>>>> fighting over a too-small pool of pages.  Really sucks when you
>>>> actually have
>>>> to read  from disk every I/O because you no longer have an in-core file
>>>> cache :)
>>>>
>>>> I'm solving a huge alpha-beta search problem, and want a huge cache
>>> table tocheck permutations.  That's why I increased RAM from 8 to 32 GB,
>>> and would
>>> be happy to use the extra 24GB all for the cache.  (Cache table is like
>>> a hash table but with collisions causing replacement rather than chaining).
>>>
>>> Even in my small tests, the cache is giving me speedups in the
>>> neighborhood of
>>> a million, so I'm going big. Because this is about speed, I do not
>>> *ever* want it to
>>> swap.
>>>
>>>
>>>> > I wrote a little test program to try this out, and it fails.  As a
>>>> regular
>>>> > user, perror() tells me some "resource is temporarily unavailable".
>>>> As
>>>> > root, it says it "cannot allocate memory".  I'm only asking for 1
>>>> byte.
>>>>
>>>> >   where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE |
>>>> > MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
>>>>
>>>> Remember that no matter what "length" you pass, even if it's 1 byte, the
>>>> kernel has to allocate at least enough integer pages to cover the
>>>> request.
>>>> You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't
>>>> quite big enough.
>>>>
>>>> And if you ask for MAP_HUGETLB, you're going to get at least 1 page.
>>>> Which
>>>> is probably 2M in size.  And that can be painful if the default max
>>>> lockable
>>>> memory is less than 2M (at least on my Fedora Rawhide box, it's all of
>>>> 64K).  There's also some interaction danger with MAP_POPULATE and memory
>>>> fragmentation.  But both POPULATE and HUGETLB have a high chance of
>>>> not playing well with LOCKED.
>>>>
>>>
>>> Good catch.  Indeed, removing MAP_HUGETLB allows the program to
>>> succeed.  I'll try increasing the resource limit, but may have to have to
>>> sudo it, do the mmap() and then chusr to my real ID.
>>>
>>> The fd of -1 that some folks thought might be a problem is suggested in
>>> the man page for use with MAP_ANONYMOUS.  That flag means I'm not using a
>>> backing file, after all.
>>>
>>> --
>>> Kevin O'Gorman
>>>
>>> programmer, n. an organism that transmutes caffeine into software.
>>>  Please consider the environment before printing this email.
>>>
>>
>>
>>
>> --
>> Kevin O'Gorman
>>
>> programmer, n. an organism that transmutes caffeine into software.
>>  Please consider the environment before printing this email.
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140818/731e05cf/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 441 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140818/731e05cf/attachment.gif 