Why I can't map memory with mmap

[Kevin O'Gorman]

Now I'm befuddled.  I modified the source so that it was easy to try
different combinations.  If I sudo it, I can map a 1GiB (1<<30) private
anonymous region if I sudo the program.  I cannot find any combination with
MAP_HUGETLB that works at all.

Here it is in  working form, with an anonymous map and no backing file.
The mmap request is for a single byte, so it works for ordinary users  Try
uncommenting the MAP_HUGETLB line, and I'd love to hear about a version
that works, even if it's only for root.

Since I'm aiming for a huge memory region, I figured HUGETLB would make
sense to limit the overhead.

============================ CUT HERE ================
/**
 * @file
 * <pre>"Find out the limits on locked memory"
 * Last Modified: Mon Aug 18 11:21:23 PDT 2014
 * @author Kevin O'Gorman
 */

#define _GNU_SOURCE /* enable some of the mmap flags */
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>

/* for getrlimit(2) */
#include <sys/time.h>
#include <sys/resource.h>

/* for open(2) */
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/fcntl.h>

int
main(int argc, char *argv[])
{
  void *where;
  size_t length = 1;
  struct rlimit rlim;
  int flags = 0;
  int fd = -1;

//  fd = open("solvedtaq", O_RDWR);  if (fd == -1) perror("open");
  printf("fd is %d\n", fd);

//  flags |= MAP_HUGETLB;
  flags |= MAP_PRIVATE;
  flags |= MAP_LOCKED;
  flags |= MAP_ANONYMOUS;

  if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) {
    printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long
long)rlim.rlim_max, (long long)rlim.rlim_cur);
  } else {
    perror("getrlimit failed");
  }

  where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0);
  if (where != MAP_FAILED) {
    printf("Mapped at %p\n", where);
  } else {
    perror("Mapping failed");
  }

  return EXIT_SUCCESS;
}
============================ CUT HERE ================



On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman at gmail.com> wrote:

>
> On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks at vt.edu> wrote:
>
>> On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
>>
>> > maybe someone here can help me figure out how to map a big (really big)
>> > work area and lock it in memory.
>>
>> What problem are you trying to solve by doing this?  It usually doesn't
>> make
>> sense to lock more than a small chunk in memory - that's mostly for crypto
>> programs like PGP that are paranoid about keys being written to swap
>> spaces,
>> and a *very* few highly performance oriented programs.  99.8% of the time,
>> the kernel will do a reasonable job of making sure the right pages are in
>> memory.
>>
>> Plus, programmers almost always over-estimate what they *really* need to
>> lock
>> down.  The end result is that performance ends up being *worse*, because
>> they
>> lock out more than needed, leaving all the rest of the processes on the
>> system
>> fighting over a too-small pool of pages.  Really sucks when you actually
>> have
>> to read  from disk every I/O because you no longer have an in-core file
>> cache :)
>>
>> I'm solving a huge alpha-beta search problem, and want a huge cache table
> tocheck permutations.  That's why I increased RAM from 8 to 32 GB, and would
> be happy to use the extra 24GB all for the cache.  (Cache table is like a
> hash table but with collisions causing replacement rather than chaining).
>
> Even in my small tests, the cache is giving me speedups in the
> neighborhood of
> a million, so I'm going big. Because this is about speed, I do not *ever*
> want it to
> swap.
>
>
>> > I wrote a little test program to try this out, and it fails.  As a
>> regular
>> > user, perror() tells me some "resource is temporarily unavailable".  As
>> > root, it says it "cannot allocate memory".  I'm only asking for 1 byte.
>>
>> >   where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE |
>> > MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
>>
>> Remember that no matter what "length" you pass, even if it's 1 byte, the
>> kernel has to allocate at least enough integer pages to cover the request.
>> You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't
>> quite big enough.
>>
>> And if you ask for MAP_HUGETLB, you're going to get at least 1 page.
>> Which
>> is probably 2M in size.  And that can be painful if the default max
>> lockable
>> memory is less than 2M (at least on my Fedora Rawhide box, it's all of
>> 64K).  There's also some interaction danger with MAP_POPULATE and memory
>> fragmentation.  But both POPULATE and HUGETLB have a high chance of
>> not playing well with LOCKED.
>>
>
> Good catch.  Indeed, removing MAP_HUGETLB allows the program to succeed.
> I'll try increasing the resource limit, but may have to have to sudo it, do
> the mmap() and then chusr to my real ID.
>
> The fd of -1 that some folks thought might be a problem is suggested in
> the man page for use with MAP_ANONYMOUS.  That flag means I'm not using a
> backing file, after all.
>
> --
> Kevin O'Gorman
>
> programmer, n. an organism that transmutes caffeine into software.
>  Please consider the environment before printing this email.
>



-- 
Kevin O'Gorman

programmer, n. an organism that transmutes caffeine into software.
Please consider the environment before printing this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140818/5f6fa068/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 441 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140818/5f6fa068/attachment.gif