Now I'm befuddled.  I modified the source so that it was easy to try different combinations.  If I sudo it, I can map a 1GiB (1<<30) private anonymous region if I sudo the program.  I cannot find any combination with MAP_HUGETLB that works at all.

Here it is in  working form, with an anonymous map and no backing file.  The mmap request is for a single byte, so it works for ordinary users  Try uncommenting the MAP_HUGETLB line, and I'd love to hear about a version that works, even if it's only for root.

Since I'm aiming for a huge memory region, I figured HUGETLB would make sense to limit the overhead. 

============================ CUT HERE ================
/**
 * @file
 * <pre>"Find out the limits on locked memory"
 * Last Modified: Mon Aug 18 11:21:23 PDT 2014
 * @author Kevin O'Gorman
 */

#define _GNU_SOURCE /* enable some of the mmap flags */
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>

/* for getrlimit(2) */
#include <sys/time.h>
#include <sys/resource.h>

/* for open(2) */
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/fcntl.h>

int
main(int argc, char *argv[])
{
  void *where;
  size_t length = 1;
  struct rlimit rlim;
  int flags = 0;
  int fd = -1;

//  fd = open("solvedtaq", O_RDWR);  if (fd == -1) perror("open");
  printf("fd is %d\n", fd);

//  flags |= MAP_HUGETLB;
  flags |= MAP_PRIVATE;
  flags |= MAP_LOCKED;
  flags |= MAP_ANONYMOUS;

  if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) {
    printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long long)rlim.rlim_max, (long long)rlim.rlim_cur);
  } else {
    perror("getrlimit failed");
  }

  where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0);
  if (where != MAP_FAILED) {
    printf("Mapped at %p\n", where);
  } else {
    perror("Mapping failed");
  }
 
  return EXIT_SUCCESS;
}
============================ CUT HERE ================



On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman@gmail.com> wrote:

On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:

> maybe someone here can help me figure out how to map a big (really big)
> work area and lock it in memory.

What problem are you trying to solve by doing this?  It usually doesn't make
sense to lock more than a small chunk in memory - that's mostly for crypto
programs like PGP that are paranoid about keys being written to swap spaces,
and a *very* few highly performance oriented programs.  99.8% of the time,
the kernel will do a reasonable job of making sure the right pages are in
memory.

Plus, programmers almost always over-estimate what they *really* need to lock
down.  The end result is that performance ends up being *worse*, because they
lock out more than needed, leaving all the rest of the processes on the system
fighting over a too-small pool of pages.  Really sucks when you actually have
to read  from disk every I/O because you no longer have an in-core file cache :)

I'm solving a huge alpha-beta search problem, and want a huge cache table tocheck permutations.  That's why I increased RAM from 8 to 32 GB, and would
be happy to use the extra 24GB all for the cache.  (Cache table is like a hash table but with collisions causing replacement rather than chaining).

Even in my small tests, the cache is giving me speedups in the neighborhood of
a million, so I'm going big. Because this is about speed, I do not *ever* want it to
swap.
 
> I wrote a little test program to try this out, and it fails.  As a regular
> user, perror() tells me some "resource is temporarily unavailable".  As
> root, it says it "cannot allocate memory".  I'm only asking for 1 byte.

>   where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE |
> MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);

Remember that no matter what "length" you pass, even if it's 1 byte, the
kernel has to allocate at least enough integer pages to cover the request.
You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't
quite big enough.

And if you ask for MAP_HUGETLB, you're going to get at least 1 page.  Which
is probably 2M in size.  And that can be painful if the default max lockable
memory is less than 2M (at least on my Fedora Rawhide box, it's all of
64K).  There's also some interaction danger with MAP_POPULATE and memory
fragmentation.  But both POPULATE and HUGETLB have a high chance of
not playing well with LOCKED.

Good catch.  Indeed, removing MAP_HUGETLB allows the program to succeed.  I'll try increasing the resource limit, but may have to have to sudo it, do the mmap() and then chusr to my real ID.

The fd of -1 that some folks thought might be a problem is suggested in the man page for use with MAP_ANONYMOUS.  That flag means I'm not using a backing file, after all.

--
Kevin O'Gorman

programmer, n. an organism that transmutes caffeine into software.
Please consider the environment before printing this email.




--
Kevin O'Gorman

programmer, n. an organism that transmutes caffeine into software.
Please consider the environment before printing this email.