Why I can't map memory with mmap

[Kevin O'Gorman]

On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks at vt.edu> wrote:

> On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
>
> > maybe someone here can help me figure out how to map a big (really big)
> > work area and lock it in memory.
>
> What problem are you trying to solve by doing this?  It usually doesn't
> make
> sense to lock more than a small chunk in memory - that's mostly for crypto
> programs like PGP that are paranoid about keys being written to swap
> spaces,
> and a *very* few highly performance oriented programs.  99.8% of the time,
> the kernel will do a reasonable job of making sure the right pages are in
> memory.
>
> Plus, programmers almost always over-estimate what they *really* need to
> lock
> down.  The end result is that performance ends up being *worse*, because
> they
> lock out more than needed, leaving all the rest of the processes on the
> system
> fighting over a too-small pool of pages.  Really sucks when you actually
> have
> to read  from disk every I/O because you no longer have an in-core file
> cache :)
>
> I'm solving a huge alpha-beta search problem, and want a huge cache table
tocheck permutations.  That's why I increased RAM from 8 to 32 GB, and would
be happy to use the extra 24GB all for the cache.  (Cache table is like a
hash table but with collisions causing replacement rather than chaining).

Even in my small tests, the cache is giving me speedups in the neighborhood
of
a million, so I'm going big. Because this is about speed, I do not *ever*
want it to
swap.


> > I wrote a little test program to try this out, and it fails.  As a
> regular
> > user, perror() tells me some "resource is temporarily unavailable".  As
> > root, it says it "cannot allocate memory".  I'm only asking for 1 byte.
>
> >   where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE |
> > MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
>
> Remember that no matter what "length" you pass, even if it's 1 byte, the
> kernel has to allocate at least enough integer pages to cover the request.
> You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't
> quite big enough.
>
> And if you ask for MAP_HUGETLB, you're going to get at least 1 page.  Which
> is probably 2M in size.  And that can be painful if the default max
> lockable
> memory is less than 2M (at least on my Fedora Rawhide box, it's all of
> 64K).  There's also some interaction danger with MAP_POPULATE and memory
> fragmentation.  But both POPULATE and HUGETLB have a high chance of
> not playing well with LOCKED.
>

Good catch.  Indeed, removing MAP_HUGETLB allows the program to succeed.
I'll try increasing the resource limit, but may have to have to sudo it, do
the mmap() and then chusr to my real ID.

The fd of -1 that some folks thought might be a problem is suggested in the
man page for use with MAP_ANONYMOUS.  That flag means I'm not using a
backing file, after all.

-- 
Kevin O'Gorman

programmer, n. an organism that transmutes caffeine into software.
Please consider the environment before printing this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140818/03e77fe4/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 441 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140818/03e77fe4/attachment.gif