Why I can't map memory with mmap
maybe someone here can help me figure out how to map a big (really big) work area and lock it in memory. I wrote a little test program to try this out, and it fails. As a regular user, perror() tells me some "resource is temporarily unavailable". As root, it says it "cannot allocate memory". I'm only asking for 1 byte. What's the problem? BTW it also fails if it asks for a mibibyte (1<<20). Here's the whole program /** * @file * <pre>"Find out the limits on locked memory" * Last Modified: Mon Aug 18 07:31:01 PDT 2014</pre> * @author Kevin O'Gorman */ #define _GNU_SOURCE /* enable some of the mmap flags */ #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include <sys/mman.h> int main(int argc, char *argv[]) { void *where; size_t length = 1; where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0); if (where != MAP_FAILED) { printf("Mapped at %p\n", where); } else { perror("Mapping failed"); } return EXIT_SUCCESS; } -- Kevin O'Gorman programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies-bounces@kernelnewbies.org] On Behalf Of Kevin O'Gorman Sent: Monday, August 18, 2014 8:15 AM To: kernelnewbies@kernelnewbies.org Subject: Why I can't map memory with mmap maybe someone here can help me figure out how to map a big (really big) work area and lock it in memory. I wrote a little test program to try this out, and it fails. As a regular user, perror() tells me some "resource is temporarily unavailable". As root, it says it "cannot allocate memory". I'm only asking for 1 byte. What's the problem? BTW it also fails if it asks for a mibibyte (1<<20). Here's the whole program /** * @file * <pre>"Find out the limits on locked memory" * Last Modified: Mon Aug 18 07:31:01 PDT 2014</pre> * @author Kevin O'Gorman */ #define _GNU_SOURCE /* enable some of the mmap flags */ #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include <sys/mman.h> int main(int argc, char *argv[]) { void *where; size_t length = 1; where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0); if (where != MAP_FAILED) { printf("Mapped at %p\n", where); } else { perror("Mapping failed"); } return EXIT_SUCCESS; } I'm pretty sure you need to pass the file descriptor of an open file or device as the fd (second parameter from the end). -1 is not a valid fd. Jeff Haran
On Mon, 18 Aug 2014 16:42:33 -0000, Jeff Haran said:
I'm pretty sure you need to pass the file descriptor of an open file or device as the fd (second parameter from the end). -1 is not a valid fd.
Good catch. But that should throw a different error than "resource unavailable" - I'd expect to get EBADF back for that and it sounds like he's getting EAGAIN back... But yeah, that needs fixing too. A common trick is to fd = open("/dev/zero",.... to get an fd that can be used for mmap of anonymous space. Linux old-timers will remember that /lib/ld-linux.so does this - with the non-obvious result that trying to run a program that was linked against shared libraries would fail in a chroot even if all the libraries were present, if /dev/zero wasn't in the chroot as well....)
On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
maybe someone here can help me figure out how to map a big (really big) work area and lock it in memory.
What problem are you trying to solve by doing this? It usually doesn't make sense to lock more than a small chunk in memory - that's mostly for crypto programs like PGP that are paranoid about keys being written to swap spaces, and a *very* few highly performance oriented programs. 99.8% of the time, the kernel will do a reasonable job of making sure the right pages are in memory. Plus, programmers almost always over-estimate what they *really* need to lock down. The end result is that performance ends up being *worse*, because they lock out more than needed, leaving all the rest of the processes on the system fighting over a too-small pool of pages. Really sucks when you actually have to read from disk every I/O because you no longer have an in-core file cache :)
I wrote a little test program to try this out, and it fails. As a regular user, perror() tells me some "resource is temporarily unavailable". As root, it says it "cannot allocate memory". I'm only asking for 1 byte.
where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
Remember that no matter what "length" you pass, even if it's 1 byte, the kernel has to allocate at least enough integer pages to cover the request. You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't quite big enough. And if you ask for MAP_HUGETLB, you're going to get at least 1 page. Which is probably 2M in size. And that can be painful if the default max lockable memory is less than 2M (at least on my Fedora Rawhide box, it's all of 64K). There's also some interaction danger with MAP_POPULATE and memory fragmentation. But both POPULATE and HUGETLB have a high chance of not playing well with LOCKED.
On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
maybe someone here can help me figure out how to map a big (really big) work area and lock it in memory.
What problem are you trying to solve by doing this? It usually doesn't make sense to lock more than a small chunk in memory - that's mostly for crypto programs like PGP that are paranoid about keys being written to swap spaces, and a *very* few highly performance oriented programs. 99.8% of the time, the kernel will do a reasonable job of making sure the right pages are in memory.
Plus, programmers almost always over-estimate what they *really* need to lock down. The end result is that performance ends up being *worse*, because they lock out more than needed, leaving all the rest of the processes on the system fighting over a too-small pool of pages. Really sucks when you actually have to read from disk every I/O because you no longer have an in-core file cache :)
I'm solving a huge alpha-beta search problem, and want a huge cache table tocheck permutations. That's why I increased RAM from 8 to 32 GB, and would be happy to use the extra 24GB all for the cache. (Cache table is like a hash table but with collisions causing replacement rather than chaining).
Even in my small tests, the cache is giving me speedups in the neighborhood of a million, so I'm going big. Because this is about speed, I do not *ever* want it to swap.
I wrote a little test program to try this out, and it fails. As a regular user, perror() tells me some "resource is temporarily unavailable". As root, it says it "cannot allocate memory". I'm only asking for 1 byte.
where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
Remember that no matter what "length" you pass, even if it's 1 byte, the kernel has to allocate at least enough integer pages to cover the request. You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't quite big enough.
And if you ask for MAP_HUGETLB, you're going to get at least 1 page. Which is probably 2M in size. And that can be painful if the default max lockable memory is less than 2M (at least on my Fedora Rawhide box, it's all of 64K). There's also some interaction danger with MAP_POPULATE and memory fragmentation. But both POPULATE and HUGETLB have a high chance of not playing well with LOCKED.
Good catch. Indeed, removing MAP_HUGETLB allows the program to succeed. I'll try increasing the resource limit, but may have to have to sudo it, do the mmap() and then chusr to my real ID. The fd of -1 that some folks thought might be a problem is suggested in the man page for use with MAP_ANONYMOUS. That flag means I'm not using a backing file, after all. -- Kevin O'Gorman programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
Now I'm befuddled. I modified the source so that it was easy to try different combinations. If I sudo it, I can map a 1GiB (1<<30) private anonymous region if I sudo the program. I cannot find any combination with MAP_HUGETLB that works at all. Here it is in working form, with an anonymous map and no backing file. The mmap request is for a single byte, so it works for ordinary users Try uncommenting the MAP_HUGETLB line, and I'd love to hear about a version that works, even if it's only for root. Since I'm aiming for a huge memory region, I figured HUGETLB would make sense to limit the overhead. ============================ CUT HERE ================ /** * @file * <pre>"Find out the limits on locked memory" * Last Modified: Mon Aug 18 11:21:23 PDT 2014 * @author Kevin O'Gorman */ #define _GNU_SOURCE /* enable some of the mmap flags */ #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include <sys/mman.h> /* for getrlimit(2) */ #include <sys/time.h> #include <sys/resource.h> /* for open(2) */ #include <sys/types.h> #include <sys/stat.h> #include <sys/fcntl.h> int main(int argc, char *argv[]) { void *where; size_t length = 1; struct rlimit rlim; int flags = 0; int fd = -1; // fd = open("solvedtaq", O_RDWR); if (fd == -1) perror("open"); printf("fd is %d\n", fd); // flags |= MAP_HUGETLB; flags |= MAP_PRIVATE; flags |= MAP_LOCKED; flags |= MAP_ANONYMOUS; if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) { printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long long)rlim.rlim_max, (long long)rlim.rlim_cur); } else { perror("getrlimit failed"); } where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0); if (where != MAP_FAILED) { printf("Mapped at %p\n", where); } else { perror("Mapping failed"); } return EXIT_SUCCESS; } ============================ CUT HERE ================ On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman@gmail.com> wrote:
On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
maybe someone here can help me figure out how to map a big (really big) work area and lock it in memory.
What problem are you trying to solve by doing this? It usually doesn't make sense to lock more than a small chunk in memory - that's mostly for crypto programs like PGP that are paranoid about keys being written to swap spaces, and a *very* few highly performance oriented programs. 99.8% of the time, the kernel will do a reasonable job of making sure the right pages are in memory.
Plus, programmers almost always over-estimate what they *really* need to lock down. The end result is that performance ends up being *worse*, because they lock out more than needed, leaving all the rest of the processes on the system fighting over a too-small pool of pages. Really sucks when you actually have to read from disk every I/O because you no longer have an in-core file cache :)
I'm solving a huge alpha-beta search problem, and want a huge cache table tocheck permutations. That's why I increased RAM from 8 to 32 GB, and would be happy to use the extra 24GB all for the cache. (Cache table is like a hash table but with collisions causing replacement rather than chaining).
Even in my small tests, the cache is giving me speedups in the neighborhood of a million, so I'm going big. Because this is about speed, I do not *ever* want it to swap.
I wrote a little test program to try this out, and it fails. As a regular user, perror() tells me some "resource is temporarily unavailable". As root, it says it "cannot allocate memory". I'm only asking for 1 byte.
where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
Remember that no matter what "length" you pass, even if it's 1 byte, the kernel has to allocate at least enough integer pages to cover the request. You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't quite big enough.
And if you ask for MAP_HUGETLB, you're going to get at least 1 page. Which is probably 2M in size. And that can be painful if the default max lockable memory is less than 2M (at least on my Fedora Rawhide box, it's all of 64K). There's also some interaction danger with MAP_POPULATE and memory fragmentation. But both POPULATE and HUGETLB have a high chance of not playing well with LOCKED.
Good catch. Indeed, removing MAP_HUGETLB allows the program to succeed. I'll try increasing the resource limit, but may have to have to sudo it, do the mmap() and then chusr to my real ID.
The fd of -1 that some folks thought might be a problem is suggested in the man page for use with MAP_ANONYMOUS. That flag means I'm not using a backing file, after all.
-- Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
-- Kevin O'Gorman programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
I tried your program and got the same result, i.e "Resource temporarily unavailable" for an ordinary user and "Cannot allocate memory" if run as root. However, reading the Huge TLB pages doc (Documentation/vm/hugetlbpage.txt), I saw the following para which might explain the error: ... Once a number of huge pages have been pre-allocated to the kernel huge page pool, a user with appropriate privilege can use either the mmap system call or shared memory system calls to use the huge pages. See the discussion of Using Huge Pages, below. The administrator can allocate persistent huge pages on the kernel boot command line by specifying the "hugepages=N" parameter, where 'N' = the number of huge pages requested. This is the most reliable method of allocating huge pages as memory has not yet become fragmented. ... ... I don't have my kernel booted with it, so that might explain why mmap fails for MAP_HUGETLB. I can't restart my machine with the new flag right now, but maybe you can try. HTH, -mandeep On Mon, Aug 18, 2014 at 11:30 AM, Kevin O'Gorman <kogorman@gmail.com> wrote:
Now I'm befuddled. I modified the source so that it was easy to try different combinations. If I sudo it, I can map a 1GiB (1<<30) private anonymous region if I sudo the program. I cannot find any combination with MAP_HUGETLB that works at all.
Here it is in working form, with an anonymous map and no backing file. The mmap request is for a single byte, so it works for ordinary users Try uncommenting the MAP_HUGETLB line, and I'd love to hear about a version that works, even if it's only for root.
Since I'm aiming for a huge memory region, I figured HUGETLB would make sense to limit the overhead.
============================ CUT HERE ================ /** * @file * <pre>"Find out the limits on locked memory" * Last Modified: Mon Aug 18 11:21:23 PDT 2014 * @author Kevin O'Gorman */
#define _GNU_SOURCE /* enable some of the mmap flags */ #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include <sys/mman.h>
/* for getrlimit(2) */ #include <sys/time.h> #include <sys/resource.h>
/* for open(2) */ #include <sys/types.h> #include <sys/stat.h> #include <sys/fcntl.h>
int main(int argc, char *argv[]) { void *where; size_t length = 1; struct rlimit rlim; int flags = 0; int fd = -1;
// fd = open("solvedtaq", O_RDWR); if (fd == -1) perror("open"); printf("fd is %d\n", fd);
// flags |= MAP_HUGETLB; flags |= MAP_PRIVATE; flags |= MAP_LOCKED; flags |= MAP_ANONYMOUS;
if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) { printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long long)rlim.rlim_max, (long long)rlim.rlim_cur); } else { perror("getrlimit failed"); }
where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0); if (where != MAP_FAILED) { printf("Mapped at %p\n", where); } else { perror("Mapping failed"); }
return EXIT_SUCCESS; } ============================ CUT HERE ================
On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman@gmail.com> wrote:
On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
maybe someone here can help me figure out how to map a big (really big) work area and lock it in memory.
What problem are you trying to solve by doing this? It usually doesn't make sense to lock more than a small chunk in memory - that's mostly for crypto programs like PGP that are paranoid about keys being written to swap spaces, and a *very* few highly performance oriented programs. 99.8% of the time, the kernel will do a reasonable job of making sure the right pages are in memory.
Plus, programmers almost always over-estimate what they *really* need to lock down. The end result is that performance ends up being *worse*, because they lock out more than needed, leaving all the rest of the processes on the system fighting over a too-small pool of pages. Really sucks when you actually have to read from disk every I/O because you no longer have an in-core file cache :)
I'm solving a huge alpha-beta search problem, and want a huge cache table tocheck permutations. That's why I increased RAM from 8 to 32 GB, and would be happy to use the extra 24GB all for the cache. (Cache table is like a hash table but with collisions causing replacement rather than chaining).
Even in my small tests, the cache is giving me speedups in the neighborhood of a million, so I'm going big. Because this is about speed, I do not *ever* want it to swap.
I wrote a little test program to try this out, and it fails. As a regular user, perror() tells me some "resource is temporarily unavailable". As root, it says it "cannot allocate memory". I'm only asking for 1 byte.
where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
Remember that no matter what "length" you pass, even if it's 1 byte, the kernel has to allocate at least enough integer pages to cover the request. You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't quite big enough.
And if you ask for MAP_HUGETLB, you're going to get at least 1 page. Which is probably 2M in size. And that can be painful if the default max lockable memory is less than 2M (at least on my Fedora Rawhide box, it's all of 64K). There's also some interaction danger with MAP_POPULATE and memory fragmentation. But both POPULATE and HUGETLB have a high chance of not playing well with LOCKED.
Good catch. Indeed, removing MAP_HUGETLB allows the program to succeed. I'll try increasing the resource limit, but may have to have to sudo it, do the mmap() and then chusr to my real ID.
The fd of -1 that some folks thought might be a problem is suggested in the man page for use with MAP_ANONYMOUS. That flag means I'm not using a backing file, after all.
-- Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
-- Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
I sent the previous mail too early! I got it to work for root user by setting "/proc/sys/vm/nr_hugepages" to 20 as suggested in Documentation/vm/hugetlbpage.txt. Now I'm able to mmap it and it reflects the same in /proc/meminfo as well! :) HTH, -mandeep On Mon, Aug 18, 2014 at 1:23 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com
wrote:
I tried your program and got the same result, i.e "Resource temporarily unavailable" for an ordinary user and "Cannot allocate memory" if run as root.
However, reading the Huge TLB pages doc (Documentation/vm/hugetlbpage.txt), I saw the following para which might explain the error: ... Once a number of huge pages have been pre-allocated to the kernel huge page pool, a user with appropriate privilege can use either the mmap system call or shared memory system calls to use the huge pages. See the discussion of Using Huge Pages, below.
The administrator can allocate persistent huge pages on the kernel boot command line by specifying the "hugepages=N" parameter, where 'N' = the number of huge pages requested. This is the most reliable method of allocating huge pages as memory has not yet become fragmented. ... ...
I don't have my kernel booted with it, so that might explain why mmap fails for MAP_HUGETLB.
I can't restart my machine with the new flag right now, but maybe you can try.
HTH,
-mandeep
On Mon, Aug 18, 2014 at 11:30 AM, Kevin O'Gorman <kogorman@gmail.com> wrote:
Now I'm befuddled. I modified the source so that it was easy to try different combinations. If I sudo it, I can map a 1GiB (1<<30) private anonymous region if I sudo the program. I cannot find any combination with MAP_HUGETLB that works at all.
Here it is in working form, with an anonymous map and no backing file. The mmap request is for a single byte, so it works for ordinary users Try uncommenting the MAP_HUGETLB line, and I'd love to hear about a version that works, even if it's only for root.
Since I'm aiming for a huge memory region, I figured HUGETLB would make sense to limit the overhead.
============================ CUT HERE ================ /** * @file * <pre>"Find out the limits on locked memory" * Last Modified: Mon Aug 18 11:21:23 PDT 2014 * @author Kevin O'Gorman */
#define _GNU_SOURCE /* enable some of the mmap flags */ #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include <sys/mman.h>
/* for getrlimit(2) */ #include <sys/time.h> #include <sys/resource.h>
/* for open(2) */ #include <sys/types.h> #include <sys/stat.h> #include <sys/fcntl.h>
int main(int argc, char *argv[]) { void *where; size_t length = 1; struct rlimit rlim; int flags = 0; int fd = -1;
// fd = open("solvedtaq", O_RDWR); if (fd == -1) perror("open"); printf("fd is %d\n", fd);
// flags |= MAP_HUGETLB; flags |= MAP_PRIVATE; flags |= MAP_LOCKED; flags |= MAP_ANONYMOUS;
if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) { printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long long)rlim.rlim_max, (long long)rlim.rlim_cur); } else { perror("getrlimit failed"); }
where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0); if (where != MAP_FAILED) { printf("Mapped at %p\n", where); } else { perror("Mapping failed"); }
return EXIT_SUCCESS; } ============================ CUT HERE ================
On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman@gmail.com> wrote:
On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
maybe someone here can help me figure out how to map a big (really big) work area and lock it in memory.
What problem are you trying to solve by doing this? It usually doesn't make sense to lock more than a small chunk in memory - that's mostly for crypto programs like PGP that are paranoid about keys being written to swap spaces, and a *very* few highly performance oriented programs. 99.8% of the time, the kernel will do a reasonable job of making sure the right pages are in memory.
Plus, programmers almost always over-estimate what they *really* need to lock down. The end result is that performance ends up being *worse*, because they lock out more than needed, leaving all the rest of the processes on the system fighting over a too-small pool of pages. Really sucks when you actually have to read from disk every I/O because you no longer have an in-core file cache :)
I'm solving a huge alpha-beta search problem, and want a huge cache table tocheck permutations. That's why I increased RAM from 8 to 32 GB, and would be happy to use the extra 24GB all for the cache. (Cache table is like a hash table but with collisions causing replacement rather than chaining).
Even in my small tests, the cache is giving me speedups in the neighborhood of a million, so I'm going big. Because this is about speed, I do not *ever* want it to swap.
I wrote a little test program to try this out, and it fails. As a regular user, perror() tells me some "resource is temporarily unavailable". As root, it says it "cannot allocate memory". I'm only asking for 1 byte.
where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
Remember that no matter what "length" you pass, even if it's 1 byte, the kernel has to allocate at least enough integer pages to cover the request. You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't quite big enough.
And if you ask for MAP_HUGETLB, you're going to get at least 1 page. Which is probably 2M in size. And that can be painful if the default max lockable memory is less than 2M (at least on my Fedora Rawhide box, it's all of 64K). There's also some interaction danger with MAP_POPULATE and memory fragmentation. But both POPULATE and HUGETLB have a high chance of not playing well with LOCKED.
Good catch. Indeed, removing MAP_HUGETLB allows the program to succeed. I'll try increasing the resource limit, but may have to have to sudo it, do the mmap() and then chusr to my real ID.
The fd of -1 that some folks thought might be a problem is suggested in the man page for use with MAP_ANONYMOUS. That flag means I'm not using a backing file, after all.
-- Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
-- Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Mandeep has solved it for me. I was able to set /proc/sys/vm/nr_hugepages to 12288, corresponding to 24GiB of memory, and make a corresponding mmap request. It takes a while -- a very noticeable couple of seconds -- but it works. There are a few things still to be worked out, like why that setting is reduced when my little program runs. So that it has to be re-established in order to run it again. But it gets me started. On Mon, Aug 18, 2014 at 1:25 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com
wrote:
I sent the previous mail too early!
I got it to work for root user by setting "/proc/sys/vm/nr_hugepages" to 20 as suggested in Documentation/vm/hugetlbpage.txt. Now I'm able to mmap it and it reflects the same in /proc/meminfo as well! :)
HTH, -mandeep
On Mon, Aug 18, 2014 at 1:23 PM, Mandeep Sandhu < mandeepsandhu.chd@gmail.com> wrote:
I tried your program and got the same result, i.e "Resource temporarily unavailable" for an ordinary user and "Cannot allocate memory" if run as root.
However, reading the Huge TLB pages doc (Documentation/vm/hugetlbpage.txt), I saw the following para which might explain the error: ... Once a number of huge pages have been pre-allocated to the kernel huge page pool, a user with appropriate privilege can use either the mmap system call or shared memory system calls to use the huge pages. See the discussion of Using Huge Pages, below.
The administrator can allocate persistent huge pages on the kernel boot command line by specifying the "hugepages=N" parameter, where 'N' = the number of huge pages requested. This is the most reliable method of allocating huge pages as memory has not yet become fragmented. ... ...
I don't have my kernel booted with it, so that might explain why mmap fails for MAP_HUGETLB.
I can't restart my machine with the new flag right now, but maybe you can try.
HTH,
-mandeep
On Mon, Aug 18, 2014 at 11:30 AM, Kevin O'Gorman <kogorman@gmail.com> wrote:
Now I'm befuddled. I modified the source so that it was easy to try different combinations. If I sudo it, I can map a 1GiB (1<<30) private anonymous region if I sudo the program. I cannot find any combination with MAP_HUGETLB that works at all.
Here it is in working form, with an anonymous map and no backing file. The mmap request is for a single byte, so it works for ordinary users Try uncommenting the MAP_HUGETLB line, and I'd love to hear about a version that works, even if it's only for root.
Since I'm aiming for a huge memory region, I figured HUGETLB would make sense to limit the overhead.
============================ CUT HERE ================ /** * @file * <pre>"Find out the limits on locked memory" * Last Modified: Mon Aug 18 11:21:23 PDT 2014 * @author Kevin O'Gorman */
#define _GNU_SOURCE /* enable some of the mmap flags */ #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include <sys/mman.h>
/* for getrlimit(2) */ #include <sys/time.h> #include <sys/resource.h>
/* for open(2) */ #include <sys/types.h> #include <sys/stat.h> #include <sys/fcntl.h>
int main(int argc, char *argv[]) { void *where; size_t length = 1; struct rlimit rlim; int flags = 0; int fd = -1;
// fd = open("solvedtaq", O_RDWR); if (fd == -1) perror("open"); printf("fd is %d\n", fd);
// flags |= MAP_HUGETLB; flags |= MAP_PRIVATE; flags |= MAP_LOCKED; flags |= MAP_ANONYMOUS;
if (getrlimit(RLIMIT_MEMLOCK, &rlim) == 0) { printf("RLIMIT_MEMLOCK: hard: %lld, soft: %lld\n", (long long)rlim.rlim_max, (long long)rlim.rlim_cur); } else { perror("getrlimit failed"); }
where = mmap(NULL, length, PROT_READ | PROT_WRITE, flags, fd, 0); if (where != MAP_FAILED) { printf("Mapped at %p\n", where); } else { perror("Mapping failed"); }
return EXIT_SUCCESS; } ============================ CUT HERE ================
On Mon, Aug 18, 2014 at 10:24 AM, Kevin O'Gorman <kogorman@gmail.com> wrote:
On Mon, Aug 18, 2014 at 9:46 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 18 Aug 2014 08:15:12 -0700, "Kevin O'Gorman" said:
maybe someone here can help me figure out how to map a big (really big) work area and lock it in memory.
What problem are you trying to solve by doing this? It usually doesn't make sense to lock more than a small chunk in memory - that's mostly for crypto programs like PGP that are paranoid about keys being written to swap spaces, and a *very* few highly performance oriented programs. 99.8% of the time, the kernel will do a reasonable job of making sure the right pages are in memory.
Plus, programmers almost always over-estimate what they *really* need to lock down. The end result is that performance ends up being *worse*, because they lock out more than needed, leaving all the rest of the processes on the system fighting over a too-small pool of pages. Really sucks when you actually have to read from disk every I/O because you no longer have an in-core file cache :)
I'm solving a huge alpha-beta search problem, and want a huge cache table tocheck permutations. That's why I increased RAM from 8 to 32 GB, and would be happy to use the extra 24GB all for the cache. (Cache table is like a hash table but with collisions causing replacement rather than chaining).
Even in my small tests, the cache is giving me speedups in the neighborhood of a million, so I'm going big. Because this is about speed, I do not *ever* want it to swap.
I wrote a little test program to try this out, and it fails. As a regular user, perror() tells me some "resource is temporarily unavailable". As root, it says it "cannot allocate memory". I'm only asking for 1 byte.
where = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED | MAP_POPULATE, -1, 0);
Remember that no matter what "length" you pass, even if it's 1 byte, the kernel has to allocate at least enough integer pages to cover the request. You ask for 4100 bytes, it has to allocate 2 pages because 4096 isn't quite big enough.
And if you ask for MAP_HUGETLB, you're going to get at least 1 page. Which is probably 2M in size. And that can be painful if the default max lockable memory is less than 2M (at least on my Fedora Rawhide box, it's all of 64K). There's also some interaction danger with MAP_POPULATE and memory fragmentation. But both POPULATE and HUGETLB have a high chance of not playing well with LOCKED.
Good catch. Indeed, removing MAP_HUGETLB allows the program to succeed. I'll try increasing the resource limit, but may have to have to sudo it, do the mmap() and then chusr to my real ID.
The fd of -1 that some folks thought might be a problem is suggested in the man page for use with MAP_ANONYMOUS. That flag means I'm not using a backing file, after all.
-- Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
-- Kevin O'Gorman
programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-- Kevin O'Gorman programmer, n. an organism that transmutes caffeine into software. Please consider the environment before printing this email.
On Mon, 18 Aug 2014 10:24:45 -0700, you said:
I'm solving a huge alpha-beta search problem, and want a huge cache table tocheck permutations. That's why I increased RAM from 8 to 32 GB, and would be happy to use the extra 24GB all for the cache. (Cache table is like a hash table but with collisions causing replacement rather than chaining).
Even in my small tests, the cache is giving me speedups in the neighborhood of a million, so I'm going big. Because this is about speed, I do not *ever* want it to swap.
Throwing another 24G at that sort of program is probably a Good Idea(tm) (although I'd check how much better your hit ratio is at 8G and 32G. However, locking it into memory isn't always obviously a good idea - the problem is that it's *way* too easy to overcommit locked memory and end up starving the rest of the system to the point where your overall performance actually drops. And if you're actually hitting all the pages, and have enough RAM, they *will* stay in memory anyhow. (And 98% of the time, the delay in paging in a swapped-out page was more than made up for by the time savings in having whatever pages *was* resident). Oh, and don't forget that if you use MAP_POPULATE, you pay the costs of populating that page (and evicting whatever used to be there) *even if you never actually use the page*. So make damned sure you have a really good hash function that hits all the buckets (this is often harder to do than it looks, especially for very large numbers of buckets). We have a Honking Big box across the hall - SGI UV, 504 cores, 2.7T of RAM, and one of the *fastest* ways to bring it to a screeching halt was locking the wrong 8G of stuff into RAM. Yes, you can find a corner case where locking only 0.3% of the RAM will panic the box. NUMA memory allocations can be *so* much fun... :)
participants (4)
-
Jeff Haran -
Kevin O'Gorman -
Mandeep Sandhu -
Valdis.Kletnieks＠vt.edu