transfer physical memory page to swap disk
Hi, Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ? AFAIK, linux kernel is supposed to do this as part of memory management. I want to know if the kernel also exposes some API to enable users to control their application's memory management. I have many applications running at a time that cause too much memory consumption. I believe that experimenting with memory management can help. -- Thanks and Regards, Sumit
Hi, Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ?
I don't think such a system call exists. There might be some mechanisms that may cause your memory to be swapped out as a side effect, but I doubt you have much control over which page gets swapped.
AFAIK, linux kernel is supposed to do this as part of memory management. I want to know if the kernel also exposes some API to enable users to control their application's memory management. I have many applications running at a time that cause too much memory consumption. I believe that experimenting with memory management can help.
I really don't think this kind of management can, or should, be done from user-space. If you have too much memory consumption, try to increase system memory or optimize your applications.
On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
Hi, Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ?
would madvise not serve your needs ? — kind regards anupam -- In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda.
On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ? would madvise not serve your needs ?
There's this word "force" in the question. The problem is that madvise() is *advice* to the kernel, not a strict guarantee. There's a difference between "If you need to move pages to disk, consider these pages first" and "Move them to disk now, whether you really wanted to or not".
Thanks a lot for your responses Aleix, Anupam and Valdis. -- Thanks and Regards, Sumit On Sat, 18 Jan 2020 at 20:40, Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ? would madvise not serve your needs ?
There's this word "force" in the question.
The problem is that madvise() is *advice* to the kernel, not a strict guarantee. There's a difference between "If you need to move pages to disk, consider these pages first" and "Move them to disk now, whether you really wanted to or not".
On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ?
would madvise not serve your needs ?
There's this word "force" in the question.
The problem is that madvise() is *advice* to the kernel, not a strict guarantee. There's a difference between "If you need to move pages to disk, consider these pages first" and "Move them to disk now, whether you really wanted to or not".
ofcourse (afaik) there is no way for the application to force the *kernel* to do something like this. but if _all_ that is required is randomly unmapping some marked application pages, _that_ can be naively 'done' by the application itself :) for example, have a list of unmappable pages and an unmapper thread which does the unmapping for you (picking any element in the list at random) -- kind regards anupam In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. On Sat, Jan 18, 2020 at 3:10 PM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
On Wed, 15 Jan 2020 18:23:05 +0530, Anupam Kapoor said:
On Wed, 15 Jan 2020 at 6:02 PM Sumit Kumar <sumit686215@gmail.com> wrote:
Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it also possible to obtain the least used page using some API ? would madvise not serve your needs ?
There's this word "force" in the question.
The problem is that madvise() is *advice* to the kernel, not a strict guarantee. There's a difference between "If you need to move pages to disk, consider these pages first" and "Move them to disk now, whether you really wanted to or not".
On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said:
but if _all_ that is required is randomly unmapping some marked application pages, _that_ can be naively 'done' by the application itself :)
Note that in this case, "naively" includes "not remembering to consider that the page being unmapped may have contained data we'd rather have kept by flushing the page to disk" :)
but if _all_ that is required is randomly unmapping some marked application pages, _that_ can be naively 'done' by the application itself :)
Note that in this case, "naively" includes "not remembering to consider that the page being unmapped may have contained data we'd rather have kept by flushing the page to disk" :)
but is it that bad ? before marking a page unmappable, the application has full control over what it wants to do with the data, and can choose to dump it to the appropriate destination. or if enough information is available, the unmapper thread can itself play that role. thinking some more about it, application has full control over the unmap/resurrect behavior. though latter might not be as transparent... -- kind regards anupam In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda. On Sun, Jan 19, 2020 at 11:14 AM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
On Sun, 19 Jan 2020 10:55:44 +0000, Anupam Kapoor said:
but if _all_ that is required is randomly unmapping some marked application pages, _that_ can be naively 'done' by the application itself :)
Note that in this case, "naively" includes "not remembering to consider that the page being unmapped may have contained data we'd rather have kept by flushing the page to disk" :)
On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said:
Note that in this case, "naively" includes "not remembering to consider that the page being unmapped may have contained data we'd rather have kept by flushing the page to disk" :)
but is it that bad ?
before marking a page unmappable, the application has full control over what it wants to do with the data, and can choose to dump it to the appropriate destination.
Yes, but now you're getting into more code that has to be written, including code to marshal things like binary trees into a savable format, and more code to read them back at a later time. Plus all the fun if the tree has hundreds of thousands or millions of entries, and how to deal with it if some parts of the tree have been released and saved to disk, or if the 4K page contained members of several different data structures - in other words, you probably just decided to write your own backing store, garbage collector, and virtual object manager for your heap. As I said - it's a naive approach that ends up following the 90/10 rule: the easy 90% of it takes the first 90% of the time to code it, and the difficult 10% takes the other 90% of the time... :)
On Sun, 19 Jan 2020 at 6:48 PM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
On Sun, 19 Jan 2020 12:45:57 +0000, Anupam Kapoor said:
Note that in this case, "naively" includes "not remembering to consider that the page being unmapped may have contained data we'd rather have kept by flushing the page to disk" :)
but is it that bad ?
before marking a page unmappable, the application has full control over what it wants to do with the data, and can choose to dump it to the appropriate destination.
Yes, but now you're getting into more code that has to be written, including code to marshal things like binary trees into a savable format, and more code to read them back at a later time. Plus all the fun if the tree has hundreds of thousands or millions of entries, and how to deal with it if some parts of the tree have been released and saved to disk, or if the 4K page contained members of several different data structures - in other words, you probably just decided to write your own backing store, garbage collector, and virtual object manager for your heap.
As I said - it's a naive approach that ends up following the 90/10 rule: the easy 90% of it takes the first 90% of the time to code it, and the difficult 10% takes the other 90% of the time... :)
well sure, if you try to replicate everything that exists below libc, then there is little hope. however if your application’s data can be serialized/deserialized, then i _suspect_ it might not be too much of work. for example, if i am maintaining l2 forwarding table entries then it might be possible to have, on an average fixed number of pages representing this cache... — kind regards anupam
--
In the beginning was the lambda, and the lambda was with Emacs, and Emacs was the lambda.
Hi all! To answer it from a somewhat different angle: On Wed, 2020-01-15 at 18:01 +0530, Sumit Kumar wrote:
Hi, Does C/C++ provide any API / system call that enables user to force the application to transfer some its physical pages to swap disk ? If so, is it
Short answer: No (and you do not want such a thing in the first place). Longer answer: I don't think any normal application would force some memory pages to the swap space (as it slows down) - quite the contrary. You can use open(..., O_DIRECT) to minimize cache effects if that may help). WRF, you can implement pseudo-swapping with that within your application. madvise() has been mentioned in other mails. One application has (usually) no knowledge about the rest of the system - neither the amount of RAM nor the workload (which may change over time) - or is at least written that way. So it's IMHO better to leave the kernels heuristics work system-wide instead of "micro-optimizing" in some application which will sooner or later interfere with the kernels heuristics. And - always - the next question is: how could this API be abused/creatively used for the own benefit/...? BTW: what's a real intended real-world application?
also possible to obtain the least used page using some API ?
AFAIK, linux kernel is supposed to do this as part of memory management. I
Yes - if there actually is swap space available (und don't underestimate the number of systems running without a swap space and CONFIG_SWAP=n in /boot/*config* - desktop and servers are not everything in the world, more like a small part ....).
want to know if the kernel also exposes some API to enable users to control their application's memory management. I have many applications running at a time that cause too much memory consumption. I believe that experimenting with memory management can help.
The application can (and should) control it's memory management anyways - just allocate as much as the application needs and free it when it's no longer used. For larger memeory areas (possibly with a self-build malloc/free-equivalent or memory pools), the application can mmap() it and munmap() it simply when it's done. You can experiment with setting the process limits via setrlimit() to smaller or larger values. MfG, Bernd -- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at
participants (5)
-
aleix sanchis ramírez -
Anupam Kapoor -
Bernd Petrovitsch -
Sumit Kumar -
Valdis Klētnieks