I'm trying to think of a good way to figure out where space is going on somewhat large and busy filesystems. Often I will be notified by nagios that I am running out of space somewhere, and it will even tell me what the trend in usage is. So now I know that I lost a bunch of space recently. There are several things I can do at that point: - Check quota usage. If quotas are working correctly, this can sometimes help when one user is obviously using a ton of space. More often than not, there is nothing obvious, quotas aren't working or the space is used by system accounts that provide no help in knowing where the space went. - Use du. This is just plain painful and time consuming. Doing this usually causes too much load. Without knowing other ways to find information, I was thinking it would be really nice to have a way to see which files or directories grew or shrank since some point in the past. Would it be possible to get the kernel to tell me when files were opened for writing and closed so I could record the size difference? I know there are various file notify mechanisms, but from what I can see, they don't handle millions of files very well. Of course I'm open to other ideas, but this was my curiosity.
On Wed, Aug 10, 2011 at 07:06:29PM -0400, Chris wrote:
I'm trying to think of a good way to figure out where space is going on somewhat large and busy filesystems. Often I will be notified by nagios that I am running out of space somewhere, and it will even tell me what the trend in usage is. So now I know that I lost a bunch of space recently.
There are several things I can do at that point: - Check quota usage. If quotas are working correctly, this can sometimes help when one user is obviously using a ton of space. More often than not, there is nothing obvious, quotas aren't working or the space is used by system accounts that provide no help in knowing where the space went. - Use du. This is just plain painful and time consuming. Doing this usually causes too much load.
Without knowing other ways to find information, I was thinking it would be really nice to have a way to see which files or directories grew or shrank since some point in the past. Would it be possible to get the kernel to tell me when files were opened for writing and closed so I could record the size difference? I know there are various file notify mechanisms, but from what I can see, they don't handle millions of files very well.
Of course I'm open to other ideas, but this was my curiosity.
Will Inotify help? http://en.wikipedia.org/wiki/Inotify http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html -- Regards, Adam Lee -------------------------------------------------- E-mail: adam8157@gmail.com Website: http://www.adam8157.info --------------------------------------------------
I like "inotify". I've become somewhat familiar with the interface and I believe this will allow you to do what you want. There are plenty "C" examples out there. Cheers, -----Original Message----- From: Adam Lee Sent: Thursday, August 11, 2011 12:17 PM To: kernelnewbies@kernelnewbies.org Subject: Re: Tracing file size changes On Wed, Aug 10, 2011 at 07:06:29PM -0400, Chris wrote:
I'm trying to think of a good way to figure out where space is going on somewhat large and busy filesystems. Often I will be notified by nagios that I am running out of space somewhere, and it will even tell me what the trend in usage is. So now I know that I lost a bunch of space recently.
There are several things I can do at that point: - Check quota usage. If quotas are working correctly, this can sometimes help when one user is obviously using a ton of space. More often than not, there is nothing obvious, quotas aren't working or the space is used by system accounts that provide no help in knowing where the space went. - Use du. This is just plain painful and time consuming. Doing this usually causes too much load.
Without knowing other ways to find information, I was thinking it would be really nice to have a way to see which files or directories grew or shrank since some point in the past. Would it be possible to get the kernel to tell me when files were opened for writing and closed so I could record the size difference? I know there are various file notify mechanisms, but from what I can see, they don't handle millions of files very well.
Of course I'm open to other ideas, but this was my curiosity.
Will Inotify help? http://en.wikipedia.org/wiki/Inotify http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html -- Regards, Adam Lee -------------------------------------------------- E-mail: adam8157@gmail.com Website: http://www.adam8157.info -------------------------------------------------- _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Thu, Aug 11, 2011 at 10:47:00AM +0800, Adam Lee wrote:
Without knowing other ways to find information, I was thinking it would be really nice to have a way to see which files or directories grew or shrank since some point in the past. Would it be possible to get the kernel to tell me when files were opened for writing and closed so I could record the size difference? >
Will Inotify help?
http://en.wikipedia.org/wiki/Inotify http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html
It is pretty much what I want to do, the problem with it is that it doesn't appear to be recursive. I'd like to just put a watch on root, and get events for the entire filesystem, not just that directory. From playing with this for a couple minutes, it looks like I'd have to walk the entire filesystem setting watches on every directory. So while it is very cool for what it does, I don't think it will help me out much. Is doing something with ld_preload to get in the way of every open call the only way to do this?
Will Inotify help?
http://en.wikipedia.org/wiki/Inotify http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html
It is pretty much what I want to do, the problem with it is that it doesn't appear to be recursive.
I looked again tonight, and it appears that newer kernels with fanotify are able to do this. So the feature is available, just too new to be on my servers. Oh, well :P Thanks for the input. I will probably just end up doing what I can with ld_preload. Chris
On Fri, Aug 12, 2011 at 08:58:41PM -0400, Chris wrote:
Will Inotify help?
http://en.wikipedia.org/wiki/Inotify http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html
It is pretty much what I want to do, the problem with it is that it doesn't appear to be recursive.
I looked again tonight, and it appears that newer kernels with fanotify are able to do this. So the feature is available, just too new to be on my servers. Oh, well :P
Thanks for the input. I will probably just end up doing what I can with ld_preload.
Chris
Hi Chris, I'm not sure whether that would help or not, but you could also do it in kernel space using LSM (if you don't already use another LSM module). With the file_security hook, you can track any changes on files and store information in the extended attributes of the filesystem for example. This might be a bit more flexible than a system wide ld_preload. My two cents. -- Christophe
participants (4)
-
Adam Lee -
Chris -
Christophe Hauser -
Vaughn Clinton