vfs_cache_pressure - time to reevaluate the defaults?

[Felix Howe]

Hello,

(I'm new to this mailing list - this seemed the most appropriate place
to ask about this; apologies if it is not.)


short version:

There is a parameter called 'vfs_cache_pressure', which can be tweaked
at runtime via /proc/sys/vm/vfs_cache_pressure.

By default, file/dir metadata is not cached much, so recursive
operations (e.g. 'du -sh', 'find', ...) are always deathly slow on
spinning disks.

Using vfs_cache_pressure, it is possible to force pinning metadata of an
entire filesystem in RAM, making these operations just as fast on
spinning disk as they are on SSD.

I am wondering whether this should be a new default for slow disks on
systems with enough RAM, and some other possibilities.


long version:

I discovered the vfs_cache_pressure parameter today while searching for
solutions to a very old problem of mine: namely, "how does every
filesystem know immediately exactly how much free space it has, even on
spinning disks, but if I ask it how big this folder is it takes 15
minutes of thrashing? And why does it take so long to find files that
tools like 'locate' were developed?"

I had toyed with the idea of trying to write a filesystem that is
better at these problems, and later, of trying to change an existing
filesystem driver to favour large sequential reads to snarf up
metadata without so many seeks (even at the cost of reading a bunch of
data it doesn't need), and still later, "what if I just rewrote it to
read all the matadata immediately and pin it in RAM?", which brought me
to, "aha, someone has already done this!"

According to

https://www.kernel.org/doc/Documentation/sysctl/vm.txt

the default is to drop cached dentries/inodes at a "fair" rate relative
to other stuff, and setting vfs_cache_pressure=0 will effectively do
what I want. It then has this caution against doing that:

> "When vfs_cache_pressure=0, the kernel will never reclaim dentries and
> inodes due to memory pressure and this can easily lead to
> out-of-memory conditions."

I thought to myself, "that doesn't seem right; how much RAM could it
possibly use?" and grabbed one of my big, slow disks and tried:

echo 0 >> /proc/sys/vm/vfs_cache_pressure
find /mnt/bigdisk > /dev/null

And after a few minutes of thrashing, I checked how much RAM was
gone... practically none (less than 1 GB on a system with 16 GB
total).

How about performance? Well, instead of 15 minutes of thrashing (it's
a cheap consumer SMR disk, so extra-extra slow):

du -sh bigfolder/

now finishes in 3 seconds (0.3% of the time, or a 300x speedup). So do
things like "find -name ...". For the cost (about 800 MB of RAM for
~450,000 files), this seems like a really good thing to do for any
filesystems that aren't on SSD.

(Experiment results for a larger filesystem: 3.75 million files takes 3
minutes for an initial scan, then occupies 1.7 GB RAM and subsequent
operations take 9 seconds (20x speedup). 9 million files uses up all
available RAM and crashes the system, which isn't proportionate to the
other results, which means one should still be a bit careful with this.)

Setting vfs_cache_pressure to 0 might be a bit extreme, but I wondered
why the default would be so eager to drop caches when RAM is plentiful
now, so I looked at the dates on that vm.txt document: 1998, 1999, and
2008.

Is it possible that this parameter just hasn't been reconsidered since
then? The kernel main mailing list archives only make 8 mentions of it
overall, and almost all of the discussion I've been able to find is
about *increasing* the parameter in attempts to solve low-memory
situations.

Some other possibilities occur to me:

- Could dentry/inode entries be cached in compressed form (e.g.
  with a fast compression method like zstd)? This would make it much
  more feasible to cache entire filesystems, even larger ones. (Looks
  like zram/zswap might be able to do this, but probably not easy to
  single out just the dentry/inode data?)

- Could the vfs_cache_pressure parameter be made per-backing-device so
  that, for example, one could set up rules to aggressively cache
  filesystems on spinning disks and network mounts, and aggressively
  drop caches for those residing on SSDs?

- Could dir/inode cache also be assigned a different swappiness
  parameter, to allow something silly like making a poor-man's SSD
  cache by putting a swapfile on an SSD, caching inode data from a
  spinning disk, then forcing it out to that swapfile? Still faster
  than seeking the heads on the spinning disk. (And yes I think this is
  a silly idea; the only advantage over things like bcachefs,
  lvm/device mapper caching is that you could set it up on an
  already-mounted filesystem, and I think data integrity could be a
  problem...)

- It looks like low non-zero values of vfs_cache_pressure have broken
  behaviour - a value of 1 acts identically to a value of 0 as far as
  I can tell. The system freezes, OOM-killer is invoked, and it starts
  killing many processes while the cache memory is left alone instead
  of being dropped. I would expect a value of 1 to mean "only drop this
  cache as a last resort, but definitely drop it before you even think
  about killing processes to regain RAM" - is there other documentation
  besides vm.txt that explains this?

I would like to put in some time to learn about how some of this might
be implemented or tweaked - does that seem worthwhile, or are there
reasons/problems/existing fixes that I'm not seeing?

~Felix.