Performance Degradation with rsync in container or cgroupv2 with MEM limit

[Alexandru Lazarev]

Hi everyone,
I'm experiencing a significant performance degradation when using
"rsync" to copy files over the network from within a container or
cgroup with memory limits on Oracle Linux 9.2. The issue occurs with
the Red Hat Compatible Kernel (RHCK) 5.14.0-284.11.1.el9_2.x86_64 but
not with the Unbreakable Enterprise Kernel (UEK)
5.15.0-101.103.2.1.el9uek.x86_64.

Details:
Setup: Oracle Linux 9.2 with containers/cgroups having memory limits.
Issue: Network file copying speed drops drastically when memory limits
are hit, specifically when the page cache (inactive files) fills up.
Tests:
- Using "rsync" from within a container or a cgroup to copy data from
remote source.
- Using "pg_basebackup" PostgreSQL data replication between two PG
Containers (Leader vs Replica).
Results:
- Initial high speeds (~100MBps) drop significantly (to ~1MBps) once
memory limits are reached.

Commands to Reproduce:
1. Create cgroup with memory limit and run rsync:
sudo systemd-run --scope --property=MemoryMax=1G rsync -av --progress
rsync://<source_ip>/files /destination_path

2. Test with drop_caches on hosting OS during slow rsync:
free && sync && echo 3 > /proc/sys/vm/drop_caches && free
After cache is dropped, rsync is again fast until MEM limit is reached again

Observations:
- When the container's memory limit is reached, the page cache
(inactive files) fills up, leading to network bandwidth degradation.
- This affects, for example, PostgreSQL replication, causing lag and
potential data loss.

Has anyone else encountered this issue? Any insights or suggestions on
how to address correctly (or maybe workarounds) this would be greatly
appreciated!

Thanks in advance!