Swap causing huge latency on disk IO operations
Hi All, We are running a pretty (rather a very old kernel) 2.6.32. Nevertheless we do see huge latency on disk IO once system starts using swap. Tweaking of various swap related variables under /proc/sys/vm did not make a big difference. The idea is we are having big memory requirement, hence using swap is expected. Unexpected thing is that this is causing very high latency. The precise latency happens as follows swap_readpage () invokes lock_page() and then the process(thread) of interest gets scheduled out through io_schedule(), it does not run again for several seconds. This occurs due to the fact another thread is writing to disk using ext3 functions, wherein it took lock_page() and spends lot of time to write data to disk before finally invoking unlock_page(). top shows huge iowaits (nr_iowait). Is it expected design behavior that Linux performs bad with swap particularly for disk operations? Tweaking elevator algorithm, io size, block size also did not help. Now we are changing ext3 mounts from data=ordered to data=writeback to see if it helps to reduce disk latency when we run into swap usage. We use RAID along with LVM. We are also experimenting by removing LVM to see if improves the behaviour. Any inputs will be immense help Thanks, Sudharsan
On Sun, Sep 10, 2017 at 12:06:36AM +0530, Sudharsan Vijayaraghavan wrote:
Hi All,
We are running a pretty (rather a very old kernel) 2.6.32. Nevertheless we do see huge latency on disk IO once system starts using swap.
That's your problem, 2.6.32 is _very_ old and obsolete. No one in the community can help you out with that release anymore, sorry. Please go get support from the company that is forcing you to stay at such an old kernel version, as you are already paying them for that support. I strongly recommend you use a more modern kernel release, you will find that things should be _much_ better... good luck! greg k-h
Hi Greg, My general question is that has the community seen "swap latency issues with disk IO" in general. Further can we assume that later kernels (like 4.4) does not exhibit such a behavior. Thanks, Sudharsan On Sun, Sep 10, 2017 at 10:32 AM, Greg KH <greg@kroah.com> wrote:
On Sun, Sep 10, 2017 at 12:06:36AM +0530, Sudharsan Vijayaraghavan wrote:
Hi All,
We are running a pretty (rather a very old kernel) 2.6.32. Nevertheless we do see huge latency on disk IO once system starts using swap.
That's your problem, 2.6.32 is _very_ old and obsolete. No one in the community can help you out with that release anymore, sorry.
Please go get support from the company that is forcing you to stay at such an old kernel version, as you are already paying them for that support.
I strongly recommend you use a more modern kernel release, you will find that things should be _much_ better...
good luck!
greg k-h
On Sun, Sep 10, 2017 at 11:26:14AM +0530, Sudharsan Vijayaraghavan wrote:
Hi Greg,
My general question is that has the community seen "swap latency issues with disk IO" in general. Further can we assume that later kernels (like 4.4) does not exhibit such a behavior.
"later" kernels are hundreds of thousands of changes newer than the kernel you are using, so really, the responsibility is for you to try it out on your workload to see if it works better or not (and if not, then let the developers know, after you have tried 4.13). There is nothing we can do to go back in time and change an obsolete, old, kernel version :) good luck! greg k-h
On 09/10/2017 01:56 AM, Sudharsan Vijayaraghavan wrote:
My general question is that has the community seen "swap latency issues with disk IO" in general. Further can we assume that later kernels (like 4.4) does not exhibit such a behavior.
the problems with swap is not limited to any specific OS, and in fact, Linux does it fairly well. You're dealing with a Hardware device which is suseptable to numerous troubles, -- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 09/10/2017 01:02 AM, Greg KH wrote:
That's your problem, 2.6.32 is _very_ old and obsolete. No one in the community can help you out with that release anymore, sorry.
that is true, but swapping at times is still a huge problem. I tend to just turn it off. -- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
participants (3)
-
Greg KH -
Ruben Safir -
Sudharsan Vijayaraghavan