Hello, I am measuring system latency using latencytop. I am using kernel version 3.0 with some minor changes. The latencytop output for the process doing the IO (qemu) is below: Process qemu-system-x86 (6239) Total: 31363.7 msec [sleep_on_page] 1966.5 msec 81.4 % Waiting for event (select) 5.0 msec 7.8 % [kvm_vcpu_block] 5.0 msec 8.5 % synchronous write 1.7 msec 0.0 % Userspace lock contention 1.5 msec 2.3 % Is the 2000 msec sleep_on_page delays normal ? I would also like to know what sleep_on_page exactly corresponds to. My understanding is that it is the time spent waiting for grabbing a lock on a page. Is this correct ? Thanks.
Hi Prateek... On Mon, Jan 16, 2012 at 11:48, Prateek Sharma <prateek3.14@gmail.com> wrote:
The latencytop output for the process doing the IO (qemu) is below:
Process qemu-system-x86 (6239) Total: 31363.7 msec [sleep_on_page] 1966.5 msec 81.4 % Waiting for event (select) 5.0 msec 7.8 % [kvm_vcpu_block] 5.0 msec 8.5 % synchronous write 1.7 msec 0.0 % Userspace lock contention 1.5 msec 2.3 %
Which qemu version do you use now? AFAIK certain new qemu versions already uses iothreads by default. That should reduce I/O latency... AFAIK too iothreads is not enabled by default...but for qemu 1.0 above it's enabled by default. Are you compiling from source? Regarding the function sleep_on_page(). It's in turn io_schedule(). And here's the comment above the function declaration of io_schedule(): /* * This task is about to go to sleep on IO. Increment rq->nr_iowait so * that process accounting knows that this is a task in IO wait state. */ You can confirm it by yourself in: http://lxr.linux.no/#linux+v3.2.1/kernel/sched.c#L5872 Hope it helps.... -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
On Mon, Jan 16, 2012 at 10:41 AM, Mulyadi Santosa <mulyadi.santosa@gmail.com> wrote:
Hi Prateek...
On Mon, Jan 16, 2012 at 11:48, Prateek Sharma <prateek3.14@gmail.com> wrote:
The latencytop output for the process doing the IO (qemu) is below:
Process qemu-system-x86 (6239) Total: 31363.7 msec [sleep_on_page] 1966.5 msec 81.4 % Waiting for event (select) 5.0 msec 7.8 % [kvm_vcpu_block] 5.0 msec 8.5 % synchronous write 1.7 msec 0.0 % Userspace lock contention 1.5 msec 2.3 %
Which qemu version do you use now? AFAIK certain new qemu versions already uses iothreads by default. That should reduce I/O latency... AFAIK too iothreads is not enabled by default...but for qemu 1.0 above it's enabled by default. Are you compiling from source?
I am using qemu-kvm 0.15, with KVM.
Regarding the function sleep_on_page(). It's in turn io_schedule(). And here's the comment above the function declaration of io_schedule(): /* * This task is about to go to sleep on IO. Increment rq->nr_iowait so * that process accounting knows that this is a task in IO wait state. */
You can confirm it by yourself in: http://lxr.linux.no/#linux+v3.2.1/kernel/sched.c#L5872
My initial understanding was that sleep_on_page is 'called' by __lock_page, which is usually called by file_read . So, i assumed that sleep_on_page is for the page-lock contention. My primary confusion is with the interpretation of latencytop output. 2 seconds even for I/O to complete seems awfully long.
Hope it helps....
-- regards,
Mulyadi Santosa Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Hi Prateek... On Mon, Jan 16, 2012 at 12:54, Prateek Sharma <prateek3.14@gmail.com> wrote:
I am using qemu-kvm 0.15, with KVM.
Hmm, not sure whether it already enables iothreads or not. Sometimes KVM changes certain things from main Qemu tree.
My initial understanding was that sleep_on_page is 'called' by __lock_page, which is usually called by file_read . So, i assumed that sleep_on_page is for the page-lock contention. My primary confusion is with the interpretation of latencytop output. 2 seconds even for I/O to complete seems awfully long.
Apologize, I haven't done complete call trace that leads to sleep_on_page(), so I can't offer my own personal view. The thing that cross my mind is to use tool like blktrace to confirm those numbers. AFAIK too, sometimes Qemu (or KVM) assume doing synchronized I/O. Maybe that also explain the numbers you get. Try to read qemu/kvm manual page and switch to async mode. NB: are you doing read from journalling filesystem? I guess so...if yes, which mode do you use i.e ext3 write back etc. -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
participants (2)
-
Mulyadi Santosa -
Prateek Sharma