<br><br><div class="gmail_quote">On Sun, Jan 15, 2012 at 4:22 PM, Raghavendra D Prabhu <span dir="ltr"><<a href="mailto:raghu.prabhu13@gmail.com">raghu.prabhu13@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Zheng,<br>
<br>
Interesting analysis. <br><div class="im">
* On Sun, Jan 15, 2012 at 03:17:12PM -0500, Zheng Da <<a href="mailto:zhengda1936@gmail.com" target="_blank">zhengda1936@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks. I was reading the code of kernel 3.0. XFS starts to support<br>
concurrent direct IO since kernel 3.1.5.<br>
But concurrent direct IO write still doesn't work well in kernel 3.2. <br>
</blockquote></div>
>From what I have heard it has supported it from sometime back. I think you may need to ask in xfs general ML about this.</blockquote><div>I didn't know this ML. I'll ask them for help. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I wrote a test program that accesses a 4G file randomly (read and write), and<br>
I ran it with 8 threads and the machine has 8 cores. It turns out that only<br>
1 core is running. I'm pretty sure xfs_rw_ilock is locked<br>
with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write.<br>
<br>
lockstat shows me that there is a lot of wait time in ip->i_lock. It seems<br>
the lock is locked exclusively.<br>
&(&ip->i_lock)->mr_lock-W: 31568 36170<br>
0.24 20048.25 7589157.99 130154 3146848<br>
0.00 217.70 1238310.72<br>
&(&ip->i_lock)->mr_lock-R: 11251 11886<br>
0.24 20043.01 2895595.18 46671 526309<br>
0.00 63.80 264097.96<br>
-------------------------<br>
&(&ip->i_lock)->mr_lock 36170<br>
[<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]<br>
&(&ip->i_lock)->mr_lock 11886<br>
[<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]<br>
-------------------------<br>
&(&ip->i_lock)->mr_lock 38555<br>
[<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]<br>
&(&ip->i_lock)->mr_lock 9501<br>
[<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]<br>
<br>
Then I used systemtap to instrument xfs_ilock and find there are at least 3<br>
functions that lock ip->i_lock exclusively during write.<br>
</blockquote>
<br></div>
>From what I saw in xfs_file_dio_aio_write code, it uses EXCL only if there is unaligned IO or there are cached pages to be invalidated after shared lock is obtained *but* it demotes that lock to SHARED just before generic_file_direct_write. </blockquote>
<div> Actually, there are two locks for an inode, i_lock and i_iolock. systemtap shows me that i_iolock is already locked to SHARED, but i_lock is locked exclusively somewhere else. Even though I don't think I have found the right spot that hurts concurrency so badly.</div>
<div><br></div><div>Thanks,</div><div>Da</div></div>