On Thu, Apr 11, 2013 at 2:50 PM, neha naik <nehanaik27@gmail.com> wrote:
Yes. Interestingly my direct write i/o performance is better than my direct read i/o performance for my passthrough device... And that doesn't make any kind of sense to me.
pdev0 = pass through device on top of lvm
root@voffice-base:/home/neha/sbd# time dd if=/dev/pdev0 of=/dev/null bs=4096 count=1024 iflag=direct 1024+0 records in 1024+0 records out 4194304 bytes (4.2 MB) copied, 4.09488 s, 1.0 MB/s
real 0m4.100s user 0m0.028s sys 0m0.000s
root@voffice-base:/home/neha/sbd# time dd if=/dev/shm/image of=/dev/pdev0 bs=4096 count=1024 oflag=direct 1024+0 records in 1024+0 records out 4194304 bytes (4.2 MB) copied, 0.0852398 s, 49.2 MB/s
real 0m0.090s user 0m0.004s sys 0m0.012s
Thanks, Neha
I assume your issue is caching somewhere. If in the top levels of the kernel, dd has various fsync, fdatasync, etc. options that should address that. I note you aren't using any of them. You mention LVM. It should pass cache flush commands down, but some flavors of mdraid will not the last I knew. ie. Raid 6 used to discard cache flush commands iirc. I don't know if that was ever fixed or not. If the cache is in hardware, then dd's cache flushing calls may or may not get propagated all the way to the device. Some battery backed caches actually intentionally reply ACK to a cache flush command without actually doing it. Further, you're only writing 4MB. Not much of a test for most devices. A sata drive will typically have at least 32MB of cache. One way to ensure that results are not being corrupted by the various caches up and down the storage stack is to write so much data you overwhelm the caches. That can be a huge amount of data in some systems. ie. A server with 128 GB or ram may use 10's of GB for cache. As you can see, testing of the write path for performance can take a significant effort to ensure caches are not biasing your results. HTH Greg