Forum for asking questions related to block device drivers

[Greg Freemyer]

On Thu, Apr 11, 2013 at 2:50 PM, neha naik <nehanaik27 at gmail.com> wrote:
> Yes. Interestingly my direct write i/o performance is better than my direct
> read i/o performance for my passthrough device... And that doesn't make any
> kind of sense to me.
>
> pdev0 = pass through device on top of lvm
>
> root at voffice-base:/home/neha/sbd# time dd if=/dev/pdev0 of=/dev/null bs=4096
> count=1024 iflag=direct
> 1024+0 records in
> 1024+0 records out
> 4194304 bytes (4.2 MB) copied, 4.09488 s, 1.0 MB/s
>
> real    0m4.100s
> user    0m0.028s
> sys    0m0.000s
>
> root at voffice-base:/home/neha/sbd# time dd if=/dev/shm/image of=/dev/pdev0
> bs=4096 count=1024 oflag=direct
> 1024+0 records in
> 1024+0 records out
> 4194304 bytes (4.2 MB) copied, 0.0852398 s, 49.2 MB/s
>
> real    0m0.090s
> user    0m0.004s
> sys    0m0.012s
>
> Thanks,
> Neha

I assume your issue is caching somewhere.

If in the top levels of the kernel, dd has various fsync, fdatasync,
etc. options that should address that.  I note you aren't using any of
them.

You mention LVM.  It should pass cache flush commands down, but some
flavors of mdraid will not the last I knew.  ie. Raid 6 used to
discard cache flush commands iirc.  I don't know if that was ever
fixed or not.

If the cache is in hardware, then dd's cache flushing calls may or may
not get propagated all the way to the device.  Some battery backed
caches actually intentionally reply ACK to a cache flush command
without actually doing it.

Further, you're only writing 4MB.  Not much of a test for most
devices.  A sata drive will typically have at least 32MB of cache.
One way to ensure that results are not being corrupted by the various
caches up and down the storage stack is to write so much data you
overwhelm the caches.  That can be a huge amount of data in some
systems.  ie. A server with 128 GB or ram may use 10's of GB for
cache.

As you can see, testing of the write path for performance can take a
significant effort to ensure caches are not biasing your results.

HTH
Greg