Split RAID: Proposal for archival RAID using incremental batch checksum

[Anshuman Aggarwal]

N pass through but with their own filesystems. Concatenation is via
some kind of union fs solution not at the block level. Data is not
supposed to be striped (this is critical so as to prevent all drives
to be required to be accessed for consecutive data)

Idea is that each drive can work independently and the last drive
stores parity to save data in case of failure of any one drive.

Any suggestions from anyone on where to start with such a driver..it
seems like a block driver for the parity drive but which depends on
intercepting the writes to other drives.

On 21 November 2014 17:11, Greg Freemyer <greg.freemyer at gmail.com> wrote:
>
>
> On November 21, 2014 5:15:43 AM EST, Anshuman Aggarwal <anshuman.aggarwal at gmail.com> wrote:
>>I'd a appreciate any help/pointers in implementing the proposal below
>>including the right path to get this into the kernel itself.
>>----------------------------------
>>I'm outlining below a proposal for a RAID device mapper virtual block
>>device for the kernel which adds "split raid" functionality on an
>>incremental batch basis for a home media server/archived content which
>>is rarely accessed.
>>
>>Given a set of N+X block devices (of the same size but smallest common
>>size wins)
>>
>>the SplitRAID device mapper device generates virtual devices which are
>>passthrough for N devices and write a Batched/Delayed checksum into
>>the X devices so as to allow offline recovery of block on the N
>>devices in case of a single disk failure.
>>
>>Advantages over conventional RAID:
>>
>>- Disks can be spun down reducing wear and tear over MD RAID Levels
>>(such as 1, 10, 5,6) in the case of rarely accessed archival content
>>
>>- Prevent catastrophic data loss for multiple device failure since
>>each block device is independent and hence unlike MD RAID will only
>>lose data incrementally.
>>
>>- Performance degradation for writes can be achieved by keeping the
>>checksum update asynchronous and delaying the fsync to the checksum
>>block device.
>>
>>In the event of improper shutdown the checksum may not have all the
>>updated data but will be mostly up to date which is often acceptable
>>for home media server requirements. A flag can be set in case the
>>checksum block device was shutdown properly indicating that  a full
>>checksum rebuild is not required.
>>
>>Existing solutions considered:
>>
>>- SnapRAID (http://snapraid.sourceforge.net/) which is a snapshot
>>based scheme (Its advantages are that its in user space and has cross
>>platform support but has the huge disadvantage of every checksum being
>>done from scratch slowing the system, causing immense wear and tear on
>>every snapshot and also losing any information updates upto the
>>snapshot point etc)
>>
>>I'd like to get opinions on the pros and cons of this proposal from
>>more experienced people on the list to redirect suitably on the
>>following questions:
>>
>>- Maybe this can already be done using the block devices available in
>>the kernel?
>>
>>- If not, Device mapper the right API to use? (I think so)
>>
>>- What would be the best block devices code to look at to implement?
>>
>>
>>Regards,
>>
>>Anshuman Aggarwal
>>
>>_______________________________________________
>>Kernelnewbies mailing list
>>Kernelnewbies at kernelnewbies.org
>>http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
> I think I understand the proposal.
>
> You say N pass-through drives.  I assume concatenated?
>
> If the N drives were instead in a Raid-0 stripe set and your X drives was just a single parity drive, then you would have described Raid-4.
>
> There are use cases for raid 4 and you have described a good one (rarely used data where random w/o performance is not key).
>
> I don't know if mdraid supports raid-4 or not.  If not I suggest adding raid-4 support is something else you might want to look at.
>
> Anyway, at a minimum add raid-4 to the existing solutions considered section.
>
> Greg
>
>
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.