Kernel Testing (was Re: Is the Kernel Janitors project still alive?)

[Greg Freemyer]

On Thu, Jul 18, 2013 at 1:58 PM, Arlie Stephens <arlie at worldash.org> wrote:
> On Jul 18 2013, Valdis.Kletnieks at vt.edu wrote:
> [snip, snip, snip]
>> A good place to start is to just use git to suck down the current linux-next
>> tree, build it, run it, and report all problems you encounter. Most code only
>> gets tested on the 3-4 boxes the code author has access to before it gets into
>> the linux-next tree. I usually manage to trip over anywhere from 1 to 5 bugs
>> per kernel release, just because nobody else has actually tried running the
>> code on a Dell Latitude with the same .config as I have and the same
>> workflow.
>
> One of the things I really miss in the open source kernel world
> (coming from the commercial world) is the lack of test packages that
> everyone automatically runs, just in case they broke
> something. FreeBSD had nothing, AFAICT, with occassional painful
> results.
>
> Does the linux kernel have any kind of regression test package? If so,
> where can I find it? If not, does anyone know of ongoing attempts to
> create one?
>
> A quick web search gave me a few pointers to attempts at this:
>
> pdf about a regression test framework:
>   https://www.kernel.org/doc/ols/2007/ols2007v2-pages-285-296.pdf
>
> A project called crackerjack:
>   http://sourceforge.net/projects/crackerjack/
>
> A paper from a 2008 linux symposium:
>
>    http://ols.fedoraproject.org/OLS/Reprints-2008/babulal-reprint.pdf
>
> What I don't see is any kind of definitive answer - as in, "just clone
> this git repo and build/run it". And the bug reports I see discussed
> tend to come from people trying to use linux to accomplish real work.
>
> So is there something current, live, and useful?

For filesystems, the xfstests package is heavily used and has over 200
separate test modules.  The name "xfs" is due to its heritage a decade
or more ago.

It now has extensive support for other filesystems including ext4 and
btrfs in particular.  ie. There are generic tests that all filesystems
should pass and there are unique tests that target xfs OR ext4 OR
btrfs (and possibly others).

I know the xfs and ext4 maintainers run extended xfstest runs prior to
merging major changes.  Further when a new bug is reported, they often
implement a new xfstest module to catch it as part of the bug fix
process.  That ensures that bug won't be coming back in the future.

In general xfstests is not a performance suite, it just tests for
functionality.  Also, xfstests has been under aggressive revision for
at least the last couple years, so it is definitely maintained.

For block level testing as a whole, including performance, Jens Axboe
supports fio.  I think a lot of fio users actually test their overall
storage stack, not just one aspect.  For developers, by keeping the
stack constant other than the one component which is under test you
can evaluate if the change helped or hurt.

Greg