Best tests to measure Kernel Performance

[Victor Rodriguez]

On Wed, Dec 2, 2015 at 6:36 PM, Greg KH <gregkh at linuxfoundation.org> wrote:
> On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
>> On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh at linuxfoundation.org> wrote:
>> > On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
>> >> Hi
>> >>
>> >> Despite the fact that this is not a well formulated question. I wonder
>> >> what tests could be a good subset to measure the performance of the
>> >> kernel . I have some approaches like phoronix does here :
>> >>
>> >> http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
>> >>
>> >> I am sure postmark/ John the ripper/ Apache are good candidates but I
>> >> want to ask the community if there is some specific test that you
>> >> recommend
>> >
>> > It depends on what you want to test, specifically.  The "kernel" isn't a
>> > very specific thing, what most of those tests test is the speed of the
>> > hardware, not specifically the kernel itself.
>> >
>> > good luck,
>> >
>> > greg k-h
>>
>> Thanks for the feedback . You are right they test the speed of the HW
>> however I have seen that when there is a change in the kernel for
>> network the performance of apache is changed, which make total sense .
>
> Maybe, maybe not, depending on if "apache" is cpu or hardware bound
> (networking hardware has physical limits...) again, you have to be very
> sure about exactly what you are wanting to test before using such a test
> to try to "validate" anything other than just raw hardware speed.
>

Agree , you are right

> Take a look at the "old" lmbench set of benchmarks for valid things that
> a kernel change can affect, it's much different from what you might be
> thinking of as a test.
>

I will do

>> I think that LTSI should have kind of a test suite with significant
>> test that could help the developer to detect those perf changes. Is
>> very common that one as OS developer make a change in one package (
>> important one as the kernel ) and do not check how this affect the
>> performance of the OS ( I know is too general , but we might show
>> BKM's)
>
> WHat is "BKM"?
>

Sorry , too many years at intel :)

Best Known Method

>> I think this might be a good topic to discuss with the community and
>> we could came with a solid recommended test suite in the LTSI project.
>
> LTSI already has a "test suite" that is uses to test the releases,
> what's wrong with that?  I'm sure the developers would be glad to add
> any additional tests that you want added to it that you find missing and
> useful.

Do we have a subset of test inside LTSI test suite just for
performance ? What i am thinking is that in LTSI suite we could have
subset of tests:

-> Unit tests
-> Functional Tests
-> Performance tests

So if that could exist would be amazing for a OS development team.

Also to give the possibility to have CI natural in the LTSI test suite
:) . Imagine that :

-> when you have a new commit you can run specific test and you can
run the test you want or all the test suite without merging your
change
-> you could make a new release and then the Performance test be
executed and then the graphs show of ( are we better or worst ? )

We in Clear Linux ( as in many other projects ) are working to make
this a full CI for the entire OS , not just for Kernel , but would be
nice if LTSI system could have that and the kernel team of many
companies could use it as a full test suite / CI

It might be just a naive idea but I think it  would be very helpful

As always thanks a lot for the feedback Greg I lear a lot from this :)

Best Regards

Victor Rodriguez


> Also note that the upstream kernel is tested by a huge test suite of
> performance tests and static analysis tools for every commit in all
> development branches by the wonderful 0-day bot system.  That's been
> helping prevent regressions for a long time now.
>

> thanks,
>
> greg k-h