Best tests to measure Kernel Performance
Hi Despite the fact that this is not a well formulated question. I wonder what tests could be a good subset to measure the performance of the kernel . I have some approaches like phoronix does here : http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1 I am sure postmark/ John the ripper/ Apache are good candidates but I want to ask the community if there is some specific test that you recommend Thanks a lot for the help Regards Victor Rodriguez
On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
Hi
Despite the fact that this is not a well formulated question. I wonder what tests could be a good subset to measure the performance of the kernel . I have some approaches like phoronix does here :
http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
I am sure postmark/ John the ripper/ Apache are good candidates but I want to ask the community if there is some specific test that you recommend
It depends on what you want to test, specifically. The "kernel" isn't a very specific thing, what most of those tests test is the speed of the hardware, not specifically the kernel itself. good luck, greg k-h
On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
Hi
Despite the fact that this is not a well formulated question. I wonder what tests could be a good subset to measure the performance of the kernel . I have some approaches like phoronix does here :
http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
I am sure postmark/ John the ripper/ Apache are good candidates but I want to ask the community if there is some specific test that you recommend
It depends on what you want to test, specifically. The "kernel" isn't a very specific thing, what most of those tests test is the speed of the hardware, not specifically the kernel itself.
good luck,
greg k-h
Thanks for the feedback . You are right they test the speed of the HW however I have seen that when there is a change in the kernel for network the performance of apache is changed, which make total sense . I think that LTSI should have kind of a test suite with significant test that could help the developer to detect those perf changes. Is very common that one as OS developer make a change in one package ( important one as the kernel ) and do not check how this affect the performance of the OS ( I know is too general , but we might show BKM's) I think this might be a good topic to discuss with the community and we could came with a solid recommended test suite in the LTSI project. Feedback more than welcome Regards Victor Rodriguez
On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
Hi
Despite the fact that this is not a well formulated question. I wonder what tests could be a good subset to measure the performance of the kernel . I have some approaches like phoronix does here :
http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
I am sure postmark/ John the ripper/ Apache are good candidates but I want to ask the community if there is some specific test that you recommend
It depends on what you want to test, specifically. The "kernel" isn't a very specific thing, what most of those tests test is the speed of the hardware, not specifically the kernel itself.
good luck,
greg k-h
Thanks for the feedback . You are right they test the speed of the HW however I have seen that when there is a change in the kernel for network the performance of apache is changed, which make total sense .
Maybe, maybe not, depending on if "apache" is cpu or hardware bound (networking hardware has physical limits...) again, you have to be very sure about exactly what you are wanting to test before using such a test to try to "validate" anything other than just raw hardware speed. Take a look at the "old" lmbench set of benchmarks for valid things that a kernel change can affect, it's much different from what you might be thinking of as a test.
I think that LTSI should have kind of a test suite with significant test that could help the developer to detect those perf changes. Is very common that one as OS developer make a change in one package ( important one as the kernel ) and do not check how this affect the performance of the OS ( I know is too general , but we might show BKM's)
WHat is "BKM"?
I think this might be a good topic to discuss with the community and we could came with a solid recommended test suite in the LTSI project.
LTSI already has a "test suite" that is uses to test the releases, what's wrong with that? I'm sure the developers would be glad to add any additional tests that you want added to it that you find missing and useful. Also note that the upstream kernel is tested by a huge test suite of performance tests and static analysis tools for every commit in all development branches by the wonderful 0-day bot system. That's been helping prevent regressions for a long time now. thanks, greg k-h
On Wed, Dec 02, 2015 at 04:36:50PM -0800, Greg KH wrote:
On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
Hi
Despite the fact that this is not a well formulated question. I wonder what tests could be a good subset to measure the performance of the kernel . I have some approaches like phoronix does here :
http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
I am sure postmark/ John the ripper/ Apache are good candidates but I want to ask the community if there is some specific test that you recommend
It depends on what you want to test, specifically. The "kernel" isn't a very specific thing, what most of those tests test is the speed of the hardware, not specifically the kernel itself.
good luck,
greg k-h
Thanks for the feedback . You are right they test the speed of the HW however I have seen that when there is a change in the kernel for network the performance of apache is changed, which make total sense .
Maybe, maybe not, depending on if "apache" is cpu or hardware bound (networking hardware has physical limits...) again, you have to be very sure about exactly what you are wanting to test before using such a test to try to "validate" anything other than just raw hardware speed.
Take a look at the "old" lmbench set of benchmarks for valid things that a kernel change can affect, it's much different from what you might be thinking of as a test.
We also still use lmbench as the usual first level of assessment as it gives a lot of information about the change set impact on low-level functions (system-calls, IPC, allocation...) was. It is much more precise than trying to detect changes in complex applications that might only be making a handful of a affected system call and thus look like performance did not change while it actually did - just its in some hard to reach corner case. As with all testing - you need layers of testing to get a usable picture of what is going on and lmbench is a good candidate for the lowest level. Deducing system level changes from looking at complex application performance changes is alost impossible. Specifically lmbench has a simple make results; make rerun which can give a good overview of differences - but actually the tests default runs are only a small part of what the tests can uncover so looking at individual microbenchmarks to discover latency/bandwidth changes can be very helpful also to uncover odd hardware behavior. Some other low-level benchmarks we use are: rt-tests - scheduling, pi NetPIPE - network bandwidth bonnie++ - filesystem thx! hofrat
On Thu, Dec 3, 2015 at 2:10 AM, Nicholas Mc Guire <der.herr@hofr.at> wrote:
On Wed, Dec 02, 2015 at 04:36:50PM -0800, Greg KH wrote:
On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
Hi
Despite the fact that this is not a well formulated question. I wonder what tests could be a good subset to measure the performance of the kernel . I have some approaches like phoronix does here :
http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
I am sure postmark/ John the ripper/ Apache are good candidates but I want to ask the community if there is some specific test that you recommend
It depends on what you want to test, specifically. The "kernel" isn't a very specific thing, what most of those tests test is the speed of the hardware, not specifically the kernel itself.
good luck,
greg k-h
Thanks for the feedback . You are right they test the speed of the HW however I have seen that when there is a change in the kernel for network the performance of apache is changed, which make total sense .
Maybe, maybe not, depending on if "apache" is cpu or hardware bound (networking hardware has physical limits...) again, you have to be very sure about exactly what you are wanting to test before using such a test to try to "validate" anything other than just raw hardware speed.
Take a look at the "old" lmbench set of benchmarks for valid things that a kernel change can affect, it's much different from what you might be thinking of as a test.
We also still use lmbench as the usual first level of assessment as it gives a lot of information about the change set impact on low-level functions (system-calls, IPC, allocation...) was. It is much more precise than trying to detect changes in complex applications that might only be making a handful of a affected system call and thus look like performance did not change while it actually did - just its in some hard to reach corner case.
As with all testing - you need layers of testing to get a usable picture of what is going on and lmbench is a good candidate for the lowest level. Deducing system level changes from looking at complex application performance changes is alost impossible.
Specifically lmbench has a simple make results; make rerun which can give a good overview of differences - but actually the tests default runs are only a small part of what the tests can uncover so looking at individual microbenchmarks to discover latency/bandwidth changes can be very helpful also to uncover odd hardware behavior.
Some other low-level benchmarks we use are: rt-tests - scheduling, pi NetPIPE - network bandwidth bonnie++ - filesystem
Thanks a lot hofrat I really appreciate all the help I think that is time to turn my eyes to lmbench for sure as well to the tools you mention :) Yes a lot of layers are necessary to measure the QA of an OS , we need full image test as well as cloud tests ( since our OS is designed for Cloud ) . lmbench will be amazing for low level I really appreciate all the help
thx! hofrat
On Wed, Dec 2, 2015 at 6:36 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
Hi
Despite the fact that this is not a well formulated question. I wonder what tests could be a good subset to measure the performance of the kernel . I have some approaches like phoronix does here :
http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
I am sure postmark/ John the ripper/ Apache are good candidates but I want to ask the community if there is some specific test that you recommend
It depends on what you want to test, specifically. The "kernel" isn't a very specific thing, what most of those tests test is the speed of the hardware, not specifically the kernel itself.
good luck,
greg k-h
Thanks for the feedback . You are right they test the speed of the HW however I have seen that when there is a change in the kernel for network the performance of apache is changed, which make total sense .
Maybe, maybe not, depending on if "apache" is cpu or hardware bound (networking hardware has physical limits...) again, you have to be very sure about exactly what you are wanting to test before using such a test to try to "validate" anything other than just raw hardware speed.
Agree , you are right
Take a look at the "old" lmbench set of benchmarks for valid things that a kernel change can affect, it's much different from what you might be thinking of as a test.
I will do
I think that LTSI should have kind of a test suite with significant test that could help the developer to detect those perf changes. Is very common that one as OS developer make a change in one package ( important one as the kernel ) and do not check how this affect the performance of the OS ( I know is too general , but we might show BKM's)
WHat is "BKM"?
Sorry , too many years at intel :) Best Known Method
I think this might be a good topic to discuss with the community and we could came with a solid recommended test suite in the LTSI project.
LTSI already has a "test suite" that is uses to test the releases, what's wrong with that? I'm sure the developers would be glad to add any additional tests that you want added to it that you find missing and useful.
Do we have a subset of test inside LTSI test suite just for performance ? What i am thinking is that in LTSI suite we could have subset of tests: -> Unit tests -> Functional Tests -> Performance tests So if that could exist would be amazing for a OS development team. Also to give the possibility to have CI natural in the LTSI test suite :) . Imagine that : -> when you have a new commit you can run specific test and you can run the test you want or all the test suite without merging your change -> you could make a new release and then the Performance test be executed and then the graphs show of ( are we better or worst ? ) We in Clear Linux ( as in many other projects ) are working to make this a full CI for the entire OS , not just for Kernel , but would be nice if LTSI system could have that and the kernel team of many companies could use it as a full test suite / CI It might be just a naive idea but I think it would be very helpful As always thanks a lot for the feedback Greg I lear a lot from this :) Best Regards Victor Rodriguez
Also note that the upstream kernel is tested by a huge test suite of performance tests and static analysis tools for every commit in all development branches by the wonderful 0-day bot system. That's been helping prevent regressions for a long time now.
thanks,
greg k-h
On Thu, Dec 03, 2015 at 10:51:09AM -0600, Victor Rodriguez wrote:
Do we have a subset of test inside LTSI test suite just for performance ?
First off, please go look at the test suite that LTSI uses for testing before asking this type of thing, I think that will answer all of your questions already...
We in Clear Linux ( as in many other projects ) are working to make this a full CI for the entire OS , not just for Kernel , but would be nice if LTSI system could have that and the kernel team of many companies could use it as a full test suite / CI
LTSI is just about the kernel, not the "full system". If you wish to help work on something larger, wonderful, but please be aware of the resources that LTSI currently has for any "new" work (i.e. very limited.) thanks, greg k-h
On Tue, 01 Dec 2015 18:45:51 -0600, Victor Rodriguez said:
Despite the fact that this is not a well formulated question.
Exactly why it's hard to give a good answer. It *really* depends on what aspect you're interested in - I/O bandwidth, throughput, latency, or other. What I currently do at $DAYJOB is worry how fast I can get a cluster of servers to flip blocks to and from disk to 40Gbit network interfaces, and we've found a combo of iozone's and 'tar xf linux-2.6.32.tar.gz' running at the same time is a good stand-in for our production workload. Of course, your mileage will vary. :)
On Tue, Dec 1, 2015 at 7:38 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Tue, 01 Dec 2015 18:45:51 -0600, Victor Rodriguez said:
Despite the fact that this is not a well formulated question.
Exactly why it's hard to give a good answer. It *really* depends on what aspect you're interested in - I/O bandwidth, throughput, latency, or other. What I currently do at $DAYJOB is worry how fast I can get a cluster of servers to flip blocks to and from disk to 40Gbit network interfaces, and we've found a combo of iozone's and 'tar xf linux-2.6.32.tar.gz' running at the same time is a good stand-in for our production workload.
Of course, your mileage will vary. :)
Iozone is fine and I think you give me a good reason to turn my eyes into it again . I decided not to check it due to the fact that is a micro bench and sometimes users want numbers of something they can feel ( how fast can I de compres , how many transactions per second can my DB manage , how fast can I build my product ) I was answering in Greg mail , that it might be good to came with a BKM of tests for the kernel , a list of linked test ( if you move this you will affect that , please test before commit ) , it might be just an initial idea but sure for industry might be something helpful Feedback more than welcome Regards Victor Rodriguez
On Wed, 02 Dec 2015 17:57:23 -0600, Victor Rodriguez said:
Iozone is fine and I think you give me a good reason to turn my eyes into it again . I decided not to check it due to the fact that is a micro bench and sometimes users want numbers of something they can feel
I'm not sure you can call "see how fast the system can serve a terabyte from disk over NFS" is a micro bench :)
participants (4)
-
Greg KH -
Nicholas Mc Guire -
Valdis.Kletnieks＠vt.edu -
Victor Rodriguez