Testing the performance impact of kernel modifications
Hi, I was wondering what are some good ways to assess the performance impact of kernel modifications. Are there some papers in the literature where this is done? Does one need to differentiate between CPU bound and different types of I/O bound processes etc? Regards, Carter.
On Mon, 15 Oct 2018 23:42:03 +0800, Carter Cheng said:
I was wondering what are some good ways to assess the performance impact of kernel modifications. Are there some papers in the literature where this is done? Does one need to differentiate between CPU bound and different types of I/O bound processes etc?
That is *so* totally dependent on exactly what the modification is, that there's no right answer here. The things you will want to measure for a new TCP flow control module (to measure the difference between, say, cubic and new_reno and fq_codel and your new module) will be *totally* different from changes to an LSM, which again will be different from an overhaul of a disk I/O scheduler. And then, the environment matters as well. The performance metrics that I care about on my laptop (which is used as a desktop replacement) are "can I do a kernel build and my desktop environment still work well" type things. But the numbers I care about on the machines I maintain across the hall in the data center are different - those are disk storage, backup, and archive - so I'm willing to burn a lot of CPU in both kernel and userspace if it gets me more IOPs and throughput - important when you have 500+ million files in a single petabyte-plus file system. Meanwhile, the guys a few cubicles down are doing HPC, which means they want as little kernel CPU usage as possible because that gets in the way of user computations. And sometimes, it doesn't matter in the slightest what the performance impact is, because the change is required for correctness - running incorrect code faster is still running incorrect code. See the recent Spectre patches for an example.
I am actually looking at some changes that litter the kernel with short code snippets and thus according to papers i have read can result in CPU hits of around 48% when applied is userspace. I am curious how you would best measure the impact of similar modifications (since obviously one isn't always in the kernel code when executing a process). My interest is in testing different approaches of making pointer runtime checks. My theory would be that perhaps writing some test code that exercises different extremes (calling the syscall api continually to do a range of tasks) to see how much slower the code would be between the safer version and the old version. This however might not reflect real world performance for I/O (sockets) such as web servers that spend less time in the kernel. I have seen a paper that benchmarked kvm against xen server but I haven't seen any kernel space papers measuring degradations in overall system performance when adding safety checks(perhaps redundant sometimes) into the kernel. On Tue, Oct 16, 2018 at 1:08 AM <valdis.kletnieks@vt.edu> wrote:
On Mon, 15 Oct 2018 23:42:03 +0800, Carter Cheng said:
I was wondering what are some good ways to assess the performance impact of kernel modifications. Are there some papers in the literature where this is done? Does one need to differentiate between CPU bound and different types of I/O bound processes etc?
That is *so* totally dependent on exactly what the modification is, that there's no right answer here.
The things you will want to measure for a new TCP flow control module (to measure the difference between, say, cubic and new_reno and fq_codel and your new module) will be *totally* different from changes to an LSM, which again will be different from an overhaul of a disk I/O scheduler.
And then, the environment matters as well. The performance metrics that I care about on my laptop (which is used as a desktop replacement) are "can I do a kernel build and my desktop environment still work well" type things. But the numbers I care about on the machines I maintain across the hall in the data center are different - those are disk storage, backup, and archive - so I'm willing to burn a lot of CPU in both kernel and userspace if it gets me more IOPs and throughput - important when you have 500+ million files in a single petabyte-plus file system. Meanwhile, the guys a few cubicles down are doing HPC, which means they want as little kernel CPU usage as possible because that gets in the way of user computations.
And sometimes, it doesn't matter in the slightest what the performance impact is, because the change is required for correctness - running incorrect code faster is still running incorrect code. See the recent Spectre patches for an example.
On Tue, 16 Oct 2018 01:23:45 +0800, Carter Cheng said:
I am actually looking at some changes that litter the kernel with short code snippets and thus according to papers i have read can result in CPU hits of around 48% when applied is userspace.
You're going to need to be more specific. Note that 48% increase in a micro-benchmark doesn't necessarily translate to a measurable performance change - for example, I have a kernel build running right now with a cold file cache, and it's only using 6-8% of the CPU in kernel mode (the rest being gcc in userspace and waiting for the spinning-oxide disk). If the entire kernel slowed down by 50% that would only be 3-4% change visible at the macro level.
but I haven't seen any kernel space papers measuring degradations in overall system performance when adding safety checks(perhaps redundant sometimes) into the kernel
Well.. here's the thing. Papers are usually written by academics and trade journal pundits, not people who write code for a living. As a result, they end up comparing released code versions. As a worked example, see how the whole Spectre thing turned out - the *initial* fears were that we'd see a huge performance drop. But the patches that finally shipped for the Linux kernel were after a bunch of clever people had thought about it and come up with less intrusive ways to close the security issue. (Having said that, the guys at Phoronix do a reasonable job of doing macro-level benchmarks of each kernel release and pointing out if there's a big hit in a subsystem). And as I said earlier - sometimes it doesn't matter, because correctness trumps performance.
Basically I am looking for methodology guidelines for doing my own testing on a bunch of techniques in different papers and seeing what the performance impact is overall. Are there guidelines for doing such things? On Tue, Oct 16, 2018 at 3:19 AM <valdis.kletnieks@vt.edu> wrote:
On Tue, 16 Oct 2018 01:23:45 +0800, Carter Cheng said:
I am actually looking at some changes that litter the kernel with short code snippets and thus according to papers i have read can result in CPU hits of around 48% when applied is userspace.
You're going to need to be more specific. Note that 48% increase in a micro-benchmark doesn't necessarily translate to a measurable performance change - for example, I have a kernel build running right now with a cold file cache, and it's only using 6-8% of the CPU in kernel mode (the rest being gcc in userspace and waiting for the spinning-oxide disk). If the entire kernel slowed down by 50% that would only be 3-4% change visible at the macro level.
but I haven't seen any kernel space papers measuring degradations in overall system performance when adding safety checks(perhaps redundant sometimes) into the kernel
Well.. here's the thing. Papers are usually written by academics and trade journal pundits, not people who write code for a living. As a result, they end up comparing released code versions. As a worked example, see how the whole Spectre thing turned out - the *initial* fears were that we'd see a huge performance drop. But the patches that finally shipped for the Linux kernel were after a bunch of clever people had thought about it and come up with less intrusive ways to close the security issue.
(Having said that, the guys at Phoronix do a reasonable job of doing macro-level benchmarks of each kernel release and pointing out if there's a big hit in a subsystem).
And as I said earlier - sometimes it doesn't matter, because correctness trumps performance.
I believe there is no definite methodology, it is all experimental and dependent on the applications you are running and as Valdis told on the changes you would like to make. Basically, when a paper offers a new algorithm or design then they should have tested it on their own testbed. They may report their experimental methodology in the paper or even have the experiment scripts on the Github. However, other than the applications, and your changes, the testbed itself is also important. Their system may work with a CPU frequency different than yours. So you might not see the performance gain as they reported and achieved in the paper. For instance, concerning some network enhancement in TCP stack, some people may improve the end to end latency just 10s of microseconds and you suppose to capture those minor microseconds in your experiments. It is really hard but not impossible, it basically takes time and effort + *extensive evaluations*. I know that nowadays system software papers are pretty practical, and they try to build a working systems. I'm particularly talking about SOSP and OSDI papers. On Tue, Oct 16, 2018 at 3:50 AM Carter Cheng <cartercheng@gmail.com> wrote:
Basically I am looking for methodology guidelines for doing my own testing on a bunch of techniques in different papers and seeing what the performance impact is overall. Are there guidelines for doing such things?
On Tue, Oct 16, 2018 at 3:19 AM <valdis.kletnieks@vt.edu> wrote:
On Tue, 16 Oct 2018 01:23:45 +0800, Carter Cheng said:
I am actually looking at some changes that litter the kernel with short code snippets and thus according to papers i have read can result in CPU hits of around 48% when applied is userspace.
You're going to need to be more specific. Note that 48% increase in a micro-benchmark doesn't necessarily translate to a measurable performance change - for example, I have a kernel build running right now with a cold file cache, and it's only using 6-8% of the CPU in kernel mode (the rest being gcc in userspace and waiting for the spinning-oxide disk). If the entire kernel slowed down by 50% that would only be 3-4% change visible at the macro level.
but I haven't seen any kernel space papers measuring degradations in overall system performance when adding safety checks(perhaps redundant sometimes) into the kernel
Well.. here's the thing. Papers are usually written by academics and trade journal pundits, not people who write code for a living. As a result, they end up comparing released code versions. As a worked example, see how the whole Spectre thing turned out - the *initial* fears were that we'd see a huge performance drop. But the patches that finally shipped for the Linux kernel were after a bunch of clever people had thought about it and come up with less intrusive ways to close the security issue.
(Having said that, the guys at Phoronix do a reasonable job of doing macro-level benchmarks of each kernel release and pointing out if there's a big hit in a subsystem).
And as I said earlier - sometimes it doesn't matter, because correctness trumps performance.
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
participants (3)
-
Carter Cheng -
SeyedAlireza Sanaee -
valdis.kletnieks＠vt.edu