Hi all, I’ve two identical Linux systems with only kernel differences. While doing kernel profiling with perf, I got the below mentioned metrics for Scheduler benchmarks. 1st system (older kernel version compared to the other system) benchmark result: $ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run Total time: 2.936 [sec] 2nd system benchmark result: $ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run Total time: 10.074 [sec] So as per scheduler benchmark results, clearly a huge difference between two systems. Can anyone suggest to me how to dive deeper to know the root cause for it. Also are there any tunable kernel parameters related to this one? -- Thanks, Sekhar
On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
Hi all,
I’ve two identical Linux systems with only kernel differences.
What are the differences in the kernels?
While doing kernel profiling with perf, I got the below mentioned metrics for Scheduler benchmarks.
1st system (older kernel version compared to the other system) benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 2.936 [sec]
2nd system benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 10.074 [sec]
So as per scheduler benchmark results, clearly a huge difference between two systems. Can anyone suggest to me how to dive deeper to know the root cause for it.
Look a the differences between your different kernels, that would be a great start :) good luck! greg k-h
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
Hi all,
I’ve two identical Linux systems with only kernel differences.
What are the differences in the kernels?
While doing kernel profiling with perf, I got the below mentioned metrics for Scheduler benchmarks.
1st system (older kernel version compared to the other system) benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 2.936 [sec]
2nd system benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 10.074 [sec]
So as per scheduler benchmark results, clearly a huge difference between two systems. Can anyone suggest to me how to dive deeper to know the root cause for it.
Look a the differences between your different kernels, that would be a great start :)
I created the difference between two kernel config files and then tried to spot the CONFIG*SCHED* differences. Interestingly I see the difference in I/O scheduler config, 1st system is set to “deadline” and other one is set to “cfq”. So I made it equal by echoing to “/sys/block/<disk device>/queue/scheduler" but still no change in scheduler benchmark metrics. Is it the correct way to find the differences between kernels? If so, what other important CONFIG_* variables need to consider? $ cat config.patch | grep -i sched CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_CGROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y # CONFIG_RT_GROUP_SCHED is not set # IO Schedulers @@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y CONFIG_CFQ_GROUP_IOSCHED=y -CONFIG_DEFAULT_IOSCHED="deadline" +CONFIG_DEFAULT_IOSCHED="cfq" +CONFIG_MQ_IOSCHED_DEADLINE=m +CONFIG_MQ_IOSCHED_KYBER=m +CONFIG_IOSCHED_BFQ=m +CONFIG_BFQ_GROUP_IOSCHED=y CONFIG_SCHED_SMT=y CONFIG_SCHED_MC=y +CONFIG_SCHED_MC_PRIO=y +# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set +CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
good luck!
greg k-h
-- Thanks, Sekhar
On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
Hi all,
I’ve two identical Linux systems with only kernel differences.
What are the differences in the kernels?
You didn't answer this question, is this the same kernel source being compared here? Same version? Same compiler? Everything identical?
While doing kernel profiling with perf, I got the below mentioned metrics for Scheduler benchmarks.
1st system (older kernel version compared to the other system) benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 2.936 [sec]
2nd system benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 10.074 [sec]
So as per scheduler benchmark results, clearly a huge difference between two systems. Can anyone suggest to me how to dive deeper to know the root cause for it.
Look a the differences between your different kernels, that would be a great start :) I created the difference between two kernel config files and then tried to spot the CONFIG*SCHED* differences. Interestingly I see the difference in I/O scheduler config, 1st system is set to “deadline” and other one is set to “cfq”. So I made it equal by echoing to “/sys/block/<disk device>/queue/scheduler" but still no change in scheduler benchmark metrics.
Is it the correct way to find the differences between kernels? If so, what other important CONFIG_* variables need to consider?
$ cat config.patch | grep -i sched
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_CGROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y # CONFIG_RT_GROUP_SCHED is not set # IO Schedulers @@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y CONFIG_CFQ_GROUP_IOSCHED=y -CONFIG_DEFAULT_IOSCHED="deadline" +CONFIG_DEFAULT_IOSCHED="cfq" +CONFIG_MQ_IOSCHED_DEADLINE=m +CONFIG_MQ_IOSCHED_KYBER=m +CONFIG_IOSCHED_BFQ=m +CONFIG_BFQ_GROUP_IOSCHED=y CONFIG_SCHED_SMT=y CONFIG_SCHED_MC=y +CONFIG_SCHED_MC_PRIO=y +# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set +CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
There's lots of other options that affect performance, depending on your specific benchmark, other than these. good luck! greg k-h
On Tue, Aug 18, 2020 at 10:44 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
Hi all,
I’ve two identical Linux systems with only kernel differences.
What are the differences in the kernels?
You didn't answer this question, is this the same kernel source being compared here? Same version? Same compiler? Everything identical?
Both systems are having exactly the same hardware configuration. Compiler and kernel versions are different. One system has Ubuntu 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc version 7.5.0).
While doing kernel profiling with perf, I got the below mentioned metrics for Scheduler benchmarks.
1st system (older kernel version compared to the other system) benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 2.936 [sec]
2nd system benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 10.074 [sec]
So as per scheduler benchmark results, clearly a huge difference between two systems. Can anyone suggest to me how to dive deeper to know the root cause for it.
Look a the differences between your different kernels, that would be a great start :) I created the difference between two kernel config files and then tried to spot the CONFIG*SCHED* differences. Interestingly I see the difference in I/O scheduler config, 1st system is set to “deadline” and other one is set to “cfq”. So I made it equal by echoing to “/sys/block/<disk device>/queue/scheduler" but still no change in scheduler benchmark metrics.
Is it the correct way to find the differences between kernels? If so, what other important CONFIG_* variables need to consider?
$ cat config.patch | grep -i sched
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_CGROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y # CONFIG_RT_GROUP_SCHED is not set # IO Schedulers @@ -369,10 +434,14 @@ CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y CONFIG_CFQ_GROUP_IOSCHED=y -CONFIG_DEFAULT_IOSCHED="deadline" +CONFIG_DEFAULT_IOSCHED="cfq" +CONFIG_MQ_IOSCHED_DEADLINE=m +CONFIG_MQ_IOSCHED_KYBER=m +CONFIG_IOSCHED_BFQ=m +CONFIG_BFQ_GROUP_IOSCHED=y CONFIG_SCHED_SMT=y CONFIG_SCHED_MC=y +CONFIG_SCHED_MC_PRIO=y +# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set +CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
There's lots of other options that affect performance, depending on your specific benchmark, other than these.
good luck!
greg k-h
-- Thanks, Sekhar
On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 10:44 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
Hi all,
I’ve two identical Linux systems with only kernel differences.
What are the differences in the kernels?
You didn't answer this question, is this the same kernel source being compared here? Same version? Same compiler? Everything identical?
Both systems are having exactly the same hardware configuration. Compiler and kernel versions are different. One system has Ubuntu 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc version 7.5.0).
Those are _very_ different kernel versions, with many years and tens of thousands of different changes between them. Hopefully the newer kernel is faster, so just stick with that :) greg k-h
On Tue, Aug 18, 2020 at 11:06 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 10:44 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
Hi all,
I’ve two identical Linux systems with only kernel differences.
What are the differences in the kernels?
You didn't answer this question, is this the same kernel source being compared here? Same version? Same compiler? Everything identical?
Both systems are having exactly the same hardware configuration. Compiler and kernel versions are different. One system has Ubuntu 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc version 7.5.0).
Those are _very_ different kernel versions, with many years and tens of thousands of different changes between them.
Hopefully the newer kernel is faster, so just stick with that :)
But unfortunately the newer kernel is very slow, that is the reason for starting this investigation :) Any type of help, and guidelines to dive deeper will be highly appreciated.
greg k-h
-- Thanks, Sekhar
On 8/18/20 7:53 PM, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 11:06 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 10:44 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote:
On Tue, Aug 18, 2020 at 8:06 PM Greg KH <greg@kroah.com> wrote:
On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote:
Hi all,
I’ve two identical Linux systems with only kernel differences. What are the differences in the kernels? You didn't answer this question, is this the same kernel source being compared here? Same version? Same compiler? Everything identical? Both systems are having exactly the same hardware configuration. Compiler and kernel versions are different. One system has Ubuntu 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc version 7.5.0). Those are _very_ different kernel versions, with many years and tens of thousands of different changes between them.
Hopefully the newer kernel is faster, so just stick with that :) But unfortunately the newer kernel is very slow, that is the reason for starting this investigation :) Any type of help, and guidelines to dive deeper will be highly appreciated.
On the 4.4 kernel you dont have +CONFIG_RETPOLINE=y +CONFIG_INTEL_RDT=y And your base is very different two. Try to use mainline on both system and see. You can also use the same base kernel version from ubuntu and run your test.
greg k-h
On Tue, Aug 18, 2020 at 9:32 PM Muni Sekhar <munisekharrms@gmail.com> wrote:
Hi all,
I’ve two identical Linux systems with only kernel differences. While doing kernel profiling with perf, I got the below mentioned metrics for Scheduler benchmarks.
1st system (older kernel version compared to the other system) benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 2.936 [sec]
2nd system benchmark result:
$ perf bench sched messaging -g 64 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 64 groups == 2560 processes run
Total time: 10.074 [sec]
So as per scheduler benchmark results, clearly a huge difference between two systems. Can anyone suggest to me how to dive deeper to know the root cause for it. Also are there any tunable kernel parameters related to this one?
-- Thanks, Sekhar
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Aside from very far different kernel version, I think you need to use perf record and visualized it with heat map too (check Brendan Gregg blog) On the lower level side, check cache miss on L1/L2/L3 rate on both configuration. That might reveal something -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaig...> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaig...> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
participants (4)
-
Greg KH -
Mulyadi Santosa -
Muni Sekhar -
peter enderborg