On Mon, Oct 18, 2021 at 04:41:14PM +0800, Dongliang Mu wrote:

On Mon, Oct 18, 2021 at 4:07 PM FMDF <fmdefrancesco@gmail.com> wrote:

...

On Mon, 18 Oct 2021, 04:46 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:

...

Hi all,

I am writing to kindly ask one question: is there any tracing mechanism in Linux kernel that can trace all the executed instructions of a user process? If this user process is run on different processors, traces of this process on different processors should be also recorded.

You've not explained in detail what is the purpose of the tracing that you want to do. Missing this information I can only provide you a list of links to various tools and methods. Take a look by yourself and try to figure out what is better suited for your needs...

I want to log all the executed instructions of a user process (e.g., poc.c in syzkaller) in the kernel mode and then would like to leverage backward analysis to capture the root cause of kernel panic/crash.

Therefore, I need the instruction-level tracing mechanisms or tools.

Then use a userspace debugger like gdb, that is what they are designed for. good luck! greg k-h

On Mon, 18 Oct 2021, 11:18 Dongliang Mu, <mudongliangabcd@gmail.com> wrote:

+Brendan Gregg

Good catch! Take a look at his "System Performance, 2nd ed., ISBN 978-0-13-682015-4. He has an interesting blog at https://www.brendangregg.com/blog/index.html As Greg, Pavel, and I said, use GDB if you need it. However I'd still consider using the "func_graph" tracer of ftrace in order to know where to attach GDB in the calls chain. With ftrace you'll also get info about which CPU is running (you wrote that you need to know this information). Now I recall that, by using Perf, somehow I was able to go interactively down to the assembly code and see where it was stuck in an endless loop. Unfortunately at this moment I cannot remember how I did that and I cannot try to reproduce it for you because these days I'm away from my PC (I'm writing with a smartphone). Please try to figure it out by your own. Regards, Fabio

On Mon, 18 Oct 2021 16:41:14 +0800, Dongliang Mu said:

I want to log all the executed instructions of a user process (e.g., poc.c in syzkaller) in the kernel mode and then would like to leverage backward analysis to capture the root cause of kernel panic/crash.

Therefore, I need the instruction-level tracing mechanisms or tools.

Tracing just the instructions won't get you where you want to be if you're going through this approach. You *also* need to track all the data - the instruction path inside two different runs of syzkaller may be essentially identical, but pass 2 different values as the 3rd parameter of a syscall. You may also have to deal with insane amounts of data - the actual error could have been minutes or even hours before, or the interaction between two different processes. You probably want to take a *really* close look at how prof and friends avoid infinite regress when code execution drops inside the prof code, because you're going to hit the same issues. Or.... You can work smarter rather than harder, and ask yourself what's the minimum amount and type of additional information to make a significant improvement in the debugging of system crashes. For example, 95% of the time, you can figure out what the bug is by merely looking at the stack traceback. For most of the rest of the cases, simply capturing the parameter values from the syscall and the basic info for page faults and other interrupts is probably sufficient, and you can probably leverage the audit subsystem for most of that. It can already record syscall parameters, while logging page faults and other interrupts can probably be done with prof. At that point, you don't actually *need* every instruction - only tracing branch and call instructions is sufficient, because you already know that each instruction between the target of a branch/call and the next branch/call will be executed. Similarly, the lockdep code will catch most locking issues. But it won't flag issues with data that should be protected with a lock, but are bereft of any locking. So ask yourself: What ways are there to analyze the code and detect critical sections prone to race conditions? Is there a sparse-on-steroids approach that wil do the heavy lifting for those? (Note that this isn't an easy task for the general case, but identifying two or three specific common patterns and finding a way to detect them may be worthwhile) And many of the rest of crashes are timing related, and "let's trace every single instruction" is almost guaranteed to make things slow enough to change/bypass the timing issue. So... What's left that would be the most helpful with the least amount of data? Go look at some threads on linux-kernel. Look at the kernel bugs that were the result of a Homer Simpson "D'oh!" moment. What can we do to make those bugs less likely to make it into the code in the first place? For the more subtle bugs, what data finally made the debugging come together?