how to debug... program hangs while downing for write a rwsem (busybox shell hangs/stucks after boot, a case related to u-boot)

[Chan Kim]

Hi, Greg K-H and all,

This is just to let you know how I solved this problem recently(I succeeded
in booting linux today and the shell works ok now).
I want to share it so that someone in the future can get help from this
email.

The major things I fixed in u-boot was (I didn't know bootloader should be
doing these things)

1. I added CONFIG_GICV3 in the u-boot because there are some GICv3 init
stuff in u-boot.
Also I added arm gic-600's redistributor power-on procedure in the u-boot
code. 
(it was powered off so the timer interrupt was not passed to the PE. 
the u-boot didn't have this code yet, arm-tf bl31 code has this part.
GICR_PWRR register related.)

2. Even with correct GIC setup in u-boot, timer interrupt didn't occur
(system counter didn't increment).
So I had to setup the system counter (frequency, and enable).  It's memory
mapped registers.
And I also setup the cntp_el1 (TVAL and enable, and masked interrupt).
  In arm64, the system counter is provided to all PEs and each PE has timers
which generates interrupt when the system counter reaches comparator. Linux
Timer ISR adds TVAL to the comparator for next interrupt.
With this fix, the scheduling by timer worked and problems during syscall in
the shell (like rwsem hang) disappeared.

Thank you!

Chan Kim

>-----Original Message-----
>From: Chan Kim <ckim at etri.re.kr>
>Sent: Thursday, September 22, 2022 10:23 PM
>To: 'Greg KH' <greg at kroah.com>
>Cc: kernelnewbies at kernelnewbies.org
>Subject: RE: how to debug... program hangs while downing for write a rwsem
>
>And I am also trying to do the experiment with linux5.15.68.
>Somehow linux doesn't start with this new setup and I need some time.
>Thank you for the advices.
>
>Chan Kim
>
>>-----Original Message-----
>>From: Chan Kim <ckim at etri.re.kr>
>>Sent: Thursday, September 22, 2022 10:18 PM
>>To: 'Greg KH' <greg at kroah.com>
>>Cc: 'kernelnewbies at kernelnewbies.org' <kernelnewbies at kernelnewbies.org>
>>Subject: RE: how to debug... program hangs while downing for write a
>>rwsem
>>
>>Hi, Greg K-H,
>>
>>I found the scheduler isn't working correct.
>>When I run an application instead of the shell, and when I call sleep()
>>inside the application, it hangs.
>>So I have to investigate about the clock settings first(in dts, etc).
>>(The serial port is 8250 compatible uart that we've been using).
>>Thank you.
>>
>>Chan Kim
>>
>>>-----Original Message-----
>>>From: Greg KH <greg at kroah.com>
>>>Sent: Wednesday, September 21, 2022 8:16 PM
>>>To: Chan Kim <ckim at etri.re.kr>
>>>Cc: kernelnewbies at kernelnewbies.org
>>>Subject: Re: how to debug... program hangs while downing for write a
>>>rwsem
>>>
>>>On Wed, Sep 21, 2022 at 08:02:50PM +0900, Chan Kim wrote:
>>>> Hello kernel experts and users,
>>>>
>>>> I'm trying to boot linux-5.10.0-rc5 on our arm64 board (using
>>>> neoverse-v1 core).
>>>
>>>5.10-rc5 is very very very old and obsolete, and not even a real release.
>>>Why are you using that specific kernel version?
>>>
>>>At the very least, use a released and supported 5.10.y tree, but even
>>>better, use the latest 5.19.y release.
>>>
>>>> After the shell comes up, when I press enter key at the prompt, it
>>>> gives me another prompt.
>>>> But when I press enter key again, it stops responding.
>>>> After inspecting where the program hangs, I found it hangs in
>>>> function tty_set_termios while downing the rwsem.
>>>> Please see below where I marked "<===== this line "". (in file
>>>> drivers/tty/tty_ioctl.c)
>>>>
>>>> int tty_set_termios(struct tty_struct *tty, struct ktermios
>>>> *new_termios) {
>>>>     struct ktermios old_termios;
>>>>     struct tty_ldisc *ld;
>>>>
>>>>     WARN_ON(tty->driver->type == TTY_DRIVER_TYPE_PTY &&
>>>>         tty->driver->subtype == PTY_TYPE_MASTER);
>>>>     /*
>>>>      *  Perform the actual termios internal changes under lock.
>>>>      */
>>>>
>>>>
>>>>     /* FIXME: we need to decide on some locking/ordering semantics
>>>>        for the set_termios notification eventually */
>>>>     down_write(&tty->termios_rwsem);                        <======
this
>>>> line
>>>
>>>What serial driver are you using that is causing this problem?
>>>
>>>thanks,
>>>
>>>greg k-h
>
>
>
>
>
>_______________________________________________
>Kernelnewbies mailing list
>Kernelnewbies at kernelnewbies.org
>https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies