[ARM64] Trap and Forward CPU_ON psci smc

[Mushahid Hussain]

Hi,
I'm working on a hobby project: AARCH64 Hypervisor on Raspberry Pi 4b, to
learn about kernel internals, boot process and ARM64 baremetal. I have a
problem with trapping and forwarding a psci smc generated by linux kernel.
I'll explain everything and what steps I have followed.
I'm also using Trusted Firmware-A at EL3 using armstub config. More about
TF-A here: https://trustedfirmware-a.readthedocs.io/en/latest/plat/rpi4.html

I have posted this in TF-A mailing list as well. I would love to learn from
experienced kernel and baremetal hackers here as well.

Right now, I'm implementing SMC trapping. I can successfully forward almost
all SMCs except for PSCI_CPU_ON_AARCH64. Linux makes these SMCs to bring up
secondary CPUs during boot. Here's what I'm trying to do:

   -  trap the PSCI_CPU_ON_AARCH64 SMC,
   - preserve the entry_point address in global variable
   - replace the entrypoint with my entrypoint and make the smc to tf-a(or
   simply forward it.)
   - when secondary cpus come online at the given address, where I set
   their stack point and then eret the original address.

Secondary cpus won't come online at the given address. Even if I don't
change any arguments of CPU_ON smc and forward it as it is, the secondary
cpus still won't come online. However, without trapping
enabled(HCR_EL2.TSC=0), everything works fine.

I tried to debug inside Trusted Firmware. I know that overall path for
secondary CPU hotplug in is:

CPU released from reset -> (ROM and possibly some other bootloader) ->
bl31/aarch64/bl31_entrypoint.S:bl31_warm_entrypoint() ->
lib/psci/psci_common.c:psci_warmboot_entrypoint() ->
lib/psci/psci_on.c:psci_cpu_on_finish() -> rpi3_pwr_domain_on_finish()

I printed at all these points in Trusted Firmware with and without trapping
enabled. Here's what I found: Nothing gets printed anywhere in that path if
trapping is enabled. However, without trapping enabled, I can print
anywhere even in bl31_entrypoint.S:bl31_warm_entrypoint().  What could be
the problem?

Here's my code:
https://github.com/SikkiLadho/Leo/blob/4f272eff39934058a7f989c91aad82eab81041a1/src/exceptions.S#L39
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20220416/47eead1f/attachment.html>