smp_processor_id used in preemptable context ?
Hi Linux hackers ! I'm reading some code in net/core/dev.c and something puzzles me : somewhere in __dev_queue_xmit, we have : int cpu = smp_processor_id(); /* ok because BHs are off */ and, indeed, a few lines up there is : /* Disable soft irqs for various locks below. Also * stops preemption for RCU. */ rcu_read_lock_bh(); Now I'm wondering : is this really ok ?
From what I understand, this code can very well run in user context, with hard IRQs ON, and in fact it should, according to the following comment :
* When calling this method, interrupts MUST be enabled. This is because * the BH enable code must have IRQs enabled so that it will not deadlock. * --BLG Then I guess it could be preempted at any time, especially with aggressive versions of preemptions ? And if so, are we not at risk that our thread is migrated to an other CPU just after smp_processor_id returned ? Cheers, V Ray
On Thu, 28 Apr 2022 11:15:27 +0200, Vincent Ray said:
Then I guess it could be preempted at any time, especially with aggressive versions of preemptions ? And if so, are we not at risk that our thread is migrated to an other CPU just after smp_processor_id returned ?
Often, we don't actually *care* if it gets migrated. A *lot* of uses of smp_processor_id() are just to make statistic gathering more efficient. Rather than all the CPUs do all sorts of locking to avoid race conditions between different threads updating the same variable, and avoid massive cache line ping-ponging, the code just gets a pointer to a per_cpu area where we update some statistic or counter without worrying about that stuff because we know no other processor should be updating *this* processor's per_cpu area. Then anything that cares about the *total* across all CPUs can iterate across all the per_cpu numbers - and that (a) usually happens much less frequently than updates and (b) only needs to read all the per_cpu area without updating them. Yes, there's a very small chance that the "wrong" CPU's stats will be updated. But let's be realistic - for a lot of statistics, you'll never notice.
Thanks for your answer Valdis. So, if I understand well : 1) the comment in net/core/dev.c:__dev_queue_xmit is indeed misleading : int cpu = smp_processor_id(); /* ok because BHs are off */ as there is a small chance that we get the wrong info. Disabled BHs are not enough. Disabling preemption would be needed to be 100% correct. You agree ? 2) You're saying that, most of the time it's ok, as, at worst, we will be a little wrong on some per-cpu stats. I'm sure it does not matter if cpu0 gets 45 'events' instead of 46, and cpu1 gets 1 extra for free, but that's only if atomic_inc or something similar is used. If not, can't we end up with completely broken values, with e.g. cpu0 more or less writing its own count, totally different from that of cpu1, to cpu1's area ? 3) Finally it looks to me that this particular usage of smp_processor_id() in __dev_queue_xmit() is more about preventing a dead loop than just gathering statistics. Although I don't understand everything going on here, it looks like cpu id is directly used to detect a bad scenario, leading to a critical message : if (dev->flags & IFF_UP) { int cpu = smp_processor_id(); /* ok because BHs are off */ /* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */ if (READ_ONCE(txq->xmit_lock_owner) != cpu) { [...] } else { /* Recursion is detected! It is possible, * unfortunately */ recursion_alert: net_crit_ratelimited("Dead loop on virtual device %s, fix it urgently!\n", dev->name); So if our thread is migrated just after smp_processor_id() and lands on the cpu occupied by the thread that is the txq->xmit_lock_owner, itself just scheduled out, can't we go to the net_crit_ratelimited() ? Thanks, V
participants (2)
-
Valdis Klētnieks -
Vincent Ray