arm64: Question about barriers with the mmu off
Hi, I have a question about dmb barriers in arm64's head.S. In the head.S, I could see the pattern below several times. str w0, [x1] dmb sys dc ivac, x1 // Invalidate potentially stale cache line I found that, Commit(fix cache flushing and barriers in set_cpu_boot_mode_flag) explained the code.
This patch reworks the broken flushing code so that we:
(1) Use a DMB to order the strongly-ordered write of the cacheline against the subsequent cache-maintenance operation (by-VA operations only hazard against normal, cacheable accesses).
(2) Use a single dc ivac instruction to invalidate any clean lines containing a stale copy of the line after it has been updated. Use a DMB to order the strongly-> ordered write of the cacheline
But I can't understand why the store operation should precede the dc operation. Is there any problem, if the dc operation precedes the store operation?
On Mon, 16 Nov 2020 20:58:52 +0900, Wonhyuk Yang said:
str w0, [x1] So we dirtied the cache line. dmb sys dc ivac, x1 // Invalidate potentially stale cache line
So we invalidate it.
Is there any problem, if the dc operation precedes the store operation?
If you swap them, you get... dc ivac,x1 // invalidate a cache line that's probably OK str w0,[x1 // and now we do a store that leaves a possibly stale cache line In other words, if you swap them you may leave an un-invalidated stale cache line.
On Tue, Nov 17, 2020 at 11:14 AM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
If you swap them, you get...
dc ivac,x1 // invalidate a cache line that's probably OK str w0,[x1 // and now we do a store that leaves a possibly stale cache line
In other words, if you swap them you may leave an un-invalidated stale cache line.
You mean, even if STCLR_EL1.{C, M} is cleared, store doesn't bypass the cache?
On Tue, 17 Nov 2020 12:00:32 +0900, Wonhyuk Yang said:
If you swap them, you get...
dc ivac,x1 // invalidate a cache line that's probably OK str w0,[x1 // and now we do a store that leaves a possibly stale cache line
In other words, if you swap them you may leave an un-invalidated stale cache line.
You mean, even if STCLR_EL1.{C, M} is cleared, store doesn't bypass the cache?
That's the problem. The store bypasses the cache line, and the next reference that uses the cache can get stale data. So you have to flush the cache line so the next reference has to refresh the cache on the memory read.
On Tue, Nov 17, 2020 at 12:45 PM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
If you swap them, you get...
dc ivac,x1 // invalidate a cache line that's probably OK str w0,[x1 // and now we do a store that leaves a possibly stale cache line
In other words, if you swap them you may leave an un-invalidated stale cache line.
You mean, even if STCLR_EL1.{C, M} is cleared, store doesn't bypass the cache?
That's the problem. The store bypasses the cache line, and the next reference that uses the cache can get stale data. So you have to flush the cache line so the next reference has to refresh the cache on the memory read.
Yes, I understood that I have to flush the cache before the cacheable read(mmu on). But I'm not fully understand your explanation below.
dc ivac,x1 // invalidate a cache line that's probably OK str w0,[x1 // and now we do a store that leaves a possibly stale cache line
Could you explain me why the store still leaves stale cache? We invalidated the cacheline and store will not make footprint in the cache.
On Tue, 17 Nov 2020 14:08:02 +0900, Wonhyuk Yang said:
dc ivac,x1 // invalidate a cache line that's probably OK str w0,[x1 // and now we do a store that leaves a possibly stale cache line
Could you explain me why the store still leaves stale cache? We invalidated the cacheline and store will not make footprint in the cache.
There's a race condition... Invalidate the cache line.... then another CPU manages to fetch the cache line. and then we do a store that doesn't update the cache - and the other CPU is still looking at the old data.
On Tue, Nov 17, 2020 at 3:25 PM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
dc ivac,x1 // invalidate a cache line that's probably OK str w0,[x1 // and now we do a store that leaves a possibly stale cache line
Could you explain me why the store still leaves stale cache? We invalidated the cacheline and store will not make footprint in the cache.
There's a race condition...
Invalidate the cache line.... then another CPU manages to fetch the cache line. and then we do a store that doesn't update the cache - and the other CPU is still looking at the old data.
Oh, I didn't consider that another cpu read with cacheable. Now I understand why the barrier is here. Thank you for your help.
participants (2)
-
Valdis Klētnieks -
Wonhyuk Yang