x86: mm: Read cr2 before prefetching the mmap_lock

Prefetch instructions can generate spurious faults on certain
models of older CPUs. The faults themselves cannot be stopped
and they can occur pretty much anywhere - so the way we solve
them is that we detect certain patterns and ignore the fault.

There is one small path of code where we must not take faults
though: the #PF handler execution leading up to the reading
of the CR2 (the faulting address). If we take a fault there
then we destroy the CR2 value (with that of the prefetching
instruction's) and possibly mishandle user-space or
kernel-space pagefaults.

It turns out that in current upstream we do exactly that:

	prefetchw(&mm->mmap_sem);

	/* Get the faulting address: */
	address = read_cr2();

This is not good.

So turn around the order: first read the cr2 then prefetch
the lock address. Reading cr2 is plenty fast (2 cycles) so
delaying the prefetch by this amount shouldnt be a big issue
performance-wise.

[ And this might explain a mystery fault.c warning that sometimes
  occurs on one an old AMD/Semptron based test-system i have -
  which does have such prefetch problems. ]

Cc: Mathieu Desnoyers <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Nick Piggin <[email protected]>
Cc: Pekka Enberg <[email protected]>
Cc: Vegard Nossum <[email protected]>
Cc: Jeremy Fitzhardinge <[email protected]>
Cc: Hugh Dickins <[email protected]>
LKML-Reference: <20090616030522.GA22162@Krystal>
Signed-off-by: Ingo Molnar <[email protected]>

diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index c6acc632637417c193394da4881fa19112ace761..0482fa649738718ba5fa6903d45b1862510527e0 100644 (file)
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -951,11 +951,11 @@ do_page_fault(struct pt_regs *regs, unsigned long error_code)
        tsk = current;
        mm = tsk->mm;
 
-       prefetchw(&mm->mmap_sem);
-
        /* Get the faulting address: */
        address = read_cr2();
 
+       prefetchw(&mm->mmap_sem);
+
        if (unlikely(kmmio_fault(regs, address)))
                return;