include/linux/page_ref.h: ensure page_ref_unfreeze is ordered against prior accesses

page_ref_freeze and page_ref_unfreeze are designed to be used as a pair,
wrapping a critical section where struct pages can be modified without
having to worry about consistency for a concurrent fast-GUP.

Whilst page_ref_freeze has full barrier semantics due to its use of
atomic_cmpxchg, page_ref_unfreeze is implemented using atomic_set, which
doesn't provide any barrier semantics and allows the operation to be
reordered with respect to page modifications in the critical section.

This patch ensures that page_ref_unfreeze is ordered after any critical
section updates, by invoking smp_mb() prior to the atomic_set.

Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Will Deacon <[email protected]>
Acked-by: Steve Capper <[email protected]>
Acked-by: Kirill A. Shutemov <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>

diff --git a/include/linux/page_ref.h b/include/linux/page_ref.h
index 610e1327191849bf6ef5564ec1d2ddef690a0123..1fd71733aa68efb95b390eb708616213271a0fd9 100644 (file)
--- a/include/linux/page_ref.h
+++ b/include/linux/page_ref.h
@@ -174,6 +174,7 @@ static inline void page_ref_unfreeze(struct page *page, int count)
        VM_BUG_ON_PAGE(page_count(page) != 0, page);
        VM_BUG_ON(count == 0);
 
+       smp_mb();
        atomic_set(&page->_refcount, count);
        if (page_ref_tracepoint_active(__tracepoint_page_ref_unfreeze))
                __page_ref_unfreeze(page, count);