sched/completion: Add lock-free checking of the blocking case

The "thread would block" case can be checked without grabbing ->wait.lock.

[ If the check does not return early then grab the lock and recheck.
  A memory barrier is not needed as complete() and complete_all() imply
  a barrier.

  The ACCESS_ONCE() is needed for calls in a loop that, if inlined, could
  optimize out the re-fetching of x->done. ]

Signed-off-by: Nicholas Mc Guire <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: Linus Torvalds <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>

diff --git a/kernel/sched/completion.c b/kernel/sched/completion.c
index 9d1fe32da232a098934d5a19e5dd3d71290bc6db..7052d3fd4e7bd87a29bd144cbca1086621040251 100644 (file)
--- a/kernel/sched/completion.c
+++ b/kernel/sched/completion.c
@@ -268,6 +268,15 @@ bool try_wait_for_completion(struct completion *x)
        unsigned long flags;
        int ret = 1;
 
+       /*
+        * Since x->done will need to be locked only
+        * in the non-blocking case, we check x->done
+        * first without taking the lock so we can
+        * return early in the blocking case.
+        */
+       if (!ACCESS_ONCE(x->done))
+               return 0;
+
        spin_lock_irqsave(&x->wait.lock, flags);
        if (!x->done)
                ret = 0;