memcg: optimize the "Search everything else" loop in mm_update_next_owner()

for_each_process_thread() is sub-optimal. All threads share the same
->mm, we can swicth to the next process once we found a thread with
->mm != NULL and ->mm != mm.

Signed-off-by: Oleg Nesterov <[email protected]>
Reviewed-by: Michal Hocko <[email protected]>
Cc: Balbir Singh <[email protected]>
Cc: Johannes Weiner <[email protected]>
Cc: KAMEZAWA Hiroyuki <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Peter Chiang <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>

diff --git a/kernel/exit.c b/kernel/exit.c
index 5ac3c19c245c855cb195c20843c76bdef4e74014..750c2e5946170fbc07f102a18db8334f036d9f78 100644 (file)
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -397,9 +397,15 @@ retry:
        /*
         * Search through everything else, we should not get here often.
         */
-       for_each_process_thread(g, c) {
-               if (!(c->flags & PF_KTHREAD) && c->mm == mm)
-                       goto assign_new_owner;
+       for_each_process(g) {
+               if (g->flags & PF_KTHREAD)
+                       continue;
+               for_each_thread(g, c) {
+                       if (c->mm == mm)
+                               goto assign_new_owner;
+                       if (c->mm)
+                               break;
+               }
        }
        read_unlock(&tasklist_lock);
        /*