usb_gadget_ops :: udc_stop might be called with driver = NULL since 511f3c5326eabe1ece35202a404c24c0aeacc246
(usb: gadget: udc-core: fix a regression during gadget driver unbinding)
Because of that, 2nd times insmod goes fail.
This patch fixes it up.