U Jc: @sUddlmZddlZddlmZmZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZiZeeefed <d d ZGd ddeZee GdddeZedddZdS))ABCN)DictType)FullyShardedDataParallel)DistributedDataParallel) Optimizer)as_functional_optim)allreduce_hook)_OptimizerHookState_hook_then_optimizer_registered_overlapped_optimscsfdd}|S)Nc s8|tkr,t|dtddd|t<|S)Nz# already registered with optim_cls  z, trying tore-register it for z is not supported.)r ValueError)Ztarget_overlapped_optim_cls optim_clse/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/algorithms/_optimizer_overlap/optimizer_overlap.py decorators z®ister_overlapped..decoratorr)rrrrrregister_overlappeds rc@s<eZdZeddddZeddddZeddd d ZdS) OverlappedOptimizerNrreturncCs ||_dS)z OverlappedOptimizer is a base class that child classes can implement to specify how different optimizers will register themselves with DDP. Nr)selfrrrr__init__#szOverlappedOptimizer.__init__)ddprcCst|jjddS)z,Registers the overlapped optimizer with DDP.z! does not support overlapped DDP.NNotImplementedError __class____name__)rrrrr register_ddp*s z OverlappedOptimizer.register_ddp)fsdprcCst|jjddS)z-Registers the overlapped optimizer with FSDP.z" does not support overlapped FSDP.Nr)rr rrr register_fsdp0s z!OverlappedOptimizer.register_fsdp) r __module__ __qualname__rrrrrr!rrrrr"srcs6eZdZdZeddfdd ZedddZZS) _OverlappedStandardOptimizerz!Overlaps a regular ``Optimizer``.Nrcs.t|t|jf||}t|||_dSN)superrrrr _opt_hook_state)rrparamsZ optim_argsZ optim_kwargsZf_optimrrrr;s z%_OverlappedStandardOptimizer.__init__)ddp_instcCs|dtt|jdSr%)Zregister_comm_hookr r r')rr*rrrr@s z)_OverlappedStandardOptimizer.register_ddp) rr"r#__doc__rrrr __classcell__rrr)rr$7sr$rc OsTt|D]6}zt|||f||WStk r>Yq Xq t||f||S)zU Returns a new ``OverlappedOptimizer`` instance that supports ``optim_cls``. )inspectgetmror KeyErrorr$)rr(argskwargsZclzrrr_as_overlapped_optimKs r2)abcrr-typingrrZtorch.distributed.fsdprZtorch.nn.parallelrZ torch.optimrZtorch.distributed.optimrZ9torch.distributed.algorithms.ddp_comm_hooks.default_hooksr ZCtorch.distributed.algorithms.ddp_comm_hooks.optimizer_overlap_hooksr r r __annotations__rrr$r2rrrrs