U Jºc^ã@sVddlmZmZmZmZmZddlmZddlm Z ddl m Z Gdd„dej ƒZ dS)é)ÚListÚUnionÚMappingÚDictÚAnyN)ÚTensor)Ú ShardedTensorc@s~eZdZeeeeeffdœdd„Zde dœdd„Z dd d „Z e ee fd œd d „Zeee fdœdd„Ze dœdd„ZdS)ÚShardedOptimizer)Ú named_paramscOsrg}| ¡D]4}t|tƒr6| ¡D]}| |j¡q"q | |¡q ||_||f|ž|Ž|_|jj|_|jj |_ dS)aª ShardedOptimizer collects all tensors and local shard tensors of ShardedTensor, then use these tensors as ``params`` for optimizers Args: named_params (Dict[str, Union[Tensor, ShardedTensor]]) : a Dict of parameters, where key is the parameter key, value is either Tensor or ShardedTensor parameter. optimizer_class (torch.optim.Optimizer): the Optimizer to use locally, i.e. torch.optim.SGD, torch.optim.Adagrad, etc. *optimizer_args: the arguments to initialize the optimizer. **optimizer_kwargs: the key-word arguments to initialize the optimizer. N) ÚvaluesÚ isinstancerZ local_shardsÚappendZtensorr Ú_optimZ param_groupsÚstate)Úselfr Zoptimizer_classZoptimizer_argsZoptimizer_kwargsZtensorsÚvalueZ local_shard©rúN/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/_shard/sharded_optim/api.pyÚ__init__ s     zShardedOptimizer.__init__F)Ú set_to_nonecCs|j |¡dS)a¶Sets the gradients of all optimized :class:`torch.Tensor` s to zero. Args: set_to_none (bool): instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests ``zero_grad(set_to_none=True)`` followed by a backward pass, ``.grad``\ s are guaranteed to be None for params that did not receive a gradient. 3. ``torch.optim`` optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether). N)rÚ zero_grad)rrrrrr+szShardedOptimizer.zero_gradNcCs|j |¡dS)afPerforms a single optimization step (parameter update). Args: closure (Callable): A closure that reevaluates the model and returns the loss. Optional for most optimizers. .. note:: Unless otherwise specified, this function should not modify the ``.grad`` field of the parameters. N)rÚstep)rÚclosurerrrr<s zShardedOptimizer.step)ÚreturncCs tdƒ‚dS)zë Returned state and param_groups will contain parameter keys instead of parameter indices like torch.optim.Optimizer. This allows for advanced functionality like optimizer re-sharding to be implemented. z0ShardedOptimizer state_dict not implemented yet!N©ÚNotImplementedError)rrrrÚ state_dictIszShardedOptimizer.state_dict)rcCs tdƒ‚dS)zÀLoads the ShardedOptimizer state. Args: state_dict (dict): ShardedOptimizer state. Should be an object returned from a call to :meth:`state_dict`. z5ShardedOptimizer load_state_dict not implemented yet!Nr)rrrrrÚload_state_dictSsz ShardedOptimizer.load_state_dict)Ú param_groupcCs tdƒ‚dS)zAdd a new param group z5ShardedOptimizer add_param_group not implemented yet!Nr)rrrrrÚadd_param_group]sz ShardedOptimizer.add_param_group)F)N)Ú__name__Ú __module__Ú __qualname__rÚstrrrrrÚboolrrrrrrrrrrrr sþ "   r )ÚtypingrrrrrZ torch.optimZoptimZtorchrZ'torch.distributed._shard.sharded_tensorrZ Optimizerr rrrrÚs