U <c:@sxddlmZmZddlZddlmZddlmZddl Z ddl Z dgZ Gddde Z e Zdd ZGd dde ZdS) ) defaultdictabcN)deepcopy)chain Optimizerc@seZdZdZddZdS)_RequiredParameterzCSingleton class representing a required parameter for an Optimizer.cCsdS)Nzselfrr9/tmp/pip-unpacked-wheel-gikjz4vx/torch/optim/optimizer.py__repr__ sz_RequiredParameter.__repr__N)__name__ __module__ __qualname____doc__r rrrr r srcsfdd}|S)Nc s>t}z$t|jd|f||}W5t|X|S)Ndifferentiable)torchZis_grad_enabledZset_grad_enableddefaults)r argskwargsZ prev_gradretfuncrr _use_grads  z/_use_grad_for_differentiable.._use_gradr)rrrrr _use_grad_for_differentiables rc@sxeZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ de dddZddZddZdS)raXBase class for all optimizers. .. warning:: Parameters need to be specified as collections that have a deterministic ordering that is consistent between runs. Examples of objects that don't satisfy those properties are sets and iterators over values of dictionaries. Args: params (iterable): an iterable of :class:`torch.Tensor` s or :class:`dict` s. Specifies what Tensors should be optimized. defaults: (dict): a dict containing default values of optimization options (used when a parameter group doesn't specify them). cCstjd||_|t|tjr8tdt|t t |_ g|_ t |}t|dkrdtdt|dt s|d|ig}|D]}||qd|_dS)Nzpython.optimizerzZparams argument given to the optimizer should be an iterable of Tensors or dicts, but got rz%optimizer got an empty parameter listparamsT)rZ_CZ_log_api_usage_oncer_hook_for_profile isinstanceTensor TypeErrortypenamerdictstate param_groupslistlen ValueErroradd_param_group$_warned_capturable_if_run_uncaptured)r rrr# param_grouprrr __init__-s"      zOptimizer.__init__cCs|j|j|jdS)Nrr"r#r+r rrr __getstate__JszOptimizer.__getstate__cCs&|j|||jdddS)NrF)__dict__updaterr setdefault)r r"rrr __setstate__Qs zOptimizer.__setstate__cCsp|jjd}t|jD]L\}}|d7}|d|7}t|D] }|dkr@|d|||7}q@q|d7}|S)Nz ( zParameter Group {0} rz {0}: {1} )) __class__r enumerater#formatsortedkeys)r format_stringigroupkeyrrr r Vs zOptimizer.__repr__cCshtjrdtjrdtj}|r<|jds.profile_hook_step..wrapper) functoolswraps)rrGrrr profile_hook_stepsz6Optimizer._hook_for_profile..profile_hook_stephookedT)r5r3r _zero_grad_profile_namer=steprK)r rJrKrrr rs  zOptimizer._hook_for_profilecsLidfddfdd|jD}fdd|jD}||dS) aKReturns the state of the optimizer as a :class:`dict`. It contains two entries: * state - a dict holding current optimization state. Its content differs between optimizer classes. * param_groups - a list containing all parameter groups where each parameter group is a dict rcsbdd|D}fddt|dDfdd|dD|d<t|d7|S)NcSsi|]\}}|dkr||qS)rr.0kvrrr sz.pack_group..cs&i|]\}}t|krt||qSrid)rOr9pparam_mappingsrr rRs rcsg|]}t|qSrrS)rOrUrVrr sz.pack_group..)itemsr.r4r%)r:packed)rW start_indexrr pack_groups "z(Optimizer.state_dict..pack_groupcsg|] }|qSrrrOg)r\rr rXsz(Optimizer.state_dict..cs.i|]&\}}t|tjr$t|n||qSr)rrrrTrNrVrr rRsz(Optimizer.state_dict..r"r#)r#r"rY)r r#Z packed_stater)r\rWr[r state_dicts  zOptimizer.state_dictc s&t|}|j}|d}t|t|kr.tddd|D}dd|D}tddt||Drjtddd ttd d|Dtd d|DD}dfd d tt }|d D]0\}} ||kr||} | | || <q| ||<qddfddt||D} | || dd S)zLoads the optimizer state. Args: state_dict (dict): optimizer state. Should be an object returned from a call to :meth:`state_dict`. r#zsz,Optimizer.load_state_dict..css|]}t|dVqdSrarbr]rrr rcscss|]\}}||kVqdSNr)rOZp_lenZs_lenrrr rcsz]loaded state dict contains a parameter group that doesn't match the size of optimizer's groupcSsi|]\}}||qSrr)rOZold_idrUrrr rRsz-Optimizer.load_state_dict..css|]}|dVqdSrarr]rrr rcscss|]}|dVqdSrarr]rrr rcsNcst|tjr8|dkr4r(|j}|j}|St|trZfdd|DSt|t j rt |fdd|DS|SdS)zBMake a deep copy of value, casting all tensors to device of param.rMcs i|]\}}|||dqS))r;rrNcastparamrr rRsz;Optimizer.load_state_dict..cast..c3s|]}|VqdSrdr)rOrQrerr rcsz:Optimizer.load_state_dict..cast..N) rrrZis_floating_pointtodtypedevicer!rYcontainer_abcsIterabletype)rgvaluer;)rf)rgr rfs     z'Optimizer.load_state_dict..castr"cSs|d|d<|S)Nrr)r:Z new_grouprrr update_groups z/Optimizer.load_state_dict..update_groupcsg|]\}}||qSrr)rOr^Zng)rorr rXsz-Optimizer.load_state_dict..r_)N) rr#r%r&anyzipr from_iterablerr!rYr0) r r`groupsZ saved_groupsZ param_lensZ saved_lensZid_mapr"rPrQrgr#r)rfror load_state_dicts4  zOptimizer.load_state_dictF) set_to_nonec Cs|jdd}t|ds ||r0tdd}tjj|j |j D]}|dD]r}|j dk rT|rnd|_ qT|j j dk r|j n |j d|r|j jr|j qT||j j|j j|j qTqH|r|D] \}}|D]}t|qqW5QRXdS)aSets the gradients of all optimized :class:`torch.Tensor` s to zero. Args: set_to_none (bool): instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests ``zero_grad(set_to_none=True)`` followed by a backward pass, ``.grad``\ s are guaranteed to be None for params that did not receive a gradient. 3. ``torch.optim`` optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether). foreachFrLcSsttSrd)rr$rrrr  z%Optimizer.zero_grad..rN)rgethasattrrrrrArBrCrLr#ZgradZgrad_fnZdetach_Zrequires_grad_Z is_sparseZzero_rjriappendrYvaluesZ_foreach_zero_) r rurvZper_device_and_dtype_gradsr:rUrEZper_dtype_gradsZgradsrrr zero_grads*            zOptimizer.zero_gradcCstdS)afPerforms a single optimization step (parameter update). Args: closure (Callable): A closure that reevaluates the model and returns the loss. Optional for most optimizers. .. note:: Unless otherwise specified, this function should not modify the ``.grad`` field of the parameters. N)NotImplementedError)r closurerrr rMs zOptimizer.stepcCs\t|tstd|d}t|tjr2|g|d<n t|trFtdn t||d<|dD]D}t|tjs|tdt||j ddsZ|j sZ|j sZt dqZ|j D]2\}}|tkr||krt d|q|||q|d}t|tt|kr tjd d d t}|jD]}|t|dq|t|dsLt d |j|dS) aAdd a param group to the :class:`Optimizer` s `param_groups`. This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the :class:`Optimizer` as training progresses. Args: param_group (dict): Specifies what Tensors should be optimized along with group specific optimization options. zparam group must be a dictrzoptimizer parameters need to be organized in ordered collections, but the ordering of tensors in sets will change between runs. Please use a list instead.z>optimizer can only optimize Tensors, but one of the params is rNz can't optimize a non-leaf TensorzJparameter group didn't specify a value of required optimization parameter zoptimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see github.com/pytorch/pytorch/issues/40967 for more information) stacklevelz7some parameters appear in more than one parameter group)rr!AssertionErrorrrsetrr$r rryZis_leafZ retains_gradr&rYrequiredr/r%warningswarnr#r. isdisjointr{)r r)rrgnamedefaultZ param_setr:rrr r',s>          zOptimizer.add_param_groupN)F)r rrrr*r,r0r r?r@rr`rtboolr}rMr'rrrr rs  @) ) collectionsrrrkrcopyr itertoolsrrrH__all__objectrrrrrrrr s