U <cj@s@ddlmZddlZddlmZmZmZmZmZddl Z ddl m Z ddl m Z m Z ddgZGd d d Zdd e ee jjje jeed d dZGddde Zdee ee ee ee ee ee eeeeeeeeeeeeeeeedddZee ee ee ee ee ee eeeeeeeeeeeeedddZee ee ee ee ee ee eeeeeeeeeeeeedddZe ee ee ee ee ee ee eeee jfeee fdddZee ee ee ee ee ee eeeeeeeeeeeeeddddZdS)) defaultdictN)castListOptionalDictTuple)Tensor) Optimizer_use_grad_for_differentiableAdamadamc@sDeZdZUeed<eeefed<eddddZeddd ZdS) _MultiDeviceReplicator main_tensor_per_device_tensorsN)rreturncCs||_t|j|i|_dSN)rstrdevicer)selfrr4/tmp/pip-unpacked-wheel-gikjz4vx/torch/optim/adam.py__init__sz_MultiDeviceReplicator.__init__rcCs4||jkr|j|S|jj|ddd}||j|<|S)NT)r non_blockingcopy)rrto)rrtensorrrrgets    z_MultiDeviceReplicator.get) __name__ __module__ __qualname__r__annotations__rrrrrrrrrs r grad_scaler) optimizerr$rrc sl|dkr dS||}fdd|D}t|dks@tdtttjt|}W5QRXt |S)Ncsg|]}|jddqS)T)r)r).0frrr ,sz'_get_fp16AMP_params..rz5No inf checks were recorded in _check_inf_per_device.) Z_check_inf_per_devicevalueslenAssertionErrortorchno_gradrrsumr)r%r$rZfound_inf_dictZ found_infsZfound_inf_combinedrrr_get_fp16AMP_params s  r/csbeZdZdZdddddddeeeeeedfd d Zfd d Zeddd ddZ Z S)r auImplements Adam algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)}, \: \textit{amsgrad}, \:\textit{maximize} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0\leftarrow 0 \text{ (second moment)},\: \widehat{v_0}^{max}\leftarrow 0\\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}g_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\textbf{if} \: amsgrad \\ &\hspace{10mm}\widehat{v_t}^{max} \leftarrow \mathrm{max}(\widehat{v_t}^{max}, \widehat{v_t}) \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}^{max}} + \epsilon \big) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_. Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (bool, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) foreach (bool, optional): whether foreach implementation of optimizer is used (default: None) maximize (bool, optional): maximize the params based on the objective, instead of minimizing (default: False) capturable (bool, optional): whether this instance is safe to capture in a CUDA graph. Passing True can impair ungraphed performance, so if you don't intend to graph capture this instance, leave it False (default: False) fused (bool, optional): whether fused implementation of optimizer is used. Currently, `torch.float64`, `torch.float32`, `torch.float16`, and `torch.bfloat16` are supported. (default: False) .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ MbP?g?g+?:0yE>rFN)foreachmaximize capturabledifferentiablefusedc sd|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dd|kstd |t|||||||| | | d } tt||| | r| rtd d |_td d|j DstddS)NzInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}r z%Invalid beta parameter at index 1: {}zInvalid weight_decay value: {}) lrbetaseps weight_decayamsgradr4r3r5r6r7z"`fused` cannot be `differentiable`Tcss,|]$}|dD]}|jo t|VqqdS)paramsN)is_cudar,Zis_floating_point)r&Zpgprrr s z Adam.__init__..zr9r:r;r<r=r3r4r5r6r7defaults __class__rrrws8z Adam.__init__cst||jD]L}|dd|dd|dd|dd|dd|ddqt|j}t|dkot |dd }|s|D]}t t |d |d <qdS) Nr=Fr4r3r5r6r7rstep) rE __setstate__rH setdefaultliststater)r*r,Z is_tensorrfloat)rrPgroupZ state_valuesZstep_is_tensorsrJrrrMs       zAdam.__setstate__r#cCs*|d}|dk r.t |}W5QRX|jD]}g}g}g}g}g} g} |d\} } d} d}|dr|dk r|} | j}t| } t|||d}|dD],}|jdk r| ||jj rt d| |j|j |}t |dkrp|jds|jdrtjd tj|jd ntd |d <tj|tjd |d<tj|tjd |d<|drptj|tjd |d<| |d| |d|dr| |d|dr|d jrt d| |d qt||||| | |d| | |d|d|d|d|d|d|d|d| |dq4|S)aNPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. grad_scaler (:class:`torch.cuda.amp.GradScaler`, optional): A GradScaler which is supplied from ``grad_scaler.step(optimizer)``. Nr:r7)r%r$rr>zJAdam does not support sparse gradients, please consider SparseAdam insteadrr5r )dtyperr8rL)Z memory_formatexp_avg exp_avg_sqr=Zmax_exp_avg_sqr6zB`requires_grad` is not supported for `step` in differentiable moder9r<r;r4r3) r=beta1beta2r9r<r;r4r3r5r6r7 grad_scale found_inf)Z _cuda_graph_capture_health_checkr,Z enable_gradrHZ_get_scale_asyncrrr/gradappendZ is_sparserFrPr*rIzerosrQrZ zeros_likeZpreserve_formatZ requires_gradr )rclosurer$ZlossrRZparams_with_gradgradsexp_avgs exp_avg_sqsmax_exp_avg_sqs state_stepsrXrYrZr[rr@rPrrrrLs          z Adam.step)r0r1r2rF)N) rr r!__doc__rboolrrMr rL __classcell__rrrJrr 2s(D" F)r>r`rarbrcrdr3r5r6r7rZr[r=rXrYr9r<r;r4c Cstdd|Dstd|dkr&d}|r.zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNFz6torch.jit.script not supported with foreach optimizers) r=rXrYr9r<r;r4r5r6rZr[)rGrFr,ZjitZ is_scripting_multi_tensor_adam _fused_adam_single_tensor_adam)r>r`rarbrcrdr3r5r6r7rZr[r=rXrYr9r<r;r4funcrrrr s:)r>r`rarbrcrdrZr[r=rXrYr9r<r;r4r5r6c CsJ|dkr|dkstt|D]&\}}|s2||n|| }||}||}||}|rl|jrd|jsltd|d7}| dkr|j|| d}t|rt|}t|}t|}t|}|| j|d| d|| j || d| d|s|r|}dt | |}dt | |}| |}| }| }|r|rR||}n||}||t|||| ||| |}n| ||| |}|||q|}d| |}d| |}| |}t |}|r tj|||||d|| || }n| || }|j||| dqdS)N@If capturable=True, params and state_steps must be CUDA tensors.r ralpha)value)out)r+ enumerater?addr, is_complex view_as_realZmul_Zadd_Zaddcmul_ZconjpownegsqrtcloneZcopy_maximumZaddcdiv_itemmath)r>r`rarbrcrdrZr[r=rXrYr9r<r;r4r5r6iparamr\rVrWZstep_trLbias_correction1bias_correction2 step_sizeZ step_size_negbias_correction2_sqrtZmax_exp_avg_sqs_idenomrrrrl?sV          rlc st|dkrdS|r4tddt||Ds4td|dkrD|dksHt|rZtt|}|rftddd|D}dd|D}d d|D}d d|D}t|d | dkrtj||| d t|tj||d d t|t |||d |r&fd d|D}fdd|D}t |d t |d t |t |t |}t |t |t|}|rt||t|}t|t||t || }t |t||}n@t|}t|t||t || }t |t||}t|||nfdd|D}fdd|D}fdd|D}dd|D}|rt||t|}t||t|| }n"t|}t||t|| }t||||dS)Nrcss|]\}}|jo|jVqdSr)r?)r&r@rLrrrrAsz%_multi_tensor_adam..rnz#_foreach ops don't support autogradcSs$g|]}t|rt|n|qSrr,rurvr&xrrrr(sz&_multi_tensor_adam..cSs$g|]}t|rt|n|qSrrrrrrr(scSs$g|]}t|rt|n|qSrrrrrrr(scSs$g|]}t|rt|n|qSrrrrrrr(sr rocsg|]}t|qSrr,rwr&rLrXrrr(scsg|]}t|qSrrrrYrrr(scsg|]}d|qSrTr|rrrrr(scsg|]}d|qSrTrrrrrr(scsg|]}|dqS)rr&Zbc)r9rrr(scSsg|]}t|qSr)r}ryrrrrr(s)r*rGzipr+r,Z _foreach_negtuple _foreach_add_Z _foreach_mul_Z_foreach_addcmul_ _foreach_sub_Z _foreach_neg_Z _foreach_divZ_foreach_reciprocal_Z _foreach_sqrtZ_foreach_maximum_Z _foreach_div_Z _foreach_mulZ _foreach_addZ_foreach_addcdiv_)r>r`rarbrcrdrZr[r=rXrYr9r<r;r4r5r6Zparams_rrrrZmax_exp_avg_sq_sqrtZeps_over_step_sizerZexp_avg_sq_sqrtr)rXrYr9rrjst                           rj)r>r`rarbrcrdrc Cstdd}tt||D]\}\}} t|j|jf} || d||| d|||| d|||| d|||r|| d|||| d| q|S) NcSsddtdDS)NcSsg|]}gqSrr)r&_rrrr(szG_group_params_by_device_and_dtype....)rangerrrrz3_group_params_by_device_and_dtype..rr )rrsrrrrUr]) r>r`rarbrcrdZper_device_and_dtype_tensorsr~r@rLkeyrrr!_group_params_by_device_and_dtypes r)r>r`rarbrcrdrZr[r=rXrYr9r<r;r4r5r6rc Cst||||||}|D]\}}|||f\}}}}}}|dk r\|dk r\||}||}nd}d}t|dtj|||||||| | | | | |||d|dk rt||gt|qdS)Nr ) r=r9rXrYr<r;r4rZr[)rrr,rZ _fused_adam_rr*)r>r`rarbrcrdrZr[r=rXrYr9r<r;r4r5r6Zgrouped_tensorsrrUZ device_paramsZ device_gradsZdevice_exp_avgsZdevice_exp_avg_sqsZdevice_max_exp_avg_sqsZdevice_state_stepsZdevice_grad_scaleZdevice_found_infrrrrk'sF     rk)NFFFNN) collectionsrr}typingrrrrrr,rr%r r __all__rZcudaampZ GradScalerrr/r rfrQr rlrjr-rrUrrkrrrrs    X > ` r