U <c@sddlZddlZddlmZddlmZmZeejeejfZdddgZ dee e e ejd d dZ dee e e ejd d dZ ee dd d dZdS)N)inf)UnionIterableclip_grad_norm_clip_grad_normclip_grad_value_@F) parametersmax_norm norm_typeerror_if_nonfinitereturnc st|tjr|g}dd|D}t|}tt|dkrFtdS|djtkrfdd|D}t|dkr~|dntt |}n"t t fdd|D}|rt | | rtdd ||d }tj|d d }|D]} | || jq|S) aAClips gradient norm of an iterable of parameters. The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float or int): max norm of the gradients norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False (will switch to True in the future) Returns: Total norm of the parameter gradients (viewed as a single vector). cSsg|]}|jdk r|jqSNgrad).0pr!s z#clip_grad_norm_..rgcs"g|]}|qSr)detachabsmaxtorg)devicerrr(scs"g|]}t|qSr)torchnormrrrrr rrr+szThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`gư>g?)r) isinstancerTensorfloatlenZtensorrrrstackr logical_orisnanisinf RuntimeErrorclamprZmul_r) r r r r ZgradsZnormsZ total_normZ clip_coefZclip_coef_clampedrrr rr s*    &"  cCstjdddt||||S)zClips gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. z[torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.) stacklevel)warningswarnr)r r r r rrrr<s )r clip_valuer cCsFt|tjr|g}t|}tdd|D]}|jjj| |dq(dS)aClips gradient of an iterable of parameters at specified value. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized clip_value (float or int): maximum allowed value of the gradients. The gradients are clipped in the range :math:`\left[\text{-clip\_value}, \text{clip\_value}\right]` cSs |jdk Srr)rrrrYz"clip_grad_value_..)minrN)r!rr"r#filterrdataZclamp_)r r/rrrrrJs )rF)rF)r-rZ torch._sixrtypingrrr"Z_tensor_or_tensors__all__r#boolrrrrrrrs0   3