U Kc;)@sddlZGdddejjZGdddejjZGdddejjZGdd d ejjZGd d d ejjZGd d d ejjZGdddej jj j Z Gdddejj Z dS)Ncs>eZdZdZd fdd ZddZddZed d ZZ S) ReLU6aApplies the element-wise function: :math:`\text{ReLU6}(x) = \min(\max(x_0, x), q(6))`, where :math:`x_0` is the zero_point, and :math:`q(6)` is the quantized representation of number 6. Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input .. image:: ../scripts/activation_images/ReLU6.png Examples:: >>> m = nn.quantized.ReLU6() >>> input = torch.randn(2) >>> # xdoctest: +SKIP >>> input = torch.quantize_per_tensor(input, 1.0, 0, dtype=torch.qint32) >>> output = m(input) Fcstt||||_dSN)superr__init__inplace)selfr __class__L/tmp/pip-unpacked-wheel-gikjz4vx/torch/ao/nn/quantized/modules/activation.pyrszReLU6.__init__cCstjj||jSr)torchops quantizedZrelu6rrinputr r r forwardsz ReLU6.forwardcCsdS)NZQuantizedReLU6r rr r r _get_name"szReLU6._get_namecCs t|jSr)rr)modr r r from_float%szReLU6.from_float)F) __name__ __module__ __qualname____doc__rrr staticmethodr __classcell__r r rr rs rcsHeZdZdZfddZddZddZedd Ze d d Z Z S) HardswishzThis is the quantized version of :class:`~torch.nn.Hardswish`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor cstt|||_||_dSr)rrrscale zero_point)rrrrr r r0szHardswish.__init__cCstjjjjj||j|jdS)N)rr)r aonnr functionalZ hardswishrrrr r r r5s  zHardswish.forwardcCsdS)NZQuantizedHardswishr rr r r r9szHardswish._get_namecCs |j\}}tt|t|Sr)activation_post_processcalculate_qparamsrfloatintrrrr r r r<szHardswish.from_floatcCs|t|t|Sr)r$r%clsrrrr r r from_referenceAszHardswish.from_reference rrrrrrrrr classmethodr)rr r rr r)s  rcsJeZdZdZd fdd ZddZddZed d Ze d d Z Z S)ELUzThis is the quantized equivalent of :class:`~torch.nn.ELU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor alpha: the alpha constant ?cs tt||||_||_dSr)rr,rrr)rrralpharr r rMsz ELU.__init__cCstjjjj||j|j|jSr) r rr rr!Zelurrr.rr r r rRs  z ELU.forwardcCsdS)NZ QuantizedELUr rr r r rVsz ELU._get_namecCs$|j\}}tt|t||jSr)r"r#r,r$r%r.r&r r r rYszELU.from_floatcCs|t|t||jSr)r$r%r.r'r r r r)^szELU.from_reference)r-r*r r rr r,Es r,csXeZdZdZdeeeeddfdd Zdd Zd d Z e d d Z e ddZ Z S) LeakyReLUaThis is the quantized equivalent of :class:`~torch.nn.LeakyReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor negative_slope: Controls the angle of the negative slope. Default: 1e-2 {Gz?FN)rrnegative_sloperreturncsH||d}t|||dtj|f||dtj|f|dS)N)devicedtyperr)rrZregister_bufferr Ztensor)rrrr1rr3r4Zfactory_kwargsrr r rjs zLeakyReLU.__init__cCstjj||j|j|j|jSr)r r rZ leaky_relur1rrrrr r r rqszLeakyReLU.forwardcCsdS)NZQuantizedLeakyReLUr rr r r ruszLeakyReLU._get_namecCs(|j\}}|t|t||j|jSr)r"r#r$r%r1rr'r r r rxszLeakyReLU.from_floatcCs|t|t||j|jSr)r$r%r1rr'r r r r)}szLeakyReLU.from_reference)r0FNN)rrrrr$r%boolrrrr+rr)rr r rr r/bs r/cs<eZdZdZeedfdd ZddZeddZ Z S) SigmoidzThis is the quantized equivalent of :class:`~torch.nn.Sigmoid`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor ) output_scaleoutput_zero_pointcst||_||_dSr)rrr7r8)rr7r8rr r rs zSigmoid.__init__cCstjj||j|jSr)r r rZsigmoidr7r8rr r r rszSigmoid.forwardcCs |j\}}|t|t|Sr)r"r#r$r%)r(rr7r8r r r rszSigmoid.from_float) rrrrr$r%rrr+rrr r rr r6s r6csJeZdZdZdfdd ZddZd d Zed d Ze d dZ Z S)Softmaxa,This is the quantized version of :class:`~torch.nn.Softmax`. Args: dim: A dimension along which Softmax will be computed (so every slice along dim will sum to 1). scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor Nr-rcs t||_||_||_dSr)rrdimrr)rr:rrrr r rs zSoftmax.__init__cCs@|j}|dkr(d}tjjd||}tjj|||j|j S)Nsoftmax) r:r r r!Z_get_softmax_dimr rr<rr)rrr: stacklevelr r r rszSoftmax.forwardcCsdS)NZQuantizedSoftmaxr rr r r rszSoftmax._get_namecCs$|j\}}t|jt|t|Sr)r"r#r9r:r$r%r&r r r rszSoftmax.from_floatcCs||jt|t|Sr)r:r$r%r'r r r r)szSoftmax.from_reference)Nr-rr*r r rr r9s  r9c@s8eZdZejjjjZddZ e ddZ e ddZ dS)MultiheadAttentioncCsdS)NZQuantizedMultiheadAttentionr rr r r rszMultiheadAttention._get_namecCs tddS)NzpIt looks like you are trying to convert a non-observed MHA module. Please, see the examples on quantizable MHAs.)NotImplementedError)r(otherr r r rszMultiheadAttention.from_floatcCstjjj|ddddd}||_|jdk rd|jd}tj|dd\}}t |||tj }t |d||j dk r|jd}tj|dd\}}t |||tj }t |d||S)NFT)mappingrZremove_qconfigZconvert_custom_config_dictbias_k)Z reduce_rangebias_v) r rZ quantizationconvertr rB _parameterspopZ_choose_qparams_per_tensorquantize_per_tensorquint8setattrrC)r(r@Z convertedrBZscZzprCr r r from_observeds*         z MultiheadAttention.from_observedN) rrrr rr quantizabler>Z _FLOAT_MODULErr+rrJr r r r r>s   r>csteZdZdZdeeeddfdd Zejdddd Z ejejd d d Z d dZ e ddZ e ddZZS)PReLUa%This is the quantized equivalent of :class:`~torch.nn.PReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor num_parameters: number of parameters: 1, or the number of channels at input. Default: 1 N)r7r8num_parametersr2csNt||_||_||_tj|tjd}tj|ddtj d}| |dS)N)r4r-r)rrr4) rrrNrrr Zrandnr$rGrH set_weight)rr7r8rNwZqwrr r rs zPReLU.__init__)rPr2cCs ||_dSr)weight)rrPr r r rOszPReLU.set_weight)rr2cCstjj||j|j|jSr)r r rZprelurQrrrr r r rsz PReLU.forwardcCsdS)NZQuantizedPReLUr rr r r rszPReLU._get_namec Csl|j\}}|t|t||j}|j}|j}|\}}t|t|t|tj } | | |Sr) r"r#r$r%rNrQqconfigr rGrHrO r(rrrZqpreluZfloat_wtZobserverZwt_scaleZwt_zpZqweightr r r rs    zPReLU.from_floatc Cs^|t|t||j}|j}|j}|\}}t|t|t|tj} | | |Sr) r$r%rNrQrRr#r rGrHrOrSr r r r) s    zPReLU.from_reference)rM)rrrrr$r%rr ZTensorrOrrr+rr)rr r rr rLs  rL)r r ZReLUrrr,r/r6r9rrKr>ModulerLr r r r s&&*