U <c@srddlmZddlZddlmZddZddZd d Zd d Zd dZ ddfddZ ej e ej dddZ dS))OptionalN)ExpandedWeightcCsL|t|t|d}|dt|t|}ddt||D}||fS)zMost `__torch_function__`s standardize the kwargs that they give, so this will separate the args and kwargs they pass. Functions that don't are linear and convND NcSsi|]\}}||qSr).0namevaluerr[/tmp/pip-unpacked-wheel-gikjz4vx/torch/nn/utils/_expanded_weights/expanded_weights_utils.py sz#standard_kwargs..)lenzip)Z kwarg_names expanded_argsZ kwarg_valuesZexpanded_args_without_kwargsexpanded_kwargsrrr standard_kwargssrcCst|||\}}|||S)aForward helper computes the forward pass for a function that has expanded weight(s) passed to it. It will run the forward pass where all ExpandedWeights are their original weight. It runs checks on the given arguments and detaches the outputs. .. note:: First argument in :attr:`expanded_args` must be the input with the batch dimension as the first element of the shape .. note:: :attr:`func` must return a Tensor or tuple of Tensors Args: func: The function to be called ctx: The context from the autograd.Function object. Will be used to save computed state from the forward pass expanded_args: Arguments to be passed to :attr:`func`. Will include arguments that need to be unpacked because they are ExpandedWeights num_true_outs: The number of outputs seen by the user since some functions return auxillary data that is only used in the backward pass )_check_and_unexpand_args)funcr runexpanded_argsunexpanded_kwargsrrr forward_helpersrc Csd|d}t|tr.tdt|jd|jt|tjsVtdt|jd|jt|jdkrttd|j|jddkrtd|d|j|jd}|t | D].}t|tr|j |krtd|d|j qd}|t | D]@}t|tr|dkr|j }q||j krtd |d |j qt d d |D}d d| D}||fS)NrzbExpanded Weights do not support inputs that are also ExpandedWeights. Input must be a Tensor, got z in function zVExpanded Weights requires a Tensor as the first input to get the batch dimension, got zSExpanded Weights requires a batch dimension but got an input of size 0 in function zI0 is not a valid batch size for Expanded Weights but got input tensor of zWExpected ExpandedWeights to have batch size matching input but got input batch size of z# with ExpandedWeight of batch size zVExpected ExpandedWeights to all have the same loss_reduction argument but got onewith z and one with css"|]}t|tr|jn|VqdSN isinstancer orig_weight)rargrrr Bsz+_check_and_unexpand_args..cSs&i|]\}}|t|tr|jn|qSrr)rrrrrr r Csz,_check_and_unexpand_args..)rr RuntimeErrortype__name__torchTensorr shapetuplevalues batch_sizeloss_reductionitems) rr rinputr#rr$rrrrr r%s2      rcCs|jdkr||jS|SdS)NZmean)r$r#) grad_sampleZexpanded_weightrrr maybe_scale_by_batch_sizeGs  r(cCsLt|}t|trHt|||}t|drB|jdk rB|j||_n||_dS)Nr') unpack_expanded_weight_or_tensorrrr(hasattrr')maybe_expanded_weightZper_sample_grad_fnunpackedZgrad_sample_contributionrrr set_grad_sample_if_existsMs  r-cCs|Srr)xrrr Vr/cCsJt|tr|j}||St|tjr2|js2||St|tjrFtddS)NzExpandedWeights currently does not support a mixture of ExpandedWeight parameters and normal Parameters. Please file and issue with pytorch/pytorch)rrrrrZ requires_gradr)r+rrrrr r)Vs  r))tensorn_dimsreturncCs:||dkr|Sttd||}|j|dSdS)aF Calculates the sum over all dimensions, except the first (batch dimension), and excluding the last n_dims. This function will ignore the first dimension and it will not aggregate over the last n_dims dimensions. Args: tensor: An input tensor of shape ``(B, ..., X[n_dims-1])``. n_dims: Number of dimensions to keep. Example: >>> tensor = torch.ones(1, 2, 3, 4, 5) >>> sum_over_all_but_batch_and_last_n(tensor, n_dims=2).shape torch.Size([1, 4, 5]) Returns: A tensor of shape ``(B, ..., X[n_dims-1])`` r)dimN)r4listrangesum)r1r2Zdimsrrr !sum_over_all_but_batch_and_last_nbsr8)typingrrZexpanded_weights_implrrrrr(r-r)rintr8rrrr s   "