U Kc"@slddlmZddlmZmZmZmZddlZddlm m Z ddl m Z dgZddZGd dde ZdS) )reduce)CallableOptionalTupleUnionN)BaseSparsifierWeightNormSparsifiercCs ||d}||d}||fS)Nr)idxshapeZrowscolsr r W/tmp/pip-unpacked-wheel-gikjz4vx/torch/ao/sparsity/sparsifier/weight_norm_sparsifier.py_flat_idx_to_2d s  rcsleZdZdZdeeeefeeeee efdfdd Z ddd Z dd d Z dd d Z ddZZS)r aWeight-Norm Sparsifier This sparsifier computes the norm of every sparse block and "zeroes-out" the ones with the lowest norm. The level of sparsity defines how many of the blocks is removed. This sparsifier is controlled by three variables: 1. `sparsity_level` defines the number of *sparse blocks* that are zeroed-out 2. `sparse_block_shape` defines the shape of the sparse blocks. Note that the sparse blocks originate at the zero-index of the tensor. 3. `zeros_per_block` is the number of zeros that we are expecting in each sparse block. By default we assume that all elements within a block are zeroed-out. However, setting this variable sets the target number of zeros per block. The zeros within each block are chosen as the *smallest absolute values*. Args: sparsity_level: The target level of sparsity sparse_block_shape: The shape of a sparse block (see note below) zeros_per_block: Number of zeros in a sparse block norm: Norm to use. Could be either `int` or a callable. If `int`, only L1 and L2 are implemented. Note:: The `sparse_block_shape` is tuple representing (block_ROWS, block_COLS), irrespective of what the rows / cols mean in the data tensor. That means, if you were to sparsify a weight tensor in the nn.Linear, which has a weight shape `(Cout, Cin)`, the `block_ROWS` would refer to the output channels, while the `block_COLS` would refer to the input channels. Note:: All arguments to the WeightNormSparsifier constructor are "default" arguments and could be overriden by the configuration provided in the `prepare` step. ?rN)sparsity_levelsparse_block_shapezeros_per_blocknormcs|dkrtdd|}|||d}|dkr.d}t|r>||_n8|dkrRdd|_n$|dkrfdd|_ntd|d tj|d dS) NcSs||SNr xyr r r;z/WeightNormSparsifier.__init__..)rrrrcSs|Sr)absTr r rrFrcSs||Srr rr r rrHrzL-z is not yet implemented.)defaults)rcallablenorm_fnNotImplementedErrorsuper__init__)selfrrrrr! __class__r rr&5s   zWeightNormSparsifier.__init__cCsJ|dkr"|dk sttj||d}|j||ddtj||||d|_|S)zCCreates patches of size `block_shape` after scattering the indices.Ndevicer)dimindexvalue)Z output_size kernel_sizestride)AssertionErrortorchonesZscatter_Ffolddata)r' output_shaper,indices block_shapemask input_shaper+r r r_scatter_fold_block_maskMs  z-WeightNormSparsifier._scatter_fold_block_maskcCsh|jdd\}}|\}} ||||} | || | } |dkrTtj|||jd}|dkrlt||_|S|dkrt||_|Stdd|} | dkrtj |ddddf||d d }| }t |} | d| d}t t|| }td t| d|}tj||d d d\}}||j}|jd || || f|||d||jd|d|f|_|S)aCreates a tensor-level mask. Tensor-level mask is described as a mask, where the granularity of sparsification of the smallest patch is the sparse_block_shape. That means, that for a given mask and a sparse_block_shape, the smallest "patch" of zeros/ones could be the sparse_block_shape. In this context, `sparsity_level` describes the fraction of sparse patches. Nr*?gcSs||Srr rr r rrorz8WeightNormSparsifier._make_tensor_mask..rT)r/r0Z ceil_moderrFkr,largest)r,r7r8r9r:)r r2r3r+ zeros_liker6 ones_likerr4Z avg_pool2dflattenlenrepeatintroundmaxmintopkreshaper<squeeze contiguous)r'r6r;rrr:hwblock_hblock_wdhdwvalues_per_blockZ num_blocksZ threshold_idx_ sorted_idx mask_reshaper r r_make_tensor_maskWsH    &z&WeightNormSparsifier._make_tensor_maskcCs8|dkrtj|j|jd}|jdd\}}|\}}||||} ||||} tdd|} | |krzt||_|Stj|| || |j|jd} | tj || d|d|f<t j | ddddf||d} | | j}tj | |dd d \}}|jd|| j||d | |jd|d|f|_|S) aCreates a block-level mask. Block-level mask is described as a mask, where the granularity of sparsification of the largest patch is the sparse_block_shape. That means that for a given mask and a sparse_block_shape, the sparsity is computed only within a patch of a size sparse_block_shape. In this context the `zeros_per_block` describes the number of zeroed-out elements within a patch. Nr*r=cSs||Srr rr r rrrz7WeightNormSparsifier._make_block_mask..)dtyper+)r/r0rFr?)r,r8r7r9r:)r2r3r r+rrBr6rZZfill_nanr4ZunfoldrLrKr<rMrN)r'r6rrr:rOrPrQrRrSrTrUZ padded_dataZ unfolded_datarXrVrWr r r_make_block_masks2    &z%WeightNormSparsifier._make_block_maskc Kstdd|}||krtd|dkr.tdt|j|dj}|dksP|dkr^t||_nl|dkr|||kr|t||_nN| t||} |j | | j ||d} ||kr|j | ||d} t | | } | |_dS) NcSs||Srr rr r rrrz2WeightNormSparsifier.update_mask..zYNumber of zeros per block cannot be more than the total number of elements in that block.rz-Number of zeros per block should be positive.r>)r6r;rr)r6rr)r ValueErrorgetattrZparametrizationsr:r2rCr6rBr#rYr r\ logical_or) r'moduleZ tensor_namerrrkwargsrUr:ZwwZ tensor_maskZ block_maskr r r update_masks2 z WeightNormSparsifier.update_mask)rrNN)NNN)N)N)__name__ __module__ __qualname____doc__floatrrGrrrr&r<rYr\rb __classcell__r r r(rr s$%  0 ') functoolsrtypingrrrrr2Ztorch.nn.functionalnnZ functionalr4Zbase_sparsifierr__all__rr r r r rs