U KcF@slddlmZmZmZddlZddlmZddlmZddlZddl m Z m Z ddl Z dgZ GdddZdS) )DictAnyListN) defaultdict)nn) fqn_to_module module_to_fqnActivationSparsifierc@seZdZdZd,ejdddZeddZdd Z d-ejd d d Z d.e ejd ddZ ddZ ddZddZddZd/ddZddZd0ddZee efdd d!Zee efdd"d#d$Zee efdd%d&Zee efdd'd(d)Zd*d+ZdS)1r aL The Activation sparsifier class aims to sparsify/prune activations in a neural network. The idea is to attach the sparsifier to a layer (or layers) and it zeroes out the activations based on the mask_fn (or sparsification function) input by the user. The mask_fn is applied once all the inputs are aggregated and reduced i.e. mask = mask_fn(reduce_fn(aggregate_fn(activations))) Note:: The sparsification mask is computed on the input **before it goes through the attached layer**. Args: model (nn.Module): The model whose layers will be sparsified. The layers that needs to be sparsified should be added separately using the register_layer() function aggregate_fn (Optional, Callable): default aggregate_fn that is used if not specified while registering the layer. specifies how inputs should be aggregated over time. The aggregate_fn should usually take 2 torch tensors and return the aggregated tensor. Example def add_agg_fn(tensor1, tensor2): return tensor1 + tensor2 reduce_fn (Optional, Callable): default reduce_fn that is used if not specified while registering the layer. reduce_fn will be called on the aggregated tensor i.e. the tensor obtained after calling agg_fn() on all inputs. Example def mean_reduce_fn(agg_tensor): return agg_tensor.mean(dim=0) mask_fn (Optional, Callable): default mask_fn that is used to create the sparsification mask using the tensor obtained after calling the reduce_fn(). This is used by default if a custom one is passed in the register_layer(). Note that the mask_fn() definition should contain the sparse arguments that is passed in sparse_config arguments. features (Optional, list): default selected features to sparsify. If this is non-empty, then the mask_fn will be applied for each feature of the input. For example, mask = [mask_fn(reduce_fn(aggregated_fn(input[feature])) for feature in features] feature_dim (Optional, int): default dimension of input features. Again, features along this dim will be chosen for sparsification. sparse_config (Dict): Default configuration for the mask_fn. This config will be passed with the mask_fn() Example: >>> # xdoctest: +SKIP >>> model = SomeModel() >>> act_sparsifier = ActivationSparsifier(...) # init activation sparsifier >>> # Initialize aggregate_fn >>> def agg_fn(x, y): >>> return x + y >>> >>> # Initialize reduce_fn >>> def reduce_fn(x): >>> return torch.mean(x, dim=0) >>> >>> # Initialize mask_fn >>> def mask_fn(data): >>> return torch.eye(data.shape).to(data.device) >>> >>> >>> act_sparsifier.register_layer(model.some_layer, aggregate_fn=agg_fn, reduce_fn=reduce_fn, mask_fn=mask_fn) >>> >>> # start training process >>> for _ in [...]: >>> # epoch starts >>> # model.forward(), compute_loss() and model.backwards() >>> # epoch ends >>> act_sparsifier.step() >>> # end training process >>> sparsifier.squash_mask() N)modelcKsb||_t|_||jd<||jd<||jd<||jd<||jd<||jd<tt|_tt|_dS)N sparse_config aggregate_fn reduce_fnmask_fnfeatures feature_dim)r rdefaultsdict data_groupsstate)selfr r rrrrr ro/tmp/pip-unpacked-wheel-gikjz4vx/torch/ao/sparsity/_experimental/activation_sparsifier/activation_sparsifier.py__init__Vs       zActivationSparsifier.__init__cCsZ|d|d}}|dk r*|dk s*tddddg}|D]}||}t|s8tdq8dS) zXMakes sure that some of the functions and attributes are not passed incorrectly rrNz#need feature dim to select featuresr rrzfunction should be callable)AssertionErrorcallable)argsrrZfn_keyskeyfnrrr_safe_rail_checksis z&ActivationSparsifier._safe_rail_checkscsHjdjdjdddfdd }|S)zMReturns hook that computes aggregate of activations passing through. rrr Nreturnc s,|d}jd}dkrR|dkrFt|}t|jd<||}n|dkrddtdtD}ddtdtDjd<n|}ttD]r}t|g |j }t ||}|dkrt|}t|jd|<n||}||||<q|jd<dS)NrdatamaskcSsg|]}dqSrr.0_rrr szFActivationSparsifier._aggregate_hook..hook..cSsg|]}dqSr$rr%rrrr(s) rgettorchZ zeros_likeZ ones_likerrangelenTensorlongtodevice index_select) moduleinput input_datar"Zout_data feature_idxZfeature_tensor data_featureZ curr_dataZagg_fnrrnamerrrhooks(  $  z2ActivationSparsifier._aggregate_hook..hook)rrr8r9rr7r_aggregate_hookys z$ActivationSparsifier._aggregate_hook)layerc Kst|j|}|dk std||jkrsz6ActivationSparsifier.register_layer..r r#r9 aggregate hook_state)r r rrwarningswarnunregister_layercopydeepcopyrupdateitemsrregister_forward_pre_hookr;r) rr<r rrrrr r8Z local_argsZ update_dictZagg_hookrrrregister_layers*       z#ActivationSparsifier.register_layer)r8r<cCs|dk s|dk std|dkrH|dk s,tt|j|}|dk sHtd||jkrZtd|j|dd}|dkr|td|S)a Returns mask associated to the layer. The mask is - a torch tensor is features for that layer is None. - a list of torch tensors for each feature, otherwise Note:: The shape of the mask is unknown until model.forward() is applied. Hence, if get_mask() is called before model.forward(), an error will be raised. Nz0Need at least name or layer obj to retrieve maskz&layer not found in the specified modelz*Error: layer with the given name not foundr#zFError: shape unknown, call layer() routine at least once to infer mask)rr r r ValueErrorr))rr8r<r#rrrget_masks    zActivationSparsifier.get_maskcCs.|j|d|j||j|dS)z/Detaches the sparsifier from the layer r9N)rremoverpop)rr8rrrrDs z%ActivationSparsifier.unregister_layerc CsRt@|jD].\}}|d}|||||j|dqW5QRXdS)zCInternally calls the update_mask() function for each layer r"N)r*Zno_gradrrH update_maskrN)rr8configsr"rrrsteps  zActivationSparsifier.stepc Cs~||}|d}|d}|d}|d}|dkrJ||}||f||_n0tt|D]"} ||| } || f||| _qVdS)a( Called for each registered layer and does the following- 1. apply reduce_fn on the aggregated activations 2. use mask_fn to compute the sparsification mask Note: the reduce_fn and mask_fn is called for each feature, dim over the data r rrrN)rLr"r+r,) rr8r"rPr#r rrrr5r6rrrrOs  z ActivationSparsifier.update_maskcs:|||j|d|j|dfdd}|S)z[Returns hook that applies sparsification mask to input entering the attached layer rrcst|d}dkr|StdtD]D}t|g|j}t|||}|||q&|SdS)Nr) r+r,r*r-r.r/r0r1Z index_copy_)r2r3r4r5ZfeatureZ sparsifiedrrr#rrr9!sz1ActivationSparsifier._sparsify_hook..hook)rLrr:rrRr_sparsify_hooks   z#ActivationSparsifier._sparsify_hookTcKs`|jD]P\}}|d|dd|j|d<|rR|d|||d<d|d<q dS)z Unregisters aggreagate hook that was applied earlier and registers sparsification hooks if attach_sparsify_hook = True. r9NonerAr<sparsifyN)rrHrMrNrIrS)rZattach_sparsify_hookkwargsr8rPrrr squash_mask/s  z ActivationSparsifier.squash_maskcCs8t}|jD]"\}}dd|D}|||<q|S)a Exclude hook and layer from the config keys before serializing TODO: Might have to treat functions (reduce_fn, mask_fn etc) in a different manner while serializing. For time-being, functions are treated the same way as other attributes cSsi|]\}}|dkr||qS))r9r<r)r&rvaluerrr EszFActivationSparsifier._get_serializable_data_groups..)rrrH)rrr8configZ new_configrrr_get_serializable_data_groups=s  z2ActivationSparsifier._get_serializable_data_groupscCst|}|D]\}}|ddk rt|dtrtt|dD]:}|rf|d||d|<qD|d||d|<qDq|r|d|d<q|d|d<q|S)zConverts the mask to sparse coo or dense depending on the `sparse_coo` argument. If `sparse_coo=True`, then the mask is stored as sparse coo else dense tensor r#N) rErFrH isinstancerr+r,Z to_sparse_cooZto_dense)rZ states_dict sparse_cooZstatesr'ridxrrr _convert_maskIs  z"ActivationSparsifier._convert_maskr cCs"|}||j}|||jdS)a0Returns the state of the sparsifier as a :class:`dict`. It contains: * state - contains name -> mask mapping. * data_groups - a dictionary containing all config information for each layer * defaults - the default config while creating the constructor rrrr[r_rrrrrrrr state_dict]s  zActivationSparsifier.state_dict)rcr!cCs0|d}|d|d}}||||ddS)zThe load_state_dict() restores the state of the sparsifier based on the state_dict Args: * state_dict - the dictionary that to which the current sparsifier needs to be restored to rrrr`N) __set_state__)rrcrrrrrrload_state_dictnsz$ActivationSparsifier.load_state_dictcCs"|}||j}|j||dS)N)rrrrarbrrr __get_state__ys  z"ActivationSparsifier.__get_state__)rr!cCs|j|ddd|d<|j||jD]z\}}t|j|}|dk sLtd|krr|ddkrr|| |}n$d|kr|ddkr|| |}||d<||d<q,dS) NrF)r]rAr@rUr<r9) r___dict__rGrrHrr rrIr;rS)rrr8rZr<r9rrrrds   z"ActivationSparsifier.__set_state__cCs|jjd}|jD]^\}}|d7}|d7}|d|d7}t|D](}|dkrXqJ|d|d||d7}qJq|d7}|S) Nz ( z Data Group z name: )r"r9rrr z z: )) __class____name__rrHsortedkeys)r format_stringr8rZrrrr__repr__s zActivationSparsifier.__repr__)NNNNN)NNNNN)NN)T)T)rk __module__ __qualname____doc__rModuler staticmethodrr;rJstrrLrDrQrOrSrWr[r_rrrcrerfrdrorrrrr s4I ).      )typingrrrr* collectionsrrrEZsparsifier.utilsrr rB__all__r rrrrs