U JcV@sddlZddlmZmZmZmZddlZddlmZddl m Z ddl m Z m Z dZdZdgZejeeefeeeeefd d d Zeeefeedd d dZGdddejZdS)N)AnyDict GeneratorList)_replace_by_prefix)FlatParamHandle HandleConfig flat_param _fpw_moduleFlattenParamsWrapper)module state_dictprefixargsreturncGst||td||S)z _post_state_dict_hook() is called after the state_dict() is executed and before returning the state_dict to the users. This API post-processes the keys of the state_dict to remove the FlattenParamsWrapper internal prefix. .)r FPW_MODULE)r rrrrQ/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/fsdp/flatten_params_wrapper.py_post_state_dict_hooks r)rrrrcGs~t|||td|tdt}t|D]D}||r4|dd}|tshtd|t||||q4dS)z _pre_load_state_dict_hook() is called before the _load_from_state_dict() is executed. This API pre-processes the keys of the state_dict to add the FlattenParamsWrapper internal prefix. rz4Expected key to contain flat_param, but key name is N)rr FLAT_PARAMlistkeys startswithsplitAssertionError)rrrZflat_param_keykZ last_partrrr_pre_load_state_dict_hook&s  rcseZdZdZejeejej e ddfdd Z e e dddZe edd d Ze edd d Zejedd dZeedfdd ZeedddZeeedddZZS)r an This is a wrapper for flattening parameters in a ``nn.Module`` 's subtree into a single flattened parameter and is based on [1]. This is used for :class:`FullyShardedDataParallel` 's recursive wrapping. [1] https://github.com/SsnL/PyTorch-Reparam-Module Args: module (nn.Module): Module to wrap. params (List[nn.Parameter]): Parameters in ``module`` 's subtree to flatten into a single flattened parameter. device (torch.device): The compute and communication device for this wrapper's handle. config (HandleConfig): A config customizing this wrapper's handle based on FSDP's available features. Attributes: flat_param (Optional[FlatParameter]): The flattened parameter. ``flat_param`` is ``None`` either when (1) this wrapper manages no parameters or (2) the wrapped module's parameters are unflattened. _fpw_module (nn.Module): The wrapped module. _flat_param_handle (FlatParamHandle): A handle for the flattened parameter; only present if this wrapper manages parameters. N)r paramsdeviceconfigrcst||_d|_|t|tt|dkr:dSt |||||_ |j j|_t |t |jksht t |t|jks|t dS)Nr)super__init__r r Z_register_state_dict_hookrZ"_register_load_state_dict_pre_hookrlenr_flat_param_handlegetattrrrr)selfr r r!r" __class__rrr$Vs     zFlattenParamsWrapper.__init__)rcCs t|dS)z4Returns whether this wrapper manages any parameters.r&)hasattrr(rrr has_paramsnszFlattenParamsWrapper.has_paramscCst|dstd|jS)Nr&zTAccessing the handle of a `FlattenParamsWrapper` that does not manage any parameters)r+rr&r,rrrhandless zFlattenParamsWrapper.handlecCs|jS)z&Returns the wrapped module (like DDP).)r r,rrrr {szFlattenParamsWrapper.modulec csNt|dddkrdVn2|`z |j dVW5QRXW5|jj|_XdS)a Assumes that the flattened parameter is unsharded. When in the context, unflattens the original parameters as ``nn.Parameter`` views into the flattened parameter and de-registers the flattened parameter. After the context, restores the original parameters as ``Tensor`` views into the flattened parameter and re-registers the flattened parameter. r N)r'r r&unflatten_as_paramsr,rrrr/s  z(FlattenParamsWrapper.unflatten_as_params)namercs4zt|WStk r.t|j|YSXdS)zAForward missing attributes of this wrapper to the wrapped module.N)r# __getattr__AttributeErrorr'r )r(r0r)rrr1sz FlattenParamsWrapper.__getattr__)keyrcCs |j|S)zhForward indexing calls to the wrapped module in case the wrapped module is an ``nn.Sequential``.)r __getitem__)r(r3rrrr4sz FlattenParamsWrapper.__getitem__)inputskwinputsrcOs$|jdk r|jjdd|j||S)NF)Z as_params)r r&Z _unflattenr )r(r5r6rrrforwards zFlattenParamsWrapper.forward)__name__ __module__ __qualname____doc__nnModuler Parametertorchr!r r$propertyboolr-rr.rr contextlibcontextmanagerrr/strr1intr4r7 __classcell__rrr)rr =s$)rBtypingrrrrr?Ztorch.nnr<Ztorch.distributed.utilsrr rr rr__all__r=rDrrr rrrr s(