U Jc*@slddlZddlmZddlmZmZddlmZmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZmZddlZddlmZddlmZddlmmZddlmZddl m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'd d d d ddddgZ(Gdd d eZ)GdddeZ*Gdd d eZ+GdddeZ,GdddeZ-eGdddZ.Gdd d ej/Z0Gdd d Z1dS)N) dataclass)autoEnum) accumulatechain) AnycastDict GeneratorIteratorList NamedTupleOptionalSequenceSetTupleUnion)Tensor)_ext_post_unflatten_transform_ext_pre_flatten_transform)_alloc_storage _free_storage_set_fsdp_flattenedp_assert FlatParameterFlatParamHandleFlatParamShardMetadata ParamInfoSharedParamInfo HandleConfigHandleShardingStrategyHandleTrainingStatec@s,eZdZUdZeed<ejed<eed<dS)rz-Information for an original module parameter. param_namemodule module_nameN__name__ __module__ __qualname____doc__str__annotations__nnModuler/r/E/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/fsdp/flat_param.pyr)s  c@sFeZdZUdZeed<ejed<eed<eed<ejed<eed<dS) rai Additional information for a shared parameter. For each shared parameter, we designate one module and its parameter variable to be the primary owner, determined as the first one encountered in the parameter walk. These are prefixed with "prim". The primary module and parameter do not have their own :class:`SharedParamInfo` instance. r#r$r%prim_param_name prim_moduleprim_module_nameNr&r/r/r/r0r1s    c@s\eZdZUdZeedfed<eejdfed<ee dfed<eee e fdfed<dS)ra This holds metadata specific to this rank's shard of the flattened parameter. Attributes: param_names (Tuple[str, ...]): Prefixed parameter names of this rank's shard of the parameters; see :class:`FlatParameter`. param_shapes (Tuple[torch.Size, ...]): Parameter shapes of this rank's shard of the parameters; see :class:`FlatParameter`. param_numels (Tuple[int, ...]): Parameter numels of this rank's shard of the parameters; see :class:`FlatParameter`. param_offsets (Tuple[Tuple[int, int], ...]): [start, end] offsets (in units of numels) giving this rank's part of each flattened original module parameter. .Z param_namesZ param_shapesZ param_numels param_offsetsN) r'r(r)r*rr+r,torchSizeintr/r/r/r0rCs c@seZdZeZeZeZdS)r!N)r'r(r)rZ FULL_SHARDZ SHARD_GRAD_OPNO_SHARDr/r/r/r0r!]sc@s*eZdZeZeZeZeZeZdS)r"N) r'r(r)rIDLEFORWARD BACKWARD_PREZ BACKWARD_POSTSUMMON_FULL_PARAMSr/r/r/r0r"cs c@sJeZdZUeed<eed<eejed<eejed<dZ eeed<dS)r sharding_strategyoffload_params param_dtypeZ reduce_dtypeFkeep_low_precision_gradsN) r'r(r)r!r,boolrr5dtyper@r/r/r/r0r ks c@sDeZdZdZeeeeeejee ee ee ddddZ dS)ra This is the flattened parameter used by :class:`FullyShardedDataParallel`. It is comprised of one or more original parameters, which are flattened and concatenated to construct the flattened parameter. Under the current design, this parameter logically represents both the unsharded and sharded flattened parameter, and its data changes storages dynamically. - In the :class:`FullyShardedDataParallel` constructor, the parameter is initialized as unsharded and then sharded in-place. - At runtime, the parameter is lazily (re)-initialized. The sharded parameter data is saved in ``self._local_shard``, and a new ``Tensor`` ``self._full_param_padded`` is created, which is the all-gather destination and owns the unsharded parameter storage thereafter. (See :meth:`FullyShardedDataParallel._init_param_attributes`.) - Throughout runtime, the parameter data changes storages as needed, e.g. to the sharded flattened parameter, reduced-precision sharded flattened parameter, or the unsharded flattened parameter. Attributes: _unpadded_unsharded_size (torch.Size): Unsharded flattened parameter's size without padding. _padded_unsharded_size (torch.Size): Unsharded flattened parameter's size with padding. This is only set for sharded strategies since they require padding for the all-gather. _param_infos (Tuple[ParamInfo, ...]): Each parameter's parameter info entry; see :class:`ParamInfo`. _numels (Tuple[int, ...]): Each parameter's numel. _shapes (Tuple[torch.Size, ...]): Each parameter's shape. _prefixed_param_names (Tuple[str, ...]): Each parameter's name prefixed with the parent module names starting from the module passed to construct this flattened parameter via :class:`FlatParamHandle`; the prefixed names are guaranteed to be unique within the subtree rooted in that module. _num_params (int): Number of original parameters flattened into this flattened parameter; this is the length of ``_param_infos``, ``_numels``, ``_shapes``, and ``_prefixed_param_names``. _shared_param_infos (Tuple[SharedParamInfo, ...]): Shared parameter info entries; see :class:`SharedParamInfo`. _param_extensions (Tuple[Optional[Any], ...]): Parameter extensions (i.e. some per-parameter state) used to customize pre-flatten and post-unflatten behavior. This is experimental, and users should not depend on its existence in the future. _shard_param_offsets (List[Tuple[int, int])): [start, end] offsets (in units of numel) giving this rank's part of each flattened original module parameter; for any parameter ``p`` that is not sharded across ranks, this will be [0, ``p.numel()``-1]. _shard_indices (Tuple[int, int]): [start, end] indices (in units of parameters) for this rank's shard of the original model parameters, where the parameters follow the order in which they were originally flattened; this indexes appropriately into any data structure that follows the flattening order (e.g. ``_param_infos``, ``_numels``, etc.). _shard_numel_padded (int): Numel padded for this rank's sharded flattened parameter. _local_shard (Tensor): Sharded flattened parameter with padding if using a sharded strategy. If using ``NO_SHARD``, then this is the unpadded unsharded flattened parameter, and there is no notion of a sharded flattened parameter or padded unsharded flattened parameter. _full_param_padded (Tensor): Unsharded flattened parameter with padding. This is not defined for ``NO_SHARD``. When using mixed precision for parameters, this has the low precision. _full_prec_full_param_padded (Tensor): Full precision unsharded flattened parameter with padding. This is used for unsharding outside of computation when using mixed precision for parameters. This is never defined for ``NO_SHARD``. _post_backward_hook_state (Tuple[AccumulateGrad, RemovableHandle]): Flattened parameter's :class:`AccumulateGrad` object and post-backward hook handle. _mp_shard (Tensor): Low precision sharded flattened parameter with padding. This is only defined when parameter mixed precision is enabled. For ``NO_SHARD``, this is used for computation. _cpu_grad (Tensor): Sharded gradient with padding stored on CPU. This is only defined when offloading parameters is enabled. _saved_grad_shard (Tensor): Sharded gradient with padding from previous iterations for gradient accumulation without :meth:`no_sync`. N) param_infosnumelsshapesprefixed_param_namesshared_param_infosparam_extensionsreturncCst|t|kstt|t|ks(tt|t|ksd?Z2d@dAZ3dddBdCZ4edcee eje5edDdEdFZ6eddGdHdIZ7e.j/e8ddJdKZ9dLdMZ:e;ejddNdOZdRdSZ?edTdUdVZ@eedTdWdXZAeedTdYdZZBd[d\ZCeDedd]d^ZEeDedd_d`ZFeDeddadbZGZHS)dra This handle manages a flattened parameter (:class:`FlatParameter`). This includes sharding and view management. Args: params (Sequence[nn.Parameter]): The parameters to use for the flattened parameter. module (nn.Module): A module that is the root of the subtree containing all parameters in ``params``; for non-recursive wrapping, this must be the top-level module, while for recursive wrapping, this may not necessarily be the top-level module. device (torch.device): The compute and communication device, which should be a non-CPU device. We refer to it as the compute device. config (HandleConfig): A config customizing the handle based on FSDP's available features. N)paramsr$deviceconfigrIcs:t||_||_tj|_||||jdddS)NF as_params) super__init__rX_configr"r9_training_state_init_flat_param _unflatten)rUrWr$rXrY __class__r/r0r]s   zFlatParamHandle.__init__)rWr$rIc Cst|}|dt|dks&tdg}g}g}g}g}i} g} g} d} d} |D]V\}}|jddD]>\}}||krql|| kr| |\}}}|t||||||qlt|t krt d| dk r|j | krt d| d|j | dkr| st d | dk r&|j | kr&t d t|\}}| ||j } |j } |||f| |<| ||t|||||||j|r|d |n|}||qlqV| dk stt| | |_|j|||||| dS) a Initializes the flattened parameter ``self.flat_param`` by flattening the parameters in ``params`` into a single :class:`FlatParameter` and saves relevant metadata. Shared parameters are only included in the flattened parameter once. This checks that all comprising parameters have the same dtype and ``requires_grad`` and does not support nested construction of :class:`FlatParameter` s. Args: See the Args in the class docstring. Nrz@Cannot initialize a `FlatParameter` from an empty parameter listF)Zrecursez(`FlatParameter` does not support nestingz/`FlatParameter` requires uniform dtype but got z and z"Integer parameters are unsupportedz0`FlatParameter` requires uniform `requires_grad`.)setdiscardrJrKZ named_modulesZnamed_parametersappendrtyper ValueErrorrBZis_floating_point requires_gradrrnumelshaperflatten_params flat_paramrV)rUrWr$Z params_setrCrDrErFrGZshared_param_memoZparams_to_flattenrHrBrjZsubmodule_name submoduler#paramr2r3r1 extensionZprefixed_param_namer/r/r0r`s        z FlatParamHandle._init_flat_param)rWrjrIc Cs@t"dd|D}tj|dd}W5QRXt||d}|S)a Flattens the parameters in ``params`` into a single :class:`FlatParameter`. This should be the only way used to construct :class:`FlatParameter` s. We expose this factory method for checkpointing (e.g. sharded state dict). The flattened parameter's metadata should only be initialized once (see :meth:`_init_metadata`), but its tensor data may be reloaded. cSs0g|](}t|tjr"|dn|dqS)) isinstancer- ParameterdetachZreshape).0pr/r/r0 sz2FlatParamHandle.flatten_params..rZdim)rj)r5no_gradcatr)rWrjZ flat_paramsZflat_param_datarnr/r/r0rmqs  zFlatParamHandle.flatten_params) process_groupcCs|js dS|j}||_||_||_|dks>td|}t ||j|j\}}| || | ||j|dkr|ddS)a Shards the handle's ``FlatParameter``. In terms of memory, this allocates new memory for the sharded flattened parameter and frees the unsharded flattened parameter's storage. Postcondition: ``self.flat_param`` is the sharded flattened parameter. ``process_group``, ``rank``, and ``world_size`` attributes are set. TODO (awgu): Once we retire ``FlattenParamsWrapper``, we should pass the process group directly to the ``FlatParamHandle`` constructor. For now, we decouple ``FlattenParamsWrapper` from a process group, but this makes the process-group-related attributes not necessarily defined. Nrz;The `FlatParameter` is not the sole occupant of its storage)uses_sharded_strategyrnr|rankrS world_sizeZstorage_offsetrKstorager _get_shardset__init_shard_metadatarkZresize_)rUr|rnZ orig_storageZ local_shard numel_paddedr/r/r0shards(     zFlatParamHandle.shard)sharded_flat_param_numelrr~rIcCsZ||krtd|d|d||}||dd}|||\|j_|j_||j_dS)a Initializes shard-related metadata for this rank's shard of the flattened parameter: ``_shard_param_offsets``, ``_shard_indices``, and ``_shard_numel_padded``. Args: sharded_flat_param_numel (int): Numel of each rank's sharded flattened parameter with padding (i.e. including ``numel_padded``). numel_padded (int): Numel padded for this rank's sharded flattened parameter. rank (int): Caller's rank. z!Sharded flattened parameter with z numel cannot have z numel paddedrN)ri_get_shard_metadatarn_shard_param_offsets_shard_indicesZ_shard_numel_padded)rUrrr~startendr/r/r0rs z$FlatParamHandle._init_shard_metadata.)rrrIc Cs|}g}g}t|D]Z\}\}}||ks||kr6q||krDd} n||} t|||} |||| | fqt|dkrd} t|dkstn0|d|df} t|| d| ddkstt|| fS)a Computes the shard metadata based on ``start`` and ``end``, which give the closed interval of the unsharded flattened parameter specifying the shard. Args: start (int): Start index (in units of numel) of this rank's shard of the flattened parameter. end (int): End index (in units of numel and inclusive) of this rank's shard of the flattened parameter. Return: Tuple[Tuple[Tuple[int, int], ...], Tuple[int, int]]: See ``_shard_param_offsets`` and ``_shard_indices`` in :class:`FlatParameter` 's docstring. r)rrrrr)_get_flat_param_offsets enumerateminrgrJrKrL) rUrrZflat_param_offsetsZshard_param_indices_rangeZshard_param_offsetsiZ param_startZ param_endZintra_param_startZintra_param_endZshard_param_indicesr/r/r0rs2  z#FlatParamHandle._get_shard_metadata)tensorr~rrIcCsdt||}t||dkr0|dd}n||}|d|}|dks\td||fS)aa Returns the shard of ``tensor`` without any padding for the given ``rank`` and ``world_size`` and the numel to pad for that shard. If ``tensor`` is already flattened or may be viewed in the flattened shape (which is true in the expected usage), then this method does not allocate any new tensor memory. rrz5Chunk's size should be at most the first chunk's size)r5flattenchunkrJZ new_emptyrkrK)rr~rchunksr numel_to_padr/r/r0_get_unpadded_shardsz#FlatParamHandle._get_unpadded_shardcCs:t|||\}}|}|dkr2t|d|g}||fS)a( Returns the shard of ``tensor`` with padding for the given ``rank`` and ``world_size`` and the numel padded for that shard. This method allocates new memory (via :meth:`clone`) since the unsharded ``tensor`` may be deallocated after this method returns. r)rrcloneFpad)rr~rrrrr/r/r0rs zFlatParamHandle._get_shardcCs^t|jdkst|jt|||\}}|}t|dksJt|t|d|gS)z Returns the shape of ``tensor`` after sharding including padding. This requires ``tensor`` to have 1D shape and ensures that the returned shape is 1D. rr)rJrlrKrrrSr5r6)rr~rZunpadded_sharded_tensorrZunpadded_sharded_sizer/r/r0_get_sharded_size3sz!FlatParamHandle._get_sharded_size)rIcCsBtt|jj}dg|dd}dd|D}tt||}|S)zReturns [start, end] offsets of each original parameter's flattened data in the unsharded flattened parameter (without padding).rNrrcSsg|] }|dqS)rr/)rvrr/r/r0rxGsz;FlatParamHandle._get_flat_param_offsets..)listrrnrNzip)rUZcumulative_sumZstartsZendsr4r/r/r0rBs z'FlatParamHandle._get_flat_param_offsetscCst|jdrt|jds td|jjd}|jjd}||krNt||dntdd}t|jj||jj||jj||jj ddS)z`Returns shard-related metadata specific to this rank's shard of the flattened parameter.rrz'Shard metadata has not been initializedrrN) hasattrrnrKrslicerrPrOrNr)rUZshard_param_start_indexZshard_param_end_indexslr/r/r0shard_metadataKs$     zFlatParamHandle.shard_metadatacCspd}|jr|jjs|snD|jr6|js6|d}n*|jjr`|jj|jkr`|j |jddd}| |j|S)a$ Returns: ``False`` if this is a no-op and ``True`` otherwise. Postcondition: ``self.flat_param`` 's data is on the device for communication and is what should be all-gathered. This means that it matches the dtype of the expected unsharded parameter. FTZ non_blocking) r}r^r> needs_unshard_uses_param_mixed_precision_force_full_precision_use_low_precision_shardrnrX_flat_param_to_check_on_compute_device)rUretr/r/r0 pre_unshardds   zFlatParamHandle.pre_unshardcCsF||j}t|j|j|j|jj|jdd|j|_ dS)z Allocates the low precision shard directly on the compute device and switches to using the low precision sharded flattened parameter. TrN) _check_low_precision_shardrnr _mp_shard _local_shardrScopy_torXdata)rUrnr/r/r0r}sz(FlatParamHandle._use_low_precision_shardcCs:|s$|jr |}||dS|}||dS)a Runs the unshard logic. This includes all-gathering the flattened parameter and switching to using the unsharded flattened parameter. If the handle does not need unsharding, then this only switches to using the unsharded flattened parameter. For ``NO_SHARD``, this is a no-op. If FSDP is in :meth:`summon_full_params` and the handle uses parameter mixed precision, then the parameter is forced to full precision. N)rr} _get_padded_unsharded_flat_param_use_unsharded_flat_param"_alloc_padded_unsharded_flat_param_all_gather_flat_paramrUunsharded_flat_paramr/r/r0unshards  zFlatParamHandle.unshardcCs,|js dS|}||k}| S)zBReturns if the handle's flattened parameter needs to be unsharded.F)r}rrrSrk)rUrZalready_unshardedr/r/r0rs zFlatParamHandle.needs_unshardcCs0||j}|}||t||j|S)a* Allocates the *padded* unsharded flattened parameter. The unpadded unsharded flattened parameter is always a view into the padded one. This padded parameter is saved to a different attribute on the ``FlatParameter`` depending on if we force full precision. )_check_sharded_strategyrnr_check_storage_freedr_padded_unsharded_sizerUrnrr/r/r0rs   z2FlatParamHandle._alloc_padded_unsharded_flat_paramcCsD||j}|jr:|j}t|j|jjkd|jjn|j}|S)z Returns a reference to the padded unsharded flattened parameter depending on the calling context. This should only be called if using a sharded strategy. zExpects full precision but got ) rrnrZ_full_prec_full_param_paddedrrBr^r?Z_full_param_paddedrr/r/r0rs  z0FlatParamHandle._get_padded_unsharded_flat_param)padded_unsharded_flat_paramrIcCsptt|dot|dd|jj}||j}t||kd|d|t|||j| |dS)z All-gathers the handle's flattened parameter to the destination ``padded_unsharded_flat_param``, and switches to using the all-gathered tensor. r|rzEExpects a process group and world size to have been set via `shard()`Expects  numel but got N) rrrnrrkrdistZ_all_gather_baser|r)rUrZsharded_flat_paramZexpected_numelr/r/r0rs  z&FlatParamHandle._all_gather_flat_paramcCs&|jj}|d|||j_dS)z Switches to using the *unpadded* unsharded flattened parameter, which is a view into the *padded* unsharded flattened parameter. N)rnrTrkviewr)rUrZunsharded_sizer/r/r0rs z)FlatParamHandle._use_unsharded_flat_paramcCs$|jr|jr|||jdS)zo Runs the post-unshard logic. This includes freeing the low precision shard if needed. N)rr}!_free_low_precision_sharded_paramrrnrUr/r/r0 post_unshards zFlatParamHandle.post_unshardcCs|t|jjdS)z4Frees the low precision sharded flattened parameter.N)rrrnrrr/r/r0rsz1FlatParamHandle._free_low_precision_sharded_paramcCst|jtjtjfkd|j}|jdk r|j|jksJ|jj |j kr| |j|jj |j k}t| pr|j j d|j d|jj |j|j k}|r|s|jj|_|j jr|jj|j jkr|j|j j|_n,|j}t|j|kd|d|jd|_dS)z Prepares the gradient for the backward computation by saving and clearing any existing sharded gradient in ``.grad`` to enable computing a new unsharded gradient. z:Expects to be in `BACKWARD_PRE` or `IDLE` (if prefetching)Nz&Expects the sharded gradient to be on but got zFExpects `.grad` to be the unsharded gradient in `no_sync()` with size z but got size )rr_r"r;r9rnZgradrSrTrXrr^r>rrZ_saved_grad_shardr@rBrr)rUrnZgrad_offloadedZprev_iter_synced_gradientsZpadded_unsharded_sizer/r/r0prepare_gradientsP       z FlatParamHandle.prepare_gradientc cs|t|j|jjkd|jjd|j||j|j}|}t||kd| t d| z dVW5t|j|jjkd|jjd|j| }|d|j|j||XdS)ao Moves the unpadded unsharded flattened parameter to CPU while in the context and moves it back to the previous device upon exit. For now, this assumes the ``FlatParameter`` is the unpadded unsharded flattened parameter since (1) there is no reason to include the padding in the copy and (2) there is no use case for the sharded flattened parameter. Precondition: ``self.flat_param`` 's data is the unpadded unsharded flattened parameter on the compute device, and the handle uses a sharded strategy. Postcondition: Same as the precondition. z Expects size rzEExpects the unpadded parameter to be a view into the padded parametercpuN)rrrnrSrTrrZdata_ptrrrr5rX_free_unsharded_flat_paramrrkrr)rUZunpadded_storage_ptrZpadded_storage_ptrrr/r/r0to_cpuDs4  zFlatParamHandle.to_cpu)free_unsharded_flat_paramcCs|r ||dS)z Runs the reshard logic. This includes freeing the unsharded flattened parameter if ``free_unsharded_flat_param`` and switching to using the sharded flattened parameter. N)r_use_sharded_flat_param)rUrr/r/r0reshardtszFlatParamHandle.reshardcCs|jr|js|js|dS)aE Runs the post-reshard logic. This includes freeing any memory that can now be freed given that the ``FlatParameter`` points to the full precision sharded flattened parameter. Precondition: ``self.flat_param`` 's data points to the full precision sharded flattened parameter. N)rr}rrrr/r/r0 post_reshard~s zFlatParamHandle.post_reshardcCsJ||}|||||ttjjtj t |dS)z Frees the padded unsharded flattened parameter. The tensor to free depends on the calling context since the unshard may have forced full precision, in which case a different tensor is used. N) rr_check_storage_allocatedrZ record_streamrr5Z_CZStreamZcudaZcurrent_streamrrr/r/r0rs  z*FlatParamHandle._free_unsharded_flat_paramcCs<|j}|jjr0|jj}t|tdkd||j|_dS)z2Switches to using the sharded flattened parameter.rz-Expects the local shard to be on CPU but got N)rnr^r>rrXrr5r)rUrnrXr/r/r0rs z'FlatParamHandle._use_sharded_flat_param)rnrrIcCsh|dkr |}t||jkd|jd|dddttj||jdd|j|jD}|S) a Returns unflattened ``Tensor`` views into ``tensor`` if it is not ``None`` or ``flat_param`` otherwise, where the unflattening is based on ``flat_param`` 's metadata. In other words, to get views into the unsharded flattened parameter, pass ``tensor`` as ``None``, but to get views into tensor optimizer state, pass ``tensor`` as the optimizer state tensor. Nrrz numelcss$|]\}}}t|||VqdSN)rr)rvZ subtensorrlZparam_extensionr/r/r0 sz4FlatParamHandle._get_unflat_views..rry) rrkrTrr5splitrNrOrR)rnrviewsr/r/r0_get_unflat_viewssz!FlatParamHandle._get_unflat_views)r[rIc Cs||j}t||jjD]F\}\}}}t||r.css|] }|jVqdSrr)rvZspir/r/r0r s)rernrMunionrQrr/r/r0 _get_modulesszFlatParamHandle._get_modulesccs:dd|jjD}t|jj|D]\}}}||fVq dS)NcSs$g|]\}}}}}}t|||qSr/)r)rvr#r$r%rr/r/r0rxsz:FlatParamHandle.parameter_module_names..)rnrQrrM)rUrGr#rr%r/r/r0parameter_module_namess  z&FlatParamHandle.parameter_module_namescCst|jddS)NzExpects sharded strategy)rr}rr/r/r0r#sz'FlatParamHandle._check_sharded_strategy)rcCst|j|jkd|jdS)Nz+Expects tensor to be on the compute device )rrX)rUrr/r/r0r&s  z(FlatParamHandle._check_on_compute_devicecCs$|}t|dkd|dS)Nrz6Expects storage to be freed but got storage with size rrSrrZ storage_sizer/r/r0r,s  z$FlatParamHandle._check_storage_freedcCs|}t|dkddS)NrzExpects storage to be allocatedrrr/r/r0r4s z(FlatParamHandle._check_storage_allocatedcCsPt|jdtt|jdddk d|jjj}t||jkd|jd|dS)Nz&Not using low precision for parametersrzExpects `_mp_shard` to existz)Expects the low precision shard to be on r)rrrrnrrX)rUrXr/r/r0r9s z*FlatParamHandle._check_low_precision_shardcCs|jjtjkSr)r^r=r!r8rr/r/r0r}Ksz%FlatParamHandle.uses_sharded_strategycCs |jjdk Sr)r^r?rr/r/r0rOsz+FlatParamHandle._uses_param_mixed_precisioncCs|jtjko|jSr)r_r"r<rrr/r/r0rSs z%FlatParamHandle._force_full_precision)N)Ir'r(r)r*rr-rtr.r5rXr r]rr` staticmethodrrArrmrzrZ ProcessGrouprr7rrrrrr6rr rrrrrrrrrrrrrr contextlibcontextmanagerrrrrrr rrar rrrrr+rrrrrrpropertyr}rr __classcell__r/r/rbr0rs  `" "" 7        = / &)2rZ dataclassesrenumrr itertoolsrrtypingrrr r r r r rrrrrr5Ztorch.distributedZ distributedrZtorch.nnr-Ztorch.nn.functionalZ functionalrrZ_fsdp_extensionsrr_utilsrrrr__all__rrrr!r"r rtrrr/r/r/r0s: 8    z