U Jc @sddlZddlZddlZddlmZmZmZmZmZm Z m Z m Z m Z m Z mZddlZddlmZddlmmmZddlmZddlmZddlmZddlmZmZddl m!Z!ee"efee e"efddd Z#Gd d d Z$Gd d d e Z%Gddde Z&eee"efe'e'eee"efdddZ(eee"efe'e$dddZ)ee$e'eee"efdddZ*ee"efejj+e'ee"efdddZ,ee"ee"effee"ee'ee"efdddZ-e"eej.ee"e ej/eej.dd d!Z0e"eej.ee"ej.d"d#d$Z1e"eeee"ed%d&d'Z2ee"efe3ee"efd(d)d*Z4e ee"efe3ee"efd+d,d-Z5ee"efe ee"efe3e3ej6ee"efd.d/d0Z7e ej.ee"efe"ej/ej8ej6e3e3dd1 d2d3Z9e ej.ee"efe"ej/ej8ej6e3dd4d5d6Z:ee"efejj+ej;jejj+d:d;d<Z?ej;jd?Z@dQejj+e e eee"efeejj=feejj=d@dAdBZAej;j        r4)optim_state_dictmodelr2rcCs|}d|ksd|krtdt|}t|}i}|d}|D]\}} t|tr||ksjtd|||} t|| | ||} t t | d} | || <qBt | dkst| d} | |krqBt t | d} t || || <qBt |d}||d S) a: Flattens the full optimizer state dict, still keying by unflattened parameter names. If ``shard_state=True``, then FSDP-managed ``FlatParameter`` 's optimizer states are sharded, and otherwise, they are kept unsharded. Returns: Dict[str, Any]: The flattened optimizer state dict. r< param_groupszc`optim_state_dict` must have the keys "state" and "param_groups" to be a valid optimizer state dictz:Check the `flat_param_to_fsdp_module` construction param: Tr>rFr<rY) ValueError_get_flat_param_to_fsdp_modulerE _get_param_to_unflat_param_namesitemsr6rAssertionError_flatten_optim_stater+tuplelencopydeepcopy)rWrXr2Z unflat_osdflat_param_to_fsdp_moduleparam_to_unflat_param_namesZflat_osd_stateunflat_osd_stateparamr,r8 flat_stater;unflat_param_nameZflat_osd_param_groupsrrr_flatten_optim_state_dicts@    rk)rgr,r/r2rcst|}|dkstd|j}t|}||ksBtd|d|fdd|D}t|s`iSfdd|D} d} | D]B} | dkrq|| dkrt| } q|| t| kr|td |q|| dk sti} | D]8fd d| D} d d| D}d }}}|D]>}|t|o&| dkM}|t |M}|t| M}q td d|D}t|dks||s|s|stdd|d||rt | |||}|rt |jj\}}|| <n|| <q|rt| || <q|stt| || <q| S)ax Flattens the optimizer state in ``full_optim_state_dict`` for a single flattened parameter ``flat_param`` in ``fsdp_module`` corresponding to the unflattened parameter names in ``unflat_param_names``. Args: unflat_osd_state (Dict[str, Dict[str, Any]]): The "state" part of the optimizer state dict corresponding to the unflattened parameters. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the flattened parameter ``flat_param``. fsdp_module (FullyShardedDataParallel): FSDP module owning the flattened parameter. flat_param (FlatParameter): The flattened parameter. shard_state (bool): Whether to shard flattened positive-dimension tensor state; if ``False``, then the full flattened tensor is kept in the returned :class:`dict. Returns: Dict[str, Any]: A :class:`dict` mapping state names to their values for a particular flattened parameter. The sharded optimizer state dict's "state" part will map a key to this returned value. rzSExpects at least one unflattened parameter corresponding to the flattened parameterzExpects z shapes but got csg|]}t|kqSr)r..0rj)rgrr tsz(_flatten_optim_state..cs*g|]"}|kr"t|jdndqS))ZpgN)rrArl)r8rgrrrn~sNz@Differing optimizer state names for the unflattened parameters: cs g|]}|dk r|ndqSrr)rmr9)rLrrrnscSsg|]}|dk r|qSrrrmvrrrrnsTcss|]}t|VqdSr)typerorrr sz'_flatten_optim_state..r>z*Differing optimizer state types for state z , values z", and unflattened parameter names )rbr_Z_shapesanysetrr[r$rBrCrK_flatten_tensor_optim_stater _get_shardrQrD$_flatten_zero_dim_tensor_optim_state_flatten_non_tensor_optim_state)rgr,r8r/r2rSunflat_param_shapesZnum_unflat_param_shapesZ has_stateZunflat_param_statesZ state_namesr9riZ state_valuesZnon_none_state_valuesZare_pos_dim_tensorsZare_zero_dim_tensorsZare_non_tensorsrptypesrUZsharded_flat_tensorrTr)r8rLrgrr`Js              r`)rLpos_dim_tensorsr,ryr/rc sdd|D}tdd|D}t|dkrFtd|d|d|tt|t||D]N\}}|d krt|d krtd q\|d k r\|j|kr\td |jd |q\tdfddt||D} t | } |j } | j| kst d| jd| | S)aW Flattens the positive-dimension tensor optimizer state given by the values ``tensors`` for the state ``state_name`` for a single flattened parameter ``flat_param`` corresponding to the unflattened parameter names ``unflat_param_names`` and unflatted parameter shapes ``unflat_param_shapes``. This flattens each unflattened parameter's tensor state into one tensor. NOTE: We use zero tensors for any unflattened parameters without state since some value is required to fill those entries. This assumes that the zero tensor is mathematically equivalent to having no state, which is true for Adam's "exp_avg" and "exp_avg_sq" but may not be true for all optimizers. Args: state_name (str): Optimizer state name. pos_dim_tensors (List[torch.Tensor]): Positive-dimension tensor optimizer state values for the unflattened parameters corresponding to the single flattened parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flattened parameter. unflat_param_shapes (List[torch.Size]): Unflattened parameter shapes corresponding to the single flattened parameter. flat_param (FlatParameter): The flattened parameter. Returns: torch.Tensor: A flattened tensor containing the optimizer state corresponding to ``state_name`` constructed by concatenating the unflattened parameter tensor states in ``pos_dim_tensors`` (using zero tensors for any unflattened parameters without the state). cSsg|]}|dk r|qSrrrmtrrrrnsz/_flatten_tensor_optim_state..css|] }|jVqdSrr)r|rrrrrsz._flatten_tensor_optim_state..r>zAll unflattened parameters comprising a single flattened parameter must have positive-dimension tensor state with the same dtype but got dtypes for state ! and unflattened parameter names Nrz6Flattening a zero-dimension parameter is not supportedzBTensor optimizer state does not have same shape as its parameter:  r7c s>g|]6\}}|dk r$t|nttj|dqS)N)rGr)device)r$flattenrFzeros)rmZ state_valuer(Z cpu_devicer)rrrn s ztensor optim state: z flattened parameter: ) rtrbr[rPiterzipr(r$rcatrJr_) rLr{r,ryr/non_none_tensorsdtypestensorr(ZtensorsrUZflat_param_shaperrrrus0&      ru)rLzero_dim_tensorsr,rc Csdd|D}tdd|D}tdd|D}t|t|ksZt|dksZt|dkrztd|d|d |d |tt|}tt|}tj||td d S) a Flattens the zero-dimension tensor optimizer state given by the values ``zero_dim_tensors`` for the state ``state_name`` for a single flattened parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all tensors are the same and using that common value. NOTE: The requirement that the tensors are the same across all unflattened parameters comprising the flattened parameter is needed to maintain the invariant that FSDP performs the same computation as its non-sharded equivalent. This means that none of the unflattened parameters can be missing this state since imposing a value may differ from having no value. For example, for Adam's "step", no value means maximum bias correction, while having some positive value means less bias correction. Args: state_name (str): Optimizer state name. zero_dim_tensors (List[torch.Tensor]): Zero-dimension optimizer state for the unflattened parameters corresponding to the single flattened parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flattened parameter. Returns: torch.Tensor: A zero-dimensional tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. cSsg|]}|dk r|qSrrr|rrrrnCsz8_flatten_zero_dim_tensor_optim_state..css"|]}|dk r|ndVqdSr)itemr|rrrrrEsz7_flatten_zero_dim_tensor_optim_state..css |]}|dk r|jndVqdSrr~r|rrrrrFsr>All unflattened parameters comprising a single flattened parameter must have scalar state with the same value and dtype but got values z and dtypes rrr7r)r)rtrbr[rPrr$rr)rLrr,rZ values_setrrMr)rrrrw"s!    rw)rL non_tensorsr,rcCs\dd|D}t|}t|t|ks2t|dkrLtd|d|d|tt|}|S)a Flattens the non-tensor optimizer state given by the values ``non_tensors`` for the state ``state_name`` for a single flattened parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all values are the same and using that common value. See the note in :func:`_flatten_zero_dim_tensor_optim_state`. Args: state_name (str): Optimizer state name. non_tensors (List[Any]): Non-tensor optimizer state for the unflattened parameters corresponding to the single flattened parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flattened parameter. Returns: Any: A non-tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. cSsg|]}|dk r|qSrr)rmntrrrrnqsz3_flatten_non_tensor_optim_state..r>rrz" and unflattened parameter names )rtrbr[rPr)rLrr,Znon_none_non_tensorsZnon_tensor_setrVrrrrxXs rx)flat_optim_state_dictrDrc Cs|}dii}|dD]\}}i|d|<t|D]\}}t|oP|dk}|sh||d||<q4|jrtj|d|d} t| dkst | t | |j } nt |j |j } | |d||<q4q|d|d<|S)a^ Processes positive-dimension tensor states in ``flat_optim_state_dict`` by replacing them with metadata. This is done so the processed optimizer state dict can be broadcast from rank 0 to all ranks without copying those tensor states, and thus, this is meant to only be called on rank 0. Args: flat_optim_state_dict (Dict[str, Any]): Flattened optimizer state dict with the positive-dimension tensor states unsharded. Returns: Dict[str, Any]: The flattened optimizer state dict with positive- dimension tensor states replaced by metadata. r<r)rQrDr>rY) r^rr$rBrCr-rZ_get_sharded_sizerbr_r'r)r() rrDflat_osd no_tensor_osdr; param_staterLrMis_pos_dim_tensor_stateZ sharded_sizeinforrr_process_pos_dim_tensor_states*  r)processed_optim_state_dictrQrcCs<|dkr|gndg}tj|d|d|d}|dk s8t|S)a~ Broadcasts the processed optimizer state dict from rank 0 to all ranks. Args: processed_optim_state_dict (Optional[Dict[str, Any]]): The flattened optimizer state dict with positive-dimension tensor states replaced with metadata if on rank 0; ignored otherwise. Returns: Dict[str, Any]: The processed optimizer state dict. rNsrcr@)rHbroadcast_object_listr_)rrQr@obj_listrrr%_broadcast_processed_optim_state_dicts  r)rrrQrDbroadcast_devicerc Cs|dks|dk std|}|}|dD]\}} t| D]\} } t| t} | sTq<|dkrz|dk sht|d|| } nd} | j| j}}|jrt| | | |||||| qFZ requires_gradr)rr) r_ functoolspartialrrvrOrFr$rrH broadcast) rrrLr(r)rrQrDr@Z get_shardZ target_rankZsharded_tensorrrrrs6  r)rrrLr(r)rrQrcCs|dkrd|dk std||jks8td|d|j||jksXtd|d|j||}ntj|d||d}tj|d|d |||<dS) aZ Broadcasts positive-dimension tensor state for the state ``state_name`` corresponding to an unsharded non-FSDP parameter from rank 0 to all ranks. This modifies ``param_state`` destructively. Args: unsharded_tensor (Optional[torch.Tensor]): Unsharded tensor to broadcast if on rank 0; ignored otherwise. rNrzShape mismatch: rzdtype mismatch: Frr)r_r(r)rFr$rrHr)rrrLr(r)rrQr@rrrr=s, r) sharded_osdrXoptim optim_inputusing_optim_inputrcs|rt||nt|}t|}t|t|ks4ti}i|D]8\}} ||krVqD||} | |t| <| D] } | | <qnqD|d} i} | D]\}}||j} || | <qg}|dD]>}t |}t t fdd|dD}||d<| |q| |dS)a Rekeys the optimizer state dict from unflattened parameter names to flattened parameter IDs according to the calling rank's ``optim``, which may be different across ranks. In particular, the unflattened parameter names are represented as :class:`_OptimStateKey` s. r<rYc3s|]}|VqdSrrrlZ"unflat_param_name_to_flat_param_idrrrrsz2_rekey_sharded_optim_state_dict..paramsrZ)'_get_param_to_param_id_from_optim_input_get_param_to_param_idrEr]rbr_r^rar,rcrdrrtrR)rrXrrrZparam_to_flat_param_idrfZ#unflat_param_names_to_flat_param_idrhr, flat_param_idrjZsharded_osd_stateZrekeyed_osd_stater;rZrekeyed_osd_param_groupsunflat_param_groupflat_param_groupZflat_param_idsrrr_rekey_sharded_optim_state_dictgs>          r)rXcCs>i}|D],}t|tjr ||jD] }|||<q*q |S)a Constructs a mapping from FSDP flattened parameters to their owning FSDP modules and ensures that all FSDP modules are initialized. Args: model (torch.nn.model): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance). Returns: Dict[FlatParameter, FullyShardedDataParallel]: Mapping from FSDP flattened parameters to their owning FSDP modules. )modulesr6rEZFullyShardedDataParallelZ _lazy_initr)rXremodulerhrrrr\s     r\)rcCs,g}|jD]}|dD]}||qq |S)z Constructs a mapping from parameter IDs to parameters. This may be used both for models with ``FlatParameter`` s and without. r)rYrR)rparam_id_to_param param_grouprhrrr_get_param_id_to_params   r)rXrrc Cs|dkrt|Sz t|}Wn"tk rBtd|YnXt|dkrXtdd}d}|D]"}|t|tjM}|t|tM}qd|s|std|r|S|st g}|D]0}d|k}|st d|dD]}| |qq|S) an Constructs a mapping from parameter IDs to parameters. This may be used both for models with ``FlatParameter`` s and without. NOTE: This method is only preserved for backward compatibility. The method :meth:`_get_param_id_to_param` is the preferred code path that does not rely on ``optim_input``. NOTE: We critically assume that, whether the optimizer input is a list of parameters or a list of parameter groups, :class:`torch.optim.Optimizer` enumerates the parameter IDs in order. In other words, for a parameter list input, the parameter IDs should be in that list order, and for a parameter groups input, the parameter IDs should be in order within each parameter group and in order across parameter groups. Args: model (torch.nn.Module): Model whose parameters are passed into the optimizer. optim_input (Optional[Union[List[Dict[str, Any]], Iterable[torch.nn.Parameter]]]): Input passed into the optimizer representing either a :class:`list` of parameter groups or an iterable of parameters; if ``None``, then this method assumes the input was ``model.parameters()``. (Default: ``None``) Returns: List[torch.nn.Parameter]: Mapping from parameter IDs to parameters, where the parameter ID is implicitly the index in the :class:`list`. NzCOptimizer input should be an iterable of Tensors or dicts, but got rz#Optimizer input should not be emptyTz9Optimizer input should be an iterable of Tensors or dictsrzNA parameter group should map "params" to a list of the parameters in the group) r5 parameters TypeErrorrbr[r6r$r%dictr_rR) rXrrZ all_tensorsZ all_dictsrhrrZhas_params_keyrrr'_get_param_id_to_param_from_optim_inputs:'     r)rrcCst|}ddt|DS)AConstructs the inverse mapping of :func:`_get_param_id_to_param`.cSsi|]\}}||qSrrrmZparam_idrhrrr sz*_get_param_to_param_id..)r enumerate)rrrrrrsrcCst||}ddt|DS)rcSsi|]\}}||qSrrrrrrr'sz;_get_param_to_param_id_from_optim_input..)rr)rXrrrrrrs r)flat_to_unflat_param_idsrcsi|D]2\}}|D]$}|ks4td|d||<qq t}t}|tt|kstdt|dd|fddt|DS)a Inverts the mapping ``flat_to_unflat_param_ids`` to be from unflattened parameter ID to flattened parameter ID, where the unflattened parameter ID is the index in the returned :class:`list`. There may be multiple unflattened parameter IDs mapping to the same flattened parameter ID. Args: flat_to_unflat_param_ids (Dict[int, List[int]]): A mapping from flattened parameter ID to a :class:`list` of corresponding unflattened parameter IDs. Returns: List[int]: A mapping from unflattened parameter ID to flattened parameter ID, where the unflattened parameter ID is the index in the :class:`list`. z<`flat_to_unflat_param_ids` has the unflattened parameter ID z+ mapped to multiple flattened parameter IDsz8The set of unflattened parameter IDs should be {0, ..., r>z } but got csg|] }|qSrr)rmunflat_param_idZunflat_to_flat_param_idsrrrnOsz1_get_unflat_to_flat_param_ids..)r^r_rbrtrrOr#)rrZunflat_param_idsrZnum_unflat_param_idsZunflat_param_ids_setrrr_get_unflat_to_flat_param_ids*s*      r)xrcCst|o|dkS)Nr)r$rBrC)rrrrrKUsrK)rXrr rank0_onlyr2r@rrc's|}|d|d}} t|} | p4| dkp4|} | rDigdni} | rT| dnd} t||rpt||nt|i}t}t D]D\}}||krqt t |t |t d}| dkr|||<|||<q| dkr|gndg}tj|d|d|ddk st|d}g}|D]F}||kr8||q||}|dkrX|tkstdqtd tj}tjt|gtj|d }tj||d |dkr,d d tt|D}tj|||d d}t |D]D\} }ttt |}t|dkr|d| ddd |D7}qt |t!|}|D]}||}|}|j"r||}t#tt ||||| |}| r$t|t|j$kstt%|j$|D]\}}|| |<qnj| r| S)a Consolidates the optimizer state and returns it as a :class:`dict` following the convention of :meth:`torch.optim.Optimizer.state_dict`, i.e. with keys ``"state"`` and ``"param_groups"``. The flattened parameters in ``FSDP`` modules contained in ``model`` are mapped back to their unflattened parameters. Args: model (torch.nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters were passed into the optimizer ``optim``. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. rank0_only (bool): If ``True``, saves the populated :class:`dict` only on rank 0; if ``False``, saves it on all ranks. (Default: ``True``) shard_state (bool): If ``True``, shard and distribute all non-zero-dimension states. Returns: Dict[str, Any]: A :class:`dict` containing the optimizer state for ``model`` 's original unflattened parameters and including keys "state" and "param_groups" following the convention of :meth:`torch.optim.Optimizer.state_dict`. If ``rank0_only=False``, then nonzero ranks return an empty :class:`dict`. r<rYrrZN)r,r-rz/Check the `flat_param_id_to_param` constructionrIrr?cSsg|]}dqSrr)rmrTrrrrnsz%_optim_state_dict..zFSDP currently requires each rank to have at least the optimizer states needed by rank 0's optimizer but some ranks are missing some of those statesz Rank z' is missing states for the parameters: cSsg|] }|jqSr)r,)rmr;rrrrnsr>csg|] }|qSrr)rmr)flat_param_id_to_paramrrrnsrcsg|] }|qSrr)rmrh)rfrrrnscSsg|]}|D]}|q qSrr)rmr,rjrrrrns)+Z state_dictrHZget_rankrEr]rr collections OrderedDictrr+rar6rrr_valuesrRrbr$rrIZcurrent_devicerZint32Z all_reducerrOZget_world_sizeZall_gather_objectr r RuntimeErrorr\r-r=r,rrcrrBr7rd)'rXrrrr2r@rZosdZ osd_stateZosd_param_groupsrQr1Zfsdp_osdZfsdp_osd_stateZ optim_state_key_to_flat_param_idZ#r0_flat_param_id_to_optim_state_keyrrhZoptim_state_keyZ key_obj_listZ missing_keysZr0_optim_state_keyrZ num_missingr error_msgrrer8Z unflat_staterjr9rLrMZfsdp_osd_param_groupsrrZparam_group_paramsZnested_unflat_param_namesr)rrfr_optim_state_dictYs(                r)N)N)Hrrcrtypingrrrrrrrr r r r r$Ztorch.distributedZ distributedrHZ2torch.distributed.fsdp.fully_sharded_data_parallelZfsdpZfully_sharded_data_parallelrEZtorch.nnnnZ'torch.distributed._shard.sharded_tensorr Z#torch.distributed.fsdp._shard_utilsrZ!torch.distributed.fsdp.flat_paramrrZ'torch.distributed.fsdp._fsdp_extensionsrr#rrr'r+r.r=r3r4Modulerkr`r%r*rurwrxintrrrrr)rrrZ Optimizer Parameterrr\rrrrrrKZ ProcessGrouprrrrrs84     $   8  F ?  ;  T 7 (  )   H  7  +   =   M    +