U Jc G @sddlZddlmZddlmZmZmZmZmZm Z m Z m Z m Z ddl mZddlmZddddd d d gZed d dZejeeeedddZejeee e ejedddZejeedddZejeeedddZedddfejeeeee e ejee e ejedddZejejhe_ejhe_ej eeedddd Z!ejeejddd Z"eGd d d Z#ejeejd!d"d#Z$d*ejeee eje ej%eee ejefd%d&d'Z&Gd(d)d)Z'dS)+N) dataclass) AnyCallableDict GeneratorOptionalSetTupleTypecast) _BatchNormalways_wrap_policylambda_auto_wrap_policytransformer_auto_wrap_policysize_based_auto_wrap_policy enable_wrapwrapParamExecOrderWrapPolicyreturncOsdS)z A simple wrapper policy that always returns ``True``, i.e. when passed as the `auto_wrap_policy` into FSDP, this will result in all submodules being wrapped as distinct FSDP instances. T)argskwargsrr?/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/fsdp/wrap.pyr #s)modulerecurseunwrapped_params lambda_fnrcCs|rdS||SdS)a A convenient auto wrap policy to wrap submodules based on an arbitrary user function. If `lambda_fn(submodule) == True``, the submodule will be wrapped as a `wrapper_cls` unit. Return if a module should be wrapped during auto wrapping. The first three parameters are required by :func:`_recursive_wrap`. Args: module (nn.Module): The module to be considered in this decision. recurse (bool): Indicate if this is called to make a decision on whether we should recurse down a subgraph of the module structure. If False, it means this function is called to make a decision on whether we should wrap the said module. unwrapped_params (int): The number of parameters yet to be wrapped in this module. lambda_fn (Callable[nn.Module] -> bool): If this returns ``True``, this module will be wrapped by wrapper_cls individually. TNr)rrrrrrrr,s)rrrtransformer_layer_clsrcCs|rdSt|t|SdS)ae A convenient auto wrap policy for transformer models. If the submodule is an instance of transformer_layer_cls, the submodule will be wrapped as a FSDP unit. Otherwise, all the other remainder submodules are wrapped by the outermost FSDP unit. Right now, FSDP requires submodules that share weights to be wrapped in the same FSDP unit, this auto wrap policy can conviniently wrap the shared embeddings into the same FSDP unit for transformer models. In the near future, FSDP will support submodules that share weights to be wrapped in the separated FSDP units. Return if a module should be wrapped during FSDP auto wrapping. The first three parameters are required by :func:`_recursive_wrap`. Args: module (nn.Module): The module to be considered in this decision. recurse (bool): Indicate if this is called to make a decision on whether we should recurse down a subgraph of the module structure. If False, it means this function is called to make a decision on whether we should wrap the said module. unwrapped_params (int): The number of parameters yet to be wrapped in this module. transformer_layer_cls (int): Submodules with one of the `transformer_layer_cls` names will be wrapped as separated FSDP units TN) isinstancetuple)rrrrrrrrQs$)rrrcOs|rdSt|tSdS)zM A policy that wraps ``BatchNorm`` instances in their own FSDP unit. TN)rr )rrrrrrr_wrap_batchnorm_individually|s r!)rrrrcstfdd|DS)zv A policy that wraps ``module`` if any policy in the passed in iterable of ``policies`` returns ``True``. c3s|]}|VqdSNr).0policyrrrrr sz_or_policy..)any)rrrZpoliciesrr%r _or_policys r(gחA)rrrmin_num_paramsforce_leaf_modulesexclude_wrap_modulesrcCs\|dkrtjn|}|dkr tjn|}||k}|rD|oBt|t| S|oVt|t| SdS)aA size based auto_wrap_policy function for FSDP API. Return if a module should be wrapped during FSDP auto wrapping. The first three parameters are used by :func:`_recursive_wrap`. If you write a custom version of this policy function, your version needs to at least accept the first three parameters and free to do whatever you want in the function. Args: module (nn.Module): The module to be considered in this decision. recurse (bool): Indicate if this is called to make a decision on whether we should recurse down a subgraph of the module structure. If False, it means this function is called to make a decision on whether we should wrap the said module. unwrapped_params (int): The number of parameters yet to be wrapped in this module. min_num_params (int): Customizable policy input. It controls the size threshold on how big should a module be to be considered wrapped. force_leaf_modules (Set[Type[nn.Module]]): set of module types to keep as leaves, i.e., their children will never be wrapped. exclude_wrap_modules (Set[Type[nn.Module]]): Customizable set of module types to be excluded in wrapping. N)rFORCE_LEAF_MODULESEXCLUDE_WRAP_MODULESrr )rrrr)r*r+Zis_largerrrrs')NNN) wrapper_clswrapper_kwargsrc ks,d|i|}tf| dVW5QRXdS)a Context manager to wrap modules using a wrapper. Useful for when you'd like to apply the same configuration arguments to all child modules that you wrap. A particularly important use case is wrapping large layers so that they get sharded (in-place) during initialization, to avoid running out of system memory. Large layers can indicate that they should be sharded via the ``wrap`` annotation and this context manager can provide the exact configuration for these nested instances. Usage:: with enable_wrap(wrapper_cls, **params): # Wraps layer in FSDP by default if within context self.l1 = wrap(torch.nn.Linear(5, 5)) Args: wrapper_cls: Class that `wrap` annotation will `wrap` modules with, such as `FullyShardedDataParallel`. **wrapper_kwargs: Configuration settings that will be passed to all ``wrap`` instances inside the context r.N)_ConfigAutoWrap)r.r/rrrrrs  )rwrap_overridesrcKs2tjr.tjdk sttj|}t|tjf|S|S)a Annotate that a module should be wrapped. Annotated modules will only be wrapped if inside of an :func:`enable_wrap` context manager. This allows a module to be initialized both with and without a wrapper without code change. The class that this function wraps the passed in ``nn.Module`` with is the passed in ``wrapper_cls`` argument into ``enable_wrap``. Both ``enable_wrap`` and ``wrap`` can take in kwargs specifying how to construct the ``wrapper_cls`` instance. In the case of duplicate kwargs in ``enable_wrap`` and ``wrap``, the argument passed into ``wrap`` will be respected. Usage:: with enable_wrap(wrapper_cls=FSDP, **fsdp_config): # Wraps layer in FSDP by default if within context self.l1 = wrap(torch.nn.Linear(5, 5)) Args: module (nn.Module): module to wrap (if in :func:`enable_wrap` context) **wrap_overrides: configuration overrides that will take priority over the values provided by the :func:`enable_wrap` context N)r0in_autowrap_contextr.AssertionErrorr_wrap)rr1rrrrs c@s*eZdZUdZeZeed<dZe ed<dS)rav This is the class used for the wrapping policy that wraps parameters and performs the communication scheduling based on the parameter execution order in the forward pass (also called non-recursive wrapping policy). The policy contains multiple wraps. Each wrap contains original parameters that will be executed together, and the wrap transfers these parameters into one ``FlattenParameter``. In both forward and the backward passes, the sharded parameters in each wrap will be gathered just before these parameters are used in the passes. These parameters will then be reshaded once they have been used. TODO (linjianma): For now, the parameters contained in each wrap of ``ParamExecOrderWrapPolicy`` are the parameters in each wrap of the ``init_policy`` (a recursive wrapping policy). Later we will wrap parameters based on bucket size. Args: init_policy (Callable): The initial recursive wrapping policy used to guide the wrapping of this policy. If tracing_config is none, in the first forward and backward iteration, ``init_policy`` is used to record parameter execution order. Otherwise, init_policy is only used in FSDP constructor for module level wrapping. The default ``always_wrap_policy`` might not be the best choice for every model. For example, for transformer based models, setting ``transformer_auto_wrap_policy`` as the ``init_policy`` will guarantee wrapping each transformer layer into one FSDP unit, and can be easily combined with checkpointing within each transformer layer. tracing_config (Optional[TracingConfig]): The configuration used to perform symbolic tracing at FSDP constructor to get the module and parameter execution order. The type of ``tracing_config`` needs to be either ``None`` or ``TracingConfig``. If set as ``None``, then symbolic tracing is not enabled, and one forward as well as backward iteration are needed to get the parameter execution order. ..warning :: Note that not all modules can be successfully traced when ``tracing_config`` is not None and symbolic tracing is enabled. The two cases below may be unable to trace: 1. when there is a data-dependent branch, 2. when the forward pass contains operators that don't support ``torch.fx.Proxy`` as the input type (e.g. ``arange``, ``zeros``, ``ones``, ``full``, ``full_like``, ``eye``, ``empty``, ``tensor``). For those cases, users can set ``tracing_config = None`` to disable symbolic tracing. init_policyNtracing_config) __name__ __module__ __qualname____doc__r r5r__annotations__r6rrrrrr#s + )rr.rcKs8|dk s tt|dr,||j}||f|S||f|S)N_wrap_overrides)r3hasattrr<)rr.rZ overridesrrrr4Ts     r4F)rauto_wrap_policyr.ignored_modulesignored_paramsonly_wrap_childrenrrc s2|dk std|dk s td|D]@\}}||kr:q(zt|tt|rPtWq(tk rfYq(Xq(tfdd|D} |dk st||d| dr*d} |D]D\} }||krqt f||||d |\} } t || | | | 7} q| | }|s"||d |dr"t ||f|| fS|| fS|dfS) a Automatically wrap child modules of *module* that meet the given criteria with :func:`auto_wrap`. Does not rely on _ConfigAutoWrap. Args: module (nn.Module): module to recursively wrap auto_wrap_policy (Callable): A callable specifying a policy to recursively wrap layers with FSDP. ignored_modules (Set[torch.nn.Module]): Modules to ignore when wrapping. ignored_params (Set[torch.nn.Parameter]): Parameters to ignore when wrapping; these should be the parameters contained in the modules in ``ignored_modules``. Returns: (nn.Module, int): Wrapped module and the number parameters wrapped recursively. NzMust specify auto_wrap_policy.zMust specify wrapper_clsc3s|]}|kr|VqdSr")Znumel)r#pr@rrr&sz"_recursive_wrap..Tr%r)rr>r.r?r@F) r3Z named_modulesrr type TypeErrorsum parametersZnamed_children_recursive_wrapsetattrr4)rr>r.r?r@rAr_childZ num_paramsZtotal_wrapped_paramsnameZ wrapped_childZnum_wrapped_params remainderrrCrrHasL     rHc@seZdZUdZdZeed<dZee ed<iZ e e e fed<e e e fddd Zee dd d d Zedd ddZdd ddZe e e ddddZdS)r0z Helper class to wrap modules based on default config args via a context manager. See :func:`enable_wrap` for more information. Fr2Nr.rrcKs ||_dSr"rN)selfrrrr__init__sz_ConfigAutoWrap.__init__)rrcCsHtjrtddt_d|ks(tdtt|dt_|d=|t_dS)Nz]You are already within an autowrap context and we currently do not supported nested autowrap.Tr.z9Expected to pass in wrapper_cls arg into _ConfigAutoWrap.) r0r2NotImplementedErrorkeysr3r rr.rrNrrrenable_autowrap_contexts z'_ConfigAutoWrap.enable_autowrap_contextrcCsdt_dt_it_dS)NF)r0r2r.rrrrrdisable_autowrap_contextsz(_ConfigAutoWrap.disable_autowrap_contextcCs||jdSr")rSr)rOrrr __enter__sz_ConfigAutoWrap.__enter__)exc_typeexc_valexc_tbrcCs |dSr")rT)rOrVrWrXrrr__exit__sz_ConfigAutoWrap.__exit__)r7r8r9r:r2boolr;r.rrrrstrrrP staticmethodrSrTrUrYrrrrr0s  r0)F)( contextlibZ dataclassesrtypingrrrrrrr r r Ztorch.nnnnZtorch.nn.modules.batchnormr __all__rZr Moduleintrrr!r(rZ ModuleListZ ModuleDictr-ZMultiheadAttentionr,contextmanagerrrrr4 ParameterrHr0rrrrs ,     &  ,   : #%0 J