U Jc' @sJddlZddlZddlmZmZddlmZmZmZm Z m Z m Z m Z ddl Z dgZeGdddZeGdddZe jjedd d Zdeeeee jjfee jjje ed feeefe ee eee jjge jjfe jjd d dZeee jjed efe ed feeefedddZeje jje jjee dddZdS)N) dataclassfield)AnyCallableDict GeneratorListOptionalTuple TracingConfigc@s@eZdZUdZejZejjed<dZ e e e e fed<dS)r a Configurations used in ``ParamExecOrderWrapPolicy`` for symbolic tracing of a model. Args: tracer (torch.fx.Tracer): An instance of ``torch.fx.Tracer`` that will be used to perform symbolic tracing. ``tracer`` is default to be ``torch.fx.Tracer()``, but can also be instance of some child class of ``torch.fx.Tracer``. For example, one may want to use ``HFTracer`` for models in Transformers: .. _Transformers: https://huggingface.co/docs/transformers/index concrete_args (Optional[Dict[str, Any]]): Concrete arguments that should not be treated as ``torch.fx.Proxy`` when tracing the forward function. ``concrete_args`` allows one to partially specialize the forward function, including removing control flow or data structures. ``concrete_args`` is also the argument used in :meth:`~torch.fx.Tracer.trace`. tracerN concrete_args)__name__ __module__ __qualname____doc__torchfxTracerr __annotations__r r rstrrrrJ/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/fsdp/_symbolic_trace.pyr s c @s|eZdZUdZejjed<eejjed<e ejjee ejjee e ejj fffed<e edZeejj ed<dS)_ExecutionInfoa Contains the execution order information in the model forward pass. Attributes: current_module: record the module that is currently being traced. module_forward_order: a list of modules, where the ordering is based on when their forward function is called. ``module_forward_order`` includes the info of how many times a module is called + used to check the forward order in different iterations. param_exec_order: a list of parameters ordered based on their execution order. module_to_execution_infos: a dict that maps each module to a list of tuples each containing a module and a list of named parameters. ``module_execution_info_dict`` is used as the parameter execution order info. For a given module, each tuple: 1. either contains this module and part of its ``named_parameters`` that will be executed together, 2. or contains one of its child modules and all of the child module's ``named_parameters``. The list of tuples is ordered based on the parameter execution order. current_modulemodule_forward_ordermodule_to_execution_infos)default_factoryparam_exec_orderN)rrrrrnnModulerrrr r Parameterrlistrrrrrr%s  "r) root_modulereturncCst||g|gidS)z Create an instance of _ExecutionInfo with initialization based on ``root_module``. Args: root_module (torch.nn.Module): the module to get the execution information via ``tracer.trace()`` inside ``_patch_tracer``. )rrr)r)r#rrr_init_execution_infoHs r%.) create_proxyexecution_infoprefixed_param_name_to_paramkindtargetargskwargsname type_exprproxy_factory_fnr$c Cs|||||||| } |j} |dkr|dk rg} |D]V} t| tjjr2| jj|kr2|| jj}| | jj|f|t|j kr2|j |q2| r|j | | | fnT|dkrt | } | r|j | | | f| D]"\}}|t|j kr|j |q| S)aH Override of :meth:`~torch.fx.Tracer.create_proxy`. ``Tracer.create_proxy`` is called in symbolic tracing for each leaf function/method/module. This override intercepts the recording of each of these operations to update ``execution_info.module_to_execution_infos``. Args: create_proxy (Callable): The ``create_proxy`` function to be patched. execution_info (_ExecutionInfo): Used to record the execution information. prefixed_param_name_to_param (Dict[str, torch.nn.Parameter]): A dict that maps each prefixed parameter name to the parameter. kind (str): The type of the target method. One of 'call_function', 'call_method', 'get_attr', 'call_module', 'placeholder', or 'output'. The semantics of these opcodes are described in the ``torch.fx.Graph`` docstring. This is the input to ``create_proxy``. target (torch.fx.node.Target): Contains the string name of the method. This is the input to ``create_proxy``. args (Tuple[Any, ...]): Arguments of the method. This is the input to ``create_proxy``. kwargs (Dict[str, Any]): Keyword arguments of the method. This is the input to ``create_proxy``. name (Optional[str]): An optional string name for the ``Node`` created in ``create_proxy``. This is the input to ``create_proxy``. type_expr (Optional[Any]): An optional type annotation representing the Python type the output of a node will have. This is the input to ``create_proxy``. proxy_factory_fn (Callable[[torch.fx.Node], torch.fx.Proxy]): An alternative proxy constructor used in ``create_proxy``. This is the input to ``create_proxy``. ) call_functionZ call_methodN call_module) r isinstancerrProxynoder*appendsetrrr"named_parameters)r&r'r(r)r*r+r,r-r.r/proxymodule named_paramsargparam_prrr_patched_create_proxyXs.0    r?)r1r'r9forwardr+r,r$c Csh|j|t|}|r:|j|j|t|f|j}||_g|j|<|||||}||_|S)ag Override of :meth:`~torch.fx.Tracer.call_module`. ``Tracer.call_module`` is called in symbolic tracing for each non-root module. This override intercepts the recording of each operation to update ``execution_info.module_forward_order`` and ``execution_info.module_to_execution_infos``. Args: call_module (Callable): The ``call_module`` function to be patched. execution_info (_ExecutionInfo): Used to repord the execution information. module (torch.nn.Module): The module for which a call is being emitted. forward (Callable[..., Any]): The ``forward()`` method of the ``torch.nn.Module`` to be invoked. args (Tuple[Any, ...]): ``args`` of the module callsite. kwargs (Dict[str, Any]): ``kwargs`` of the module callsite. )rr5r"r7rr) r1r'r9r@r+r,r:Zprev_current_moduleoutputrrr_patched_call_modules    rB)r r#r'r$ccsX|j}|j}tt|||_t|}tt||||_z dVW5||_||_XdS)a Within the context manager, patches the input tracer so that during ``tracer.trace()``, the forward order of all modules and the parameter execution information are recorded. The patches of the input tracer will be removed after the context manager exits. Args: tracer (torch.fx.Tracer): the input ``tracer`` whose member functions will be patched within the context manager. root_module (torch.nn.Module): the top-level module to be traced and should not contain any FSDP modules. execution_info (_ExecutionInfo): used to record the execution order information when performing ``tracer.trace()`` within the context manager. N)r1r& functoolspartialrBdictr7r?)r r#r'Zoriginal_call_moduleZoriginal_create_proxyr(rrr _patch_tracers$  rF)NNN) contextlibrCZ dataclassesrrtypingrrrrrr r r__all__r rrr r%rr!rr4ZTargetNoder3r?rBcontextmanagerrrFrrrrsP$"   K    .