U Jc)@sddlmZmZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl mZddlmZmZmZddlmZdZGd d d eZGd d d e j jZejd dfe j jeee j jdddZeddfddZdS))autoEnum)partial)AnyDictIteratorTupleN) save_on_cpu) _pack_kwargs_replace_by_prefix_unpack_kwargs) checkpoint_checkpoint_wrapped_modulec@seZdZeZeZdS)CheckpointImplN)__name__ __module__ __qualname__r REENTRANTZ NO_REENTRANTrr_/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/algorithms/_checkpoint/checkpoint_wrapper.pyr srcseZdZdZejddfejjee dfdd Z e e dfdd Z ee d d d Zd dZeee ejjfdfdd Zeejee e fe e ee e fdddZeejee e fe e ddddZZS)CheckpointWrapperz An nn.Module that wraps another nn.Module with checkpointing. Note that this module is not meant to be used directly, but instead it is to be used through the ``checkpoint_wrapper`` function. FN)modcheckpoint_imploffload_to_cpucs|t||_||_||_|jr*d|_n2|dkrJtt|jtj kd|_nt|f|||_| |j |j |j dddS)N)Z use_reentrantT)Z with_module)super__init__rrr checkpoint_fnrr rrZ_register_state_dict_hook_post_state_dict_hookZ"_register_load_state_dict_pre_hook_pre_load_state_dict_hook)selfrrrrcheckpoint_fn_argscheckpoint_fn_kwargs __class__rrrs.   zCheckpointWrapper.__init__)namereturncs4zt|WStk r.t|j|YSXdS)z-Forward missing attributes to wrapped module.N)r __getattr__AttributeErrorgetattrr)rr$r"rrr&?szCheckpointWrapper.__getattr__)keyr%cCs |j|S)z=Forward indexing calls in case the module is a nn.Sequential.)r __getitem__)rr)rrrr*FszCheckpointWrapper.__getitem__c sjr2tddj||W5QRSQRXnTjtjkrr|ikrrt||\}fdd}j|f|Sjjf||SdS)NT)Z pin_memorycst|\}}j||S)N)r r)inputsZ unpacked_argsZunpacked_kwargsZ kwarg_keysrrr my_functionYsz.CheckpointWrapper.forward..my_function)rr rrrrr r)rargskwargsZ flat_argsr-rr,rforwardJs$   zCheckpointWrapper.forward)r%c/s4tj||D] \}}|tdd|fVqdS)z Overrides :meth:`named_parameters()` to intercept parameter names and remove all occurrences of _CHECKPOINT_PREFIX. .N)rnamed_parametersreplace_CHECKPOINT_PREFIX)rr.r/ param_nameparamr"rrr3ps z"CheckpointWrapper.named_parameters)module state_dictprefixr.r%cGst||td||S)a _post_state_dict_hook() is called after the state_dict() of this FSDP module is executed. For ``checkpoint_wrapper``, it will strip checkpoint-wrapped module prefix so that this module can be loaded into non-checkpointed modules. It would still be able to be loaded into checkpoint-wrapped modules as this class adds the prefix back before loading the state_dict. r1r r5r8r9r:r.rrrr|sz'CheckpointWrapper._post_state_dict_hookcGst|||tddS)a ``_pre_state_dict_hook` is called before ``self._load_from_state_dict()`` is called. For ``checkpoint_wrapper``, it will add back the module prefix so that non-checkpointed modules can be loaded into checkpoint_wrapper modules properly. r1Nr;r<rrrrs z+CheckpointWrapper._pre_load_state_dict_hook)rrr__doc__rrtorchnnModuleboolrstrrr&intr*r0rr Parameterr3 staticmethodrrr __classcell__rrr"rrs8'*    rF)r8rrr%cOst||||||S)a A convenience wrapper for activation checkpointing. If the module is wrapped with this function, all subsequent calls to the module will automatically perform checkpointing without the user having to explicitly call ``checkpoint`` function. Usage:: checkpointed_module = checkpoint_wrapper(module) outputs = checkpointed_module(inputs) Args: module (nn.Module): The module to be wrapped checkpoint_impl (Optional[CheckpointImpl]): The checkpointing implementation to use. Note that this will only be passed into the ``torch.utils.checkpoint.checkpoint`` implementation, and is ignored if a custom ``checkpoint_fn`` is specified. Note that for implementations using reentrant checkpoint from ``torch.utils.checkpoint``, keyword arguments will only be supported if ``checkpoint_impl`` is passed as ``CheckpointImpl.REENTRANT`. offload_to_cpu (Optional[bool]): Whether to offload activations of this wrapped module to CPU. Note that if this is specified, ``checkpoint_impl`` and ``checkpoint_fn`` arguments will be ignored in favor of the activations being offloaded to CPU. Default is ``False``. Wrappers with activation offload can be composed with ones that do recomputation-based checkpoint to trade off increased compute versus increased CPU memory usage and additional H2D transfers. checkpoint_fn (Optional[Callable]): Functional checkpoint implementation to use. If this is specified, it will be used over the default ``torch.utils.checkpoint.checkpoint`` implementation and the `checkpoint_impl` argument will be ignored. *checkpoint_fn_args: (Sequence[Any]): Arguments to pass into `checkpoint_fn`. **checkpoint_fn_kwargs: (Dict[str, Any]): Keyword arguments to pass into `checkpoint_fn`. Returns: (nn.Module): Wrapped module )r)r8rrrr r!rrrcheckpoint_wrappers.rGcCsdS)NTr)_rrrrIcCs0ddlm}m}||t||d|ttddS)aK Applies :func:`checkpoint_wrapper` to modules within `model` based on a user-defined configuration. For each module within `model`, the `check_fn` is used to decide whether `module` should be wrapped with :func:`checkpoint_wrapper` or not. Note:: This function modifies `model` in place and replaces appropriate layers with their checkpoint-wrapped modules. Note:: This function will not wrap the overall root module. If this is needed, please directly use :class:`CheckpointWrapper`. Usage:: model = nn.Sequential( nn.Linear(10, 10), nn.Linear(10, 10), nn.Linear(10, 10) ) check_fn = lambda l: isinstance(l, nn.Linear) apply_activation_checkpointing(model, checkpoint_wrapper_fn=checkpoint_wrapper, check_fn=check_fn) Args: model (nn.Module): The model whose submodules should be wrapped with activation checkpointing. checkpoint_wrapper_fn (Optional[Callable[nn.Module]]) A ``Callable`` which will wrap modules check_fn (Optional[Callable[nn.Module, nn.Module]]) A lambda function which will be passed each child submoule of ``model`` and returns ``True`` or ``False`` depending on whether the submodule should be wrapped. Returns: None (`model` is modified inplace) r)_recursive_wraplambda_auto_wrap_policy)Z lambda_fnT)r8Zauto_wrap_policyZ wrapper_clsZignored_modulesZignored_paramsZonly_wrap_children)Ztorch.distributed.fsdp.wraprKrLrset)modelZcheckpoint_wrapper_fnZcheck_fnrKrLrrrapply_activation_checkpointings  rO)enumrr functoolsrtypingrrrrr>Ztorch.nnr?Ztorch.autograd.graphr Ztorch.distributed.utilsr r r Ztorch.utils.checkpointr r5rr@rrrArGrOrrrrs.     4