U Jc @sddlZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmZddlmZdgZdd Zd d Zee e jeee e eefed d dZe jjeedddZe e je jedddZe e jedddZdeeeddddZ dS)N) OrderedDict)AnyCallableDictListSetTupleUnion) _BatchNorm)_is_namedtuple)PackedSequencep_assertcCstdd|DS)Ncss|]}t|tVqdSN) isinstancer ).0modr@/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/fsdp/utils.py sz&_contains_batchnorm..)anymodules)modulerrr_contains_batchnormsrcCs&|D]}t|trddi|_qdS)NZmixed_precision)rrr Z_wrap_overrides)rrrrr#_override_batchnorm_mixed_precisions  r)fn containerreturncs2ttjttttttft dfdd |S)zFRecursively apply to all tensor in different kinds of container types.)xrcst|r|St|drVt|}t|D] }|j}t||t||q0|St |t r| }| D]\}}|||<qp|St |t r|j|St |tr‡fdd| DSt|rfdd|D}t||St |tttfrt|fdd|DS|SdS)NZ__dataclass_fields__csi|]\}}||qSrr)rkeyvalueapplyrr 4sz4_apply_to_tensors..apply..c3s|]}|VqdSrrrelr rrr6sz3_apply_to_tensors..apply..c3s|]}|VqdSrrr#r rrr9s)torchZ is_tensorhasattr dataclassesreplacefieldsnamesetattrgetattrrr __class__itemsr datadictr typelisttupleset)rdcfr*Zodrrresr!rrrr!"s0        z _apply_to_tensors..apply) r r%Tensorrrrrrr r)rrrr8r_apply_to_tensorss*r:) root_module module_fn return_fncs6tjjtdfdd |df|||||S)ae Performs a pre-order traversal of the modules in the hierarchy rooted at ``root_module``, applying ``module_fn`` at each module and finally returning a value using ``return_fn``. The traversal constructs the full module prefix name (e.g. "module.submodule." just like in model state dict) and makes that available to ``module_fn``. )rprefixcsN||f|||D].\}}|dk r||d}||f||qdS)N.)Znamed_children)rr>argskwargsZsubmodule_name submodule new_prefixr6r<rrr6Ns  z_apply_to_modules..f)r%nnModulestr)r;r<r=r@rArrDr_apply_to_modules@srI)tensorsizercCsP||k}|sJ|}t|dkd|||| S)z Allocate storage for ``tensor`` with the given size. Returns: bool: ``True`` if this method allocated storage and ``False`` if the storage was already allocated. rz8Tensor storage should have been resized to be 0 but got )storagerKZnumelr resize_)rJrKZalready_allocatedZtensor_storage_sizerrr_alloc_storageZs  rN)rJrcCs:|dk}|s4t|dkd|d| S)z Frees the underlying storage of ``tensor``. Returns: bool: ``True`` if the method freed the storage and ``False`` if the storage was already freed. rzEFreeing a tensor's storage is unsafe when it is not the sole occupant)rLrKr Zstorage_offsetrM)rJZ already_freedrrr _free_storagens  rOT)condsraise_assertion_errorrcCs |st|t|rtdS)zThis is used as an alternate to ``assert`` when in the backward context to print the error message ``s`` since otherwise, it is swallowed.N)print traceback print_stackAssertionError)rPrQrRrrrr s )T)!r'rT collectionsrtypingrrrrrrr r%Ztorch.nn.modules.batchnormr Z torch.nn.parallel.scatter_gatherr Ztorch.nn.utils.rnnr __all__rrr9r:rFrGrIZno_gradSizeboolrNrOr rrrrs. $    $