U Jc @s2ddlZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmZddlmZdZdd Zd d Zee e jeee e eefed d dZe jjeedddZe e je jedddZe e jedddZe jddddZ e jedddZ!d eeeddddZ"dS)!N) OrderedDict)AnyCallableDictListSetTupleUnion) _BatchNorm)_is_namedtuple)PackedSequenceZ_fsdp_flattenedcCstdd|DS)Ncss|]}t|tVqdSN) isinstancer ).0modrA/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/fsdp/_utils.py sz&_contains_batchnorm..)anymodules)modulerrr_contains_batchnormsrcCs&|D]}t|trddi|_qdS)NZmixed_precision)rrr Z_wrap_overrides)rrrrr#_override_batchnorm_mixed_precisions  r)fn containerreturncs2ttjttttttft dfdd |S)zFRecursively apply to all tensor in different kinds of container types.)xrcst|r|St|drVt|}t|D] }|j}t||t||q0|St |t r| }| D]\}}|||<qp|St |t r|j|St |tr‡fdd| DSt|rfdd|D}t||St |tttfrt|fdd|DS|SdS)NZ__dataclass_fields__csi|]\}}||qSrr)rkeyvalueapplyrr 4sz4_apply_to_tensors..apply..c3s|]}|VqdSr rrelrrrr6sz3_apply_to_tensors..apply..c3s|]}|VqdSr rr"rrrr9s)torchZ is_tensorhasattr dataclassesreplacefieldsnamesetattrgetattrrr __class__itemsr datadictr typelisttupleset)rdcfr)Zodrrresr rrrr "s0        z _apply_to_tensors..apply) r r$Tensorrrrrrr r)rrrr7r_apply_to_tensorss*r9) root_module module_fn return_fncs6tjjtdfdd |df|||||S)ae Performs a pre-order traversal of the modules in the hierarchy rooted at ``root_module``, applying ``module_fn`` at each module and finally returning a value using ``return_fn``. The traversal constructs the full module prefix name (e.g. "module.submodule." just like in model state dict) and makes that available to ``module_fn``. )rprefixcsN||f|||D].\}}|dk r||d}||f||qdS)N.)Znamed_children)rr=argskwargsZsubmodule_name submodule new_prefixr5r;rrr5Ns  z_apply_to_modules..f)r$nnModulestr)r:r;r<r?r@rrCr_apply_to_modules@srH)tensorsizercCsP||k}|sJ|}t|dkd|||| S)z Allocate storage for ``tensor`` with the given size. Returns: bool: ``True`` if this method allocated storage and ``False`` if the storage was already allocated. rz8Tensor storage should have been resized to be 0 but got )storagerJZnumelp_assertresize_)rIrJZalready_allocatedZtensor_storage_sizerrr_alloc_storageZs  rN)rIrcCs:|dk}|s4t|dkd|d| S)z Frees the underlying storage of ``tensor``. Returns: bool: ``True`` if the method freed the storage and ``False`` if the storage was already freed. rzEFreeing a tensor's storage is unsafe when it is not the sole occupant)rKrJrLZstorage_offsetrM)rIZ already_freedrrr _free_storagens  rOcCst|tddS)z Sets an attribute on ``tensor`` to mark it as flattened by FSDP. This is to avoid re-flattening it during nested construction. TN)r*FSDP_FLATTENEDrIrrr_set_fsdp_flattenedsrRcCs t|tdS)z;Returns if ``tensor`` has been marked as flattened by FSDP.F)r+rPrQrrr_is_fsdp_flattenedsrST)condsraise_assertion_errorrcCs |st|t|rtdS)zThis is used as an alternate to ``assert`` when in the backward context to print the error message ``s`` since otherwise, it is swallowed.N)print traceback print_stackAssertionError)rTrUrVrrrrLs rL)T)#r&rX collectionsrtypingrrrrrrr r$Ztorch.nn.modules.batchnormr Z torch.nn.parallel.scatter_gatherr Ztorch.nn.utils.rnnr rPrrr8r9rErFrHZno_gradSizeboolrNrOrRrSrLrrrrs2 $    $