U Jºcã ã@sôUddlmZmZddlmZmZmZmZddlZddl m Z ddl m Z Gdd„deƒZdaeeed<eddœd d „Zejeejeefd œd d „Zejeejdœdd„Zejeeee jejdœdd„Zejeejeejfd œdd„ZdS)é)ÚABCÚabstractmethod)ÚAnyÚListÚOptionalÚTupleN)Ú_create_chunk_sharded_tensorc@s–eZdZdZeejeejee fdœdd„ƒZ eeje ejdœdd„ƒZ eeje e e e jejdœd d „ƒZeejeejeejfdœd d „ƒZd S)ÚFSDPExtensionszä This enables some customizable hooks to enable composability with tensor parallelism. To activate these hooks, use :func:`_set_fsdp_extensions` to set a custom :class:`FSDPExtensions` that implements the hooks. ©ÚtensorÚreturncCsdS)z6E.g. converting ``DistributedTensor`` to local tensor.N©©Úselfr r r úK/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/fsdp/_fsdp_extensions.pyÚpre_flatten_transformsz$FSDPExtensions.pre_flatten_transform©r Úparam_extensionr cCsdS)z6E.g. converting local tensor to ``DistributedTensor``.Nr )rr rr r rÚpost_unflatten_transformsz'FSDPExtensions.post_unflatten_transform©r ÚrankÚ world_sizeÚnum_devices_per_nodeÚpgr cCsdS)z6Shards a tensor to chunks and returns the local chunk.Nr )rr rrrrr r rÚ chunk_tensor"s zFSDPExtensions.chunk_tensorcCsdS)z£ This is to be called before loading a *sharded* model state dict and should return the tensor and list of shards from which to load data. Nr rr r rÚpre_load_state_dict_transform.s z,FSDPExtensions.pre_load_state_dict_transformN)Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚtorchÚTensorrrrrrÚintÚdistÚ ProcessGrouprrrr r r rr s,ýüù ýr Ú _extensions)Ú flattenerr cCs|adS©N)r%)r&r r rÚ_set_fsdp_extensions=sr(r cCs.tdk r&t |¡\}}|dk r&||fS|dfSr')r%r)r Z new_tensorÚ extensionr r rÚ_ext_pre_flatten_transformBs r*rcCs tdk r|dk rt ||¡S|Sr')r%r)r rr r rÚ_ext_post_unflatten_transformLs r+rcCs"tdk rtjnt}||||||ƒSr')r%rr)r rrrrZchunk_tensor_fnr r rÚ_ext_chunk_tensorUs ÿýûr,cCs"tdk rt |¡S| ¡}||fSr')r%rZ local_shards)r Zshardsr r rÚ"_ext_pre_load_state_dict_transformjs r-)ÚabcrrÚtypingrrrrr Ztorch.distributedZ distributedr#Z#torch.distributed.fsdp._shard_utilsrr r%Ú__annotations__r(r!r*r+r"r$r,r-r r r rÚs2  0þ ý ú þ