U Jc$@s2UddlmZmZddlmZddlZddlmZmZm Z m Z ddl Z ddl m Z mZmZmZddlmZddlmmmmZddlmZe rdd lmZGd d d eZeGd d d eZGdddeZiZ ee!eeeffe"d<ddZ#edddZ$ddZ%eGdddeZ&ddZ'dS))ABCabstractmethod) dataclassN)CallableDictList TYPE_CHECKING) check_tensorget_chunked_dim_sizeget_split_size(validate_non_overlapping_shards_metadata) ShardMetadata)_decorator_func) ShardedTensorc@seZdZdZdS) PlacementSpecz Base class representing the placement of an entity. Subclasses of this class can be used to specify customized placements which might not be covered by existing APIs. N)__name__ __module__ __qualname____doc__rrN/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/_shard/sharding_spec/api.pyrsrc@s&eZdZUdZejjed<ddZdS)DevicePlacementSpecz Associates placement of an entity with a single device. Args: device(:class:`torch.distributed._remote_device`): The device to place the entity on. devicecCs$t|jtjjs tj|j|_dSN) isinstancertorch distributed_remote_device)selfrrr __post_init__,sz!DevicePlacementSpec.__post_init__N) rrrrrrr__annotations__r rrrrr!s  rc@sFeZdZdZeejejej dddZ ed ej e ddd d Z dS) ShardingSpecz: Base class representing sharding specifications.  tensor_sizestensor_propertiesreturncCsdS)a Given a global tensor size, define how to shard a tensor like this shape across ranks, return ShardedTensorMetadata Args: tensor_sizes (:class:`torch.Size`): The tensor shape to shard on, a `torch.Size` object that represents the tensor shape to be sharded according to the ShardingSpec. tensor_properties(:class:`torch.distributed._shard.sharded_tensor.TensorProperties): Tensor properties used to create a ShardedTensor. Returns: A :class:`ShardedTensorMetadata` object that encodes the information about the layout of the ShardedTensor and its properties. Nrrr$r%rrrbuild_metadata4szShardingSpec.build_metadatarNrtensorsrc_rankr&cCsdS)a Given a global tensor on src_rank, shard this tensor across ranks within the process group, return a ShardedTensor. Args: tensor (:class:`torch.Tensor`): Tensor needs to be sharded. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the parameter that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Returns: A :class:`ShardedTensor` sharded from the given tensor. Nrrr*r+ process_grouprrrshardGszShardingSpec.shard)rN)rrrrrrSizesharded_tensor_metaTensorPropertiesShardedTensorMetadatar(Tensorintr.rrrrr"0sr"_CUSTOM_SHARDING_SPEC_OPScCst|j}|tko|t|kS)zQ Returns whether or not the ShardingSpec has a custom op implementation. )typerr5) sharding_specop class_namerrr_has_custom_op\s r:)r8cCsBt|j}t||s(td|d|t||}|||||S)zA Calls the custom op for this ShardingSpec if it exists. z Custom op: z not registered for )r6rr: RuntimeErrorr5)r7r8typesargskwargsr-r9funcrrr_dispatch_custom_opcs    r@cCs*|j}|tkrit|<tjt|t|dS)z Decorator to allow custom registration of ops. Args: sharding_spec_class(type): The ShardingSpec for which we need to add this custom op. func(Callable): The op to override (ex: torch.bmm) )r8Zop_table)rr5 functoolspartialr)Zsharding_spec_classr?r9rrrcustom_sharding_spec_opmsrCc@sTeZdZUdZeeed<ddZej e j e j dddZ dejed d d d Zd S)EnumerableShardingSpeca@ This is a type of PlacementSpec that allows users to specify a generic sharding scheme by enumerating exactly how each shard is laid out. Args: shards(List[ShardMetadata]): List of :class:`ShardMetadata` objects representing each shard. Note that none of the shards should overlap. shardscCsvt|jdkrtd|jd}|jD]>}|dkr\|t|jkr\td|dt|jt|j}q(t|jdS)NrzEmpty shard list provided: z%Found inconsistent ranks for shards: z and )lenrE ValueError shard_offsetsr )rZrankr.rrrr s  z$EnumerableShardingSpec.__post_init__r#cCst|j|t|j||Sr)r rEr0r2r'rrrr(s  z%EnumerableShardingSpec.build_metadatarNrr)cCs tddS)Nz1EnumerableShardingSpec.shard not implemented yet!)NotImplementedErrorr,rrrr.szEnumerableShardingSpec.shard)rN)rrrrrrr!r rr/r0r1r2r(r3r4r.rrrrrD~s   rDc sDg}dg}g}|D]}||j|j}|t|||jddt|D}t|dkrdqt|dkrxdqs|dq|dkrdqqdk rsz=_infer_sharding_spec_from_shards_metadata..rr cSsg|] \}}|qSrr)rK_xrrrrNscSs|dS)Nrr)rMrrrz;_infer_sharding_spec_from_shards_metadata..)key)ChunkShardingSpec)Zdim placementscsg|] }|qSrr)rKrP)chunk_sharding_dimrrrNscsg|]}t|qSr)r )rKrL)shard_total_length split_sizerrrNs)appendZ placementrIsum shard_sizes enumeraterGsortedzipZchunk_sharding_specrTr rangerD) Zshards_metadatarUZchunk_offset_listZshard_size_listZshard_metadataZ local_offsetsZ shard_dimsrTZ chunk_specr[chunksZchunk_shard_sizesr)rVrWrXr)_infer_sharding_spec_from_shards_metadatasR            ra)(abcrrZ dataclassesrrAtypingrrrrrZ _internalsr r r r Z!torch.distributed._shard.metadatarZ0torch.distributed._shard.sharded_tensor.metadatarZ_shardZsharded_tensormetadatar0Z*torch.distributed._shard.op_registry_utilsrZ'torch.distributed._shard.sharded_tensorrrrr"r5strr!r:r@rCrDrarrrrs,     * *