U Jc3@sdddlmZddlmZeedddZeedddZd d d d Zd dZddZddZ d S))List) ShardMetadata)shard1shard2cCsdt|j}t|D]L}|j||j||j|kr:dS|j||j||j|krdSqdS)z' Checks if two shards overlap. FT)len shard_offsetsrange shard_sizes)rrZndimsir U/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/_shard/sharding_spec/_internals.py"_check_shard_metadata_pair_overlaps  r )shardscCs^tt|D]L}t|dt|D]4}t||||r"td||d||dq"q dS)a Ensures none of the shards overlap with each other. Args: shards(List[ShardMetadata]): List of :class:`ShardMetadata` objects representing each shard. Raises: ``ValueError`` if there's overlap in any two shards. zShards z and z overlapN)rrr ValueError)rr jr r r (validate_non_overlapping_shards_metadatas rN)returnc Cst|}t|dj}||kr2td|d|d}|D]x}d}t|jD]\\}}||9}|j||j|||krLtd|j|d|j|d||d|qL||7}q:d} |D] } | | 9} q|| krtd |d | d d S) ab Checks if the shards_metadata is compatible with the provided tensor dims. Args: shards_metadata(List[ShardMetadata]): List of :class:`ShardMetadata` objects representing each shard of the tensor. tensor_dims(Sequence of int): Dimensions of tensor to verify Raises: ``ValueError`` if not compatible. rzRank of tensor is z, but shards rank is rz Shard offset z and length z exceeds tensor dim: z for shard zTotal volume of shards: z does not match tensor volume: zI, in other words all the individual shards do not cover the entire tensorN)rrr enumerater ) Zshards_metadataZ tensor_dimsZ tensor_rankZ shards_rankZtotal_shard_volumeZshardZ shard_volumer Z shard_lengthZ tensor_volumesizer r r check_tensor(s**  rcCs||d|S)a Computes the split size inline with ``torch.chunk`` Args: dim_size(int): Size of the dimension being chunked. chunks(int): Number of chunks to create for ``dim_size``. Returns: An int indicating the split size to use. rr )dim_sizechunksr r r get_split_sizeSs rcCs tt|||d||dS)a Computes the dim size of the chunk for provided ``idx`` given ``dim_size`` and ``split_size``. Args: dim_size(int): Size of the dimension being chunked. split_size(int): The chunk size for each chunk of ``dim_size``. idx(int): The index of chunk whose dim size is being requested. Returns: An int indicating the dim size of the chunk. rr)maxmin)r split_sizeidxr r r get_chunked_dim_size`s rc CsVt||}d}|}t|jD]0\}}t|||} ||krD|}qN|| 7}q|| fS)a Generate the start pos and offset length for the current rank for chunk sharding. Args: sharding_dim_size(int): The dimension length which we shard on. world_size(int): number of ranks. spec (:class:`torch.distributed._shard.sharding_spec.ChunkShardingSpec`): sharding spec. rank(int): # of cuda process. Returns: start_pos(int): start position of sharded tensor on the given rank. chunk_size(int): chunk size of sharded tensor on the given rank. r)rrZ placementsrrank) Zsharding_dim_sizeZ world_sizespecrrZcurrent_offsetsZ start_posrZ placement chunk_sizer r r get_chunk_sharding_paramsos    r") typingrZ!torch.distributed._shard.metadatarr rrrrr"r r r r s  +