U Jc$ @shddlZddlZddlZddlmZmZmZmZmZddl Z ddl m Z ddl mmZddl mZddlmZmZmZmZddlmZmZmZmZeeeeedddZeeeeeeeeeeeefd d d Zeeeee jee j e j!d d dZ"deee j e j!dddZ#dee$efee j ee$efdddZ%e j!eeee j edddZ&dS)N)AnyDictListOptionalTuple)distributed_c10d)Shard ShardedTensorShardedTensorMetadataTensorProperties)ChunkShardingSpecEnumerableShardingSpec ShardingSpec ShardMetadata) sharding_spec tensor_numel world_sizereturncsg}t|tr,|jD]}||jdqnt|tr|jdksDtt |dkrrfddt |D}qfddt |D}t t |}ntdt|d|S)z Translates the sharding spec to a list of offsets along dim 0. If the sharding spec is ChunkShardingSpec, only the ``dim`` is used and the placement is not used. rcsg|]}|kr|nqSr.0rank)rrG/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/fsdp/_shard_utils.py (sz-_sharding_spec_to_offsets..csg|]}|dkrndqSrrr) chunk_sizerrr-sz!Un-recognized sharding spec type .) isinstancer shardsappendZ shard_offsetsr dimAssertionErrormathceilrangelist itertools accumulate ValueErrortype)rrroffsetsZshardr)rrr_sharding_spec_to_offsetss    r,) input_offsetsoutput_offsetsrrmy_rankrcsjfddddfdd}ddtD}d dtD}||||||||||fS) z Given the shard offsets for each rank of the input tensor and output tensor, this API returns the corresponding split sizes that can be passed to all_to_all_single(). cs8dkr$||ddfS|dfSdSNrrr+)r/rrrr _get_intervalBs z._offsets_to_split_sizes.._get_intervalcSslg}t|D]Z\}}|t|dkr0||dn|d}|||t||dt||ddq |S)Nrr) enumeratelenr max)r+beginendZsizesioffsetZ next_offsetrrr_offsets_to_sizesHs$ z2_offsets_to_split_sizes.._offsets_to_sizescsX|\}}t||d}t||d}|||d||}||||d<dSr0)bisect)Z from_offsetsZ to_offsetsZ split_sizesr6r7Z to_begin_rankZ to_end_rankZ _split_sizes)r2r:rr_convertSs z)_offsets_to_split_sizes.._convertcSsg|]}dqSrrr_rrrr\sz+_offsets_to_split_sizes..cSsg|]}dqSrrr=rrrr]s)r%)r-r.rrr/r<input_split_sizesoutput_split_sizesr)r2r:r/rrr_offsets_to_split_sizes5s     rA) input_tensor output_specrr/device process_grouprcCs|}|}t|tr"td|}t|||} t|||} t| | |||\} } t| } t j | |j |d}t j ||dj| | |d|S)a Resharded a sharded flatten tensor, this is used by FSDP to do sharded state_dict. But the functionaility is not supported by ShardedTensor. This API is designed to be used for FSDP; therefore this API supports only 1-D ShardedTensor (hence the naming, reshard_flatten_tensor). This API uses the ChunkShardingSpec and EnumerableShardingSpec from torch.distributed.sharding_spec but ignores the placement field in ChunkShardingSpec, as the placement requires the callees understand the number of GPUs per node. The API simply uses the semantics of the sharding specs. Args: input_tensor (ShardedTensor): the original ShardedTensor. Must be 1D. output_spec (ShardingSpec): the sharding spect for the output tensor. world_size (int): total trainer count. my_rank (int): the rank for this trainer. Returns: The local shard for the new ShardedTensor. z#The input tensor has no dimensions.dtyperDr)r?r@group)rsizerintr)numelr,rAsumtorchemptyrGdistZall_to_all_single local_shardstensor)rBrCrr/rDrEZ input_specrIrr-r.r?r@Z output_size local_shardrrr_reshard_flatten_tensords0    rS)sharded_tensorpgrc Cs(|dkrt}t|}|}|d}|}t||||}t dt j }|r|dj }|jst jd|d} | }nt jd|d} ||} | dkrt|d| g}n t j||j|d}t jd|d} t j|||j|d} tj| ||d| dd||} | S)Nrcudar)rDrF)rH)rZ_get_default_grouprOZget_world_sizerPrIrKr#r$rMrDrVZcurrent_devicerQflattenis_cudaZoneszerosFpadrGrNZ_all_gather_baseZnarrowZreshape) rTrUrrZ dim_0_sizerrZ cuda_deviceZ local_tensorZ move_to_cpuZ num_paddingrQrrr_all_gather_sharded_tensors>     r\) state_dictrUrcCs\i}|D]J\}}t|trNt||}|rF|djjrF|}n|}|||<q |S)zX Given a state_dict, this API gathers all the ShardedTensors in the state_dict. r)itemsrr r\rPrQrXcpu)r]rUZnew_state_dictkeyrQZ output_tensorrrr_gather_state_dicts   ra)rQrrnum_devices_per_noderUrc s\|j|dd}t||krh||}dd|Dt|d||d<t||g}ng}dd|D}dgtt dd|Ddd} dgt|dd fd d| D} fd dt t|D} t|t| krt| ks nt d dt | || D} t| |t|j|jd tj|dd} tj|| |dS)z Shard a tensor to chunks along the first dimension. The local rank will gets its corresponding chunk as the local shard to create a ShardedTensor. r)r!cSsg|]}dqSrrr=rrrrsz0_create_chunk_sharded_tensor..cSsg|]}t|qSr)r&rI)rchunkrrrrscSsg|] }|dqSrr)rrrrrrsNrcsg|]}|gqSrr)rZd0r1rrrscs g|]}d|d|qS)zrank:z/cuda:r)rr)rbrrrscSsg|]\}}}t|||qSr)r)rr9rIZ placementrrrrsF)rGlayoutZ requires_gradZ memory_formatZ pin_memory)Zshards_metadatarIZtensor_properties)sharded_tensor_metadatarE)rcr4clonerIr#r$rZfrom_tensor_and_offsetsr&r'r(r%r"zipr r rGrfrMZcontiguous_format is_pinnedr Z+_init_from_local_shards_and_global_metadata)rQrrrbrUchunksrRrPZ chunk_sizesZ dim0_offsetsZ chunk_offsetsZ placementsZshard_metadatargr)rbr+r_create_chunk_sharded_tensorsL     *  rl)N)N)'r;r'r#typingrrrrrrMZtorch.distributedZ distributedrOZtorch.nn.functionalnnZ functionalrZrZ'torch.distributed._shard.sharded_tensorrr r r Z&torch.distributed._shard.sharding_specr r rrrJr,rArDZ ProcessGroupZTensorrSr\strrarlrrrrs`     0 5 (