U Jcn#@sddlZddlZddlmZmZmZddlZddlm Z ddlm Z ddl m Z m Z ddlmZddlmZmZdd lmZd d Zeeejeejdd d dZejdddZdddZeeejee jedddZeeedddZdS)N)OptionalListSequence)distributed_c10d)rpc) check_tensor(validate_non_overlapping_shards_metadata) ShardMetadata)TensorPropertiesShardedTensorMetadata)ShardcCs|dkrtd|}|}|}t|s^|dk r^|dksP|t|kr^td||dk rts|t d|t }|D]}|j |kr|j |fSqtd|||fS)Nzremote device is NonerzInvalid rank: z>RPC framework needs to be initialized for using worker names: zInvalid worker name: ) ValueError worker_namerankdevicerZ_rank_not_in_groupZget_world_sizerZ_is_current_rpc_agent_set RuntimeErrorZ_get_current_rpc_agentZget_worker_infosnameid)pgZ remote_devicerrrworkersZworkerrQ/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/_shard/sharded_tensor/utils.py!_parse_and_validate_remote_devices"   r)my_rankdst_ranksize dst_tensorreturncCs`||krP|dkrtd|t||kr\tdt|dt|n |r\tddS)Nz>Argument ``dst_tensor`` must be specified on destination rank z"Argument ``dst_tensor`` have size z,but should be zGArgument ``dst_tensor`` must NOT be specified on non-destination ranks.)rtupler)rrrrrrr"_validate_output_tensor_for_gather+sr )rcCs\t|dkr(t|dtjjr(t|}nt|}|D]}t|ts4td|q4t |S)zR Checks if tensor size is valid, then flatten/return a torch.Size object. r rz*size has to be a sequence of ints, found: ) len isinstance collectionsabcrlistint TypeErrortorchSize)rZdimsZdimrrr_flatten_tensor_sizeAs  r*TcCs|rJt|tst||krtd|d|d|d|d|d|d nTt|dksZt||krtd|d |d|d |d d |d|d |d ddS)NzLocal shards' tensor z& property need to be the same on rank:z! Found one local shard tensor =z, the other local shard tensor .zShardedTensor z5 property does not match from different ranks! Found on rank:rz, and r )r"r&AssertionErrorrr!)expectedactualZ prop_nameZranksis_localrrr_raise_if_mismatchPs0r3) local_shards global_size current_rankrrcCslt|dkstdg}|djj}|djj}|djj}|dj}t|D]\} } | j} | j} | | t || j \} }| jt j ks| j|krtd| jd|d| std| |krtd|d| | j|krtd | jd |t| jt| d |t| |d |t| j|d |t| j|d|qRt|||t j|d}t|||d}|S)Nrzmust have local shards!zs2       C