U Jc@sbddlZddlmZddlmZddlmZddlmZej ej j ej j gZ Gdddej Z dS)N) ShardedTensor)distributed_c10d)get_default_nowrap_functionscspeZdZUdZejed<dgZdddZddZ fdd Z e dd d Z e d ddZddZddZZS)ReplicatedTensora  ReplicatedTensor represents a tensor which is replicated across the `world_size` and has the same value on each rank. ReplicatedTensor is a :class:`~torch.Tensor` subclass, and it could be used together with ShardedTensor/Tensor together to express different types of computation. The inter-op rules defined as (using torch.add as an example op): ReplicatedTensor + ReplicatedTensor = ReplicatedTensor ReplicatedTensor + torch.Tensor = torch.Tensor ReplicatedTensor + ShardedTensor = ShardedTensor ReplicatedTensor + other type (i.e. Scalar) = other type NOTE: We do not gurantee equal content of ReplicatedTensor across nodes after its construction. Although we defined proper inter-op rules to make sure ReplicatedTensor stays the same, there's no enforcement on it (i.e. if you manually modify content on some ranks, the modified value will not automatically get synced to other nodes). If you wish to manually validate tensors are the same across ranks, use `validate()`. _process_groupNcCs>|dkrtd}tj|||j}|dk r0|nt|_|S)Nr)torchemptyTensorZ_make_subclass requires_gradrZ_get_default_groupr)clsdataZ process_grouprrN/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/_shard/replicated_tensor.py__new__(s zReplicatedTensor.__new__cCsHt||kr|t|St||jjtjd|j}||t|<|SdS)N)Z memory_format)idtyper clonerZpreserve_formatr)selfmemoresultrrr __deepcopy__3s    zReplicatedTensor.__deepcopy__csdtt|dS)NzReplicatedTensor())superr__repr__r __class__rrr;szReplicatedTensor.__repr__rc sdkr idddfdd}D]}||\}}|r4|Sq4dk rD] \} } || \}}|rb|Sqbtjv} tkr| W5QRSt| tjot| t } pڈoڈtk} | r| r| t} | _ | W5QRSQRXdS)NTcstt|trd|fSt|trVdkr8|jqp|jkrptdjdnt|tjrlddnddS)NTzOReplicatedTensor operands must be in the same process group in torch function 'zQ', but found at least two ReplicatedTensor operands in different process groups! F)FN) isinstancer__torch_function__rr RuntimeError__name__rr )argZall_replicatedargsfunckwargsZ replicated_pgZreplicated_with_non_tensortypesrr dispatch_argJs     z9ReplicatedTensor.__torch_function__..dispatch_arg) itemsr_CDisableTorchFunctionrrr r%_REPLICATED_WITH_NON_TENSOR_ALLOWLISTZ as_subclassr)r r%r'r$r&r(r"Z redispatchedreskvrsZresult_not_replicatedZshould_convert_to_replicatedrr#rr>s6         z#ReplicatedTensor.__torch_function__)returncsvtj}tj}fddt|D}tj|jdt|D](\}}t|sHt d|d|qHdS)a Validate the ReplicatedTensor is legit by all gathering tensors on all ranks and check to make sure they are the same. If there's some ranks with different values, a ValueError will be raised. Keyword args: process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Returns: True if validation succeed. csg|]}tqSr)rZ empty_like).0_rrr sz-ReplicatedTensor.validate..)groupz/ReplicatedTensor have different values on rank z and T) distZget_world_sizerZget_rankrangeZ all_gather enumeraterZallclose ValueError)rZ world_sizeZ current_rankZtensors_on_rankZrankZtensorrrrvalidates   zReplicatedTensor.validatec Cs<tj(||_|j|_ddlm}||_W5QRXdS)Nr)_get_current_process_group)rr*r+r r Ztorch.distributed._shard.apir;r)rstater;rrr __setstate__s   zReplicatedTensor.__setstate__cCs|jS)N)r rrrr __getstate__szReplicatedTensor.__getstate__)NN)rN)r! __module__ __qualname____doc__rZ ProcessGroup__annotations__ __slots__rrr classmethodrboolr:r=r> __classcell__rrrrrs     Dr)rZtorch.distributedZ distributedr6Z+torch.distributed._shard.sharded_tensor.apirrZtorch.overridesrZ unsqueezer __getitem__r,rrrrrs