U JcH@sUddlmZddlmZddlmZmZmZmZm Z m Z m Z ddl Z ddl mZddlZddlZddlZddlmZddlmZddlmZddlmZddlmmmZdd lmZmZdd l m!Z!m"Z"d d l#m$Z$m%Z%d d l&m'Z'd dl(m)Z)m*Z*d dl+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4e5Z6da7ia8de9d<iZ:de9d<iZ;de9d<ddddddZGddde>Z?d d!d"d#Z@dS)$) annotations) dataclass)CallableDictListOptionalSequenceTuplecastN)reduce)rpc)distributed_c10d) ShardMetadata)_dispatch_custom_op_has_custom_op) check_tensor(validate_non_overlapping_shards_metadata)TensorPropertiesShardedTensorMetadata)Shard)reshuffle_local_shardreshard_local_shard)_flatten_tensor_size!_parse_and_validate_remote_device"_validate_output_tensor_for_gather build_metadata_from_local_shardsbuild_global_metadata)_remote_device)tree_mapz1Dict[int, 'weakref.ReferenceType[ShardedTensor]']_sharded_tensor_mapzDict[Callable, Callable] _SHARDED_OPS_CUSTOM_SHARDED_OPSintList[rpc.RRef[Shard]])sharded_tensor_idrrefsrpc_rankc Cs\tN|tkr&td|dtt|}|dkrBtdn |||W5QRXdS)Nz"Could not find sharded_tensor_id: z in map: z*ShardedTensor weakref has been deallocated)_sharded_tensor_lockr RuntimeErrorkeys_register_remote_shards)r%r&r'sharded_tensorr-O/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/_shard/sharded_tensor/api.pyr+8s  r+c@sveZdZUded<ded<ded<dddd Zdd d d Zdd d dZedddddddZedddZ dS)ShardedTensorBaseshard_spec.ShardingSpec_sharding_specr _metadata List[Shard] _local_shards sharding_specc Ostjdt|tjs*tdt|t|}|d}|d}|d}|d}|dkrbt }t ||||d} |j || d} tj j ||||||d } || _| | _g| _| S) Nz'torch.distributed._shard.sharded_tensorz Expecting ShardingSpec but got: dtypelayout pin_memory requires_grad)r9tensor_propertiesr7r8r9r:)torchZ_CZ_log_api_usage_once isinstance shard_specZ ShardingSpec ValueErrortyperZget_default_dtyperbuild_metadataTensorZ_make_wrapper_subclassr1r2r4) clsr6sizekwargsZsizesr7r8r9r:r<sharded_tensor_metadatarr-r-r.__new__Is@   zShardedTensorBase.__new__returncCs|jS)z~ Returns a :class:`ShardedTensorMetadata` object corresponding to the metadata for the entire tensor. r2selfr-r-r.metadatarszShardedTensorBase.metadatacCs|jS)z Returns a list of :class:`Shard' corresponding to the local shards for this rank. Returns an empty list if the current rank does not host any shards for this Tensor. )r4rNr-r-r. local_shardsyszShardedTensorBase.local_shardsN'ShardedTensor'rQrHrLc Cs`|j}|j}t|dkr td|jtjkr4td|dkrHt|}n|}t j t ||j |j |j|j |jd}ddd}|D]} | j} | j} | j} | dk std | } | }||j| jd | d | std || jt| d | ||j | d| d ||| jd| ||j | j d| d ||j| jd| d q|t|t|t|j ||_|S)ab Initialize a ShardedTensorBase with local shards and a global ShardedTensorMetadata built on each rank. Warning: This API is experimental and subject to change. It does not do cross rank validations, and fully rely on the user for the correctness of sharded_tensor_metadata on each rank rz"shards_metadata must not be empty!0Only torch.strided layout is currently supportedNr=FcSsR|rdnd}||krNtd|d|d|d|d|d|d |d|d dS) Nztensor propertyzlocal ShardMetadatazLocal shards' tensor z property is incompatible with z on rank z:  =z, local shard tensor .)rA)expectedactualZ prop_namerankZ is_propertyZtensor_property_or_metadatar-r-r._raise_if_mismatchs  4zYShardedTensorBase._init_from_local_shards_and_global_metadata.._raise_if_mismatchz#Must specify placement for `Shard`!r8TAOnly torch.contiguous_format memory_format is currently supportedrFr9devicer7r:)F)shards_metadatar<lenrAr8r>stridedr@)_infer_sharding_spec_from_shards_metadata ShardedTensorrJrFr7r9r:rPtensor placementAssertionErrorrZr] is_contiguous shard_sizeslist is_pinnedrrr4)rErQrHr6r^r<specZsharded_tensor_baser[shardZ shard_metaZlocal_shard_tensorrdrZ local_devicer-r-r.+_init_from_local_shards_and_global_metadatas      z=ShardedTensorBase._init_from_local_shards_and_global_metadatar-cCs&td|jd|jd|jddS)NzA z- object is being used from c++ while calling rWzE but the there is no custom __torch_dispatch__ implementation for it.)r)__name__ __module__)rEfunctypesargsrGr-r-r.__torch_dispatch__sz$ShardedTensorBase.__torch_dispatch__)N)r-N) rnro __qualname____annotations__rJrPrQ classmethodrmrsr-r-r-r.r/Ds )lr/cseZdZdZddfdd Zdejddejdddddd d ZdRd d Z d dZ ddZ ddZ ddddZ dSddddddZejdfddddZddejdfddd d!Zddd"d#Zeddd$d%d&d'd(Zeddd$d)dd*d+d,d-d.d/ZedTd%d0d,d1fd2d3 Zddd4d5Zddd6d7d8Zd)dd9d:ZedUddd?d@ZdAddBdCdDZdEddFdGZdHdIZdJdKZe GdLdMdMZ!dNdOZ"dPdQZ#Z$S)Vrba ShardedTensor is an torch.Tensor subclass to represent Tensors that are sharded across multiple devices and multiple processes. ShardedTensor is initialized in an SPMD like fashion where each rank initializes the ShardedTensor. The ShardedTensor object on each rank then only stores the local shard for the Tensor and provides global metadata for all the shards. ShardedTensor doesn't provide any Tensor like operations but is a wrapper providing the Tensor representing the local shard and the global metadata. Using these, users can build their custom distributed._sharded computations on top of this primitive. The local shards are all initialized using the create_op specified by tensor_init_params.create_op, e.g., torch.ones, or torch.empty Args: sharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how to shard the Tensor. size (int...): a sequence of integers defining the shape of the output tensor. Can be a variable number of arguments or a collection like a list or tuple. Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned tensor. Default: if ``None``, uses a global default (see :func:`torch.set_default_tensor_type`). layout (:class:`torch.layout`, optional): the desired layout of returned Tensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned tensor. Default: ``False``. pin_memory (bool, optional): If set, returned tensor would be allocated in the pinned memory. Works only for CPU tensors. Default: ``False``. memory_format (:class:`torch.memory_format`, optional): the desired memory format of returned Tensor. Default: ``torch.contiguous_format``. init_rrefs (bool, optional): Whether or not to initialize :class:`torch.distributed.rpc.RRef`s pointing to remote shards. Need to initialize the RPC Framework if specified as ``True``. Default: ``False``. .. note:: ShardedTensor uses collectives to do various operations, i.e. it uses all_gather to do cross rank validations. For NCCL-based process groups, internal tensor representations of objects must be moved to the GPU device before communication takes place. In this case, the device used is given by ``torch.cuda.current_device()`` and it is the user's responsibility to ensure that this is set so that each rank has an individual GPU, via ``torch.cuda.set_device()`` r0r5cstt|j||f||}|SN)superrbrJ)rEr6rFrGrO __class__r-r.rJ%szShardedTensor.__new__NF)r7r8r:r9 memory_format process_group init_rrefscGs|j||d|tjkr td|tjkr2td||jj_t |j } |jj D]D} t |j | j \} } | | krPt| j| |jjd}|jt|| qP|dS)Nr|r}rTr\)rlr<) _prepare_initr>r`rAcontiguous_formatr2r<r{distget_rank_process_groupr^rrd_create_tensor_from_paramsrgr4appendr _post_init)rOr6r7r8r:r9r{r|r}rF current_rankshard_metadatarZr] local_tensorr-r-r.__init__)s"     zShardedTensor.__init__cCs,||_d|_|dk r|nt|_i|_dSrw) _init_rrefs_sharded_tensor_idr _get_default_groupr_remote_shards)rOr|r}r-r-r.rPszShardedTensor._prepare_initc CsP|jrLt$t|_t|t|j<td7aW5QRXtsDt d| dS)NrzgRPC Framework needs to be initialized using torch.distributed.rpc.init_rpc if init_rrefs is set to True) rr(_sharded_tensor_current_idrweakrefrefr r Z_is_current_rpc_agent_setr) _init_rpcrNr-r-r.r\szShardedTensor._post_initc Cs4t&t|dr&|jtkr&t|jW5QRXdS)Nr)r(hasattrrr poprNr-r-r.__del__ks zShardedTensor.__del__c Cs t}tj}||kr.td|d|i|_t}i}i}|D]}|j ||j<|j||j <qLtj |j }g}tj}t tD]\} | tkrqt|dkrdd|D} tj| t||| | |fd} || qtj|tj ddS)NzaDefault ProcessGroup and RPC ranks must be the same for ShardedTensor, found process group rank: z and RPC rank: rcSsg|]}t|qSr-)r ZRRef).0rkr-r-r. sz+ShardedTensor._init_rpc..)rr)rrr Zget_worker_infoidrArZ_get_current_rpc_agentZget_worker_infosnameapiZ _all_gatherrrangeget_world_sizer_rQZ rpc_asyncr+rr>ZfuturesZwait_all) rOZpg_rankr'Z worker_infosZ rank_to_nameZ name_to_rankZ worker_infoZall_tensor_idsZfutsrZr&Zfutr-r-r.rus:       zShardedTensor._init_rpcz torch.devicerKcCs.t|jtjjkr$ttjStdS)z Return the prefered device to be used when creating tensors for collectives. This method takes into account the associated process group cpu) r get_backendrZBackendZNCCLr>r]cudacurrent_devicerNr-r-r._get_preferred_devicesz#ShardedTensor._get_preferred_devicerr#zOptional[torch.Tensor]None)dstoutrLc sdd}t|j}|j}t||||}t|j}ddt|D}di} |j D]R} t t | j  } | dk st| || f| | <|| || 7<t|| qd||krdk stfddt|D} nd} tVtj|d} |D]8}|j}| |jd }| ||||q W5QRXtj| | ||jd ||krpdSt tj| dk st|j}t|}|j D]r} | | \}}| |}||||| }|| j}}t|D] }||| j|| j|}q||qdS) a Creates a full :class:`Tensor` on rank ``dst`` by gathering all shards of the sharded tensor. The API needs to be called on all ranks in SPMD fashion. All ranks should have the same ``dst``. ``out`` should be a tensor of the same size as the overall size of the sharded tensor on ``dst`` and ``None`` on all other ranks. Args: dst(int): The rank where full tensor is constructed. Default: 0 out (:class `torch.Tensor`, optional): The output full tensor. Must to be provided ONLY on ``dst`` rank. Default: ``None`` cSstdd|jS)NcSs||Srwr-)xyr-r-r.z:ShardedTensor.gather..shard_size..)r rg)shard_mdr-r-r. shard_sizesz(ShardedTensor.gather..shard_sizecSsg|]}dqS)rr-r_r-r-r.rsz(ShardedTensor.gather..rNcsg|]}tjfjdqS)r])r>emptyr]rZ max_rank_sizerr-r.rsrr)rc gather_listrgroup)rrrrPrFrrQrrr^r rrdrZremaxr>Zno_gradrrrcflattenZnumelZcopy_gatherrDr_viewrgZnarrowZ shard_offsets)rOrrrrZZ full_sizerQ world_sizeZ rank_sizesZshard_placementrZ shard_rankrdatarksrcZ shard_offsetZdimsZ rank_offsetrcZout_narrow_viewdimr-rr.rsd       (      zShardedTensor.gatherc Cs|tjkr|tjkrtdd}|jD]}||jjdkM}q*|rL|Sg}|j D]<}|j j |d}t |j}td|j_|t||qVt |} | jD]"}|jjdkrtd|j_q|dkr|jn|} tj|| | |jd} | S)a Returns a copy of this object in CPU memory. If this ShardedTensor is already on CPU memory, then no copy is performed and original object is returned. .. note:: When moving a ShardedTensor from GPU to CPU, the ShardedTensor might need to be managed by a different type of ProcessGroup(i.e. ProcessGroupGloo), it is the user's responsiblity to explicitly pass in a new process_group that is compatible with CPU. GOnly `torch.contiguous_format` or `torch.preserve_format` is supported!Tr)r{NrHr|r})r>preserve_formatrr)rPr^rdr]rBr4rcrcopydeepcopy_devicerrrrbrmr) rOr{r|Z all_on_cpumeta list_shardsrkZ cpu_tensorrPst_metapgZst_cpur-r-r.rs:    zShardedTensor.cpucCs|tjkr|tjkrtd|dk r`t|tr8t|n|}t|tjrX|jtj ks`t dttj }g}|j D]:}|j j|||d}t |j} || j_|t|| qzt |} | jD]} | jjdkr|| j_q|dkr|jn|} tj|| | |jd} | S)a Returns a copy of this object in CUDA memory, if the original ShardedTensor is on CPU, we will move the local shard to the current GPU device of each process in a SPMD fashion. If this ShardedTensor is already on CUDA memory and local shards on each rank are already on current device, we still returns a new ShardedTensor object with new metadata, but no underlying data movements are performed. .. note:: When moving a ShardedTensor from CPU to GPU, the ShardedTensor might need to be managed by a different type of ProcessGroup(i.e. ProcessGroupNCCL), it is the user's responsiblity to explicitly pass in a new process_group that is compatible with GPU. rNzSOnly device without device id (e.g. "cpu" or "cuda") is expected for ShardedTensor!)r] non_blockingr{rr)r>rrr)r?strr]indexrrrer4rcrrrPrdrrrr^rBrrbrmr)rOr]rr{r|rrrkZ cuda_tensorrPrrrZst_cudar-r-r.r8sD     zShardedTensor.cudacOs<|jdjj}|j}|}|}t|dkrt|dtjrB|d}qt|dtjr\|d}qt|dttfr~t|d}qt|dtj r|dj}|dj}qt d|n.t|dkr|\}}n| d|}| d|}t|ttfrt|n|}|j dkrrr#rDr)getrBrrrwarningswarnrtorrrPrdrrrr<r^rrbrmr)rOrrrGrZ current_dtypeZ device_toZdtype_toZ current_idxrZ copy_tensorrr{r|rrkZ new_tensorrPrrrZst_tor-r-r.rxsr                      zShardedTensor.tor~r3)rQc Gs|dk r |nt}t|}t|}d}t|}t|dkrNt||||}g} |dkr~ddt|D} tj | ||dn|g} t | } | j } t | j} |j|| | j| j| j| j| jd} | j||d|| _| | S)NrrcSsg|]}dqSrwr-rr-r-r.rsz9ShardedTensor._init_from_local_shards..)rr=r~)r rrrrrr_rrZall_gather_objectrr<r@rar^rJrFr7r8r9r:rr4r)rErQr|r} global_sizerrZlocal_sharded_tensor_metadataglobal_tensor_sizeZgathered_metadatasZglobal_sharded_tensor_metadatar<rjr,r-r-r._init_from_local_shardssJ     z%ShardedTensor._init_from_local_shardsz torch.Tensorz Sequence[int]zdist.ProcessGrouprR)rr6rr|rLcGs|stdt|}t|j|j|jtj| d}| ||}|dk rN|nt }t |} g} |jD],} t|| j\} } | | krj| t|| qjtj| ||||dS)ah Initialize a ShardedTensor given only one local tensor, global sharded tensor size and sharding spec on each rank. Args: local_tensor (Tensor): Single tensor of local shard stored in each rank. sharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how to shard the Tensor. global_size (Sequence[int]): Size of the sharded tensor. process_group (ProcessGroup, optional): The process group to aggregate on. Default: None init_rrefs (bool, optional): Whether or not to initialize :class:`torch.distributed.rpc.RRef`s pointing to remote shards. Need to initialize the RPC Framework if specified as ``True``. Default: ``False``. Returns: A :class:`ShardedTensor` sharded based on the given sharding_spec with local tensor stored in the current rank. Examples: >>> # All tensors below are of torch.int64 type. >>> # We have 2 process groups, 2 ranks. >>> # xdoctest: +SKIP >>> tensor = torch.arange(2, dtype=torch.int64) + 1 + 2 * rank >>> local_tensor = torch.unsqueeze(torch.cat([tensor, tensor + 2])) >>> local_tensor tensor([[1, 2, 3, 4]]) # Rank 0 tensor([[3, 4, 5, 6]]) # Rank 1 >>> sharding_dim = 0 >>> sharding_spec = ChunkShardingSpec( dim=sharding_dim, placements=[ "rank:0/cuda:0", "rank:1/cuda:1", ], ) >>> st = ShardedTensor._init_from_local_tensor(local_tensor, sharding_spec, [2, 4]) >>> st ShardedTensor( ShardedTensorMetadata( shards_metadata=[ ShardMetadata(shard_offsets=[0, 0], shard_sizes=[1, 4], placement=rank:0/cuda:0), ShardMetadata(shard_offsets=[1, 0], shard_sizes=[1, 4], placement=rank:1/cuda:1), ], size=torch.Size([2, 4]) ) >>> st.local_tensor() tensor([1, 2, 3, 4]) # Rank 0 tensor([3, 4, 5, 6]) # Rank 1 Warning: This API is experimental and subject to change. It lacks of a fully across rank validations, and we only validate the local shard on the current rank. We fully rely on the user to ensure local tensor is sharded based on the sharding spec. z(local_tensor is not a contiguous Tensor.)r7r8r:r{r9N)r|r}r6)rfrArrr7r8r:r>rrirCr rrrr^rrdrrrbrm)rErr6r|r}rrr<rHrrQrrZr]r-r-r._init_from_local_tensors>A  z%ShardedTensor._init_from_local_tensorrrSc s|dk r |nt}t|}|j}g}|D]&} t|| j\} } || kr,|| q,t|t|krt dt|dt|d|dt t |j |||d} | j ||d| | S)a_ Initialize a ShardedTensor with local shards and a global ShardedTensorMetadata built on each rank. Warning: This API is experimental and subject to change. It does not do cross rank validations, and fully rely on the user for the correctness of sharded_tensor_metadata on each rank NzNumber of local shards (zM) does not match number of local shards metadata in sharded_tensor_metadata (z ) on rank (z) r5r~)r rrrr^rrdrr_r)rxrbrmrr) rErQrHr|r}r6rr^Zlocal_shard_metadatasrrZrlr,ryr-r.rmks4  z9ShardedTensor._init_from_local_shards_and_global_metadatacCs|jS)z: Returns the ShardingSpec for the tensor. )r1rNr-r-r.r6szShardedTensor.sharding_spec)resharding_specrLcCst|tjrt|jtjs"tdt|dkr:td|jj|jkr||jj|jkrZ|St | | |j||j \}}n t | | |j||j \}}||_||j_||_|S)a Reshard a sharded tensor given the ``resharding_spec``. For now, we only support single local shard. If ``resharding_spec`` is same as the original one, this becomes a no-op. If only ``resharding_spec`` shares the same sharding dim with the original one, we swap local shards directly. For more generic cases, we merge different shards across different ranks and split the local shards based on the ``resharding_spec`` via `all_to_all` collective API. Args: resharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how the tensor is sharded. Returns: A :class:`ShardedTensor` object whose local shards are resharded. Examples: >>> # We have 2 process groups, 2 ranks. >>> # xdoctest: +SKIP >>> tensor = torch.arange(4, dtype=torch.int64) + 1 + 2 * rank >>> tensor = torch.stack([tensor, tensor]) >>> tensor tensor([[1, 2, 3, 4], [1, 2, 3, 4]]) # Rank 0 tensor([[3, 4, 5, 6], [3, 4, 5, 6]]) # Rank 1 tensor([[5, 6, 7, 8], [5, 6, 7, 8]]) # Rank 2 tensor([[7, 8, 9, 10], [7, 8, 9, 10]]) # Rank 3 >>> sharding_dim = 0 >>> spec = ChunkShardingSpec( dim=sharding_dim, placements=[ "rank:0/cuda:0", "rank:1/cuda:1", "rank:2/cuda:2", "rank:3/cuda:3", ], ) >>> current_offsets = [0] * 2 >>> current_offsets[0] = rank * 2 >>> shard_metadata = ShardMetadata( shard_offsets=copy.deepcopy(current_offsets), shard_sizes=tensor.size(), placement=spec.placements[rank], ) >>> local_shards = [ Shard( tensor=tensor, metadata=shard_metadata, ) ] >>> st = ShardedTensor._init_from_local_shards(local_shards, tensor.size()) >>> sharding_dim = 1 >>> resharding_spec = ChunkShardingSpec( dim=sharding_dim, placements=[ "rank:0/cuda:0", "rank:1/cuda:1", "rank:2/cuda:2", "rank:3/cuda:3", ], ) >>> st.reshard(resharding_spec) >>> tensor = st.local_shards()[0].tensor >>> tensor tensor([[1], [1], [3], [3], [5], [5], [7], [7]]) # Rank 0 tensor([[2], [2], [4], [4], [6], [6], [8], [8]]) # Rank 1 tensor([[3], [3], [5], [5], [7], [7], [9], [9]]) # Rank 2 tensor([[4], [4], [6], [6], [8], [8], [10], [10]]) # Rank 3 z-Only ChunkShardingSpec supported for reshard.rz.Only single local shard supported for reshard.)r?r@ZChunkShardingSpecr1NotImplementedErrorr_rQrZ placementsrrrFrrr4r2r^)rOrrQr^r-r-r.reshards8G   zShardedTensor.reshardcCs&t|dkrtd|djS)z Return local tensor for a sharded_tensor. For now we only support single local shard. Returns: A :class:`torch.Tensor` of the local shard. rz%Only single local shard is supported.r)r_rQrrcrNr-r-r.r szShardedTensor.local_tensorr-cspdddfdd }dfdd}t|t|dk rN||Std|jd d d dS) Nrbrstrpcs||tkrt||jSt|j|r>t|j||jS|tkrZt||jStd|jddddS)Ntorch function '', with args: and kwargs: ! not supported for ShardedTensor!)r"rrr1rr!r)rnr)rrrGrqr-r.dispatchs   z2ShardedTensor.__torch_function__..dispatchcsdkrt|tr|dSrw)r?rb)e) st_instancer-r.find_sharded_tensor2sz=ShardedTensor.__torch_function__..find_sharded_tensorrrrr)rr)rn)rErprqrrrGrrr-)rrrGrrqr.__torch_function__s    z ShardedTensor.__torch_function__boolcCs |jjjS)za Returns True if the sharded tensor (each local shard) resides in pinned memory. )r2r<r9rNr-r-r.riAszShardedTensor.is_pinnedr$) remote_shardsr'cCs||j|<dSrw)r)rOrr'r-r-r.r+Gsz%ShardedTensor._register_remote_shardsz Dict[int, List[rpc.RRef[Shard]]]cCs|jstd|jS)a Returns a Dict[int, RRef] with keys being the RPC rank and values being RRefs to shards on that rank. Need to initialize the RPC framework for this functionality. Raises an exception if ShardedTensor was created with ``init_rrefs=False`` zPShardedTensor created with init_rrefs=False, no RRefs to remote shards available)rr)rrNr-r-r.rJs zShardedTensor.remote_shardscCst|Srw)rrNr-r-r.__hash__XszShardedTensor.__hash__cCsd|jdS)NzShardedTensor()rMrNr-r-r.__repr__[szShardedTensor.__repr__c@s2eZdZUdZded<ded<ded<ded<dS)zShardedTensor.ProcessGroupStatez3 State for ser-de of process group r# local_rank global_ranklocal_world_sizeglobal_world_sizeN)rnrort__doc__rur-r-r-r.ProcessGroupState^s rcCs>tt|jtt|jt}|j|j||j|j fSrw) rbrr rrrr4r2r1r)rOpg_stater-r-r. __getstate__hs  zShardedTensor.__getstate__cCsd|_tstd|\|_|_}|_|_ddlm }||_ t |j }|j |krltd|j d|t }|j |krtd|j d|t|j }|j|krtd|jd|t}|j|krtd|jd||dS) Nz`Need to initialize default process group using "init_process_group" before loading ShardedTensorr)_get_current_process_groupzLocal rank at save time was z, but at load time was zGlobal rank at save time was z"Local world size at save time was z#Global world size at save time was )rr Zis_initializedr)r4r2r1rZtorch.distributed._shard.apirrrrrrrrr)rOstaterrrrrrr-r-r. __setstate__rs:       zShardedTensor.__setstate__)NF)rN)NFN)r-N)%rnrortrrJr>r`rrrrrrrrrrrrrvrrrmr6rrrrir+rrrrrrr __classcell__r-r-ryr.rbsl/'  - X9@N=f3f  *  rbrr;c Gs8|j}|j}|j}|j}|j}tj|||||||dS)zA Helper to construct tensor from size, device and common params. )r7r8r]r:r{r9)r7r8r:r{r9r>r)rlr<rFr7r8r:r{r9r-r-r.rsr)A __future__rZ dataclassesrtypingrrrrrr r r functoolsr r threadingr>Ztorch.distributedZ distributedrr r Z!torch.distributed._shard.metadatarZ&torch.distributed._shard.sharding_specZ_shardr6r@Z*torch.distributed._shard.sharding_spec.apirrZ1torch.distributed._shard.sharding_spec._internalsrrrPrrrkrrrrutilsrrrrrZtorch.distributed.remote_devicerZtorch.utils._pytreerLockr(rr rur!r"r+rDr/rbrr-r-r-r.sL  $             2-