U Jc)@sddlZddlZddlZddlZddlmZmZddlmZddl m Z m Z m Z m Z ddlmZmZmZddlmZmZddlmZmZmZmZdd lmZmZdd lmZmZe e!Z"Gd d d eZ#eed ddZ$eed ddZ%ee e#efd ddZ&dS)N) b64decode b64encode) timedelta)AnyOptionalTuplecast) FileStoreStoreTCPStore) NodeStateconstruct_and_record_rdzv_event)RendezvousConnectionErrorRendezvousErrorRendezvousParametersRendezvousStateError)RendezvousBackendToken)_matches_machine_hostnameparse_rendezvous_endpointc@seZdZUdZdZeed<eed<eeddddZe ed d d Z e e e efd d d Zde e ee e e eefdddZeedddZe e e e efdddZdS)C10dRendezvousBackendzRepresents a C10d-backed rendezvous backend. Args: store: The :py:class:`torch.distributed.Store` instance to use to communicate with the C10d store. run_id: The run id of the rendezvous. Z Y2FuaW1hZGFt_store_keyN)storerun_idreturncCs4|s td||_d||_|d|jd|jdS)Nz&The run id must be a non-empty string.ztorch.rendezvous. compare_set) ValueErrorrr _call_store_NULL_SENTINEL)selfrrr#`/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py__init__2s  zC10dRendezvousBackend.__init__)rcCsdS)See base class.Zc10dr#)r"r#r#r$nameBszC10dRendezvousBackend.namecCs|d|j}||S)r&get)r r _decode_state)r" base64_stater#r#r$ get_stateGszC10dRendezvousBackend.get_state)statetokenrc Cst|}|rDt|ts:|}|dk r6|d}|SdS|}n|j}|d|j||}||}|dkrrdS|\}} || ||kfS)r&NFr)F) rdecode isinstancebytesr+r!r rr)) r"r,r-Zbase64_state_strresulttmpr*Zstate_token_pairZ new_state new_tokenr#r#r$ set_stateMs     zC10dRendezvousBackend.set_state)store_oprc OsLzt|j|||WStttfk rF}ztd|W5d}~XYnXdS)NMThe connection to the C10d store has failed. See inner exception for details.)getattrrr RuntimeError TimeoutErrorr)r"r5argskwargsexcr#r#r$r osz!C10dRendezvousBackend._call_store)r*rc CsV||jkrdSz t|}Wn.tjk rL}ztd|W5d}~XYnX||fS)Nz=The state object is corrupt. See inner exception for details.)r!encoderbinasciiErrorr)r"r*r,r<r#r#r$r)ws z#C10dRendezvousBackend._decode_state)N)__name__ __module__ __qualname____doc__r!r __annotations__strr%propertyr'rrr0rr+boolr4rr r)r#r#r#r$r!s   "r)paramsrc Cst|jdd\}}|d}|dk r*|}nt|}tt|dd}|dkrTtd|dfD]}zPt|||t |d d }|rd t d }t |j |tjd t|WqWq\tttfk r} z|r|dk rtd| W5d} ~ XYq\Xq\|S)Nir) default_portis_host read_timeout<rz,The read timeout must be a positive integer.F)seconds)Z is_mastertimeoutzProcess z5 hosts the TCP store for the C10d rendezvous backend.)rmessage node_stater6)rendpointZ get_as_boolrrintZ get_as_intrr rosgetpidr rr ZINITloginfor8r9r) rHhostportZ cfg_is_hostrJrKZ is_serverrmsgr<r#r#r$_create_tcp_storesB     rZc Cs|jr|j}n>zt\}}Wn,tk rJ}ztd|W5d}~XYnXz t|}Wn0ttfk r}ztd|W5d}~XYnX|S)NzMThe file creation for C10d store has failed. See inner exception for details.r6) rQtempfilemkstempOSErrorrr rr8r)rHpath_r<rr#r#r$_create_file_stores& r`c Cs|dd}z<|dkr(t|}n|dkr:t|}ntdt||j}WnJtk r}z,t t |j dt ||jt jdW5d}~XYnX||fS)a Creates a new :py:class:`C10dRendezvousBackend` from the specified parameters. +--------------+-----------------------------------------------------------+ | Parameter | Description | +==============+===========================================================+ | store_type | The type of the C10d store. The currently supported types | | | are "tcp" and "file" which correspond to | | | :py:class:`torch.distributed.TCPStore` and | | | :py:class:`torch.distributed.FileStore`, respectively. | | | Defaults to "tcp". | +--------------+-----------------------------------------------------------+ | read_timeout | The read timeout, in seconds, for store operations. | | | Defaults to 60 seconds. | | | | | | Note this only applies to | | | :py:class:`torch.distributed.TCPStore`. It is not relevant| | | to :py:class:`torch.distributed.FileStore` which does not | | | take in timeout as a parameter. | +--------------+-----------------------------------------------------------+ | is_host | A boolean value indicating whether this backend instance | | | will host the C10d store. If not specified it will be | | | inferred heuristically by matching the hostname or the IP | | | address of this machine against the specified rendezvous | | | endpoint. Defaults to ``None``. | | | | | | Note that this configuration option only applies to | | | :py:class:`torch.distributed.TCPStore`. In normal | | | circumstances you can safely skip it; the only time when | | | it is needed is if its value cannot be correctly | | | determined (e.g. the rendezvous endpoint has a CNAME as | | | the hostname or does not match the FQDN of the machine). | +--------------+-----------------------------------------------------------+ store_typeZtcpfilez?Invalid store type given. Currently only supports file and tcp.z: )rOrrPN)r(striplowerr`rZrrr Exceptionr typer@rEr ZFAILED)rHrarbackender#r#r$create_backends %  ri)'r>loggingrSr[base64rrdatetimertypingrrrrZtorch.distributedr r r Z torch.distributed.elastic.eventsr r apirrrrZdynamic_rendezvousrrutilsrr getLoggerr@rUrrZr`rir#r#r#r$s   d1