U <ºcºwã@sJddlZddlZddlZddlZddlZddlZddlZddlm Z ddl mZddlmZddlm Z ddlmZddlZddlmZddlmZddlmmZddlmmZddlmZmZmZmZddlm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)ddl*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2dd l3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:ddl;mm?Z@dd„ZAdÃdd„ZBdddddddgZCdZDejE )¡ZFejE )¡ZGdZHdaIdd „ZJd!d"„ZKGd#d$„d$ƒZLd%d&„ZMd'd(„ZNd)d*„ZOd+d,„ZPd-d.„ZQe d/d0gƒZRGd1d2„d2ƒZSGd3d4„d4ƒZTGd5d6„d6ƒZUd7d8„ZVd9d:„ZWd;d<„ZXd=d>„ZYdÄd?d@„ZZdAdB„Z[dCdD„Z\dEdF„Z]dGdH„Z^dIdJ„Z_dÅdLdM„Z`dNdO„ZadPdQ„ZbdÆdSdT„ZcdUdV„ZddWdX„ZedYdZ„Zfd[d\„Zgd]d^„Zhd_d`„Zidadb„Zjdcdd„Zkdedf„Zldgdh„Zmdidj„Zndkdl„Zodmdn„Zpdodp„Zqdqdr„Zrejsjtdsdt„ƒZuejsjtdudv„ƒZvdwZwdxdy„Zxejsjteyejzdzœd{d|„ƒZ{d}Z|d~d„Z}da~d€d„Zd‚dƒ„Z€d„d…„Zd†d‡„Z‚dˆd‰„ZƒdŠd‹„Z„dŒd„Z…dŽd„Z†dd‘„Z‡ejˆj‰d’d“„ƒZŠejˆj‰d”d•„ƒZ‹ejˆj‰d–d—„ƒZŒdÇd™dš„Zejˆj‰dÈd›dœ„ƒZŽejˆj‰ddž„ƒZejˆj‰dŸd „ƒZejˆj‰d¡d¢„ƒZ‘ejˆj‰d£d¤„ƒZ’ejˆj‰d¥d¦„ƒZ“ejˆj‰d§d¨„ƒZ”ejˆj‰d©dª„ƒZ•Gd«d¬„d¬ƒZ–Gdd®„d®ƒZ—d¯d°„Z˜Gd±d²„d²ej™ƒZše0Z0Gd³d´„d´ejj›ƒZœGdµd¶„d¶ƒZGd·d¸„d¸ej›ƒZžGd¹dº„dºƒZŸGd»d¼„d¼eosÿz0StubRpcAgent.get_worker_infos..)ÚrangerG©rHr*r*r+Úget_worker_infosnsþzStubRpcAgent.get_worker_infosN)Ú__name__Ú __module__Ú__qualname__rIrRr*r*r*r+rDjsrDcKst ¡SrE)rZMock©Úkwargsr*r*r+Ú+_stub_construct_rpc_backend_options_handlerusrXcCs t|dS)NrF)rD)ÚstorerJrNrGÚrpc_backend_optionsr*r*r+Ú_stub_init_rpc_backend_handlerysr[cCst |¡dSrE)ÚVALUE_FUTUREÚ set_result©Úvaluer*r*r+Ú set_value}sr`cCst ¡SrE)r\Úresultr*r*r*r+Úwait_for_value_futuresrbcCst |¡t ¡SrE)r\r]ÚDONE_FUTURErar^r*r*r+Úset_and_check_done…s rdÚTensorClassÚtensorsc@s,eZdZdd„Zdd„Zdd„Zdd„Zd S) Ú MyPickleClasscCs d|_dSrE©r:rQr*r*r+rIszMyPickleClass.__init__c Cs2t ttt dd¡t dd¡fdƒ¡\}}||fS©Né)rÚ serializerÚmy_tensor_functionr'r))rHZpickled_python_udfrfr*r*r+Ú__getstate__“sÿzMyPickleClass.__getstate__cCs6t |d|d¡}| |jd|jd¡}||_dS©Nrr&)rZdeserializeÚfuncÚargsr:)rHÚobjZ python_udfrar*r*r+Ú__setstate__™szMyPickleClass.__setstate__cCs ||_dSrErh)rHÚvalr*r*r+ÚsetžszMyPickleClass.setN)rSrTrUrIrmrrrtr*r*r*r+rgsrgc@s$eZdZdd„Zdd„Zdd„ZdS)ÚSlowPickleClasscCs ||_dSrErh©rHr:r*r*r+rI£szSlowPickleClass.__init__cCst |j¡|jfSrE)ÚtimeÚsleepr:rQr*r*r+rm¦szSlowPickleClass.__getstate__cCs|d|_t |j¡dSrB)r:rwrx)rHrqr*r*r+rrªs zSlowPickleClass.__setstate__N)rSrTrUrIrmrrr*r*r*r+ru¢sruc@sNeZdZddd„Zdd„Zedd„ƒZedd „ƒZd d„Z dd „Z dd„ZdS)ÚMyClassFcCs||_|rt d¡dSri)Úarwrx)rHrzÚdelayr*r*r+rI°szMyClass.__init__cCs |j|SrE©rz)rHÚbr*r*r+Úmy_instance_method¶szMyClass.my_instance_methodcCs||SrEr*)ÚclsÚdÚer*r*r+Úmy_class_method¹szMyClass.my_class_methodcCs|dkS©Né r*)Úfr*r*r+Úmy_static_method½szMyClass.my_static_methodcCs|j|7_dSrEr|)rHÚ incrementr*r*r+Úincrement_valueÁszMyClass.increment_valuecCs|jSrEr|rQr*r*r+Ú get_valueÄszMyClass.get_valuecCst d¡t |j|¡S©Né)rwrxr'r(rz)rHZ my_tensor_argr*r*r+Úmy_slow_methodÇs zMyClass.my_slow_methodN)F)rSrTrUrIr~Úclassmethodr‚Ústaticmethodr†rˆr‰rŒr*r*r*r+ry¯s rycOs|| ¡f|ž|ŽSrE)Úlocal_value)ÚmethodÚrrefrprWr*r*r+Ú_call_method_on_rrefÌsr’cCsdd„|DƒS)NcSsg|]}tt|ƒƒ‘qSr*)rry)rMrzr*r*r+Ú Ñsz!get_rref_list..r*)Úvaluesr*r*r+Ú get_rref_listÐsr•cCs| ¡|SrE©Úto_here)r‘r_r*r*r+Úadd_rref_to_valueÔsr˜cCs |j|SrErh)Zpickle_cls_instanceÚtensorr*r*r+Úrun_nested_pickleØsršcCs<dddgdddgg}dddg}t ||d¡}|r8| ¡}|S)Nrr&rjéér‹)rjr›)r'Zsparse_coo_tensorÚcoalesce)rÚiÚvr™r*r*r+Úbuild_sparse_tensorÛs r cCs:t dd¡}||g}||g}||g}||i}|||||gS©Nr›)r'r))rzr}Úcr€rr*r*r+Úbuild_complex_tensorsãsr£cCs8| ¡rtdƒ‚| ¡s tdƒ‚t ||¡s4tdƒ‚|S)Nzt_view is contiguous!zt_cont is not contiguous!zt_view is not equal to t_cont!)Ú is_contiguousÚ Exceptionr'Úequal)Út_viewÚt_contr*r*r+Ú non_cont_testësr©cCs|||SrEr*©rzr}r¢r*r*r+Úmy_functionôsr«cCs||SrEr*)rzr}r*r*r+rløsrlcCs&|d}|dd…D]}||7}q|Srnr*)rzrar™r*r*r+Úmy_container_sumûs r¬r&cCs"t |¡t t d¡t d¡¡Sr%)rwrxr'r6r™)Úsecondsr*r*r+Ú my_sleep_funcs r®cCsR|d}|D]}||7}q| ¡D]\}}||7}q"|j}||d|d|dfS)Nrr&rj)Úitemsrf)Z list_inputZtensor_class_inputZ dict_inputÚresr:ÚkrŸZcomplex_tensorsr*r*r+Úmy_complex_tensor_functions r²cCs| ¡| ¡SrEr–)Úrref_aÚrref_br*r*r+Úmy_rref_functionsrµçš™™™™™©?cCst |¡||SrE©rwrx)rzr}rr*r*r+Údelayed_adds r¸cCs|SrEr*r|r*r*r+Úidentitysr¹cCstdƒdS)Nz do nothing)Úprintr*r*r*r+Ú no_resultsr»cCs| ¡dkrtdƒ‚|dS)NrjúExpected errorr&)ZnumelÚ ValueErrorr^r*r*r+Úraise_or_inc sr¾cCstj|tjt dd¡dfdS©Nrjr&©rp)ÚrpcÚrpc_syncr'r(r)©Údstr*r*r+Ú nested_rpc%srÅcCstj|tjtƒtƒfdS©NrÀ)rÁrÂr'r(r rÃr*r*r+Únested_rpc_sparse)s ýrÇcCs>|dkr:t|ƒ}|d|}tj|t|||dfddSdS©Nrr&rÀ)r!rÁÚ rpc_asyncÚmulti_layer_nested_async_rpc)rÄrGÚttlÚcurrent_dstÚnext_dstr*r*r+rÊ1sýrÊcCs<tj|tjt dd¡dfdtj|tjt dd¡dfdfSr¿©rÁÚremoter'r(r)rÃr*r*r+Únested_rref?sþrÐcCs4tj|tjtƒtƒfdtj|tjtƒtƒfdfSrÆ)rÁrÏr'r(r rÃr*r*r+Únested_rref_sparseFs ý ýúrÑcCs&tj|tjt dd¡dfd}| ¡S)Nrjr›rÀ)rÁrÏr'r(r)r—©rÄr‘r*r*r+Ú nested_remoteUsrÓcCs"tj|tjtƒtƒfd}| ¡SrÆ)rÁrÏr'r(r r—rÒr*r*r+Únested_remote_sparseYsrÔcCsJ|dkr>t|ƒ}|d|}tj|t||||dfd}|gS| ¡SdSrÈ)r!rÁrÏÚrref_forward_chainr—)rÄrGr‘rËrÌrÍÚret_rrefr*r*r+rÕ^sÿrÕcCstj|tjt dd¡dfdSr¿rÎrÃr*r*r+Úrpc_return_rrefjsr×cCsdSrBr*r*r*r*r+Ú light_rpcnsrØcCs(tddƒD]}||9}||d}q dS©Nr&édr©rP©r™ržr*r*r+Ú heavy_rpcrsrÝcCs(tddƒD]}||9}||d}q dSrÙrÛrÜr*r*r+Úheavy_rpc_sparseysrÞcCs(tddƒD]}||9}||d}q dSrÙrÛrÜr*r*r+Úheavy_rpc_torchscriptsrßcCst ||¡SrE)r'r(©r™r*r*r+Úmy_script_func‡srár¼cCsttƒ‚dSrE)r½Úexpected_errr*r*r*r+Ú raise_funcsrã)râÚreturncCst|ƒ‚dSrE)r½)râr*r*r+Úraise_func_scriptsråz> First line of error next line of error last line of errorcCsttƒ‚dSrE)r½Úexpected_err_escaper*r*r*r+Úraise_func_escape•srçcCs|adSrE©Úglobal_rref©r‘r*r*r+Úset_global_rrefœsrëcCsdadSrErèr*r*r*r+Úclear_global_rref¡srìcCs| ¡SrE)Zconfirmed_by_ownerrêr*r*r+Úcheck_rref_confirmed¦srícCstƒSrE)r r*r*r*r+Úget_rref_debug_infoªsrîcs@tj ¡‰‡‡fdd„}tj|tj||fd}| |¡ˆ ¡S)Ncsˆ | ¡ˆ¡dSrE©r]Úwait©Úfut©ÚoutÚzr*r+Úcallback±sz#add_use_future_cb..callbackrÀ) Ú concurrentÚfuturesrrÁrÉr'r(Úthenra)ÚtoÚxÚyrõröròr*rór+Úadd_use_future_cb®s rýcCs | ¡jSrE)rZprocess_global_function_events)Zprofile_rrefr*r*r+Úget_events_from_profile¹srþcs<tj ¡‰tj|tj||fd}| ‡‡fdd„¡ˆ ¡S)NrÀcsˆ | ¡ˆ¡SrErïrñrór*r+ÚÀóz+add_use_future_set_result..©r'rørrÁrÉr(rùrð)rúrûrürõròr*rór+Úadd_use_future_set_result½s rcsBtj ¡‰‡‡‡fdd„}tjˆtj||fd}| |¡ˆ ¡S)Ncs0tjˆtj| ¡ˆfd}| ‡fdd„¡dS)NrÀcsˆ | ¡¡SrErï)Úfut2)rôr*r+rÿÉrz.callback..©rÁrÉr'r(rðrù)Úfut1r©rôrúrõr*r+röÇsz*add_use_future_nested_cb..callbackrÀr)rúrûrürõrörr*rr+Úadd_use_future_nested_cbÄs rcCsdSrEr*rñr*r*r+Úfail_on_futÐsrcCstdƒ‚dS)Nr¼©ÚRuntimeErrorr*r*r*r+Úasync_raise_funcÔsrcCst dd¡Sri)r'Úzerosr*r*r*r+Úasync_wrong_typeÙsr cCstj|tj||fdSrÆ)rÁrÉr'r()rúrûrür*r*r+Ú async_addÞsrÚcpucCs.t d¡| |¡}| |¡}t ||¡ ¡Sr%)rwrxrúr'r(r)rûrür2r*r*r+Úslow_addãs rcCstj|t|||fdSrÆ)rÁrÉr)rúrûrür2r*r*r+Úslow_async_addêsrcs4tj ¡‰tj|tj||fd ‡‡fdd„¡ˆS)NrÀcsˆ | ¡ˆ¡SrErï©r©ròrõr*r+rÿórz,async_add_with_future_ctor..©r'rørrÁrÉr(rù©rúrûrürõr*rr+Úasync_add_with_future_ctorïs ÿrcs$tj|tj||fd ‡fdd„¡S)NrÀcs| ¡ˆSrE©rðrñ©rõr*r+rÿûrz#async_add_chained..©rÁrÉr'r(rùrr*rr+Úasync_add_chainedøs ÿrcs:tj|tj|dfd}t|ƒD]}| ‡fdd„¡}q|S)NrrÀcs| ¡ˆSrErrñ©Ústepr*r+rÿrz)async_add_chained_multi..)rÁrÉr'r(rPrù)rúrûÚnumrròÚ_r*rr+Úasync_add_chained_multiÿsrcs$tj|t|||fd ‡fdd„¡S)NrÀcs| ¡ˆSrErrñrr*r+rÿ rz"async_add_nested..)rÁrÉrrùrr*rr+Úasync_add_nesteds ÿr csžg‰t|ƒD]F}|dkr6ˆ tj|tj||fd¡qˆ tj|tjd|fd¡qtƒ‰dt |¡dœ‰tj ¡‰‡‡‡‡fdd„}ˆD]}| |¡qŠˆS)NrrÀ)ÚcntÚretc sVˆHˆdd7<ˆd| ¡7<ˆdtˆƒkrHˆ ˆd¡W5QRXdS)Nr!r&r")rðÚlenr]rñ©ÚfutsÚlockZ ret_futureÚstater*r+Úinc_and_sets z+async_add_multi_fanout..inc_and_set)rPÚappendrÁrÉr'r(rZ zeros_likerørrù)rúrûrrržr(ròr*r$r+Úasync_add_multi_fanouts r*c Cs†|j}tj |¡}tj |¡}| |¡tj |¡Htj tdt ƒƒ¡| d¡t|gd}| |¡|W5QR£SQRXdS)Néèr&©Údevices) r2r'r1Úcurrent_streamÚStreamÚwait_streamÚstreamÚ_sleepÚintrÚfill_rr])r:r2Zoriginal_streamZ new_streamròr*r*r+Úasync_cuda_sleep_and_set_to_one)s r5cs(‡fdd„}tj|tj||fd |¡S)Ncs"tj tdtƒƒ¡| ¡ˆS)Nr+)r'r1r2r3rr_rñrr*r+Úcb9sz!async_cuda_nested_add..cbrÀr)rúrûrürõr6r*rr+Úasync_cuda_nested_add7sr7c@s(eZdZdZdd„Zdd„Zdd„ZdS) Ú TensorWrapper)r™r&ÚeventÚthreadcCs6||_tƒ|_tjjdd|_t ¡|_ |j ¡dS)NT)Z enable_timing)r™rr&r'r1rr9Ú threadingÚThreadr:Ústartrvr*r*r+rIEs zTensorWrapper.__init__c Cs$|j|j|7_W5QRXdSrE)r&r™)rHrŸr*r*r+ÚincreaseMszTensorWrapper.increasec Cs2|j"|j ¡|j ¡W5QR£SQRXdSrE)r&r9Úrecordr™ÚsumrQr*r*r+r@Qs zTensorWrapper.sumN)rSrTrUÚ __slots__rIr>r@r*r*r*r+r8Bsr8c@sDeZdZeejjdd„ƒƒZeejjdd„ƒƒZ ejjdd„ƒZ dS)ÚAsyncExecutionClasscs$tj|tj||fd ‡fdd„¡S)NrÀcs| ¡ˆSrErrñrr*r+rÿ]rz6AsyncExecutionClass.static_async_add..rrr*rr+Ústatic_async_addYs ÿz$AsyncExecutionClass.static_async_addcs4tj ¡‰tj|tj||fd ‡‡fdd„¡ˆS)NrÀcsˆ | ¡ˆ¡SrErïrñ©Zret_futrõr*r+rÿerz5AsyncExecutionClass.class_async_add..r)rrúrûrürõr*rDr+Úclass_async_add`s ÿz#AsyncExecutionClass.class_async_addcs$tj|tj||fd ‡fdd„¡S)NrÀcs| ¡ˆSrErrñrr*r+rÿlrz5AsyncExecutionClass.bound_async_add..r)rHrúrûrürõr*rr+Úbound_async_addis ÿz#AsyncExecutionClass.bound_async_addN)rSrTrUrŽrÁÚ functionsÚasync_executionrCrrErFr*r*r*r+rBWsrBcCs tj ¡SrE)r'rørr*r*r*r+Ú return_futurepsrIc@seZdZdd„ZdS)ÚFooBackendOptionscCstj |¡||_dSrE)rÁÚRpcBackendOptionsrIÚinit_method)rHrLr*r*r+rIuszFooBackendOptions.__init__N)rSrTrUrIr*r*r*r+rJtsrJcs$eZdZ‡fdd„Zdd„Z‡ZS)ÚMyEmbeddingBagModelcs"tƒ ¡tjjdd|d|_dS)Nr„©Úsparse)ÚsuperrIr'ÚnnZEmbeddingBagÚeb)rHrO©Ú __class__r*r+rI‚s ýzMyEmbeddingBagModel.__init__cCs | |¡SrE)rR©rHrûr*r*r+ÚforwardŠszMyEmbeddingBagModel.forward©rSrTrUrIrVÚ __classcell__r*r*rSr+rMsrMc@s4eZdZdd„Zedd„ƒZeejjdd„ƒƒZ dS)ÚMyParameterServercCs0tƒ|_||_d|_d|_g|_d|_d|_dSrB)rr&ÚtrainersÚ iterationÚupdatesrøÚtotalÚgradient)rHrZr*r*r+rIszMyParameterServer.__init__cCs | ¡jSrE)rr^rêr*r*r+Úget_gradient˜szMyParameterServer.get_gradientc CsÆ| ¡}tj ¡}|j¤||jkr:||_d|_|j ¡|j |¡|j dkrX||_ n|j |7_ |jd7_|j |jkr¸|j t|j ƒ|_|jD]}|j t|j ƒ}| |¡q˜W5QRX|Srn)rr'rørr&r[r\Úclearr)r]rZÚfloatr^r])r‘Z riterationr™rHròrar*r*r+Úaverageœs$ zMyParameterServer.averageN) rSrTrUrIrŽr_rÁrGrHrbr*r*r*r+rYŽs rYcs.eZdZ‡fdd„Zddd„Zdd„Z‡ZS) ÚMyConvNetForMNISTcsvtƒ ¡t t dddd¡t ¡t dddd¡t ¡t d¡t d¡t dd¡t ¡t dd¡¡ |¡|_ ||_dS) Nr&ér›é rjié€r„)rPrIrQZ SequentialZConv2dZReLUZ MaxPool2dZFlattenÚLinearrúÚnetr2)rHr2rSr*r+rIµs ÷ özMyConvNetForMNIST.__init__Fc CsX|r| ¡n|}tj tj |j¡¡(tj dt¡| |¡W5QR£SQRXdSrƒ) r—r'r1r1r.r2r2ÚFIFTY_MIL_CYCLESrh)rHrûZis_rrefr*r*r+rVÄszMyConvNetForMNIST.forwardcCsiSrEr*rQr*r*r+rmËszMyConvNetForMNIST.__getstate__)F)rSrTrUrIrVrmrXr*r*rSr+rc´s rcc@sØeZdZd6dd„Zdd„Zdd„Zdd „Zd d„Zdd „Zd7dd„Z dd„Z dd„Zdd„Zd8dd„Z dd„Zdd„Zdd „d!d „fd"d#„Zd$d%„Zd&d'„Zd(d)„Zd*d+„Zd,d-„Zd.d/„Zd0d1„Zd2d3„Zd4d5„ZdS)9Ú RpcTestCommonNcCs`|tjkrtj||||dS|tjkr._rpc_synccs0|d}tjdt||fd ¡}ˆ ||¡dSr|)rÁrÉrlrðrrr}rQr*r+Ú _rpc_asyncsý z1RpcTestCommon._world_size_one.._rpc_asynccs0|d}tjdt||fd ¡}ˆ ||¡dSr|)rÁrÏrlr—rrr}rQr*r+Ú_remote sý z.RpcTestCommon._world_size_one.._remote)rNrÁÚinit_rpcÚrpc_backendrZÚshutdown)rHrzr}rr€rr*rQr+Ú_world_size_oneîs û zRpcTestCommon._world_size_onecCsŒ|jd|j}tdƒD]n}||jd}|rDtƒ|}tƒ|}nt dd¡}t dd¡}tjt|ƒtj ||fd}| ||d¡qdS)Nr&érjrÀ)rNrGrPr r'r)rÁrÂr!r(rr)rHrOÚdst_rankržÚnrûrür"r*r*r+Ú _multi_rpcs ýzRpcTestCommon._multi_rpcéc CsÆ|jdkrÂ| |jdk¡d}g}t|ƒD] }tj|||fd}| |¡q*tj |¡ ¡D]}| | ¡d¡q\d}g}t|ƒD] }tj|||fd}| |¡q‚tj |¡D]}| |d¡q°dS)Nrr›Úworker1rÀZworker2) rNÚ assertTruerGrPrÁrÉr)r'røZcollect_allrðrrÚwait_all) rHr…rûZ num_repeatrÄr%rròrsr*r*r+Ú_run_uneven_workload*s z"RpcTestCommon._run_uneven_workloadcCsbt|j|j|jƒtjd|j|j|j|j|jd| ||¡tj ¡t ¡tj dddS©Núworker%drzF©Úgraceful©rÚfile_init_methodrNrGrÁr‚rƒrZrŽÚapiÚ_wait_all_workersÚdistÚbarrierr„©rHr…rûr*r*r+r–Fsû zRpcTestCommon._wait_all_workerscCslt|j|j|jƒtjd|j|j|j|j|jd| ||¡tj ¡tj ¡t ¡tj dddSrr“r™r*r*r+Ú_wait_all_workers_twice\sû z%RpcTestCommon._wait_all_workers_twicecCs@|jd}||j}tjt|ƒ|t|jƒfd}| ||¡dS©Nr&rÀ)rNrGrÁrÂr!rr)rHr…Úexpectedrˆr‡r"r*r*r+Ú_nested_rpcss ýzRpcTestCommon._nested_rpcr+r*cCsŽ|jd}||j}g}t ¡}t|ƒD]"}tjt|ƒ||d} | | ¡q(tj |¡D]} | | d¡qXt ¡}td |j|||¡ƒdS)Nr&rÀrz0Rank {} finished testing {} times in {} seconds.)rNrGrwrPrÁrÉr!r)r'rørrrrºÚformat)rHr…Úrepeatrprˆr‡r%ÚtikrròrsÚtokr*r*r+Ú_stress_test_rpc}s" ÿÿzRpcTestCommon._stress_test_rpccCsB|jd}||j}tjt|ƒtj||fd}| | ¡|¡dSr›) rNrGrÁrÏr!r'r(rrr—)rHrûrürœrˆr‡r‘r*r*r+Ú_builtin_remote_rets ýz!RpcTestCommon._builtin_remote_retcCs0tjt|jƒtj||fd}| | ¡|¡dSrÆ)rÁrÏr!rNr'r(rrr)rHrûrürœr‘r*r*r+Ú_builtin_remote_self™sýz"RpcTestCommon._builtin_remote_selfcCsdS)Nr*r*©rûrür*r*r+rÿ¡rzRpcTestCommon.cCsiSrEr*r¥r*r*r+rÿ¡rc Cs¤d}|jd}||j}g}g} t|ƒD]P} || }| tjt|ƒ||||ƒ|||ƒd¡| ||||ƒ|||ƒŽ¡q(t|ƒD]} | || ¡| | ¡q‚dS)Nr„r&rk) rNrGrPr)rÁrÏr!rrr—)rHrorOÚargs_fnÚ kwargs_fnÚmrˆr‡Úrrefsrœržr*r*r+Ú_test_multi_remote_call¡s$ üÿz%RpcTestCommon._test_multi_remote_callcCst|jd}||j}tjt|ƒtj||fd}tjt|ƒtj||fd} tjt|ƒt|| fd} | | ¡|¡dSr›) rNrGrÁrÏr!r'r(rµrrr—)rHrzr}rûrürœrˆr‡r³r´Úrref_cr*r*r+Ú _py_rref_args¶s$ ÿÿÿzRpcTestCommon._py_rref_argscCs‚|jd}||j} |d|j} tjt| ƒt|||fd}tjt| ƒt|||fd}tjt| ƒt||fd} | | ¡|¡dSr›) rNrGrÁrÏr!r«rµrrr—)rHrzr}r¢rûrürõrœrˆÚ owner_rankZ user_rankr³r´r«r*r*r+Ú_py_rref_args_user_shareÄs& ÿÿÿz&RpcTestCommon._py_rref_args_user_sharecCsp|jd}||j} tjt| ƒt|||fd} tjt| ƒt|||fd}tjt| ƒt| |fd}| ||¡dSr›) rNrGrÁrÏr!r«rÂrµrr)rHrzr}r¢rûrürõrœrˆr‡r³r´r*r*r+Ú_py_rpc_rref_argsÓs$ ÿÿÿzRpcTestCommon._py_rpc_rref_argscCsP|jd}||j}|d|j}tjt|ƒ|t|ƒfd}| | ¡|¡dSr›)rNrGrÁrÏr!rrr—)rHr…rœrˆÚ dst_rank1Ú dst_rank2r‘r*r*r+Ú_nested_remoteâs ýzRpcTestCommon._nested_remotec Cs€|jd}||j}|d|j}tjt|ƒ|t|ƒfd}| ¡}| t|ƒd¡| |d ¡|¡| |d ¡|¡dS)Nr&rÀrjr)rNrGrÁrÏr!r—rrr#) rHr…Ú expected1Ú expected2rˆr°r±Ú rref_of_rrefsr©r*r*r+Ú_nested_rrefîs ý zRpcTestCommon._nested_rrefcCs®|jd}||j}|d|j}g}tdƒD]$}| tjt|ƒ|t|ƒfd¡q.tdƒD]L} || } | ¡}| t |ƒd¡| |d ¡|¡| |d ¡|¡q\dS)Nr&r†rÀrjr) rNrGrPr)rÁrÏr!r—rrr#)rHr…r³r´rˆr°r±Z all_rrefsrržrµr©r*r*r+Ú_nested_rref_stresss$ ýÿz!RpcTestCommon._nested_rref_stressc Cs¾t|d}t ¡}tdƒD]ž}|t dd¡ ¡ƒ}||t dd¡ƒ ¡t| ¡ƒdj }| ¡ |||¡}| ¡}|jr†| ¡ ¡}| ¡ |¡} | jr¦| ¡ ¡} | t || ¡¡qdS)NrNr„r)rMrQZMSELossrPr'ÚrandÚlongÚbackwardÚlistÚ parametersÚgradrÉrbrðZ is_sparseZto_denseÚdoublerÂr_rŒr¦) rHr‘rOr¨Zloss_fnržÚoutputsr^ròZps_gradientr*r*r+Ú _trainer_funcs zRpcTestCommon._trainer_funccCsdtt|jdƒƒ}g}td|jƒD]0}| tjt|j||jƒ|j ||fd¡q"t j |¡dSr›) rrYrGrPr)rÁrÉr!rNrÀr'rør)rHrOZps_rrefrøÚindexr*r*r+Ú_my_parameter_server&sþýÿ z"RpcTestCommon._my_parameter_serverc Csztdgd}tj d¡Vtj ¡}tj ¡}tj |¡Œ|rhtƒ d¡}tƒ d¡}|| ¡} n$tj ddd}tj ddd}||} tj tdt ƒƒ¡||7}|r¶| ¡}| ||ƒ¡W5QRXtj |¡Œ|| ¡ƒ}|rH| t | ¡| ¡¡ ¡ ¡¡| t | ¡| ¡¡ ¡ ¡¡| | ¡| ¡¡n| t || ¡ ¡ ¡¡W5QRXW5QRXdS)Núcuda:0r,©rÚ©r2r+)rr'r1r2r/r1r rúrrr)r2r3rr]rðrŒÚeqÚindicesÚallÚitemr”rrÚsize) rHÚwrapperÚ unwrapperÚ sparse_tensorÚfuturer1Úanother_streamr™Z add_tensorÚexpected_tensorr*r*r+Ú_test_cuda_future_extraction6s0 ""z*RpcTestCommon._test_cuda_future_extraction)NN)rŠ)r+r*)rSrTrUrqrtrwrxr…r‰rŽr–ršrr¢r£r¤rªr¬r®r¯r²r¶r·rÀrÂrÑr*r*r*r+rjÑs. + rjc@sêeZdZedd„ƒZedd„ƒZedd„ƒZedd„ƒZed d „ƒZedd„ƒZ ed d„ƒZ edd„ƒZedd„ƒZedd„ƒZ dd„Zedd„ƒZedd„ƒZedd„ƒZdd„Zedd „ƒZed!d"„ƒZej ejjd#¡ej ejjjd$¡ed%d&d'd(„ƒƒƒZed%d&d)d*„ƒZed%d&d+d,„ƒZed%d&d-d.„ƒZ ed%d&d/d0„ƒZ!d1d2„Z"ed%d&d3d4„ƒZ#ed5d6„ƒZ$ed7d8„ƒZ%e&d9d:„ƒZ'ed;d<„ƒZ(ed=d>„ƒZ)ed?d@„ƒZ*edAdB„ƒZ+edCdD„ƒZ,edEdF„ƒZ-edGdH„ƒZ.ed%d&dIdJ„ƒZ/dKdL„Z0dMdN„Z1edOdP„ƒZ2edQdR„ƒZ3ddSdT„Z4edUdV„ƒZ5edWdX„ƒZ6edYdZ„ƒZ7ed[d\„ƒZ8ed]d^„ƒZ9ed%d&d_d`„ƒZ:edadb„ƒZ;edcdd„ƒZdidj„Z?edkdl„ƒZ@edmdn„ƒZAedodp„ƒZBedqdr„ƒZCdsdt„ZDedudv„ƒZEedwdx„ƒZFdydz„ZGdd|d}„ZHed~d„ƒZIed€d„ƒZJed‚dƒ„ƒZKd„d…„ZLd†d‡„ZMedˆd‰„ƒZNedŠd‹„ƒZOdddŽ„ZPdd„ZQed‘d’„ƒZRed“d”„ƒZSd•d–„ZTed—d˜„ƒZUed™dš„ƒZVd›dœ„ZWeddž„ƒZXedŸd „ƒZYd¡d¢„ZZed£d¤„ƒZ[ed¥d¦„ƒZ\d§d¨„Z]ed©dª„ƒZ^ed«d¬„ƒZ_dd®„Z`ed¯d°„ƒZaed±d²„ƒZbd³d´„Zcedµd¶„ƒZded·d¸„ƒZed¹dº„Zfed»d¼„ƒZged½d¾„ƒZhd¿dÀ„ZiedÁdÂ„ƒZjedÃdÄ„ƒZkdÅdÆ„ZledÇdÈ„ƒZmedÉdÊ„ƒZnedËdÌ„ƒZoedÍdÎ„ƒZpedÏdÐ„ƒZqedÑdÒ„ƒZredÓdÔ„ƒZsedÕdÖ„ƒZted×dØ„ƒZuedÙdÚ„ƒZvedÛdÜ„ƒZwedÝdÞ„ƒZxedßdà„ƒZyedádâ„ƒZzedãdä„ƒZ{edådæ„ƒZ|edçdè„ƒZ}edédê„ƒZ~edëdì„ƒZedídî„ƒZ€edïdð„ƒZedñdò„ƒZ‚edódô„ƒZƒe&ddõdö„ƒZ„ed÷dø„ƒZ…edùdú„ƒZ†e&ddûdü„ƒZ‡edýdþ„ƒZˆedÿd„ƒZ‰edd„ƒZŠedd„ƒZ‹edd„ƒZŒedd„ƒZed d „ƒZŽedd„ƒZed d„ƒZedd„ƒZ‘edd„ƒZ’edd„ƒZ“edd„ƒZ”edd„ƒZ•edd„ƒZ–dd„Z—dd„Z˜edd „ƒZ™d!d"„Zšd#d$„Z›ed%d&„ƒZœd'd(„Zd)d*„Zže&d+d,„ƒZŸed-d.„ƒZ ed/d0„ƒZ¡ed1d2„ƒZ¢ed3d4„ƒZ£ed5d&d6d7„ƒZ¤ej ejjjd8¡d9d:„ƒZ¥ed%d&d;d<„ƒZ¦ed%d&d=d>„ƒZ§ed?d@„ƒZ¨edAdB„ƒZ©edCdD„ƒZªedEdF„ƒZ«ed%d&dGdH„ƒZ¬edIdJ„ƒZed%d&e®e¯dKƒdLdM„ƒƒZ°edNdO„ƒZ±ed%d&dPdQ„ƒZ²ed%d&dRdS„ƒZ³edTdU„ƒZ´edVdW„ƒZµdXdY„Z¶dZd[„Z·ed\d]„ƒZ¸ed^d_„ƒZ¹ed`da„ƒZºedbdc„ƒZ»eddde„ƒZ¼dŒZ½e&dfdg„ƒZ¾edhdi„ƒZ¿edjdk„ƒZÀedldm„ƒZÁedndo„ƒZÂedpdq„ƒZÃed%d&drds„ƒZÄe&dtdu„ƒZÅe&dvdw„ƒZÆe&dxdy„ƒZÇdzd{„ZÈed|d}„ƒZÉed~d„ƒZÊed€d„ƒZËed‚dƒ„ƒZÌed„d…„ƒZÍed†d‡„ƒZÎedˆd‰„ƒZÏedŠd‹„ƒZÐedŒd„ƒZÑedŽd„ƒZÒedd‘„ƒZÓed’d“„ƒZÔed”d•„ƒZÕed–d—„ƒZÖed˜d™„ƒZ×edšd›„ƒZØedœd„ƒZÙedždŸ„ƒZÚed d¡„ƒZÛd¢d£„ZÜed¤d¥„ƒZÝed¦d§„ƒZÞd¨d©„Zßedªd«„ƒZàed¬d„ƒZáed®d¯„ƒZâd°d±„Zãed²d³„ƒZäed´dµ„ƒZåed¶d·„ƒZæed¸d¹„ƒZçeèjéfdºd»„Zêed¼d½„ƒZëed¾d¿„ƒZìedÀdÁ„ƒZíedÂdÃ„ƒZîedÄdÅ„ƒZïedÆdÇ„ƒZðedÈdÉ„ƒZñedÊdË„ƒZòedÌdÍ„ƒZóedÎdÏ„ƒZôeèjéfdÐdÑ„ZõedÒdÓ„ƒZöedÔdÕ„ƒZ÷edÖd×„ƒZøeèjéfdØdÙ„ZùedÚdÛ„ƒZúedÜdÝ„ƒZûedÞdß„ƒZüedàdá„ƒZýedâdã„ƒZþedädå„ƒZÿdædç„Zedèdé„ƒZedêdë„ƒZedìdí„ƒZedîdï„ƒZed%d&e®ej dðdŒ¡dñkdòƒdódô„ƒƒZed%d&e®ej dðdŒ¡dñkdõƒdöd÷„ƒƒZ edødù„ƒZ edúdû„ƒZed%d&e®ej dðdŒ¡dñkdüƒdýdþ„ƒƒZdÿd„Z dd„Zedd„ƒZe&dd„ƒZe&dd„ƒZed d „ƒZed%d&dd„ƒZed d„ƒZdŒS(ÚRpcTestc Csv|jd}||j}t ¡}t t|ƒ¡}| |jt|jƒ¡| |jt|ƒ¡| td¡t d¡}W5QRXdS)Nr&zcould not find destinationZ WorkerUnknown) rNrGrÁrur!rrrJÚassertRaisesRegexr )rHrˆZ peer_rankrvZpeer_worker_infoZunknown_worker_idr*r*r+Útest_worker_idVs zRpcTest.test_worker_idcCshtj ¡ ¡}dd„|Dƒ}dd„t|jƒDƒ}| ||¡dd„|Dƒ}tt|jƒƒ}| ||¡dS)NcSsh|] }|j’qSr*©rJ©rMrsr*r*r+rOgsz0RpcTest.test_get_worker_infos..cSsh|]}t|ƒ’qSr*)r!rLr*r*r+rOhscSsh|] }|j’qSr*)rKrÖr*r*r+rOms)rÁr•Ú_get_current_rpc_agentrRrPrGrrrt)rHZworker_infosZworker_namesZexpected_worker_namesZ worker_idsZexpected_worker_idsr*r*r+Útest_get_worker_infoscsÿzRpcTest.test_get_worker_infoscCs†t ¡}t|jƒ}tj|tjt dd¡dfd}tj|tjt dd¡dfd}| | ¡t dd¡d¡| |t dd¡d¡dSr¿)rÁrur!rNrÉr'r(r)rÂrrrð)rHrvÚself_worker_nameròr"r*r*r+Ú test_self_addqs zRpcTest.test_self_addc Cs¤|jd|j}tjtjtjfD]<}|j|tj|t dd¡dfd}| |t dd¡d¡q tjtjtjfD]@}| t¡,|j|jdtj|t dd¡dfdW5QRXqntjtjtjfD]:}| t¡&|jdtj|t dd¡dfdW5QRXqÀtjtjtjfD]@}| t ¡*|j|dtj|t dd¡dfdW5QRXqtjtjtjfD]@}| t ¡*|j|dtj|t dd¡dfdW5QRXq^dS)Nr&rjrÀr-çà?)rNrGrrlrmrnrqr'r(r)rrÚassertRaisesr r½)rHr‡Ú exec_moder"r*r*r+Útest_send_to_rankzs 2,2zRpcTest.test_send_to_rankcCs | t ¡t dd¡dd¡dS©Nrjr&r›)rtrÁrur'r)rQr*r*r+Útest_self_py_udf_remote”s üzRpcTest.test_self_py_udf_remotecCs0t|jd|jƒ}| |t dd¡dd¡dS©Nr&rjr›)r!rNrGrwr'r)©rHrÄr*r*r+Ú test_self_remote_rref_as_rpc_args üz(RpcTest.test_self_remote_rref_as_rpc_argcCs | t ¡t dd¡dd¡dSrß)rwrÁrur'r)rQr*r*r+Ú%test_self_remote_rref_as_self_rpc_arg§s üz-RpcTest.test_self_remote_rref_as_self_rpc_argcCs0t|jd|jƒ}| |t dd¡dd¡dSrá)r!rNrGrxr'r)râr*r*r+Ú#test_self_remote_rref_as_remote_arg°s üz+RpcTest.test_self_remote_rref_as_remote_argcCs | t ¡t dd¡dd¡dSrß)rxrÁrur'r)rQr*r*r+Ú(test_self_remote_rref_as_self_remote_argºs üz0RpcTest.test_self_remote_rref_as_self_remote_argc Csªt|jd|jƒ}tj|tt dd¡ddfd}d}| t |¡| ¡ ¡W5QRX| t |¡| ¡ ¡ ¡W5QRX| t |¡| ¡ ¡W5QRXdS)Nr&rjr›rÀzhas no attribute 'non_exist')r!rNrGrÁrÏr«r'r)rÓÚAttributeErrorrÂZ non_existrÉrð)rHrÄr‘Úmsgr*r*r+Útest_rref_proxy_non_existÃsz!RpcTest.test_rref_proxy_non_existcCstj|tt dd¡ddfd}t dd¡dd}| | ¡| ¡ ¡¡| |d| ¡ d¡ ¡¡| | dd¡| ¡ dd¡ ¡¡dS)Nrjr&r›rÀrœ) rÁrÏr«r'r)rrrÊrÂrÉr(rðÚviewr—©rHrÄr‘rœr*r*r+Ú_test_rref_proxy_tensorÑs zRpcTest._test_rref_proxy_tensorcCs| t|jd|jƒ¡dSr%)rìr!rNrGrQr*r*r+Útest_rref_proxy_tensorÙszRpcTest.test_rref_proxy_tensorcCs| t ¡¡dSrE)rìrÁrurQr*r*r+Útest_rref_proxy_tensor_selfÝsz#RpcTest.test_rref_proxy_tensor_selfcCsDtjt|jd|jƒtt dd¡ddfd}t dd¡dd}| ¡}| ¡}| ¡}| | ¡| ¡¡| |d| d¡¡| | dd¡| dd¡¡| | ¡| ¡ ¡¡| |d| d¡ ¡¡| | dd¡| dd¡ ¡¡| | ¡| ¡ ¡¡| |d| d¡ ¡¡| | d¡| d¡ ¡¡dS)Nr&rjr›rÀrœr‹r-)rÁrÏr!rNrGr«r'r)rÂrÉrrrÊr(rêrðr—)rHr‘rœZproxy_rpc_syncZproxy_rpc_asyncZproxy_remoter*r*r+Útest_rref_proxy_reuseás$ý zRpcTest.test_rref_proxy_reusec Cs`tj|tdd}tdƒ}| | ¡| ¡ ¡¡| | ¡| ¡ ¡ ¡¡| | ¡| ¡ ¡ ¡¡| d¡| d| ¡ d¡¡| d| ¡ d¡ ¡¡| d| ¡ d¡ ¡¡| | ¡| ¡ ¡¡| | ¡| ¡ ¡ ¡¡| | ¡| ¡ ¡ ¡¡| | d¡| ¡ d¡¡| | d¡| ¡ d¡ ¡¡| | d¡| ¡ d¡ ¡¡| | d¡| ¡ d¡¡| | d ¡| ¡ d ¡ ¡¡| | d ¡| ¡ d ¡ ¡¡| | dt dd¡¡| ¡ dt dd¡¡¡| | dt dd¡¡| ¡ dt dd¡¡ ¡¡| | dt dd¡¡| ¡ dt dd¡¡ ¡¡dS)N)érÀrðr›r&rjrœé r„é)rÁrÏryrrr‰rÂrÉrðr—rˆr~r†r‚r'rr)rër*r*r+Ú_test_rref_proxy_classús` þþþþþþþþþzRpcTest._test_rref_proxy_classcCs| t|jd|jƒ¡dSr%)rór!rNrGrQr*r*r+Útest_rref_proxy_class1szRpcTest.test_rref_proxy_classcCs| t ¡¡dSrE)rórÁrurQr*r*r+Útest_rref_proxy_class_self5sz"RpcTest.test_rref_proxy_class_selfÚ_initZ_set_and_start_rpc_agentF©Z setup_rpcc CsZd}tj |tt¡}| td¡tj |tt¡}W5QRXtjd||j|j |j ddS)NZstub_backendz$^RPC backend .+: already registered$r‹rz)rÁZbackend_registryZregister_backendrXr[rÓr r‚rNrGrZ)rHZmock_rpc_agentZmock_dist_autograd_initÚbackend_namer{r*r*r+Ú7test_register_rpc_backend_and_set_and_start_rpc_backend9s,ýÿýûz?RpcTest.test_register_rpc_backend_and_set_and_start_rpc_backendc Cs\| td¡Fttjj|j|j|jdƒ\}}}t j |j|d|j|j|jdW5QRXdS)Nú is not unique)rNrGZduplicate_name)r{rYrJrNrGrZ) rÓr Únextr'ÚdistributedZ rendezvousrLrNrGrÁZ_init_rpc_backendrƒrZ)rHrYrr*r*r+Útest_duplicate_nameXs ÿÿ úzRpcTest.test_duplicate_namec CsH| td¡2tjt|j|jdƒ|j|j|j|jdW5QRXdS)Nrúr&rz) rÓr rÁr‚r!rNrGrƒrZrQr*r*r+Útest_duplicate_name_2isûzRpcTest.test_duplicate_name_2c Cs¨tjt|jƒ|j|j|j|jdt|j|j|jƒt ¡tj dd¡dkr^|jdkr^d}nd}| t|¡(tjt|jƒ|j|j|j|jdW5QRXt ¡dS)NrzÚRPC_INIT_WITH_TCPÚ1rzAddress already in usezis already initialized)rÁr‚r!rNrƒrGrZrr”r—r˜ÚosÚenvironÚgetrÓr r„)rHZexpected_reinit_errr*r*r+Útest_reinitts*ûûzRpcTest.test_reinitc Cs„tjd|j|j|jdGdd„dtjjƒ}|ƒ}| ¡tjj |¡}| td¡&g}| ¡D]}| t|ƒ¡qbW5QRXdS)NÚgloo©r{rLrNrGcs$eZdZ‡fdd„Zdd„Z‡ZS)z1RpcTest.test_pg_init_no_rpc_init..MyModelcstƒ ¡tj dd¡|_dS)Nr›rœ)rPrIr'rQrgÚlinrQrSr*r+rIs z:RpcTest.test_pg_init_no_rpc_init..MyModel.__init__cSs | |¡SrE)rrUr*r*r+rV¡sz9RpcTest.test_pg_init_no_rpc_init..MyModel.forwardrWr*r*rSr+ÚMyModelœsrzBCurrent RPC agent is not set! Did you initialize the RPC framework)r—Úinit_process_groupr”rNrGr'rQÚModuleZtrainÚparallelZDistributedDataParallelrÓr r¼r)r)rHrÚmodelÚparamsÚparamr*r*r+Útest_pg_init_no_rpc_init”süz RpcTest.test_pg_init_no_rpc_initcCs | t dd¡t dd¡¡dSri)r…r'r)rQr*r*r+Útest_world_size_ones þzRpcTest.test_world_size_onec Cs¤d}| td¡td|ƒ}W5QRX| td¡td|ƒ}W5QRX| td¡td|ƒ}W5QRX| td¡$td dd „td ƒDƒ¡|ƒ}W5QRXdS)NrzWorker name must matchzabc*ú zmust be non-emptyÚzshorter than 128cSsg|]}d‘qSr|r*©rMržr*r*r+r“Ãsz.RpcTest.test_invalid_names..iô)rÓr r ÚjoinrP)rHZ worker_idÚinfor*r*r+Útest_invalid_names³szRpcTest.test_invalid_namescCs@|jd|j}tj ¡}tjt|ƒt|fd}| ||¡dSr›) rNrGrÁr•rurÂr!r¹rr)rHr‡rsr"r*r*r+Útest_worker_info_pickleÆs zRpcTest.test_worker_info_picklecCsZ|jd}||j}tjt|ƒtjt ||¡t ||¡fd}| |t ||¡d¡dS©Nr&rÀrj© rNrGrÁrÂr!r'r(r)rr©rHrˆr‡r"r*r*r+Útest_addÍs ýzRpcTest.test_addcCs t ¡jSrE)rÁrurKr*r*r*r+Úreturn_callee_idØszRpcTest.return_callee_idcCs.|jd|j}t |tj¡}| ||¡dSr%)rNrGrÁrÂrÒrrr)rHr‡r"r*r*r+Útest_int_calleeÜszRpcTest.test_int_calleecCsd|jd}||j}t t|ƒ¡}tj|tjt ||¡t ||¡fd}| |t ||¡d¡dSr) rNrGrÁrur!rÂr'r(r)rr)rHrˆr‡Zworkder_infor"r*r*r+Útest_add_with_idâs ÿzRpcTest.test_add_with_idcCsR|jd}||j}tjt|ƒtjt ||¡|fd}| |t ||¡|¡dSr›rrr*r*r+Útest_scalar_addís ÿzRpcTest.test_scalar_addcCs^|jd}||j}tjt|ƒtjt ||¡t ||¡fd}| | ¡t ||¡d¡dSr) rNrGrÁrÉr!r'r(r)rrrð)rHrˆr‡ròr*r*r+Útest_async_addös ýzRpcTest.test_async_addcCs`|jd}||j}t |j|j¡}d||j|j<tjt|ƒtj|fd}| || ¡¡dS)Nr&rrÀ) rNrGr'r)rÁrÂr!Znonzerorr)rHrˆr‡rûr"r*r*r+Útest_nonzeros zRpcTest.test_nonzerocCs| d¡dS©NF©r‰rQr*r*r+Útest_multi_rpc szRpcTest.test_multi_rpcc CsŠt|jd|jƒ}g}tdƒD]}| t |t¡¡q | t d¡t j |¡W5QRX|D]$}| t d¡| ¡W5QRXq`dS)Nr&r†r¼)r!rNrGrPr)rÁrÉrãrÓr½r'rørrð)rHrÄr%ržròr*r*r+Útest_future_wait_twiceszRpcTest.test_future_wait_twicec CsŽt|j|j|jƒtjt|jƒ|j|j|j|jdtj j }dd„}|tj _ z*| td¡tj dddW5QRXW5|tj _ Xt ¡dS)Nrzc SsDztjjtdƒ|dWn&tk r>}z|‚W5d}~XYnXdS)NrÛ©Útimeout)rÁr•Ú_all_gatherrur )r'Úexr*r*r+Úwait_all_workers_sleep*szERpcTest.test_wait_all_workers_timeout..wait_all_workers_sleeprTç{®Gáz„?)r’r')rr”rNrGrÁr‚r!rƒrZr•r–rÓr r„r—r˜)rHÚog_funcr*r*r*r+Útest_wait_all_workers_timeouts û z%RpcTest.test_wait_all_workers_timeoutcCs| tt dd¡¡dS©NrÚ)r–rÝr'r)rQr*r*r+Útest_wait_all_workers_dense9sz#RpcTest.test_wait_all_workers_densecCs| tt dd¡¡dSr.)ršrÝr'r)rQr*r*r+Ú!test_wait_all_workers_twice_dense<sz)RpcTest.test_wait_all_workers_twice_densecCsHt ¡}tj |j¡}i}t ¡ ¡D]}|j||j<q&| ||¡dSrE) rÁrur•r(rKr×rRrJrr)rHrÚresultsrœr*r*r+Útest_all_gather?szRpcTest.test_all_gatherc Csrt d¡|jdkr>| td¡tj tdƒ¡W5QRXn0| ¡}| t|¡tj tdƒ¡W5QRXdS)Nçš™™™™™¹?rz,timed out in _all_gather after 0\.10 secondsrÛ) rÁÚ_set_rpc_timeoutrNrÓr r•r(ruÚget_timeout_error_regex)rHÚexpected_errorr*r*r+Útest_all_gather_timeoutIs þzRpcTest.test_all_gather_timeoutcCs|t|ƒ}|d}t |t¡|s6|j|kr6| td¡tj |¡t |t ¡tj |¡|sx|j|krx| tt |ƒ¡dSrB)ÚsortedrÁrÂrCrJrrr@r•Z_barrierrAr#)rHrÚnamesZmulti_threadedZleaderr*r*r+Ú_test_barrier_helperXszRpcTest._test_barrier_helpercCs2t ¡}t ¡ ¡}dd„|Dƒ}| ||¡dS)NcSsg|] }|j‘qSr*rÕ©rMÚworkerr*r*r+r“isz0RpcTest.test_rpc_barrier_all..)rÁrur×rRr:©rHrÚall_worker_infor9r*r*r+Útest_rpc_barrier_alldszRpcTest.test_rpc_barrier_allcCsLt ¡}t ¡ ¡}|jdr.dd„|Dƒ}ndd„|Dƒ}| ||¡dS)NrjcSsg|]}|jdr|j‘qS©rj©rKrJr;r*r*r+r“rs z3RpcTest.test_rpc_barrier_subset..cSsg|]}|jds|j‘qSr@rAr;r*r*r+r“ts ©rÁrur×rRrKr:r=r*r*r+Útest_rpc_barrier_subsetls zRpcTest.test_rpc_barrier_subsetcCsLt ¡}t ¡ ¡}|jdr.dd„|Dƒ}nd|j›g}| ||¡dS)NrjcSsg|]}|jdr|j‘qSr@rAr;r*r*r+r“}s z;RpcTest.test_rpc_barrier_partial_subset..r<rBr=r*r*r+Útest_rpc_barrier_partial_subsetws z'RpcTest.test_rpc_barrier_partial_subsetcCsrt ¡}t ¡ ¡}dd„|Dƒ}g}tdƒD],}tj|j||dfd}| |¡| ¡q.|D]}| ¡q`dS)NcSsg|] }|j‘qSr*rÕr;r*r*r+r“ˆsz:RpcTest.test_rpc_barrier_multithreaded..r›T)Útargetrp)rÁrur×rRrPr;r<r:r)r=r)rHrr>r9ÚthreadsrÚthr*r*r+Útest_rpc_barrier_multithreaded‚s z&RpcTest.test_rpc_barrier_multithreadedcCs| tt dd¡¡dS)zTest graceful termination.rÚN)rŽrÝr'r)rQr*r*r+Ú+test_graceful_shutdown_with_uneven_workload‘sz3RpcTest.test_graceful_shutdown_with_uneven_workloadc CsÆtjd|j|j|j|j|jd|jd}||j}tjt|ƒtj t ||¡t ||¡fd}| |t ||¡d¡t ¡| td¡0tjt|ƒtj t ||¡t ||¡fdW5QRXdS)Nrrzr&rÀrjz^RPC has not been initialized)rÁr‚rNrƒrGrZrÂr!r'r(r)rrr„rÓr rr*r*r+Útest_shutdown_followed_by_rpc–s,û ýýz%RpcTest.test_shutdown_followed_by_rpccCsP|jd|j}|jd|j}tjt|ƒt|jfd}t ¡}| ||¡dSr›) rNrGrÁrÂr!r`r\rarr)rHr‡Zexpected_src_rankr"r_r*r*r+Útest_expected_src²s zRpcTest.test_expected_srccCsV|jd}||j}tjt|ƒt||d|dfd}| |t||d|dƒ¡dS©Nr&rjrÀ)rNrGrÁrÂr!Úminrrrr*r*r+Útest_py_built_inºs "zRpcTest.test_py_built_incCsX|jd}||j}tjt|ƒt||d|ddœd}| |t||d|dƒ¡dS©Nr&rjrªrV)rNrGrÁrÂr!r«rrrr*r*r+Útest_py_user_definedÁs ýzRpcTest.test_py_user_definedcCsZtjtjtjfD]D}t|dddƒ}| |j|¡| d|¡| d|¡| d|¡qdS)NÚfooÚworker0r‹)rrlrmrnrÚassertInr_)rHrÝÚrpc_profiling_keyr*r*r+Útest_build_rpc_profiling_keyÌsÿz$RpcTest.test_build_rpc_profiling_keycCs~| ||jk¡| ||jk¡t|tjjƒrH| tj |¡|jk¡n| |j|jk¡| |j |jk¡| |jd¡dSr%)rŒrJÚ isinstancer'ÚjitZScriptFunctionÚ _jit_internalÚ_qualified_namerSr_rrÚcount)rHrÙÚdst_worker_nameroÚ rpc_eventÚ rpc_exec_moder*r*r+Úcheck_profiling_infoØszRpcTest.check_profiling_infoc Csä|jdkrdS|jd|j}t|ƒ}t d¡t d¡}}tdd}tj|tj||fdW5QRX|j }dd„|Dƒ}dd„|Dƒd }|j } tdd}t ||¡W5QRX|j } d d„| Dƒd }|j }| | |¡dS)Nr&rÚT)Z record_shapesrÀcSsg|]}|jr|‘qSr*©Ú is_remote©rMr9r*r*r+r“ísz;RpcTest.test_profiler_rpc_record_shapes..cSsg|]}d|jkr|‘qS©r<rÕrar*r*r+r“îs rcSsg|]}d|jkr|‘qSrbrÕrar*r*r+r“÷s )rNrGr!r'r)Ú_profilerÁrÂr(Úfunction_eventsZinput_shapesrr) rHrÄÚ dst_workerr8r9ÚprofrdÚ remote_eventsÚremote_add_eventZremote_add_input_shapesÚlocal_function_eventsZlocal_add_eventZlocal_add_input_shapesr*r*r+Útest_profiler_rpc_record_shapesâs2 ÿþÿþz'RpcTest.test_profiler_rpc_record_shapesc CsÒ|jdkrdS|jd|j}t|ƒ}tdd}tj|tdd}| ¡}W5QRX|j}t dd„|Dƒƒ}| dh|¡td d}tj|tdd}| ¡}W5QRX|j}t d d„|Dƒƒ}| dh|¡dS)Nr&T)Zprofile_memoryr*rÀcss|]}|jVqdSrE©Zcpu_memory_usagerar*r*r+Ú sz3RpcTest.test_profiler_rpc_memory..rFcss|]}|jVqdSrErkrar*r*r+rls)rNrGr!rcrÁrÉr;rðrdrtÚassertNotEqualrr)rHrÄreÚpròr°rdZevent_cpu_mem_usagesr*r*r+Útest_profiler_rpc_memoryýs z RpcTest.test_profiler_rpc_memoryc sÚ|jdkrdS|jd|j}t|ƒ}tƒ}tj|tdd}| ¡}W5QRX|j}t ƒt}|}| |¡t|ƒR} t | ¡} dd„| Dƒ}ttjjgD]$‰t‡fdd„|Dƒƒ}| |¡qœW5QRXW5QRXdS)Nr&r*rÀcSsg|]}|d‘qSrÕr*rar*r*r+r“%sz6RpcTest.test_profiler_export_trace..csg|]}ˆ|k‘qSr*r*)rMÚ event_name©Zexpected_event_namer*r+r“'s)rNrGr!rcrÁrÉr;rðrdr#Zexport_chrome_traceÚopenÚjsonÚloadÚEXPECTED_REMOTE_EVENTSrrmr_ÚanyrŒ) rHrÄrernròr°ÚeventsÚfnameÚpathr…ÚtraceZevent_namesZevent_existsr*rqr+Útest_profiler_export_traces" z"RpcTest.test_profiler_export_tracec s”ˆjdkrdS‡fdd„tdˆjƒDƒ}‡fdd„‰|D]V}t|ƒ‰d}tjj|d2‰‡‡‡fd d„t|ƒDƒ}|D]}| ¡qvW5QRXq8dS) Nr&csg|]}|ˆjkr|‘qSr*©rNrLrQr*r+r“3s z7RpcTest.test_profiler_rpc_key_names..rc sìtƒ}tj|tdd}| ¡W5QRX|j}dd„|Dƒ}ttjtj t ˆjƒ|ƒ}tt ƒ}| ¡D]f\}}ˆ | |¡¡ˆ |j¡ˆ |jt |¡jk¡|t|ƒd…‰‡fdd„|Dƒ} || 8}qfˆ |tƒkd|›d¡dS) Nr*rÀcSsi|]}|jr|j|“qSr*©r`rJrar*r*r+Ú ;szSRpcTest.test_profiler_rpc_key_names..rpc_with_profiling..csh|]}|ˆkr|’qSr*r*)rMZremote_event_name©Zoperator_name_substrr*r+rOSsþzRRpcTest.test_profiler_rpc_key_names..rpc_with_profiling..z Expected z* to be included in remote profiler output.)rcrÁrÉr;rðrdrrrmrUr!rNrtrur¯rŒÚ startswithr`Únode_idrurKr#) rerfròrwÚremote_event_namesrTZremote_event_name_setrJr9Zmatching_eventrQrr+Úrpc_with_profiling5s6ÿü þ þz?RpcTest.test_profiler_rpc_key_names..rpc_with_profilingrj)Úmax_workerscsg|]}ˆ ˆˆ¡‘qSr*)Zsubmit©rMr)reÚexecutorrƒr*r+r“gsÿ)rNrPrGr!r÷røZThreadPoolExecutorra)rHÚ dst_ranksrÄZnum_parallel_rpcsr%ròr*)rer†rƒrHr+Útest_profiler_rpc_key_names*s ,ÿþz#RpcTest.test_profiler_rpc_key_namesc sDˆjdkrdS‡fdd„tdˆjƒDƒ}|D]}t|ƒ}tƒ}tj|tdd}| ¡}W5QRX|j }t |tjj ƒ}ˆ tˆjƒ|t|tj¡dd„|Dƒ} ttjtjtˆjƒt|ƒƒ‰tD]4} ˆt| }ˆ || k¡| |}ˆ |j|¡qÀ‡fd d „‰‡fdd„|Dƒ} ˆ t| ƒttƒdt| ƒ›d ttƒ›¡q,dS)Nr&csg|]}|ˆjkr|‘qSr*r|rLrQr*r+r“us zERpcTest._run_test_profiler_remote_events_profiled..rr*rÀcSsi|]}|jr|j|“qSr*r}rar*r*r+r~‡szERpcTest._run_test_profiler_remote_events_profiled..cs"ˆt}|| |¡t|ƒd…SrE)Ú REMOTE_OP_STRÚfindr#©rpZ remote_op_key)rTr*r+Úconvert_remote_to_local˜sÿÿzRRpcTest._run_test_profiler_remote_events_profiled..convert_remote_to_localcs$g|]}ˆ|jƒtkrˆ|jƒ‘qSr*)rJrura©rŒr*r+r“Ÿsþz"Mismatch between profiled events: z and expected events: )rNrPrGr!rcrÁrÉr;rðrdrrrmr_r^rrUrur‰rŒrrrrt)rHr‡rÄrerfròr"rwr\rgZexpected_remote_event_nameÚexpected_keyÚremote_eventZremote_events_listr*)rŒrTrHr+Ú)_run_test_profiler_remote_events_profiledosL ûü þýz1RpcTest._run_test_profiler_remote_events_profiledcCs| ¡dSrE©rrQr*r*r+Ú$test_profiler_remote_events_profiledªsz,RpcTest.test_profiler_remote_events_profiledcCs| ¡dSrEr‘rQr*r*r+Ú4test_profiler_remote_events_profiled_single_threaded®sz.csg|]}|jˆkr|‘qSr*rÕra)rŽr*r+r“×s rcsg|]}|jˆkr|‘qSr*rÕra)Úremote_add_keyr*r+r“ás )rNr!rGr'r)rcrÁrÉrrðrdrrrmrUrr‰rrr#rrWZ _builtinsZ _find_builtinr()rHr2Údst1Údst2rûrürfr"rôrdZ key_prefixZnested_rpc_key_prefixrgZrpc_remote_eventrhr*)rŽr˜r+Ú!_run_rpc_profiling_async_function½sV ÿÿÿ ÿÿ ÿz)RpcTest._run_rpc_profiling_async_functioncCs<t|j|j|jƒ| ¡tj ¡r8t ¡|jdddS©NrÃrÅ© rr”rNrGr›r'r1Zis_availabler—r˜rQr*r*r+Ú!test_rpc_profiling_async_functionés z)RpcTest.test_rpc_profiling_async_functioncCs<t|j|j|jƒ| ¡tj ¡r8t ¡|jdddSrœrrQr*r*r+Ú1test_rpc_profiling_async_function_single_threadedñs z9RpcTest.test_rpc_profiling_async_function_single_threadedc s4ˆjdkrdS‡fdd„tˆjƒDƒ}|D]}t|ƒ}tƒ}tj|tdd}| ¡W5QRX|j }dd„|Dƒ}ˆ dt|ƒ¡|d}ˆ |j|¡‡fdd „‰ˆ|ƒ}tƒ}td dƒW5QRX|j } dd„| Dƒd} ˆ| ƒ}d d„|Dƒ}d‰‡fdd„} |D]}| |j ƒ}ˆ ||k¡qq*dS)Nr&csg|]}|ˆjkr|‘qSr*r|rrQr*r+r“ÿs zERpcTest.test_rpc_profiling_remote_record_function..)r-TrÀcSsg|]}d|jkr|‘qS©r.rÕ©rMÚevtr*r*r+r“s rcs.|js gS|j}|jD]}| ˆ|ƒ¡q|SrE)Úcpu_childrenÚextend)r9r£r)Úget_cpu_childrenr*r+r¥s zKRpcTest.test_rpc_profiling_remote_record_function..get_cpu_childrenr-TcSsg|]}d|jkr|‘qSr rÕr¡r*r*r+r“s cSsg|] }|j‘qSr*rÕr¡r*r*r+r“"sr>csˆ}|| |¡t|ƒd…SrE)rŠr#r‹)r‰r*r+rŒ(sÿzRRpcTest.test_rpc_profiling_remote_record_function..convert_remote_to_local)rNrPrGr!rcrÁrÉr;rðrdrrr#rrJrŒ)rHr‡r‡rerfròrdZrecord_function_remote_eventZremote_childrenriZlocal_record_function_eventZlocal_childrenZlocal_children_namesrŒr¢Z local_namer*)r‰r¥rHr+Ú)test_rpc_profiling_remote_record_functionùsD ÿÿþÿ z1RpcTest.test_rpc_profiling_remote_record_functioncshdd„‰|j}‡fdd„|Dƒ}| d|k¡|d}| |j|¡| t|jƒt|ƒtj|t j ¡dS)NcSs|| t¡ttƒd…SrE)rŠr‰r#)rpr*r*r+rŒ4szDRpcTest.validate_profiling_workload..convert_remote_to_localcsi|]}|jrˆ|jƒ|“qSr*r}rarr*r+r~8sþz7RpcTest.validate_profiling_workload..r=)rdrŒrrrr^r!rNr'r6rrm)rHrÄrfrwrgZremote_mul_eventr*rr+Úvalidate_profiling_workload2s þûz#RpcTest.validate_profiling_workloadc Cs–|jd|j}|jdkr’t ¡"}tƒ}| |¡W5QRXW5QRX| ||¡tƒ$}t ¡}| |¡W5QRXW5QRX| ||¡dSr%)rNrGÚ dist_autogradÚcontextrcr–r§)rHrÄÚ context_idrfr*r*r+Ú(_run_test_profiler_with_autograd_contextHs z0RpcTest._run_test_profiler_with_autograd_contextcCs| ¡dSrE©r«rQr*r*r+Ú3test_profiler_with_autograd_context_single_threadedZsz;RpcTest.test_profiler_with_autograd_context_single_threadedcCs| ¡dSrEr¬rQr*r*r+Ú#test_profiler_with_autograd_context^sz+RpcTest.test_profiler_with_autograd_contextNc sTˆdk rˆn|jd|j‰|s$tntjj}|jdkrP|ƒÐ}|sLt ¡ntjj d¡} | ¦} ˆt jkr€tj tˆƒ||dn€ˆt jkrÈtjtˆƒ||d}|r¾tjtˆƒ||d}| ¡| ¡n8| ˆt jk¡tjtˆƒ||d} | ¡| ¡ ¡W5QRXW5QRX|s |jn| ¡}|rT| t¡t|ˆjƒW5QRXdSt|ˆjƒ}| |j|j¡‡fdd„|Dƒ|h}| t |ƒd¡|D]}| |jˆ¡qœ|rêt|dƒ}| !|j"j#|j"j#¡| |j"j$|j"j$¡t|jƒ}tˆƒ}| %||||ˆ¡|rPt&dd„t'|ƒDƒƒ}t&‡fdd„t'|ƒDƒƒ}| (||¡dS) Nr&rQrÀcsh|]}|jˆkr|’qSr*)rrarÃr*r+rO˜s z2RpcTest._profiler_test_with_rpc..css |]\}}d|jkr|VqdS)rQNrÕ©rMržr9r*r*r+rl«s z2RpcTest._profiler_test_with_rpc..c3s"|]\}}ˆj|jkr|VqdSrE)r_rJr¯)r]r*r+rl¬s))rNrGrcr'r4r$r/r0r3r5rrlrÁrÂr!rmrÉrðrŒrnrÏr—Z_get_profiling_futurerdrwrÜÚ IndexErrorrr_rrrZassertGreaterEqualr#ZassertLessEqualÚ time_ranger=Úendr^rûÚ enumerateZ assertLess)rHr]rorpr7rÄÚkineto_profilernrfZrecord_function_ctx_mgrÚrfròrr‘rwr\rgrZscope_eventrÙr[Zfoo_event_ixZ rpc_event_idxr*)rÄr]r+Ú_profiler_test_with_rpcbsZÿ ÿý zRpcTest._profiler_test_with_rpccCs*|jtjtdd|jtjtddddS)N©r&rÀT©rpr7)r¶rrlr®rQr*r*r+Ú$_run_test_profiler_with_sync_rpc_udf¯sÿz,RpcTest._run_test_profiler_with_sync_rpc_udfcCs| ¡dSrE©r¹rQr*r*r+Útest_profiler_with_sync_rpc_udf´sz'RpcTest.test_profiler_with_sync_rpc_udfcCs| ¡dSrErºrQr*r*r+Ú/test_profiler_with_sync_rpc_udf_single_threaded¸sz7RpcTest.test_profiler_with_sync_rpc_udf_single_threadedcCsN|jtjtjt d¡t d¡fd|jtjtjt d¡t d¡fdddS©Nr&rÀTr¸)r¶rrlr'r6r)rQr*r*r+Ú(_run_test_profiler_with_sync_rpc_builtin¼sÿþz0RpcTest._run_test_profiler_with_sync_rpc_builtincCs| ¡dSrE©r¾rQr*r*r+Ú#test_profiler_with_sync_rpc_builtinÅsz+RpcTest.test_profiler_with_sync_rpc_builtincCs| ¡dSrEr¿rQr*r*r+Ú3test_profiler_with_sync_rpc_builtin_single_threadedÉsz;RpcTest.test_profiler_with_sync_rpc_builtin_single_threadedcCs>|jtjtdd|jtjtddd|jtjtddddS)Nr·rÀTr¸)rpr´)r¶rrmr®rQr*r*r+Ú%_run_test_profiler_with_async_rpc_udfÍsÿÿz-RpcTest._run_test_profiler_with_async_rpc_udfcCs| ¡dSrE©rÂrQr*r*r+Ú test_profiler_with_async_rpc_udf×sz(RpcTest.test_profiler_with_async_rpc_udfcCs| ¡dSrErÃrQr*r*r+Ú0test_profiler_with_async_rpc_udf_single_threadedÛsz8RpcTest.test_profiler_with_async_rpc_udf_single_threadedcCsN|jtjtjt d¡t d¡fd|jtjtjt d¡t d¡fdddSr½)r¶rrmr'r6r)rQr*r*r+Ú)_run_test_profiler_with_async_rpc_builtinßsÿþz1RpcTest._run_test_profiler_with_async_rpc_builtincCs| ¡dSrE©rÆrQr*r*r+Ú$test_profiler_with_async_rpc_builtinèsz,RpcTest.test_profiler_with_async_rpc_builtincCs| ¡dSrErÇrQr*r*r+Ú4test_profiler_with_async_rpc_builtin_single_threadedìsztest_async_record_function_double_end_callbacks_new_signaturesš s ÿzFRpcTest.test_async_record_function_double_end_callbacks_new_signaturesc CsÜ|jdkrØtƒ’}ttjtj t¡ddƒ}tj j |¡4}tj tdƒtt d¡fd}tjj |j|¡}W5QRX| ¡}t t d¡t d¡¡}| ||¡W5QRX|j}t|tj t¡ƒ}| tj t¡|jk¡dS)Nr&r‹rRrrÀ)rNrcrrrmr'rXrYrár3r4r5rÁrÉr!r™rèréÚhandlerðr(rrrdrrŒrJ) rHræÚkeyrµròrarœrwr\r*r*r+Ú'test_async_record_function_cbs_jit_call¬ s0 ü ÿ ÿz/RpcTest.test_async_record_function_cbs_jit_callcCs<|jd}||j}tjt|ƒt|fd}| |j|¡dSr›)rNrGrÁrÂr!ryrrrzrr*r*r+Útest_py_class_constructorÇ s z!RpcTest.test_py_class_constructorcCsJ|jd}||j}tjt|ƒtdƒj|fd}| |tdƒ |¡¡dSrL)rNrGrÁrÂr!ryr~rrrr*r*r+Útest_py_class_instance_methodÎ s ÿz%RpcTest.test_py_class_instance_methodcCsN|jd}||j}tjt|ƒtj||dfd}| |t ||d¡¡dSr›)rNrGrÁrÂr!ryr‚rrrr*r*r+Útest_py_class_method× s ÿzRpcTest.test_py_class_methodcCsJ|jd}||j}tjt|ƒtj|dfd}| |t |d¡¡dS)Nr&r„rÀ)rNrGrÁrÂr!ryr†rrrr*r*r+Útest_py_class_static_methodà s ÿz#RpcTest.test_py_class_static_methodcCs–|jd}||j}t t|ƒ¡}tj|tj|dfd}tj|t||d|dfd}| | ¡t |d¡¡| | ¡t||d|dƒ¡dS)Nr&r„rÀrj)rNrGrÁrur!rÉryr†rMrrrð)rHrˆr‡Zdst_worker_inforrr*r*r+Útest_py_multi_async_callé s z RpcTest.test_py_multi_async_callcCs6|jd}||j}t t|ƒt¡}| |tƒ¡dSr%)rNrGrÁrÂr!r»rrrr*r*r+Útest_py_no_return_resultó s z RpcTest.test_py_no_return_resultc Csb|jd}||j}tjt|ƒtt ||¡t ||¡fd}| |tt ||¡t ||¡ƒ¡dSr›) rNrGrÁrÂr!rlr'r)rrrr*r*r+Útest_py_tensorsú s ýzRpcTest.test_py_tensorsc Csœg}|jd}||j}tdƒD]6}tjt|ƒtt ||¡t ||¡fd}| |¡q d}tj |¡D].}| |tt ||¡t ||¡ƒ¡|d7}qhdS)Nr&rÚrÀr) rNrGrPrÁrÉr!rlr'r)r)rørrr)rHr%rˆr‡ržròÚjrsr*r*r+Ú test_py_tensors_multi_async_call s" ýÿz(RpcTest.test_py_tensors_multi_async_callcCs‚|jd}||j}t ||¡t ||¡g}ttƒƒ}t ||¡t ||¡dœ}tjt|ƒt |||fd}| |t |||ƒ¡dS)Nr&)rQÚbarrÀ)rNrGr'r)rer£rÁrÂr!r²rr)rHrˆr‡rzr}r¢r"r*r*r+Útest_py_tensors_in_container s ÿz$RpcTest.test_py_tensors_in_containerc Csz|jd}||j}tjt|ƒttƒt dd¡fd}tƒ}| t t dd¡t dd¡ƒ¡| |t|t dd¡ƒ¡dSrL)rNrGrÁrÂr!ršrgr'r)rtrlrr)rHrˆr‡r"r¨r*r*r+Útest_py_nested_pickle% s ý zRpcTest.test_py_nested_picklec CsB|jd}||j}| t¡tjt|ƒtdd}W5QRXdS©Nr&)r„rÀ)rNrGrÜÚ TypeErrorrÁrÂr!r»rr*r*r+Útest_py_function_exception4 s z"RpcTest.test_py_function_exceptionc Cs–tƒp\}}t|j|j|jƒt ¡|jd}||j}t t |ƒt ¡}| tt ¡| ¡W5QRXt ¡W5QRX| ¡}| t |k¡dSr%)rrr”rNrGr—r˜rÁrÉr!rãrÓr½rârðÚgetvaluerŒ)rHrÚerrrˆr‡ròZstderr_linesr*r*r+Útest_py_raise_in_user_func; s z"RpcTest.test_py_raise_in_user_funcc Cs„|jd}||j}t t|ƒt¡}z| ¡WnBtk rr}z$t|ƒ}| || d¡ d¡¡W5d}~XYnX| dd¡dS)Nr&zutf-8Zunicode_escapeFz/expected raise_func_escape to raise ValueError.) rNrGrÁrÉr!rçrðr½ÚstrrrÚencodeÚdecoderŒ)rHrˆr‡ròrrèr*r*r+Ú&test_py_raise_in_user_func_escaped_strO s *z.RpcTest.test_py_raise_in_user_func_escaped_strcCs| tt dd¡d¡dS©Nrjr&)rrÅr'r)rQr*r*r+Útest_nested_rpc] szRpcTest.test_nested_rpccCs| t¡dSrE)r¢rØrQr*r*r+Útest_stress_light_rpca szRpcTest.test_stress_light_rpccCs|jtdt dd¡fddS©Nr†rÚ©rŸrp)r¢rÝr'r)rQr*r*r+Útest_stress_heavy_rpce szRpcTest.test_stress_heavy_rpccCs|jtdt dd¡fddSr)r¢rßr'r)rQr*r*r+Ú!test_stress_heavy_rpc_torchscripti sz)RpcTest.test_stress_heavy_rpc_torchscriptcCs.| t dd¡t dd¡t dd¡d¡dSri)r£r'r)rQr*r*r+Útest_builtin_remote_retm s ýzRpcTest.test_builtin_remote_retcCs.| t dd¡t dd¡t dd¡d¡dSri)r¤r'r)rQr*r*r+Útest_builtin_remote_selfu s ýz RpcTest.test_builtin_remote_selfcCs,|rtƒtƒfSt ||¡t ||¡fSdSrE©r r'r)©rˆrOr*r*r+Ú_multi_args_fn} szRpcTest._multi_args_fncCs|jtjdtjddS)NF©r¦©rªr'r(rÒrrQr*r*r+Útest_multi_builtin_remote_ret„ s þz%RpcTest.test_multi_builtin_remote_retcCs\|jd}||j}tjt|ƒt||d|ddœd}| | ¡t||d|dƒ¡dSrO)rNrGrÁrÏr!r«rrr—©rHrˆr‡r‘r*r*r+Útest_py_udf_remote‹ s ýzRpcTest.test_py_udf_remotecCs>|rtƒtƒtƒdœSt ||¡t ||¡t ||¡dœSdS)Nrªr rr*r*r+Ú_multi_kwargs_fn– sýzRpcTest._multi_kwargs_fncCs|jtdtjddS)NF©r§©rªr«rÒrrQr*r*r+Útest_multi_py_udf_remote¡ s ýz RpcTest.test_multi_py_udf_remotec Cs6| t dd¡dt dd¡dt dd¡dd¡dSrß)r¬r'r)rQr*r*r+Útest_py_rref_args© s ûzRpcTest.test_py_rref_argscCs:| t dd¡ddt dd¡ddt dd¡dd¡dS©Nrjr&r›rœr„)r®r'r)rQr*r*r+Útest_py_rref_args_user_share² s ùz$RpcTest.test_py_rref_args_user_sharecCs:| t dd¡ddt dd¡ddt dd¡dd¡dSr)r¯r'r)rQr*r*r+Útest_py_rpc_rref_args¾ s ùzRpcTest.test_py_rpc_rref_argscCs| tt dd¡d¡dS)Nrjr›)r²rÓr'r)rQr*r*r+Útest_nested_remoteÊ sþzRpcTest.test_nested_remotecCs*| tt dd¡dt dd¡d¡dSr)r¶rÐr'r)rQr*r*r+Útest_nested_rrefÑ s ýzRpcTest.test_nested_rrefcCs*| tt dd¡dt dd¡d¡dSr)r·rÐr'r)rQr*r*r+Útest_nested_rref_stressÙ s ýzRpcTest.test_nested_rref_stresscCs*d}|jd}||j}t||j|ƒdS)Nr†r&)rNrGrÊ)rHrËrˆr‡r*r*r+Ú!test_multi_layer_nested_async_rpcá s z)RpcTest.test_multi_layer_nested_async_rpcc Csz|jd}||j}t t|ƒt¡}| t¡| ¡W5QRXtjt|jƒt dd}| t ¡| ¡W5QRXdSrú)rNrGrÁrÏr!rãrÜr½r—r»rûrr*r*r+Útest_remote_with_exceptioní s z"RpcTest.test_remote_with_exceptioncCs\|jd}||j}|d|j}tjt|ƒtt|ƒfd}| | ¡t dd¡d¡dSr) rNrGrÁrÂr!r×rrr—r'r))rHrˆr°r±r‘r*r*r+Útest_rpc_return_rrefú s ýzRpcTest.test_rpc_return_rrefc Cs˜d}|jd}||j}tjt|ƒtjt ||¡dfd}t||j||ƒ}t |ƒD] }| t|ƒd¡|d ¡}qR|}| |t t ||¡d¡¡dS)Nér&rÀr) rNrGrÁrÏr!r'r(r)rÕrPrrr#r—)rHrËrˆr‡r‘rÖržr"r*r*r+Útest_rref_forward_chains ÿzRpcTest.test_rref_forward_chaincCstdƒ}| | ¡d¡dS)Né#)rrrr)rHÚ local_rrefr*r*r+Útest_local_rref_no_forkszRpcTest.test_local_rref_no_forkcCs|jd|j}tjt|ƒtjt d¡t d¡fd}| t d|j›d|j›d|j›dt|jƒ›d|›dt|ƒ›d ¡| ¡W5QRXdS) Nr&rÀz3For UserRRef$rref_id=GloballyUniqueId\(created_on=z5, local_id=0$, fork_id=GloballyUniqueId$created_on=zC, local_id=1$\), can't call localValue on user WorkerInfo$id=z, name=z$$. Call it on owner WorkerInfo$id=z$)rNrGrÁrÏr!r'r(r)rÓr r)rHÚ next_rankr‘r*r*r+Útest_local_value_not_on_ownersÿ8þ z%RpcTest.test_local_value_not_on_ownercCsz|jd}||j}tjt|ƒtdddgfd}|D] }tj| ¡ttj |dfdq4dd„|Dƒ}| |dd d g¡dS)Nr&rjr›rÀr„cSs&g|]}tj| ¡ttj|fd‘qS)rÀ)rÁrÂÚownerr’ryr‰)rMr‘r*r*r+r“Asýÿz3RpcTest.test_return_local_rrefs..ròéé )rNrGrÁrÂr!r•r*r’ryrˆrr)rHrˆr‡Z rref_listr‘Úretsr*r*r+Útest_return_local_rrefs1s" ÿ ýüzRpcTest.test_return_local_rrefsc s†dd„}tˆjdˆjƒ}tj|tjt d¡dfd}tƒ}|j |d}|sX| ¡}W5QRXˆ ||jƒ¡t t d¡ƒ‰ˆ |ˆ¡g}‡‡fdd„}tƒL}td ƒD]<} |j |d}|sÞ| |¡| |¡| ¡}ˆ |ˆ¡q®W5QRX|s&|d } |dd…D]}ˆ || k¡qˆ ||jƒ¡ˆ |t t d¡ƒ¡tj|tdd}|j |d}|sv| ¡}ˆ |t¡dS)Ncs&dtjj›d‰t‡fdd„|DƒƒS)NZrpc_z#_rref_typeof_on_ownercsg|]}|j ˆ¡‘qSr*)rJr€)rMr©Z expected_namer*r+r“OszARpcTest._test_rref_type..launched_rpc..)rrmr_rv)rwr*r/r+Úlaunched_rpcMsz-RpcTest._test_rref_type..launched_rpcr&rjrÀ©Úblockingcsˆ | ¡ˆ¡dSrE)rrr_rñ©Z expected_typerHr*r+Úverify_sz'RpcTest._test_rref_type..verifyr„r©r)r!rNrGrÁrÏr'r(r)rcÚ _get_typerðrŒrdÚtyperrrPr)Úadd_done_callbackÚassertFalsery) rHr2r0rÄr‘rnr:r%r4rZ first_futr…Ú rref_typer*r3r+Ú_test_rref_typeJs> zRpcTest._test_rref_typecCs|jdddS©NTr1©r;rQr*r*r+Útest_rref_type_blocking|szRpcTest.test_rref_type_blockingcCs|jdddS©NFr1r=rQr*r*r+Útest_rref_type_non_blockingsz#RpcTest.test_rref_type_non_blockingc Cszt|jd|jƒ}t |t¡}|rJ| td¡|j|dW5QRXn,|j|d}| td¡| ¡W5QRXdS)Nr&r¼r1) r!rNrGrÁrÏrãrÓr½r6rð)rHr2rÄr‘ròr*r*r+Ú_test_rref_type_with_error‚sz"RpcTest._test_rref_type_with_errorcCs|jdddSr<©rArQr*r*r+Ú"test_rref_type_with_error_blocking’sz*RpcTest.test_rref_type_with_error_blockingcCs|jdddSr?rBrQr*r*r+Ú&test_rref_type_with_error_non_blocking•sz.RpcTest.test_rref_type_with_error_non_blockingcCsttt d¡dƒ}|j|d}|s*| ¡}| |tt d¡ƒ¡ttdƒƒ}|j|d}|sd| ¡}| |t¡dS)Nrjr&r1r)rr'r)r6rðrrr7ry)rHr2r‘r:r*r*r+Ú_test_rref_type_owner˜szRpcTest._test_rref_type_ownercCs|jdddSr<©rErQr*r*r+Útest_rref_type_owner_blocking¦sz%RpcTest.test_rref_type_owner_blockingcCs|jdddSr?rFrQr*r*r+Ú!test_rref_type_owner_non_blocking©sz)RpcTest.test_rref_type_owner_non_blockingcCst d¡||Sr%r·r¥r*r*r+Ú _slow_add¬s zRpcTest._slow_addcCsNt|jd|jƒ}tj|tjt d¡dfd}| | ¡tt d¡ƒ¡dSrL)r!rNrGrÁrÏrÒrIr'r)rrr6r7)rHrÄr‘r*r*r+Útest_rref_type_slow_init±sz RpcTest.test_rref_type_slow_initcCsŒtdƒ}tdƒ}|jd|j}tjt|ƒtjt d¡dfd}tjt|ƒtjt d¡dfd}| ¡| ¡| | ¡d¡| | ¡| ¡¡| | ¡| ¡¡| | ¡| ¡¡| | ¡| ¡¡| | ¡| ¡¡| | ¡| ¡¡| | ¡t ¡¡i}||| ¡<||| ¡<| || ¡|¡| || ¡|¡| || ¡|¡| || ¡|¡| t|ƒd¡dS)Né(é2r&rÀérj)rrNrGrÁrÏr!r'r(r)r—rmr*rrrur#)rHrzr}Z other_rankZother_aZother_brûr*r*r+Útest_owner_equality·s>ÿÿzRpcTest.test_owner_equalitycCs„|jd}||j}t|ƒ}tdƒ}| tj|t|dfdd¡| tj|t|dfd ¡d¡| tj |t|dfd ¡d¡dS)Nr&rKrLrÀéZ)rNrGr!rrrrÁrÂr˜rÉrðrÏr—)rHrˆr‡rer‘r*r*r+Útest_pass_local_rrefs×s ÿÿÿzRpcTest.test_pass_local_rrefscCs|jd}||j}tjt|ƒtjt ||¡dfd}tjt|ƒtjt ||¡dfd}tjt|ƒt||fd}| | ¡t ||¡d¡dS)Nr&rjrÀrœ)rNrGrÁrÏr!r'r(r)rµrrr—)rHrˆr‡r³r´r«r*r*r+Útest_remote_same_workerès$ ÿÿÿzRpcTest.test_remote_same_workerTcCsÌddddg}|jd|j}t|ƒ}tj|t|dfd}tj| ¡ttj ||dfdtj | ¡ttj ||dfd ¡tj| ¡ttj ||dfd ¡tj|ttj |fd}| |t|ƒ¡d S) z’ Tests that it is possible to call an instance method on a remote objet by using rref.owner() as destination of the call. r„rjr‹rðr&rrÀr›N)rNrGr!rÁrÏryrÂr*r’rˆrÉrðr—r‰rrr@)rHÚvalsr‡rer‘rar*r*r+Útest_call_method_on_rref÷s2ýý ý ÿz RpcTest.test_call_method_on_rrefÚ)_delete_all_user_and_unforked_owner_rrefsc CsÈtjt|jƒ|j|j|j|jdt|j|j|jƒt ¡tjt|jd|jƒtj t dd¡dfd}ddlmmm}|ršd|_tjddn*d|_| td ¡tjddW5QRXdS) Nrzr&rjrÀrTr‘FzLeaking RRef)rÁr‚r!rNrƒrGrZrr”r—r˜rÏr'r(r)Útorch.distributed.rpc.apirür•Z_ignore_rref_leakr„rÓr )rHZ._mock_delete_all_user_and_unforked_owner_rrefsÚignore_leakr‘r•r*r*r+Ú_test_rref_leak!s*ûýzRpcTest._test_rref_leakcCs|jdddS)NF©rV©rWrQr*r*r+Útest_rref_leak?szRpcTest.test_rref_leakcCs|jdddS)NTrXrYrQr*r*r+Útest_ignore_rref_leakCszRpcTest.test_ignore_rref_leakcCsxt|jƒ}d}| d ||j¡| ¡¡|jd|j}tjt|ƒt j t dd¡dfd}| | ¡d ||j¡¡dS)NZGloballyUniqueIdz(OwnerRRef({}(created_on={}, local_id=0))r&rjrÀz\UserRRef(RRefId = {0}(created_on={1}, local_id=1), ForkId = {0}(created_on={1}, local_id=2)))rrNrrržÚ__str__rGrÁrÏr!r'r(r))rHÚrref1Zid_classr‡Úrref2r*r*r+Ú test_rref_strGs& ÿÿÿþzRpcTest.test_rref_strcCs´|jdkr°tjtdƒtjdd}| ¡| ¡}| |tj j ¡tjtdƒtdd}| ¡| ¡}| |tj j ¡tjtdƒtt d¡fd}| ¡| ¡}| |tj j ¡dS)Nrr&)r&r&rÀr*)rNrÁrÏr!r'r(r—Ú_get_futureÚassertIsInstanceÚ_Crr,rár™)rHr‘ròr*r*r+Útest_rref_get_futureZs zRpcTest.test_rref_get_futurecCsÄt|j|j|jƒt|jƒ}tƒ}| d|¡| d|¡| dt|dƒ¡| dt|dƒ¡t ¡|jd|j}tjt |ƒt|fdtƒt ¡tƒ}| d|¡| dt|dƒ¡| dt|dƒ¡t ¡t t |ƒt¡t|dƒdkrtƒ}t d¡qðt ¡tjt |ƒtjt dd¡dfd}tjt |ƒtjt dd¡dfd}| ¡| ¡tƒt ¡tƒ}| d|¡| dt|dƒ¡| dt|dƒ¡t ¡dS)NZnum_owner_rrefsZnum_pending_usersrr&rÀr3rj)rr”rNrGrr rSrrr3r—r˜rÁrÂr!rërrìrwrxrÏr'r(r)r—)rHr]rr‡r^Zrref3r*r*r+Útest_rref_context_debug_inforsR ÿÿz$RpcTest.test_rref_context_debug_infocs¦|jd|j}tjt|ƒtjt d¡t d¡fdtj ¡ ¡‰| t‡fdd„¡t d¡tjt|ƒtjt d¡t d¡fdtj ¡ ¡‰| dˆ¡dS)Nr&rÀcsˆdS)Núagent.gil_average_wait_time_usr*r*©rr*r+rÿÈrz4RpcTest.test_disable_gil_profiling..Tre)rNrGrÁrÂr!r'r(r)r•r×Úget_debug_inforÜÚKeyErrorZenable_gil_profilingrS)rHr‡r*rfr+Útest_disable_gil_profiling½s ÿ ÿz"RpcTest.test_disable_gil_profilingcCs2tjd|j|j|j|j|jdtjdddSr)rÁr‚rNrƒrGrZr„rQr*r*r+Útest_local_shutdownÐsûzRpcTest.test_local_shutdownc Cs¾ddlmm}tƒ}tƒ}tj ¡ ¡}| ¡}| ¡| ¡@| ¡@}| dt|ƒ¡i}| |¡| |¡| |¡| ¡D]}| || ¡¡q†| ¡D]}| || ¡¡q¤dSrB)Útorch.distributed.autogradrür3rr rÁr•r×rgÚkeysrrr#ÚupdaterS) rHr¨rZ rref_infoZ agent_infoZ autograd_infoZcommon_keysrœrìr*r*r+Útest_debug_infoÞs zRpcTest.test_debug_infozITest is flaky on MacOS since libuv error handling is not as robust as TCPc CsÆtjd|j|j|j|j|jdt d¡t|j|j|jƒt ¡|jdkr¶|jd|j}t|ƒ}| ¡}t ||ƒtj|tjt d¡dfd}| t|¡| ¡W5QRXtjdddS) Nrrzr„r&r›rÀFr‘)rÁr‚rNrƒrGrZr4rr”r—r˜r!Zget_shutdown_error_regexrrÉr'r(r)rÓr rðr„)rHr‡reZ error_strròr*r*r+Útest_handle_send_exceptionsös&û z#RpcTest.test_handle_send_exceptionscCsv|jdkrNt|jd|jƒ}t d¡}t d¡}tj|tj||fdd ¡t ¡}|srt jd|j |j|jddS)Nr&rjér—rr)rNr!rGr'r)rÁrÉrÒrIrðr—Zis_initializedr r”)rHr™rûrüZdist_initializedr*r*r+Ú test_deadlock s üzRpcTest.test_deadlockcCsŠtjd|j|j|j|j|jd|jd}||j}tjt|ƒtj t ||¡t ||¡fdt|j|j|jƒt ¡tjdddS)Nrrzr&rÀFr‘)rÁr‚rNrƒrGrZrÂr!r'r(r)rr”r—r˜r„)rHrˆr‡r*r*r+Útest_local_shutdown_with_rpc* s"û ýz$RpcTest.test_local_shutdown_with_rpccCsPd}|j}||_tjt|jƒ|j|j|j|dt ¡}| ||¡t ¡dS)NrÛrz)rZÚrpc_timeoutrÁr‚r!rNrƒrGÚget_rpc_timeoutrrr„)rHr'rZZset_timeoutr*r*r+Ú$test_set_and_get_default_rpc_timeoutC sûz,RpcTest.test_set_and_get_default_rpc_timeoutc sú|jd|j‰t d¡‡fdd„tdƒDƒ}| ¡}|D]$}| t|¡| ¡W5QRXq.r„éÈr·rÀrr*N)rNrGrÁr4rPr5rÓr rðrÉr!r®Ú constantsÚDEFAULT_RPC_TIMEOUT_SEC)rHr%r6ròrrr*rwr+Útest_default_timeout_usedW s& þ z!RpcTest.test_default_timeout_usedc Csh|jd|j}t|ƒ}d}| ¡}tj|td|d}| t|¡| ¡W5QRXtj|tdd ¡| t|¡tj |td|dW5QRXtj |tddt d¡tj|tdd}| t|¡| ¡W5QRX| t|¡tj |tddW5QRXtj|tddd ¡tj |tdddtj|tddd ¡tj |tdddt tjj ¡dS) Nr&r3r·r—rÀrvr‹r)rNrGr!r5rÁrÉr®rÓr rðrÂr4ryrz)rHr‡rer'r6ròr*r*r+Útest_rpc_timeouts| s, zRpcTest.test_rpc_timeoutscCsBtdddd„ƒ}| ||ƒd¡tdd„ƒ}| ||ƒd¡dS)NFr÷cSsdS©Núexpected resultr*rQr*r*r+Ú test_func§ sz3RpcTest.test_dist_init_decorator..test_funcr~cSsdSr}r*rQr*r*r+r s)rrr)rHrr*r*r+Útest_dist_init_decorator¦ s z RpcTest.test_dist_init_decoratorc CsXGdd„dƒ}|ƒ}t|ƒ| tjjjj|k¡W5QRX| tjjjjtk¡dS)Nc@seZdZdS)z1RpcTest.test_use_rpc_pickler..TestPicklerN)rSrTrUr*r*r*r+ÚTestPickler´ sr)rrŒr'rürÁr•Z_default_picklerr)rHrZtest_picklerr*r*r+Útest_use_rpc_pickler³ s ÿzRpcTest.test_use_rpc_picklerc Cs¾tƒt| tjgk¡t|jd|jƒ}t |t j t dd¡df¡}| ttjƒdk¡| t tjdt jjƒ¡W5QRX| | ¡¡| | ¡t dd¡d¡| ttdƒ¡dS)Nr&rjrÚfuture_list)r rŒrrƒr!rNrGrÁrÉr'r(r)r#rVrbrÚdonerrrðr9Úhasattrr•r*r*r+Ú test_wait_all¾ s$zRpcTest.test_wait_allc CsØtƒ¸| tjgk¡t|jd|jƒ}tdƒD]p}t |t jt ||¡df¡}t |t jt ||¡df¡}| |t ||¡d¡| | ¡t ||¡d¡q4| ttjƒdk¡W5QRX| ttdƒ¡dS)Nr&r†rƒ)r rŒrrƒr!rNrGrPrÁrÉr'r(r)rÂrrrðr#r9r…)rHrÄržròr°r*r*r+Útest_wait_all_multiple_callÊ sz#RpcTest.test_wait_all_multiple_callc Cs€| ¡}| t|¡Rtƒ@| tjgk¡t|jd|j ƒ}d}t j|td|d}W5QRXW5QRX| ttdƒ¡dS)Nr&r3r·r—rƒ)r5rÓr r rŒrrƒr!rNrGrÁrÉr®r9r…)rHr6rÄr'ròr*r*r+Útest_wait_all_timeout× s&zRpcTest.test_wait_all_timeoutc Csl| t¡Htƒ6| tjgk¡t|jd|jƒ}t |t¡}W5QRXW5QRX| t tdƒ¡dS)Nr&rƒ)rÜr½r rŒrrƒr!rNrGrÁrÉrãr9r…r•r*r*r+Ú test_wait_all_raise_in_user_funcâ s z(RpcTest.test_wait_all_raise_in_user_funcc CsB| t¡tƒtƒW5QRXW5QRX| ttdƒ¡dS)Nrƒ)rÜr½r rãr9r…rrQr*r*r+Útest_wait_all_raise_in_bodyë sz#RpcTest.test_wait_all_raise_in_bodycCstj ¡dSrE)rÒÚtimed_out_rpc_eventrðr*r*r*r+Ú timed_out_rpcõ szRpcTest.timed_out_rpcc Cs”tƒt_t|j|j|jƒt ¡t |jd|jƒ}t |tj¡}t |t ¡}t |t ¡}| tt¡tj |||g¡W5QRXtj ¡dSr%)rrÒr‹rr”rNrGr—r˜r!rÁrÉrŒrãrÓr½râr'rørrt©rHrÄrrÚfut3r*r*r+Útest_wait_all_exit_early_pythonù sz'RpcTest.test_wait_all_exit_early_pythonc CsÀtƒt_t|j|j|jƒt ¡t |jd|jƒ}t |tj¡}t j|t jt d¡t d¡fd}t j|t jt d¡t d¡fd}| td¡t j |||g¡W5QRXtj ¡dS)Nr&r„r‹rÀzsize of tensor)rrÒr‹rr”rNrGr—r˜r!rÁrÉrŒr'r(r¸rÓr rørrtrr*r*r+Ú test_wait_all_exit_early_builtins""z(RpcTest.test_wait_all_exit_early_builtinc Cs tƒt_t|j|j|jƒt ¡t |jd|jƒ}t |tj¡}t j|t tfd}t j|t tfd}| tt¡tj |||g¡W5QRXtj ¡dSr›)rrÒr‹rr”rNrGr—r˜r!rÁrÉrŒrårârÓr r'rørrtrr*r*r+Ú(test_wait_all_exit_early_script_function#sz0RpcTest.test_wait_all_exit_early_script_functionc Cs†tjt}d}d}|jdkr:t|dƒtj|t|jfd|jdkr‚tƒ| t |dƒ¡| td¡tj|t ddW5QRXdS) NrRr‹r&r,rÀrzRPC pickler does not serializer*)ÚsysÚmodulesrSrNÚdelattrrÁrÂr`rbrŒr…rÓr r,)rHZthis_moduleZ caller_workerZ callee_workerr*r*r+Útest_function_not_on_callee9s ÿz#RpcTest.test_function_not_on_calleecCsZt|jd|jƒ}tdƒ}tdƒ}||_||_|j}tj|tjt ||¡dfd|_ dSrL)r!rNrGryÚotherrÁrÏr'r(r)r‘)rHr[rzr}rˆr*r*r+ÚEtest_non_garbage_collected_user_rref_due_to_local_circular_dependencyRsýzMRpcTest.test_non_garbage_collected_user_rref_due_to_local_circular_dependencyc CsÊtjd|j|j|j|j|jd|jd}||j}tjt|ƒtj t ||¡t ||¡fd}tjdd| t d¡| ¡W5QRX| t d¡(d dlmmm}| |¡W5QRXdS) Nrrzr&rÀTr‘z-Cannot call to_here on it after deletion.z,Cannot call fork an UserRRef after deletion.r)rÁr‚rNrƒrGrZrÏr!r'r(r)r„rÓr r—Útorch.distributed.rpc.internalrüÚinternalrk)rHrˆr‡r‘r™r*r*r+Útest_use_rref_after_shutdownds4û ýÿÿz$RpcTest.test_use_rref_after_shutdowncCst dd¡ d¡S)Nr›r©r'r¸r1r*r*r*r+Ú_return_gpu_tensor‚szRpcTest._return_gpu_tensorcCs$t dd¡ d¡t dd¡ d¡gS)Nr›rr&r›r*r*r*r+Ú_return_gpu_tensor_list†szRpcTest._return_gpu_tensor_listcCst dd¡Sr¡)r'r¸)Útensor_listr*r*r+Ú_gpu_tensor_list_argŠszRpcTest._gpu_tensor_list_argcCs2|jd|j}tjt|ƒtjt dd¡dfdSr¿)rNrGrÁrÏr!r'r(r)rHrr*r*r+Ú_create_rrefŽsýzRpcTest._create_rrefcCs>|jd|j}| ¡}tjt|ƒt|fd}| |d¡dS©Nr&rÀT)rNrGr rÁrÂr!rírr)rHr‡r‘r"r*r*r+Útest_user_rrefs_confirmed–sýz!RpcTest.test_user_rrefs_confirmedcCsB|jd|j}| ¡}tjt|ƒt|fd}| | ¡d¡dSr¡) rNrGr rÁrÏr!rírrr—)rHr‡r‘rÖr*r*r+Ú test_user_rrefs_confirmed_remote¡sýz(RpcTest.test_user_rrefs_confirmed_remotec CsBtdƒ}tƒ*}| td¡t ||¡W5QRXW5QRXdS)Nr%z%Can not pickle rref in python pickler)rr#rÓr r'Úsave)rHr&rxr*r*r+Ú!test_rref_py_pickle_not_supported¬sz)RpcTest.test_rref_py_pickle_not_supportedc CsLtjt|jd|jƒtt d¡fd}| t d¡| ¡W5QRXdS)Nr&rjrÀz.*Expected error.*)rÁrÏr!rNrGr¾r'r)rÓr¥r—)rHr‘r*r*r+Útest_remote_throw³s þzRpcTest.test_remote_throwcCsš|jdkr–t dd¡}| ddd¡}| | ¡¡| ¡}| | ¡¡| ||¡|jd|j }t jt|ƒt ||fd}| ||¡| | ¡¡dS)Nrr‹r&rjrÀ)rNr'r¸Znarrowr9r¤Ú contiguousrŒrrrGrÁrÂr!r©)rHr:r§r¨r(Zt_retr*r*r+Útest_non_cont_tensors»s zRpcTest.test_non_cont_tensorscsºtj ¡‰ˆjd‰‡‡‡fdd„}tjtˆˆjƒtj t ˆˆ¡t ˆˆ¡fd}| |¡ˆ | ¡t ˆˆ¡d¡ˆ ˆ ¡t ˆˆ¡dd¡ˆ | ¡t ˆˆ¡d¡dS)Nr&cs6| ¡}ˆ |t ˆˆ¡d¡ˆ | ¡d¡dSr)rðrrr'r)r]Úclone)ròr"©rˆrHÚ set_by_cbr*r+röÓsz.RpcTest.test_callback_simple..callbackrÀrj)r÷rørrNrÁrÉr!rGr'r(r)rùrrrðra)rHröròr*rªr+Útest_callback_simpleÎs ý zRpcTest.test_callback_simplec CsŽtj ¡}|jd}tjt||jƒtj t ||¡t ||¡fd}| t¡}| | ¡t ||¡d¡| td¡| ¡W5QRXdS)Nr&rÀrjz8my\_function missing 2 required positional arguments)r÷rørrNrÁrÉr!rGr'r(r)rùr«rrrðrÓr )rHr«rˆròZcb_futr*r*r+Útest_callback_wrong_arg_numäs ý þz#RpcTest.test_callback_wrong_arg_numc Csdt|jd|jƒ}tj|tjt dd¡dfd}| dd„¡}| t d¡| ¡W5QRXdS)Nr&rjrÀcSs|dSr%r*©rûr*r*r+rÿþrz6RpcTest.test_callback_wrong_arg_type..z$unsupported operand type$s$ for \+)r!rNrGrÁrÉr'r(r)rùrÓr rð)rHrÄÚfut0rr*r*r+Útest_callback_wrong_arg_typeùsþz$RpcTest.test_callback_wrong_arg_typecsâd}ˆjd‰‡‡fdd„}tjtˆˆjƒtjt ˆˆ¡t ˆˆ¡fd}g}t|ƒD]}| | t||ƒ¡¡qXˆ | ¡t ˆˆ¡d¡t|ƒD](}ˆ || ¡t ˆˆ¡d|¡q˜ˆ | ¡t ˆˆ¡d¡dS)Nr„r&cs(| ¡}ˆ |t ˆˆ¡d¡||Sri)rðrrr'r))Úidxròr"©rˆrHr*r+rösz-RpcTest.test_callback_multi..callbackrÀrj)rNrÁrÉr!rGr'r(r)rPr)rùrrrrð)rHÚnum_cbsröròZcb_futsr±r*r²r+Útest_callback_multis$ ý þzRpcTest.test_callback_multicCsˆ|jd}t||jƒ}dd„}tjt||jƒtjt ||¡dfd}d}t|ƒD]}| |¡}qT| | ¡t ||¡d|¡dS)Nr&cSs| ¡dSr%rrñr*r*r+rö)sz-RpcTest.test_callback_chain..callbackrÀr†)rNr!rGrÁrÉr'r(r)rPrùrrrð)rHrˆrÄröròr³rr*r*r+Útest_callback_chain$s ýzRpcTest.test_callback_chaincCsht|jd|jƒ}t|jd|jƒ}tj|t|t dd¡ddfd}| |t dd¡dd¡dSrL) r!rNrGrÁrÂrýr'r)rr©rHr™ršr"r*r*r+Útest_callback_in_rpc8sýzRpcTest.test_callback_in_rpccsdt|jd|jƒ‰‡fdd„}tjˆtjt dd¡dfd |¡}| | ¡t dd¡d¡dS)Nr&cs,tjˆtj| ¡dfd dd„¡}| ¡S)Nr&rÀcSs| ¡dSr%rrr*r*r+rÿMrzBRpcTest.test_callback_with_ret..callback..r)r¯rrÃr*r+röHs ýüz0RpcTest.test_callback_with_ret..callbackrjrÀr›)r!rNrGrÁrÉr'r(r)rùrrrð)rHrörŽr*rÃr+Útest_callback_with_retDs ýüzRpcTest.test_callback_with_retc sVtˆjdˆjƒ}‡fdd„}t |t¡ |¡}ˆ td¡| ¡W5QRXdS)Nr&c s,ˆ td¡| ¡W5QRXtdƒ‚dS)Nr¼úAnother expected error)rÓr½rðr )r¯rQr*r+rö]sz2RpcTest.test_callback_with_error..callbackr¹) r!rNrGrÁrÉrãrùrÓr rð)rHrÄrörr*rQr+Útest_callback_with_errorYs z RpcTest.test_callback_with_errorc CsBt|jd|jƒ}| td¡t |t¡ d¡W5QRXdS)Nr&z incompatible function arguments.) r!rNrGrÓrûrÁrÉrãrùrâr*r*r+Útest_callback_nonefsþzRpcTest.test_callback_nonecs°d‰|jd}‡fdd„}tjt||jƒtjt ||¡t ||¡fd}| |¡| dd„¡}| | ¡t ||¡d¡| ¡| ˆ¡| | ¡t ||¡d¡dS) NFr&cs| ¡d‰dS©NTrrñ©r«r*r+rötsz0RpcTest.test_add_done_callback..callbackrÀcSsdSr¼r*)rr*r*r+rÿ€rz0RpcTest.test_add_done_callback..rj) rNrÁrÉr!rGr'r(r)r8rùrrrðrŒ)rHrˆröròZfut_thenr*r½r+Útest_add_done_callbackos ý zRpcTest.test_add_done_callbackc Csptjt|jd|jƒtjt dd¡dfd}| | ¡t dd¡d¡| td¡| d¡W5QRXdS)Nr&rjrÀz(Future can only be marked completed once) rÁrÉr!rNrGr'r(rrrrðrÓr r]©rHròr*r*r+Útest_mark_future_twiceŠsýþzRpcTest.test_mark_future_twicec CsÚtj ¡}d}t|jd|jƒ}tƒ0}| t|¡t j |t|fdW5QRXW5QRXtƒ0}| t|¡t j|t|fdW5QRXW5QRXtƒ0}| t|¡t j |t|fdW5QRXW5QRXdS)Nú#Can not pickle torch.futures.Futurer&rÀ)r'rørr!rNrGr#rÓr rÁrÂrrÉrÏ)rHròÚerrMsgrÄrxr*r*r+Útest_pickle_future˜s &&zRpcTest.test_pickle_futurecCsJt|jd|jƒ}tj|tjt d¡dfd}| ¡| | ¡¡dSrL)r!rNrGrÁrÉr'r(rrðrŒr„r•r*r*r+Útest_future_doneªszRpcTest.test_future_donec CsRt|jd|jƒ}t |t¡}| td¡| ¡W5QRX| | ¡¡dS)Nr&r¼)r!rNrGrÁrÉrãrÓr½rðrŒr„r•r*r*r+Útest_future_done_exception±s z"RpcTest.test_future_done_exceptioncCsht|jd|jƒ}t|jd|jƒ}tj|||t dd¡ddfd}| |t dd¡dd¡dSrL)r!rNrGrÁrÂr'r)rr)rHror™ršr"r*r*r+Ú_test_future_cb¹sýzRpcTest._test_future_cbcCs| t¡dSrE)rÆrrQr*r*r+Útest_future_in_rpcÄszRpcTest.test_future_in_rpccCs| t¡dSrE)rÆrrQr*r*r+Útest_future_nested_callbackÈsz#RpcTest.test_future_nested_callbackc Cs:| td¡$| t|jd|jƒt|¡W5QRXdS)Nr¼r&)rÓr rqr!rNrGr©rHrpr*r*r+Ú_test_async_function_raiseÌsýz"RpcTest._test_async_function_raisecCs| tj¡dSrE)rÊrrlrQr*r*r+Útest_async_function_raiseÔsz!RpcTest.test_async_function_raisecCs| tj¡dSrE)rÊrrmrQr*r*r+Útest_async_function_raise_asyncØsz'RpcTest.test_async_function_raise_asynccCs| tj¡dSrE)rÊrrnrQr*r*r+Ú test_async_function_raise_remoteÜsz(RpcTest.test_async_function_raise_remotec Cs>d}| t|¡$| t|jd|jƒt|¡W5QRXdS)NzZFunctions decorated with @rpc\.async_function must return a torch\.futures\.Future object,r&)rÓr rqr!rNrGr )rHrprÂr*r*r+Ú&_test_async_function_wrong_return_typeàsÿýz.RpcTest._test_async_function_wrong_return_typecCs| tj¡dSrE)rÎrrlrQr*r*r+Ú%test_async_function_wrong_return_typeìsz-RpcTest.test_async_function_wrong_return_typecCs| tj¡dSrE)rÎrrmrQr*r*r+Ú+test_async_function_wrong_return_type_asyncðsz3RpcTest.test_async_function_wrong_return_type_asynccCs| tj¡dSrE)rÎrrnrQr*r*r+Ú,test_async_function_wrong_return_type_remoteôsz4RpcTest.test_async_function_wrong_return_type_remotecCsbt|jd|jƒ}t|jd|jƒ}tj|t|t dd¡dfd}| |t dd¡d¡dSrL) r!rNrGrÁrÂrr'r)rrr¶r*r*r+Útest_async_function_simpleøsz"RpcTest.test_async_function_simplecCsjt|jd|jƒ}t|jd|jƒ}|t dd¡ddf}|j||||d}| |t dd¡d¡dS)Nr&rjrÀr›©r!rNrGr'r)rqrr)rHrorpr™ršrpr"r*r*r+Ú_test_async_functions zRpcTest._test_async_functioncCs| t¡dSrE)rÔrrQr*r*r+Ú$test_async_function_with_future_ctorsz,RpcTest.test_async_function_with_future_ctorcCs| ttj¡dSrE)rÔrrrnrQr*r*r+Ú+test_async_function_with_future_ctor_remotesþz3RpcTest.test_async_function_with_future_ctor_remotecCs| t¡dSrE)rÔrrQr*r*r+Útest_async_function_chainedsz#RpcTest.test_async_function_chainedcCs| ttj¡dSrE)rÔrrrnrQr*r*r+Ú"test_async_function_chained_remotesz*RpcTest.test_async_function_chained_remotecCs| t¡dSrE)rÔr rQr*r*r+Útest_async_function_nestedsz"RpcTest.test_async_function_nestedcCs| ttj¡dSrE)rÔr rrnrQr*r*r+Ú!test_async_function_nested_remotesz)RpcTest.test_async_function_nested_remotecCs| tj¡dSrE)rÔrBrCrQr*r*r+Útest_async_static_method#sz RpcTest.test_async_static_methodcCs| tjtj¡dSrE)rÔrBrCrrnrQr*r*r+Útest_async_static_method_remote'sþz'RpcTest.test_async_static_method_remotecCs| tj¡dSrE)rÔrBrErQr*r*r+Útest_async_class_method.szRpcTest.test_async_class_methodcCs| tjtj¡dSrE)rÔrBrErrnrQr*r*r+Útest_async_class_method_remote2sþz&RpcTest.test_async_class_method_remotecCslt|jd|jƒ}t|jd|jƒ}t |t¡}t dd¡}t dd¡d}|tj kr | ¡ ||||¡}|| ¡ ||||¡7}|| ¡ ||||¡7}n¸|tjkrü| ¡ ||||¡ ¡}|| ¡ ||||¡ ¡7}|| ¡ ||||¡ ¡7}n\|tjkrX| ¡ ||||¡ ¡}|| ¡ ||||¡ ¡7}|| ¡ ||||¡ ¡7}| |d|¡dS)Nr&rjr+)r!rNrGrÁrÏrBr'r)rrlrÂrCrErFrmrÉrðrnr—rr)rHrpr™ršr‘rûrür"r*r*r+Ú!_test_test_async_class_rref_proxy9s$ z)RpcTest._test_test_async_class_rref_proxycCs| ¡dSrE)rßrQr*r*r+Útest_async_class_rref_proxyOsz#RpcTest.test_async_class_rref_proxycCs|jtjddS©N)rp)rßrrmrQr*r*r+Ú!test_async_class_rref_proxy_asyncSsz)RpcTest.test_async_class_rref_proxy_asynccCs|jtjddSrá)rßrrnrQr*r*r+Ú"test_async_class_rref_proxy_remoteWsz*RpcTest.test_async_class_rref_proxy_remotec Csvt|jd|jƒ}t|jd|jƒ}d}d}|t dd¡||f}|j||||d}| |t dd¡||¡dS)Nr&rjr†r›rÀrÓ) rHrorpr™ršrrrpr"r*r*r+Ú_test_async_function_multi[sz"RpcTest._test_async_function_multicCs| t¡dSrE)rärrQr*r*r+Ú!test_async_function_multi_chainedesz)RpcTest.test_async_function_multi_chainedcCs| ttj¡dSrE)rärrrmrQr*r*r+Ú'test_async_function_multi_chained_asyncisþz/RpcTest.test_async_function_multi_chained_asynccCs| ttj¡dSrE)rärrrnrQr*r*r+Ú(test_async_function_multi_chained_remotepsþz0RpcTest.test_async_function_multi_chained_remotecCs| t¡dSrE)rär*rQr*r*r+Ú test_async_function_multi_fanoutwsz(RpcTest.test_async_function_multi_fanoutcCs| ttj¡dSrE)rär*rrmrQr*r*r+Ú&test_async_function_multi_fanout_async{sþz.RpcTest.test_async_function_multi_fanout_asynccCs| ttj¡dSrE)rär*rrnrQr*r*r+Ú'test_async_function_multi_fanout_remote‚sþz/RpcTest.test_async_function_multi_fanout_remotec Cs:| td¡$| t|jd|jƒt|¡W5QRXdS)NrÁr&)rÓr rqr!rNrGrIrÉr*r*r+Ú_test_return_future‰sþýzRpcTest._test_return_futurecCs| tj¡dSrE)rërrlrQr*r*r+Útest_return_future”szRpcTest.test_return_futurecCs| tj¡dSrE)rërrmrQr*r*r+Útest_return_future_async˜sz RpcTest.test_return_future_asynccCs| tj¡dSrE)rërrnrQr*r*r+Útest_return_future_remoteœsz!RpcTest.test_return_future_remotec Csž|jdkrdS|jd|j}d |¡}tj|tddd}| ¡}| t|¡| ¡ ¡W5QRXtƒ| td¡| ¡W5QRXt dddddS) Nrr&zworker{}r@r+r—z RRef creationr|)rNrGržrÁrÏr®r5rÓr r`rðrr—r )rHr‡rer‘r6r*r*r+Útest_rref_timeout s zRpcTest.test_rref_timeoutrÿrzbinit_pg_then_rpc does not work with TCP init, see https://github.com/pytorch/pytorch/issues/41614.cCs˜tjd|j|j|jdtjt|jƒ|j|j|j|j d|jd|j}tj t|ƒtjt dd¡dfd}| |t dd¡d¡t ¡t ¡dS)Nrrrzr&rjrÀ)r—r rLrNrGrÁr‚r!rƒrZrÂr'r(r)rrr˜r„©rHr(r"r*r*r+Útest_init_pg_then_rpc¶s$üû "zRpcTest.test_init_pg_then_rpczbinit_rpc_then_pg does not work with TCP init, see https://github.com/pytorch/pytorch/issues/41614.cCs˜tjt|jƒ|j|j|j|jdtjd|j |j|jd|jd|j}tj t|ƒtjt dd¡dfd}| |t dd¡d¡t ¡t ¡dS)Nrzrrr&rjrÀ)rÁr‚r!rNrƒrGrZr—r rLrÂr'r(r)rrr˜r„rðr*r*r+Útest_init_rpc_then_pgÕs$ûü"zRpcTest.test_init_rpc_then_pgc Cs`g}t|jd|jƒ}tdƒD]}| t |t¡¡q | t d¡t j |¡}W5QRXdS)Nr&r„r¼) r!rNrGrPr)rÁrÉrãrÓr½r'rør©rHr%rÄrr"r*r*r+Útest_wait_all_with_exceptionôsz$RpcTest.test_wait_all_with_exceptionc Cs‚g}t|jd|jƒ}tdƒD]&}| tj|tjt d¡dfd¡q | t |t ¡¡| td¡tj |¡}W5QRXdS)Nr&r„rjrÀr¼)r!rNrGrPr)rÁrÉr'r(r)rãrÓr½rørrór*r*r+Ú$test_wait_all_with_partial_exceptionþs$z,RpcTest.test_wait_all_with_partial_exceptionzUTest does not work with TCP init, see https://github.com/pytorch/pytorch/issues/46491cCsÆt|j|j|jƒtjt|jƒ|j|j|j|jdt ¡t ¡|j}|jd7_tjt|jƒ|j|j|j|dt|jd|jƒ}tj |tjt dd¡dfdtj |tddt ¡dS)NrzZinit_2r&rjrÀr*)rr”rNrGrÁr‚r!rƒrZr„r—r˜rLrÂr'r(r)r,)rHZnew_backend_optionsrÄr*r*r+Útest_init_rpc_twice s.ûû zRpcTest.test_init_rpc_twicec Csv| td¡"tjt|jƒ|j|jddW5QRX| td¡,tjt|jƒ|j|j|jd|jidW5QRXdS)Nz0Argument backend must be a member of BackendTypeÚ TENSORPIPE)rJrNrGr{zEArgument rpc_backend_options must be an instance of RpcBackendOptionsrL©rJrNrGr{rZ) rÓrûrÁr‚r!rNrGrƒrLrQr*r*r+Útest_wrong_types2s*þüþûzRpcTest.test_wrong_typesc CsBt|jƒ}| td¡"tjt|jƒ|j|j|dW5QRXdS)Nz#Could not infer backend for options©rJrNrGrZ) rJrLrÓrûrÁr‚r!rNrG©rHrZr*r*r+Ú&test_cannot_infer_backend_from_optionsJs ûz.RpcTest.test_cannot_infer_backend_from_optionsc CsÜt|jd|jƒ}tjdddd}t | ¡| ¡¡}| ¡t |¡d}| ||j¡t ¡J}tj|tj||fd}t | ¡¡}| |¡| |t |¡|¡W5QRXt ¡\}tj|tj||fd}t | ¡¡}|j|dd| |¡| |dt |¡|¡W5QRX| td¡t t d¡¡ ¡W5QRX| td ¡t tjddd¡ ¡W5QRX| td ¡$t tjddd ¡¡ d¡W5QRX| td¡t d ¡ ¡W5QRXdS)Nr&r„Tr”rjrÀ©Zretain_graphz9tensors does not require grad and does not have a grad_fnz6grad can be implicitly created only for scalar outputsz,Could not find autograd context with id: 100rÚú,RRef should contain a tensor for .backward()rQ)r!rNrGr'r¸rÁrr@rºÚ ones_likerrr½r¨r©rÂr(Ú get_gradientsrÓr )rHrÄr8r‘Z expected_gradrªr9r*r*r+Útest_owner_rref_backwardYs2 $"(z RpcTest.test_owner_rref_backwardcCs| ¡SrE)r@r®r*r*r+Ú_sum}szRpcTest._sumcCs|SrEr*r®r*r*r+Ú _identityszRpcTest._identityc Csît|jd|jƒ}tjddd}t ¡R}tj|t j |fd}|j|dd| |¡| t |¡dt |¡|¡W5QRXt ¡Z}tj|t jdd}| td ¡| |¡W5QRX| td ¡| ¡W5QRXW5QRXdS)Nr&r„Tr”rÀrýrj)rQrþz9User RRefs require 'dist_autograd_ctx_id' to be specified)r!rNrGr'r¸r¨r©rÁrÏrÒrrºrrrÿrrrÓr )rHrÄr:rªr‘r*r*r+Útest_user_rref_backward…s * zRpcTest.test_user_rref_backwardc sØt|j|j|jƒtjt|jƒ|j|j|j|jd|jdkr¬tj j ‰tj j}‡fdd„}dd„}z4|tj _ |tj _| t d¡t ¡W5QRXW5ˆtj _ |tj _Xn | t d¡t ¡W5QRXt ¡dS) Nrzrcsˆ||ƒtdƒ‚dS)NZ simulationr )Zsequence_idZobjects_map©r,r*r+Úraise_error©s z1RpcTest.test_shutdown_errors..raise_errorcSstdƒ‚dS)Núsimulation rrefr r*r*r*r+Ú rref_error¯sz0RpcTest.test_shutdown_errors..rref_errorrztimed out in _all_gather)rr”rNrGrÁr‚r!rƒrZr•Z_broadcast_to_followersrTrÓr r„r—r˜)rHZog_rref_funcrrr*rr+Útest_shutdown_errors—s.û zRpcTest.test_shutdown_errorscCs| d¡dSr"©rÂrQr*r*r+Útest_my_parameter_serverÀsz RpcTest.test_my_parameter_server)F)r)FNF)F)F(rSrTrUrrÔrØrÚrÞràrãrärårærérìrírîrïrórôrõrÚpatchÚobjectr'rür3rÁr•rùrýrþrrrrrrrŽrrrrr r!r$r%r-r/r0r2r7r:r?rCrDrHrIrJrKrNrPrUr^rjror{rˆrr’r“r–r›ržrŸr¦r§r«rr®r¶r¹r»r¼r¾rÀrÁrÂrÄrÅrÆrÈrÉrËrÍrÎrÐrÒrÓrÔrÖr×rØrÚrÛrÜrÞrßràrärçrêrírîrïrðrñròrórôrörørùrürÿrrrr r rrrrrrrrrrrrrr r!r"r$r'r)r.r;r>r@rArCrDrErGrHrIrJrNrPrQrSrWrZr[r_rcrdrirjrnrrrorqrrrur{r|r€r‚r†r‡rˆr‰rŠr‹rŒrrr‘r•r—ršrœrrŸr r¢r£r¥r¦r¨r¬rr°r´rµr·r¸rºr»r¾rÀrÃrÄrÅrÆrÇrÈrÊrËrÌrÍrÎrÏrÐrÑrÒrrlrÔrÕrÖr×rØrÙrÚrÛrÜrÝrÞrßràrârãrärårærçrèrérêrërìrírîrïrrrrñròrôrõrörùrürrrrr rr*r*r*r+rÒUsØ 7 D; , 8 ÿ M 1 )J þ$) þþ þ##(rÒc@s eZdZedƒedd„ƒƒZdS)ÚCudaRpcTestrjc sž|jdkrdS|jd|j}|jd|j}t|ƒ}t|ƒ}tdd6}tj|tdd}tj|tdd}| ¡| ¡W5QRXdd „‰|j}|D]Î} | j rÌ| d | j¡| g| j¡| d | j ¡q–| jdkrØq–| | j||fk¡ˆ| ƒtkr–| | jd ¡| dt| jƒ¡| jd } | j|kr<| | jd ¡| j|krV| | jd¡| | j d ¡q–dd„|Dƒ}‡fd d„|Dƒ}| t|ƒttƒ¡dS)Nr&rjT)Zuse_cudar5rÀr·cSs|j|j t¡ttƒd…SrE)rJrŠr‰r#)r9r*r*r+Úget_nameØsz7CudaRpcTest.test_profiler_remote_cuda..get_namercSsg|]}|jr|‘qSr*r_rar*r*r+r“ñsz9CudaRpcTest.test_profiler_remote_cuda..cs g|]}ˆ|ƒtkrˆ|ƒ‘qSr*)rura©rr*r+r“òs)rNrGr!rcrÁrÉr;rðrdZis_asyncrrZcuda_time_totalZkernelsZ cuda_timerrŒruZ assertGreaterr#r2rt) rHZ dst_cuda_0Z dst_cuda_1Zdst_worker_cuda_0Zdst_worker_cuda_1rnrrrdr9Úkernelrgr‚r*rr+Útest_profiler_remote_cudaÇsB z%CudaRpcTest.test_profiler_remote_cudaN)rSrTrUrrrr*r*r*r+rÅsrc@seZdZdd„Zdd„Zedddd„ƒZeddd d „ƒZedddd„ƒZed d„ƒZ dd„Z dd„Zedd„ƒZdd„Z edd„ƒZedd„ƒZedd„ƒZedd„ƒZedd „ƒZed!d"„ƒZed#d$„ƒZd%d&„Zed'd(„ƒZd)d*„Zd+d,„Zed-d.„ƒZed/d0„ƒZed1d2„ƒZed3d4„ƒZed5d6„ƒZed7d8„ƒZed9d:„ƒZed;d<„ƒZ ed=d>„ƒZ!ed?d@„ƒZ"edAdB„ƒZ#edCdD„ƒZ$edEdF„ƒZ%edGdH„ƒZ&edddIdJ„ƒZ'edddKdL„ƒZ(edddMdN„ƒZ)e*dOƒedddPdQ„ƒƒZ+edddRdS„ƒZ,edddTdU„ƒZ-dVS)WÚTensorPipeAgentRpcTestc CsHt|jƒ}| td¡(tjt|jƒ|j|jtj j |dW5QRXdS)Nz=`rpc_backend_options` must be a `TensorPipeRpcBackendOptions`rø)rJrLrÓrûrÁr‚r!rNrGZBackendTyper÷rûr*r*r+Ú test_mismatched_type_for_optionsøs ÿûz7TensorPipeAgentRpcTest.test_mismatched_type_for_optionscCsFtj|jtƒd}tjt|jƒ|j|j|d| tj ¡tj¡dS)N)rLÚ_transportsrú)rÁÚTensorPipeRpcBackendOptionsrLrr‚r!rNrGrar•r×ZTensorPipeAgentrûr*r*r+Útest_infer_backend_from_optionssþûz6TensorPipeAgentRpcTest.test_infer_backend_from_optionsFr÷cCshd}tj|jj|tƒd}tjt|jƒ|j|j|j |dtj ¡ ¡}| t|dƒ|¡t ¡dS)Né)rLÚnum_worker_threadsrrzzagent.thread_pool_size)rÁrrZrLrr‚r!rNrƒrGr•r×rgrrr3r„)rHZNUM_THREADSrZrr*r*r+Ú#test_set_and_get_num_worker_threadss ýûz:TensorPipeAgentRpcTest.test_set_and_get_num_worker_threadscCs`d}tj|jj|jj|tƒd}tjt|jƒ|j |j|j |dt ¡}| ||¡t ¡dS)NrÚ)rLrrsrrz)rÁrrZrLrrr‚r!rNrƒrGrtrrr„)rHr'rZZdefault_timeoutr*r*r+Ú#test_tensorpipe_set_default_timeout.s"üûz:TensorPipeAgentRpcTest.test_tensorpipe_set_default_timeoutc CsFddlm}|ƒ}| td¡tj|jj|jj|d}W5QRXdS)Nr)Ú timedeltaz"incompatible constructor arguments)rLrrs) ÚdatetimerrÓrûrÁrrZrLr)rHrr'rZr*r*r+Ú2test_tensorpipe_options_throw_on_timedelta_timeoutFsýzITensorPipeAgentRpcTest.test_tensorpipe_options_throw_on_timedelta_timeoutc Cs¦|jd|j}t|ƒ}tj|tt dd¡dfd}d}| ¡}|rl| t |¡|j||dW5QRXn.|j||d}| t |¡| ¡W5QRX| ¡dS)Nr&rjTrÀrÛ)r'r2)rNrGr!rÁrÏryr'r)r5rÓr r6rðr—)rHr2r‡rÄÚ slow_rrefr'râròr*r*r+Ú_test_rref_get_type_timeoutSsz2TensorPipeAgentRpcTest._test_rref_get_type_timeoutcCs|jdddSr<©r rQr*r*r+Ú#test_rref_get_type_timeout_blockingjsz:TensorPipeAgentRpcTest.test_rref_get_type_timeout_blockingcCs|jdddSr?r!rQr*r*r+Ú'test_rref_get_type_timeout_non_blockingmsz>TensorPipeAgentRpcTest.test_rref_get_type_timeout_non_blockingc CsBt|jd|jƒ}| td¡tj|tjddW5QRXdS)Nr&zGOverloaded torch operator invoked from Python failed to many any schemar*rÀ) r!rNrGrÓr rÁrÂr'r(râr*r*r+Útest_op_with_invalid_argspsÿz0TensorPipeAgentRpcTest.test_op_with_invalid_argsc Cs:|jd|j}t|ƒ}tj|tt dd¡fd}| ¡t ||ƒ}| |dk d|›¡| ¡}d}| t |¡H||d t dd¡¡}||jkr | ¡n||jkr¶| ¡ ¡W5QRXtj|tt dd¡dfd} d}t | |ƒ}| t |¡2||d t dd¡¡}|| jkr$| ¡W5QRX| ¡dS)Nr&rjrÀzFailed to get RRef proxy api: r&Tr+)rNrGr!rÁrÏryr'r)r—ÚgetattrrŒr5rÓr rŒrÉrðr`r~) rHZrref_proxy_apir‡rÄr‘Zrref_apir6r'rarr*r*r+Ú_test_rref_proxy_timeoutxs, z/TensorPipeAgentRpcTest._test_rref_proxy_timeoutcCsdD]}| |¡qdS)N)rÂrÉrÏ)r&)rHZrpc_apir*r*r+Útest_rref_proxy_timeoutœsz.TensorPipeAgentRpcTest.test_rref_proxy_timeoutcCs¸|jd|j}tjtjtjfD]<}tƒ}tƒ}||}|j|tj |||fd}| ||¡q tjtjtjfD]D}tdd}tdd}||}|j|tj |||fd}| ||¡qndS)Nr&rÀT)r)rNrGrrlrmrnr rqr'r(rr)rHr‡rÝrûrürÐr"r*r*r+Útest_send_to_rank_sparse¡s z/TensorPipeAgentRpcTest.test_send_to_rank_sparsecCs| t ¡tƒtƒtƒ¡dSrE)rtrÁrur rQr*r*r+Útest_self_py_udf_remote_sparse´süz5TensorPipeAgentRpcTest.test_self_py_udf_remote_sparsecCs.t|jd|jƒ}| |tƒtƒtƒ¡dSr%)r!rNrGrwr râr*r*r+Ú'test_self_remote_rref_as_rpc_arg_sparse½süz>TensorPipeAgentRpcTest.test_self_remote_rref_as_rpc_arg_sparsecCs| t ¡tƒtƒtƒ¡dSrE)rwrÁrur rQr*r*r+Ú,test_self_remote_rref_as_self_rpc_arg_sparseÇsüzCTensorPipeAgentRpcTest.test_self_remote_rref_as_self_rpc_arg_sparsecCs.t|jd|jƒ}| |tƒtƒtƒ¡dSr%)r!rNrGrxr râr*r*r+Ú*test_self_remote_rref_as_remote_arg_sparseÐsüzATensorPipeAgentRpcTest.test_self_remote_rref_as_remote_arg_sparsecCs| t ¡tƒtƒtƒ¡dSrE)rxrÁrur rQr*r*r+Ú/test_self_remote_rref_as_self_remote_arg_sparseÚsüzFTensorPipeAgentRpcTest.test_self_remote_rref_as_self_remote_arg_sparsecCs| tƒtƒ¡dSrE)r…r rQr*r*r+Útest_world_size_one_sparseãsþz1TensorPipeAgentRpcTest.test_world_size_one_sparsecCs| d¡dSr¼r#rQr*r*r+Útest_multi_rpc_sparseész,TensorPipeAgentRpcTest.test_multi_rpc_sparsecCs| ttƒ¡dSrE)r–rÞr rQr*r*r+Útest_wait_all_workers_sparseísz3TensorPipeAgentRpcTest.test_wait_all_workers_sparsecCs| ttƒ¡dSrE)ršrÞr rQr*r*r+Ú"test_wait_all_workers_twice_sparseðsz9TensorPipeAgentRpcTest.test_wait_all_workers_twice_sparsecCsJ|jd}||j}tƒtƒg}tjt|ƒt|fd}| |t|ƒ¡dSr›)rNrGr rÁrÂr!r¬rr)rHrˆr‡rzr"r*r*r+Ú#test_py_sparse_tensors_in_containerós ÿz:TensorPipeAgentRpcTest.test_py_sparse_tensors_in_containercCs| ttƒd¡dSri)rrÇr rQr*r*r+Útest_nested_rpc_sparseýsz-TensorPipeAgentRpcTest.test_nested_rpc_sparsecCs|jtdtƒfddS)Nr†r)r¢rÞr rQr*r*r+Útest_stress_heavy_rpc_sparsesz3TensorPipeAgentRpcTest.test_stress_heavy_rpc_sparsecCs| tƒtƒtƒd¡dSri)r£r rQr*r*r+Útest_builtin_remote_ret_sparses ýz5TensorPipeAgentRpcTest.test_builtin_remote_ret_sparsecCs| tƒtƒtƒd¡dSri)r¤r rQr*r*r+Útest_builtin_remote_self_sparse s ýz6TensorPipeAgentRpcTest.test_builtin_remote_self_sparsecCs|jtjdtjddS)NTrrrQr*r*r+Ú$test_multi_builtin_remote_ret_sparses þz;TensorPipeAgentRpcTest.test_multi_builtin_remote_ret_sparsecCs|jtdtjddS)NTrrrQr*r*r+Útest_multi_py_udf_remote_sparses ýz6TensorPipeAgentRpcTest.test_multi_py_udf_remote_sparsecCs$| tƒtƒtƒtƒtƒd¡dS)Nrœ)r¬r rQr*r*r+Útest_py_rref_args_sparse$sûz/TensorPipeAgentRpcTest.test_py_rref_args_sparsec Cs,| tƒtƒtƒtƒtƒtƒtƒd¡dS©Né)r®r rQr*r*r+Ú#test_py_rref_args_user_share_sparse.sùz:TensorPipeAgentRpcTest.test_py_rref_args_user_share_sparsec Cs,| tƒtƒtƒtƒtƒtƒtƒd¡dSr:)r¯r rQr*r*r+Útest_py_rpc_rref_args_sparse:sùz3TensorPipeAgentRpcTest.test_py_rpc_rref_args_sparsecCs| ttƒtƒ¡dSrE)r²rÔr rQr*r*r+Útest_nested_remote_sparseFs þz0TensorPipeAgentRpcTest.test_nested_remote_sparsecCs| ttƒdtƒd¡dSri)r¶rÑr rQr*r*r+Útest_nested_rref_sparseMs ýz.TensorPipeAgentRpcTest.test_nested_rref_sparsecCs| ttƒdtƒd¡dSri)r·rÑr rQr*r*r+Útest_nested_rref_stress_sparseUs ýz5TensorPipeAgentRpcTest.test_nested_rref_stress_sparsecCs| d¡dSr¼r rQr*r*r+Útest_my_parameter_server_sparse]sz6TensorPipeAgentRpcTest.test_my_parameter_server_sparsecCs*tjt|jƒ|j|j|jdt ¡dS)N©rJr{rNrZ)rÁr‚r!rNrƒrZr„rQr*r*r+Útest_dynamic_rpc_init_rpcbsüz0TensorPipeAgentRpcTest.test_dynamic_rpc_init_rpccCsÄt|j|j|jƒ|jdkr:tjt|jƒ|j|j|jdt ¡|jdkr°tjt|jƒ|j|j|jdtjtdƒtj t d¡t d¡fd}| t t d¡t d¡¡|¡t ¡t ¡dS©NrrBr&rÀ)rr”rNrGrÁr‚r!rƒrZr—r˜rÂr'r(r™rrr„)rHrar*r*r+Ú=test_dynamic_rpc_new_rank_can_communicated_with_existing_rankms( ü ü& zTTensorPipeAgentRpcTest.test_dynamic_rpc_new_rank_can_communicated_with_existing_rankc Csèt|j|j|jƒ|jdkr:tjt|jƒ|j|j|jdt ¡|jdkrjtjt|jƒ|j|j|jdt ¡|jdkrÔtd|jƒD]J}tjt|ƒt jt d¡t d¡fd}| t t d¡t d¡¡|¡qˆt ¡t ¡dSrD)rr”rNrGrÁr‚r!rƒrZr—r˜rPrÂr'r(r™rrr„)rHržrar*r*r+Úr?r@rArCrErFrrJrKrLr*r*r*r+rös– $ #, rc@s€eZdZdd„Zedƒdd„ƒZedƒdd„ƒZedƒd d „ƒZedƒdd„ƒZedƒd d„ƒZ edƒdd„ƒZ edd„ƒZedƒdd„ƒZ edd„ƒZdådd„Zdd„Zedƒdd„ƒZedƒdd„ƒZedƒd d!„ƒZedƒd"d#„ƒZedƒd$d%„ƒZedƒd&d'„ƒZedƒd(d)„ƒZedƒd*d+„ƒZedƒd,d-„ƒZedƒd.d/„ƒZedƒd0d1„ƒZedƒd2d3„ƒZedƒd4d5„ƒZedƒd6d7„ƒZedƒd8d9„ƒZedƒd:d;„ƒZ edƒdd?„ƒZ"edƒd@dA„ƒZ#edƒdBdC„ƒZ$edƒdDdE„ƒZ%edƒdFdG„ƒZ&edƒdHdI„ƒZ'edƒdJdK„ƒZ(edLdM„ƒZ)dNdO„Z*edƒdPdQ„ƒZ+edƒdRdS„ƒZ,edTdU„ƒZ-edƒdVdW„ƒZ.dXdY„Z/edZƒd[d\„ƒZ0edZƒd]d^„ƒZ1ed_d`„ƒZ2dadb„Z3dcdd„Z4edƒe5dedf„ƒƒZ6edƒdgdh„ƒZ7edƒe5didj„ƒƒZ8edƒe5dkdl„ƒƒZ9edƒe5dmdn„ƒƒZ:edƒe5dodp„ƒƒZ;edƒe5dqdr„ƒƒZdwdx„Z?dydz„Z@edƒd{d|„ƒZAd}d~„ZBedƒdd€„ƒZCedd‚„ƒZDdƒd„„ZEedƒd…d†„ƒZFd‡dˆ„ZGedƒd‰dŠ„ƒZHed‹dŒ„ƒZIedƒddŽ„ƒZJdd„ZKedƒd‘d’„ƒZLedƒd“d”„ƒZMedƒd•d–„ƒZNedƒd—d˜„ƒZOdæd™dš„ZPedƒd›dœ„ƒZQedƒddž„ƒZRedƒdŸd „ƒZSedƒd¡d¢„ƒZTedƒd£d¤„ƒZUed¥d¦„ƒZVd§d¨„ZWedƒd©dª„ƒZXedƒd«d¬„ƒZYedƒdd®„ƒZZedƒd¯d°„ƒZ[d±d²„Z\edƒd³d´„ƒZ]edƒdµd¶„ƒZ^edƒd·d¸„ƒZ_edƒd¹dº„ƒZ`ed»d¼„ƒZaedƒd½d¾„ƒZbedƒd¿dÀ„ƒZcedƒdÁdÂ„ƒZdedƒdÃdÄ„ƒZeedƒdÅdÆ„ƒZfedƒdÇdÈ„ƒZgedƒdÉdÊ„ƒZhedƒdËdÌ„ƒZiedƒdÍdÎ„ƒZjedƒdÏdÐ„ƒZkedƒdÑdÒ„ƒZledƒdÓdÔ„ƒZmedƒdÕdÖ„ƒZnedƒd×dØ„ƒZoedƒdÙdÚ„ƒZpedƒdÛdÜ„ƒZqedƒdÝdÞ„ƒZredƒdßdà„ƒZsedƒdádâ„ƒZtedƒdãdä„ƒZudS)çÚTensorPipeAgentCudaRpcTestc CsL| t|¡&tjt|jƒ|j|j|j|dW5QRX| tj ¡¡dS)Nrz)rÓr½rÁr‚r!rNrƒrGr9r•Z_is_current_rpc_agent_set)rHrIrÂr*r*r+Ú_test_device_mapssûz,TensorPipeAgentCudaRpcTest._test_device_mapsrjcCs(|j}| dddi¡|j|dddS)NZ none_existrr&z=Node worker0 has invalid target node names in its device maps©rÂ)rZrHrN)rHrIr*r*r+Ú"test_device_maps_wrong_worker_name)sþz=TensorPipeAgentCudaRpcTest.test_device_maps_wrong_worker_namer&cCsB|j}t|jd|jƒ}| |tj ¡di¡|j|dddS)Nr&rzRNode worker0 has source devices with invalid indices in its device map for worker1rO© rZr!rNrGrHr'r1Zdevice_countrN©rHrIrÄr*r*r+Ú)test_device_maps_invalid_max_local_device3sþzDTensorPipeAgentCudaRpcTest.test_device_maps_invalid_max_local_devicecCsB|j}t|jd|jƒ}| |dtj ¡i¡|j|dddS)Nr&rzRNode worker0 has target devices with invalid indices in its device map for worker1rOrQrRr*r*r+Ú*test_device_maps_invalid_max_remote_device>sþzETensorPipeAgentCudaRpcTest.test_device_maps_invalid_max_remote_devicecCsL|j}t|jd|jƒ}| |ddi¡| |ddi¡|j|dddS)Nr&rzHNode worker0 has duplicated target devices in its device map for worker1rO)rZr!rNrGrHrNrRr*r*r+Útest_device_maps_many_to_oneIsþz7TensorPipeAgentCudaRpcTest.test_device_maps_many_to_onec Cs`|jdkr\|j}t|jd|jƒ}| |ddi¡| td¡| |ddi¡W5QRXdS)Nrr&z-`set_device_map` only supports 1-to-1 mapping)rNrZr!rGrHrÓr½rRr*r*r+Útest_device_maps_one_to_manyUs ÿz7TensorPipeAgentCudaRpcTest.test_device_maps_one_to_manyc Csn|j}t|jd|jƒ}| td¡| |ddi¡W5QRX| td¡| |ddi¡W5QRXdS)Nr&z!Device index must not be negativer-r)rZr!rNrGrÓr rHrRr*r*r+Ú#test_device_maps_invalid_min_device`sÿÿz>TensorPipeAgentCudaRpcTest.test_device_maps_invalid_min_devicecCs>t|j|jjdk|j|jjdkgƒr2|| d¡Stdƒ‚dS©Nr&rúWrong device affinity©rÈÚis_cudar2rÁrúr½r¥r*r*r+Ú_gpu_addns$z#TensorPipeAgentCudaRpcTest._gpu_addcCs¼|j}t|jd|jƒ}| |dddœ¡tjt|jƒ|j|j|j|dtj|t j t d¡ d¡t d¡ d¡fd}| |jt d¡¡| |t d¡t d¡ d¡¡t ¡dS)Nr&r©rr&rzrjrÀ)rZr!rNrGrHrÁr‚rƒrÂrMr\r'rrúr)rrr2r„)rHrIrÄr"r*r*r+Útest_device_maps_gpuus$ûý"z/TensorPipeAgentCudaRpcTest.test_device_maps_gpucCs`|jjdkrdn|jj}|jjdkr(dn|jj}||krT||krT| |¡| |¡Stdƒ‚dS)NrrY)r2r7rÁrúr½)rûrüÚx_toÚy_toÚz_toZx_deviceZy_devicer*r*r+Ú_gpu_add_given_devicesŒs z1TensorPipeAgentCudaRpcTest._gpu_add_given_devicesNcs|dkrtjn|}ˆ|}ˆ|}|j} |dkrDt|jd|jƒn|}| |ˆ¡tjt|jƒ|j |j|j| dt d¡ |¡} t d¡ |¡}tj||| ||||fd}‡fdd„ˆDƒ} | |}|jjdkrØdn|jj}| ||¡| |t d¡ |¡¡t ¡dS)Nr&rzrjrÀcsi|]}ˆ||“qSr*r*)rMr±©Ú device_mapr*r+r~«szDTensorPipeAgentCudaRpcTest._test_device_maps_gpu..r)rMrbrZr!rNrGrHrÁr‚rƒr'rrúr)rÂr2r7rÁrrr„)rHÚx_fromÚy_fromrardrÄror_r`rIrûrür"Zreverse_device_mapZz_fromZ ret_devicer*rcr+Ú_test_device_maps_gpu•s, ûz0TensorPipeAgentCudaRpcTest._test_device_maps_gpucCs|jddddditjddS)Nr©rerfrardro©rgrMrbrQr*r*r+Útest_device_map_cpu´sûz.TensorPipeAgentCudaRpcTest.test_device_map_cpucCs|jddddditjddS)NrrrhrirQr*r*r+Ú"test_device_map_cpu_to_gpu_default½sûz=TensorPipeAgentCudaRpcTest.test_device_map_cpu_to_gpu_defaultcCs|jddddditjddS)Nrr&rhrirQr*r*r+Ú&test_device_map_cpu_to_gpu_non_defaultÇsûzATensorPipeAgentCudaRpcTest.test_device_map_cpu_to_gpu_non_defaultcCs|jddddditjddS)NrrrhrirQr*r*r+Ú"test_device_map_gpu_to_cpu_defaultÑsûz=TensorPipeAgentCudaRpcTest.test_device_map_gpu_to_cpu_defaultcCs|jddddditjddS)Nr&rrhrirQr*r*r+Ú&test_device_map_gpu_to_cpu_non_defaultÛsûzATensorPipeAgentCudaRpcTest.test_device_map_gpu_to_cpu_non_defaultcCs|jdddddiddS)Nr©rerfrard©rgrQr*r*r+Útest_device_map_gpu_defaultåsüz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_defaultcCs|jdddddiddS)Nr&rorprQr*r*r+Útest_device_map_gpu_non_defaultîsüz:TensorPipeAgentCudaRpcTest.test_device_map_gpu_non_defaultcCs|jdddddiddS)Nrr&rorprQr*r*r+Ú*test_device_map_gpu_default_to_non_default÷süzETensorPipeAgentCudaRpcTest.test_device_map_gpu_default_to_non_defaultcCs|jdddddiddS)Nr&rrorprQr*r*r+Ú*test_device_map_gpu_non_default_to_defaultsüzETensorPipeAgentCudaRpcTest.test_device_map_gpu_non_default_to_defaultcCs|jddddddœddS©Nrr&r]rorprQr*r*r+Útest_device_map_gpu_mixed_1 süz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_1cCs|jddddddœddSrurprQr*r*r+Útest_device_map_gpu_mixed_2süz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_2cCs|jddddddœddS©Nr&rr]rorprQr*r*r+Útest_device_map_gpu_mixed_3süz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_3cCs|jddddddœddSrxrprQr*r*r+Útest_device_map_gpu_mixed_4$süz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_4cCs|jddddddœddSrurprQr*r*r+Útest_device_map_gpu_mixed_5-süz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_5cCs|jddddddœddSrurprQr*r*r+Útest_device_map_gpu_mixed_66süz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_6cCs|jddddddœddSrxrprQr*r*r+Útest_device_map_gpu_mixed_7?süz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_7cCs|jddddddœddSrxrprQr*r*r+Útest_device_map_gpu_mixed_8Hsüz6TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_8cCs$|jddddddœt|jƒddS©Nrr&r]©rerfrardrÄ©rgr!rNrQr*r*r+Ú test_device_map_gpu_mixed_self_1Qsûz;TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_self_1cCs$|jddddddœt|jƒddSrrrQr*r*r+Ú test_device_map_gpu_mixed_self_2[sûz;TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_self_2cCs$|jddddddœt|jƒddS©Nr&rr]r€rrQr*r*r+Ú test_device_map_gpu_mixed_self_3esûz;TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_self_3cCs$|jddddddœt|jƒddSr„rrQr*r*r+Ú test_device_map_gpu_mixed_self_4osûz;TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_self_4cCs$|jddddddœt|jƒddSrrrQr*r*r+Ú test_device_map_gpu_mixed_self_5ysûz;TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_self_5cCs$|jddddddœt|jƒddSrrrQr*r*r+Ú test_device_map_gpu_mixed_self_6ƒsûz;TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_self_6cCs$|jddddddœt|jƒddSr„rrQr*r*r+Ú test_device_map_gpu_mixed_self_7sûz;TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_self_7cCs$|jddddddœt|jƒddSr„rrQr*r*r+Ú test_device_map_gpu_mixed_self_8—sûz;TensorPipeAgentCudaRpcTest.test_device_map_gpu_mixed_self_8cCsLt|j|jjdk|j|jjdkgƒr@| d¡||| d¡fStdƒ‚dSrXrZr¥r*r*r+Ú_gpu_add_multi_gpu¡s$z-TensorPipeAgentCudaRpcTest._gpu_add_multi_gpucCs|j}| |ddi¡| |ddi¡tjt|jƒ|j|j|j|dt d¡ d¡}t d¡ d¡}tj|t j||fd}| |djt d¡¡| |djt d¡¡| |dt d¡t d¡ d¡¡| |dt d¡t d¡ d¡¡t ¡dS)Nrr&rzrjrÀ)rZrHrÁr‚r!rNrƒrGr'rrúr)rÂrMr‹rrr2r„©rHrÄrIrûrür-r*r*r+Ú_test_device_maps_multi_gpu¨s,ûý&&z6TensorPipeAgentCudaRpcTest._test_device_maps_multi_gpucCs"t|jd|jƒ}| |¡dSr%)r!rNrGrrâr*r*r+Útest_device_maps_multi_gpuÃsz5TensorPipeAgentCudaRpcTest.test_device_maps_multi_gpucCst|jƒ}| |¡dSrE)r!rNrrâr*r*r+Útest_device_maps_multi_gpu_selfÈs z:TensorPipeAgentCudaRpcTest.test_device_maps_multi_gpu_selfcCsX|jjdkrL|jjdkrL|| d¡|| d¡|| d¡|| d¡fStdƒ‚dS)Nrrr&rjr›rY)r2r7rúr½r¥r*r*r+Ú_gpu_add_return_to_gpuÍs4z1TensorPipeAgentCudaRpcTest._gpu_add_return_to_gpucCst|jd|jƒ}|j}tjt|jƒ|j|j|jtj|j|j |dddœit ƒddtj|tj t d¡ d¡t d¡ d¡fd}| |djt d¡¡| |djt d¡¡| |dt d¡t d¡ d¡¡| |dt d¡t d¡ d¡¡t ¡dS)Nr&rr])rLrÚdevice_mapsrrzrjrÀ)r!rNrGrZrÁr‚rƒrrLrrrÂrMr‹r'rrúr)rrr2r„)rHrÄrIr-r*r*r+Útest_device_maps_in_optionsÔs0üû ý&&z6TensorPipeAgentCudaRpcTest.test_device_maps_in_optionscCs^|j}| |ddi¡| |ddi¡| |ddi¡| |ddi¡tjt|jƒ|j|j|j|dtj|t j t d¡t d¡fd}tt|ƒƒD]$}| ||jt d|d¡¡q”| |dt d¡t d¡ d¡¡| |dt d¡t d¡ d¡¡| |dt d¡t d¡ d¡¡| |dt d¡t d¡ d¡¡t ¡dS)Nrr&rjr›rzrÀrœ)rZrHrÁr‚r!rNrƒrGrÂrMrr'rr)rPr#rrr2rúr„)rHrÄrIr-ržr*r*r+Ú_test_device_maps_return_to_gpuñs0ûý"&&&&z:TensorPipeAgentCudaRpcTest._test_device_maps_return_to_gpurœcCs"t|jd|jƒ}| |¡dSr%)r!rNrGr“râr*r*r+Útest_device_maps_return_to_gpusz9TensorPipeAgentCudaRpcTest.test_device_maps_return_to_gpucCst|jƒ}| |¡dSrE)r!rNr“râr*r*r+Ú#test_device_maps_return_to_gpu_selfs z>TensorPipeAgentCudaRpcTest.test_device_maps_return_to_gpu_selfcCs|| d¡SrB)rúr¥r*r*r+Ú_add_to_gpusz&TensorPipeAgentCudaRpcTest._add_to_gpuc CsÔt|jd|jƒ}d}| t|¡t|tjkrTtj|t j t d¡ d¡dfdn@|tj kr†tj|t j t d¡ d¡dfd ¡ntd|›ƒ‚W5QRXtj|t j t d¡dfd}| |t d¡d¡dS)Nr&znTensorPipe RPC backend only supports CPU tensors by default.*`set_device_map` on `TensorPipeRpcBackendOptions`rjrrÀúunexpected mode )r!rNrGrÓr rrlrÁrÂr'r(rrúrnrÏr—r½r)rr©rHrprÄrÂr"r*r*r+Ú _test_device_maps_missing_configsÿ $ (z;TensorPipeAgentCudaRpcTest._test_device_maps_missing_configc CsÈt|jd|jƒ}d}| t|¡h|tjkrNtj|t j t d¡dfdn:|tj krztj|t j t d¡dfd ¡ntd|›ƒ‚W5QRXtj|tjt d¡dfd}| |t d¡d¡dS)Nr&z(Response device mapping is not availablerjrÀr—)r!rNrGrÓr rrlrÁrÂrMr–r'rrnrÏr—r½r(r)rrr˜r*r*r+Ú)_test_device_maps_missing_config_response/s$ ý ýzDTensorPipeAgentCudaRpcTest._test_device_maps_missing_config_responsecCs| tj¡dSrE)r™rrlrQr*r*r+Útest_device_maps_missing_configGsz:TensorPipeAgentCudaRpcTest.test_device_maps_missing_configcCs~t|jd|jƒ}|j}tjt|jƒ|j|j|j|jdt ¡}t ¡}| t j¡t ¡t ¡}| |||k¡dS©Nr&rz)r!rNrGrZrÁr‚rƒrtrwr™rrlr„rŒ)rHrÄrIr'r r¡r*r*r+Ú+test_device_maps_missing_config_not_timeoutLsûzFTensorPipeAgentCudaRpcTest.test_device_maps_missing_config_not_timeoutcCs&t|jjdƒD]}| tj¡qdSrŠ)rPrZrr™rrl©rHrr*r*r+Ú$test_device_maps_missing_config_loopbsz?TensorPipeAgentCudaRpcTest.test_device_maps_missing_config_loopcCs| tj¡dSrE)ršrrlrQr*r*r+Ú(test_device_maps_missing_config_responsehszCTensorPipeAgentCudaRpcTest.test_device_maps_missing_config_responsecCs&t|jjdƒD]}| tj¡qdSrŠ)rPrZrršrrlržr*r*r+Ú-test_device_maps_missing_config_response_loopmszHTensorPipeAgentCudaRpcTest.test_device_maps_missing_config_response_loopcCs| tj¡dSrE)r™rrnrQr*r*r+Ú&test_device_maps_missing_config_remotesszATensorPipeAgentCudaRpcTest.test_device_maps_missing_config_remotecCs| tj¡dSrE)ršrrnrQr*r*r+Ú/test_device_maps_missing_config_remote_responsexszJTensorPipeAgentCudaRpcTest.test_device_maps_missing_config_remote_responsecCs¢|j}t|jd|jƒ}| |ddi¡tjt|jƒ|j|j|j|dtj|t j t d¡dfd}| | ¡jjd¡| | ¡t d¡ d¡¡t ¡dS)Nr&rrzrjrÀ)rZr!rNrGrHrÁr‚rƒrÏrMr–r'rrrr—r2rÁr)rúr„)rHrIrÄr‘r*r*r+Útest_device_maps_remote}s$ûýz2TensorPipeAgentCudaRpcTest.test_device_maps_remotec Cs„tj |j¡}tjj|jd}| |¡| |¡| |¡tj |¡tj dt ¡||}W5QRX| |¡| |¡|S)NrÅr„) r'r1r.r2r/r0Z record_streamr1r2ri)rûrüÚs0Ús1rõr*r*r+Ú_slow_add_on_user_stream–s z3TensorPipeAgentCudaRpcTest._slow_add_on_user_streamcCsZ|j}t|jd|jƒ}| ||¡tjt|jƒ|j|j|j|d||ƒt ¡dSrœ) rZr!rNrGrHrÁr‚rƒr„)rHrordrIrÄr*r*r+Ú_test_custom_stream¤sûz.TensorPipeAgentCudaRpcTest._test_custom_streamcCs<t dd¡ d¡}tj|tj||fd}| |d|¡dS)NrjrrÀ)r'r)rúrÁrÂrMr§rr)rHrÄrûr"r*r*r+Ú_test_stream_syncµsýz,TensorPipeAgentCudaRpcTest._test_stream_synccCs| |jddi¡dS©NrÃrG)r¨r©rQr*r*r+Útest_custom_stream¾sz-TensorPipeAgentCudaRpcTest.test_custom_streamc Cs€g}tdƒD]6}t dd¡ d¡|}| tj|tj||fd¡qtdƒD].}| || ¡dt dd¡ d¡|¡qLdS)Nr†rjrrÀ)rPr'r)rúr)rÁrÉrMr§rrrð)rHrÄr%ržrûr*r*r+Ú_test_stream_multi_asyncÂsýÿz3TensorPipeAgentCudaRpcTest._test_stream_multi_asynccCs| |jddi¡dSrª)r¨r¬rQr*r*r+Útest_custom_stream_multiÑsþz3TensorPipeAgentCudaRpcTest.test_custom_stream_multicCs"tj|tj||fd}t ||¡SrÆ)rÁrÂrMr§)rÄrûrürõr"r*r*r+Ú_nested_slow_add_on_user_streamØsýz:TensorPipeAgentCudaRpcTest._nested_slow_add_on_user_streamcCs€t dd¡ d¡}t dd¡ d¡d}t dd¡ d¡d}t|jd|jƒ}tj|tj ||||fd}| |d|¡dS)Nrjrr›rÀr;)r'r)rúr!rNrGrÁrÂrMr®rr)rHrÄrûrürõÚ nested_dstr"r*r*r+Ú_test_stream_nested_syncâs ýz3TensorPipeAgentCudaRpcTest._test_stream_nested_synccCs| |jdddœ¡dS©NrGrÃ)rÃrG)r¨r°rQr*r*r+Útest_custom_stream_nestedîsþz4TensorPipeAgentCudaRpcTest.test_custom_stream_nestedc Cs|jdkrg}d}ggg}}}t|ƒD] }t dd¡ d¡|d}t dd¡ d¡|} t dd¡ d¡|d} | |¡| | ¡| | ¡t|jd|jƒ}| tj |t j||| | fd¡q,t|ƒD],}| || ¡||||||¡qÖdS)Nrr‹rjr&rÀ)rNrPr'r)rúr)r!rGrÁrÉrMr®rrrð)rHrÄr%rˆÚxsZysZzsržrûrürõr¯r*r*r+Ú_test_stream_nested_multi_asyncõs* ýÿz:TensorPipeAgentCudaRpcTest._test_stream_nested_multi_asynccCs| |jdddœ¡dSr±)r¨r´rQr*r*r+Útest_custom_stream_nested_multi sþz:TensorPipeAgentCudaRpcTest.test_custom_stream_nested_multicCs(|jr|jr| ¡| ¡Stdƒ‚dS)NrY)r[rr1r½r¥r*r*r+Ú_gpu_add_wrong_gpussz.TensorPipeAgentCudaRpcTest._gpu_add_wrong_gpusc Cs¤t|jd|jƒ}|j}| |ddi¡tjt|jƒ|j|j|j|dt d¡ d¡}t d¡ d¡}| t d¡tj|tj||fd}W5QRXt ¡dS)Nr&rrzrjzMExpected all tensors to be on the same device, but found at least two devicesrÀ)r!rNrGrZrHrÁr‚rƒr'rrúr)rÓr rÂrMr¶r„rŒr*r*r+Útest_device_mismatchs,ûþýz/TensorPipeAgentCudaRpcTest.test_device_mismatchc CsÂt|jd|jƒ}|j}| |||i¡tjt|jƒ|j|j|j|d|jdkr¶tj|t |fd}t dƒD]F}t dddd¡ |¡}| ¡ |¡ ¡}| ¡ |¡} | || ¡qnt ¡dS)Nr&rzrÀr„rxé)r!rNrGrZrHrÁr‚rƒrÏrcrPr'ÚrandnrúrVr—rÂrrr„) rHÚlocal_deviceÚ remote_devicerÄrIr‘rrûÚactualrœr*r*r+Ú_test_rref_synchronization8s$û z5TensorPipeAgentCudaRpcTest._test_rref_synchronizationcCs| dd¡dS©NrÃ©r½rQr*r*r+Ú"test_rref_to_here_synchronization1Tsz=TensorPipeAgentCudaRpcTest.test_rref_to_here_synchronization1cCs| dd¡dS©NrGrÃr¿rQr*r*r+Ú"test_rref_to_here_synchronization2Xsz=TensorPipeAgentCudaRpcTest.test_rref_to_here_synchronization2cCs| dd¡dS©NrGr¿rQr*r*r+Ú"test_rref_to_here_synchronization3\sz=TensorPipeAgentCudaRpcTest.test_rref_to_here_synchronization3cCs| dd¡dSrªr¿rQr*r*r+Ú"test_rref_to_here_synchronization4`sz=TensorPipeAgentCudaRpcTest.test_rref_to_here_synchronization4cCst|jd|jƒ}|j}| |||i¡t|jd|j|jƒ}| |||i¡|dk rl| ||j¡tjt|jƒ|j|j|j|d|jdkrtj |t |fd}tdƒD]N}tt dddd¡ |¡ƒ} | ¡ | d¡ ¡} | ¡ | d¡}| | |¡q²t ¡dS)Nr&rzrÀr„rxr¸T)r!rNrGrZrHÚset_devicesrÁr‚rƒrÏrcrPrr'r¹rúrVr—rÂrrr„)rHrºr»ZdevicesOptionsrÄrIÚ input_srcr‘rZrref_xr¼rœr*r*r+Ú!_test_rref_as_arg_synchronizationds,ûz.rœ)rÈrPrQr*r*r+Ú!test_rref_as_arg_synchronization5›s ýz ûý z=TensorPipeAgentCudaRpcTest._test_rref_forward_synchronizationcCs| dd¡dSr¾©rÐrQr*r*r+Ú"test_rref_forward_synchronization1Ûsz=TensorPipeAgentCudaRpcTest.test_rref_forward_synchronization1cCs| dd¡dSrªrÑrQr*r*r+Ú"test_rref_forward_synchronization2ßsz=TensorPipeAgentCudaRpcTest.test_rref_forward_synchronization2cCs| dd¡dSrÁrÑrQr*r*r+Ú"test_rref_forward_synchronization3ãsz=TensorPipeAgentCudaRpcTest.test_rref_forward_synchronization3cCs| dd¡dSrÃrÑrQr*r*r+Ú"test_rref_forward_synchronization4çsz=TensorPipeAgentCudaRpcTest.test_rref_forward_synchronization4c Csº|jdkr¶|j}| d||i¡tjddd|dt dtjjd¡ ¡ |¡}t dƒD]V}t dd¡ |¡}| ¡ |¡}t |¡ ¡ ¡ ¡ ¡}| ¡ ¡} | || ¡qVt ¡dS)NrZw0r&)rNrGrZ)éi NrŠrÖ)rNrZrHrÁr‚rÏr'rQrgrúrPr¸rÂrVrr@r—rÉrrr„) rHrºr»rIrrÚdataÚoutputZv0Zv1r*r*r+Ú(_test_owner_rref_forward_synchronizationës. üÿþzCTensorPipeAgentCudaRpcTest._test_owner_rref_forward_synchronizationcCs| dd¡dSr¾©rÙrQr*r*r+Ú(test_owner_rref_forward_synchronization1szCTensorPipeAgentCudaRpcTest.test_owner_rref_forward_synchronization1cCs| dd¡dSrªrÚrQr*r*r+Ú(test_owner_rref_forward_synchronization2szCTensorPipeAgentCudaRpcTest.test_owner_rref_forward_synchronization2cCs| dd¡dSrÁrÚrQr*r*r+Ú(test_owner_rref_forward_synchronization3szCTensorPipeAgentCudaRpcTest.test_owner_rref_forward_synchronization3cCs| dd¡dSrÃrÚrQr*r*r+Ú(test_owner_rref_forward_synchronization4szCTensorPipeAgentCudaRpcTest.test_owner_rref_forward_synchronization4cCs4t dd¡ d¡|}tj dt¡| d¡dS)Nr+rxrr„rÚ)r'r)r1r2riÚsplit)ržrûr*r*r+Ú_return_tensor_viewsz.TensorPipeAgentCudaRpcTest._return_tensor_viewcCs°t|jd|jƒ}|j}| |ddi¡tjt|jƒ|j|j|j|dg}tdƒD]}| tj |tj|fd¡qVtdƒD]$}| t dd¡||| ¡¡q~t ¡dS)Nr&rrzr‹rÀrÚrx)r!rNrGrZrHrÁr‚rƒrPr)rÉrMràrrr'r)rðr„)rHrÄrIr%ržr*r*r+Ú test_tensor_view_as_return_values(ûý "z;TensorPipeAgentCudaRpcTest.test_tensor_view_as_return_valuec Csz| td¡dt|jd|jƒ}|j}| |ddi¡| dg¡tj t|jƒ|j |j|j|dt ¡W5QRXdS)NzHNode worker0 has unexpected source devices in its device map for worker1r&rrz)rÓr½r!rNrGrZrHrÆrÁr‚rƒr„©rHrÄrIr*r*r+Útest_devices_option_mismatch7s þûz7TensorPipeAgentCudaRpcTest.test_devices_option_mismatchc Cs|| td¡ft|jd|jƒ}tj|jj|jj |ddiidgd}tj t|jƒ|j|j|j|dt ¡W5QRXdS)NzHNode worker0 has unexpected target devices in its device map for worker1r&r)rLrr‘r-rz) rÓr½r!rNrGrÁrrZrLrr‚rƒr„râr*r*r+Ú$test_devices_option_mismatch_reverseLs&þ üûz?TensorPipeAgentCudaRpcTest.test_devices_option_mismatch_reversecCstdgd}dS)Nrr,rr¿r*r*r+Útest_cuda_future_device_as_intesz9TensorPipeAgentCudaRpcTest.test_cuda_future_device_as_intcCstdgd}dS)NrÃr,rr¿r*r*r+Útest_cuda_future_device_as_strisz9TensorPipeAgentCudaRpcTest.test_cuda_future_device_as_strcCstt dd¡gd}dS)Nr1rr,)rr'r2r¿r*r*r+Ú!test_cuda_future_device_as_devicemsz.cSs|SrEr*©rŸr*r*r+rÿ{rF©rËrÌrÍ©rÑrQr*r*r+Ú(test_cuda_future_can_extract_cuda_tensorxs ÿzCTensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_cuda_tensorcCs|jdd„dd„dddS)NcSs|gSrEr*rhr*r*r+rÿrz_TensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_list_with_cuda_tensor..cSs|dSrBr*rér*r*r+rÿrFrêrërQr*r*r+Ú2test_cuda_future_can_extract_list_with_cuda_tensor~s ÿzMTensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_list_with_cuda_tensorcCs|jdd„dd„dddS)NcSst|ƒSrE©r8rhr*r*r+rÿ‡rzgTensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_custom_class_with_cuda_tensor..cSs|jSrEràrér*r*r+rÿ‡rFrêrërQr*r*r+Ú:test_cuda_future_can_extract_custom_class_with_cuda_tensor„s ÿzUTensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_custom_class_with_cuda_tensorc stjddd}tjddd‰tddgd}‡fdd„}| |¡}tj d¡Rtj ¡}tj |¡0tj t dt ƒƒ¡| d ¡| |¡W5QRXW5QRXtj d¡Ftj ¡}tj |¡$| t | ¡d ¡ ¡ ¡¡W5QRXW5QRXdS) NrÄrÃrÅrGr,cs| ¡}ˆj|ddˆS)NT)Znon_blocking)r_Zcopy_)ròÚt0©Ztensor1r*r+r6’szPTensorPipeAgentCudaRpcTest.test_cuda_future_callback_changes_devices..cbr+r&)r'rrrùr1r2r/r1r2r3rr4r]rŒrÆrðrÈrÉ©rHZtensor0Z parent_futurer6Zchild_futurer1rÏr*rñr+Ú)test_cuda_future_callback_changes_devicesŠs zDTensorPipeAgentCudaRpcTest.test_cuda_future_callback_changes_devicesc stjddd}tjddd‰tdgd}‡fdd„}| |¡}tj d¡Rtj ¡}tj |¡0tj t dt ƒƒ¡| d ¡| |¡W5QRXW5QRX| td ¡| ¡W5QRXtj d¡Ftj ¡}tj |¡$| t | ¡d ¡ ¡ ¡¡W5QRXW5QRXdS)NrÄrÃrÅrGr,c sHtj d¡2tj tdtƒƒ¡ˆ d¡ˆW5QR£SQRXdS)NrGr+r&)r'r1r2r2r3rr4rñrñr*r+r6¬s zKTensorPipeAgentCudaRpcTest.test_cuda_future_value_on_bad_device..cbr+r&zoThe result contained tensors residing on device$s$ cuda:0 which are not among the expected device$s$ cuda:1)r'rrrùr1r2r/r1r2r3rr4r]rÓr½rðrŒrÆrÈrÉròr*rñr+Ú$test_cuda_future_value_on_bad_device£s( þ z?TensorPipeAgentCudaRpcTest.test_cuda_future_value_on_bad_devicec Cs¸t|jd|jƒ}|j}| |ddi¡tjt|jƒ|j|j|j|dtj ddd}tj |t|fd}tj d¡}tj |¡$| t | ¡d¡ ¡ ¡¡W5QRXt ¡dS)Nr&rÃrzrÄrÅrÀ)r!rNrGrZrHrÁr‚rƒr'rrÉr5r1r/r1rŒrÆrðrÈrÉr„)rHrÄrIr:ròrÏr*r*r+Ú%test_async_execution_with_cuda_futureÄs û(z@TensorPipeAgentCudaRpcTest.test_async_execution_with_cuda_futurec Csît|jd|jƒ}t|jd|jƒ}|j}| |ddi¡tjt|jƒ|j|j|j|dtj ddd}tj ddd}tj ddd}tj |t||||fd}tj d¡}tj |¡$| t | ¡d¡ ¡ ¡¡W5QRXt ¡dS) Nr&rjrÃrzrÄrÅrÀr›)r!rNrGrZrHrÁr‚rƒr'r)rÉr7r1r/r1rŒrÆrðrÈrÉr„) rHrÄr¯rIrzr}r¢ròrÏr*r*r+Ú,test_async_execution_nested_with_cuda_futureÚs&û(zGTensorPipeAgentCudaRpcTest.test_async_execution_nested_with_cuda_futurecCs:tjddd}tdgd}| |¡| d¡| ¡dS)NrÄrÃrÅr,r&)r'rrr]r4rð)rHr™rÎr*r*r+Ú&test_cuda_future_modify_tensor_inplaceós zATensorPipeAgentCudaRpcTest.test_cuda_future_modify_tensor_inplacecCsDtjdddg}tdgd}| |¡tjddd|d<| ¡dS)NrÄrÃrÅr,r)r'rrr]r)rð)rHržrÎr*r*r+Útest_cuda_future_replace_tensors z:TensorPipeAgentCudaRpcTest.test_cuda_future_replace_tensorcCs˜t|jd|jƒ}|j}| |ddi¡tjt|jƒ|j|j|j|dtj|t t jdddfd}| ¡ d¡| ¡ ¡}| |d¡t ¡dS)Nr&rÃrzé*rÅrÀ)r!rNrGrZrHrÁr‚rƒrÏr8r'rrÂr>r@rrr„)rHrÄrIr‘r"r*r*r+Ú&test_rref_with_unpickleable_attributessûzATensorPipeAgentCudaRpcTest.test_rref_with_unpickleable_attributescCs|jdd„dd„dddS)NcSs|SrEr*rhr*r*r+rÿ*rz\TensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_cuda_sparse_tensor..cSs|SrEr*rér*r*r+rÿ*rTrêrërQr*r*r+Ú/test_cuda_future_can_extract_cuda_sparse_tensor's ÿzJTensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_cuda_sparse_tensorcCs|jdd„dd„dddS)NcSs|gSrEr*rhr*r*r+rÿ0rzfTensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_list_with_cuda_sparse_tensor..cSs|dSrBr*rér*r*r+rÿ0rTrêrërQr*r*r+Ú9test_cuda_future_can_extract_list_with_cuda_sparse_tensor-s ÿzTTensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_list_with_cuda_sparse_tensorcCs|jdd„dd„dddS)NcSst|ƒSrErîrhr*r*r+rÿ6rznTensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor..cSs|jSrEràrér*r*r+rÿ6rTrêrërQr*r*r+ÚAtest_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor3s ÿz\TensorPipeAgentCudaRpcTest.test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor)NN)N)vrSrTrUrNrrPrSrTrUrVrWrŽr\r^rbrgrjrkrlrmrnrqrrrsrtrvrwryrzr{r|r}r~r‚rƒr…r†r‡rˆr‰rŠr‹rrŽrrr’r“r”r•r–r™ršrr›rrŸr r¡r¢r£r¤r§r¨r©r«r¬rr®r°r²r´rµr¶r·r½rÀrÂrÄrÅrÈrÊrËrÌrÍrÎrÏrÐrÒrÓrÔrÕrÙrÛrÜrÝrÞràrárãrärårærçrèrìrírïrórôrõrör÷rørúrûrürýr*r*r*r+rMs® ü ' 4 rM)r-F)F)r&)r¶)r)r)¤Úconcurrent.futuresr÷r/rsrr’r;rwÚcollectionsrÚ functoolsrrrZunittestrr'Ztorch.nnrQZtorch.distributedrür—Ztorch.distributed.rpcrÁrkr3r¨rrr r rUrrr r˜rrrrZ torch.futuresrZ*torch.testing._internal.common_distributedrrrZ$torch.testing._internal.common_utilsrrrrZ"torch.testing._internal.dist_utilsrrrrrr r!Z>torch.testing._internal.distributed.rpc.rpc_agent_test_fixturer"r#Ztorch.autograd.profiler_legacyr$rcr,r;rur‰rør\rcrir@rArCrDrXr[r`rbrdrergruryr’r•r˜ršr r£r©r«rlr¬r®r²rµr¸r¹r»r¾rÅrÇrÊrÐrÑrÓrÔrÕr×rØrÝrÞrWÚscriptrßrárârãrZTensorrårærçrérërìrírîrýrþrrrrGrHrr rrrrrrr r*r5r7r8rBrIrKrJr rMrYrcrjrÒrrrMr*r*r*r+ÚsX$ ù &1)