U <cm @sJddlZddlZddlmZddlmZddlmZddZd ddZ dS) N_get_device_index)autocast)ExceptionWrappercCs~t|tjr|St|ts$t|trHtt|D]}t|tjr.|Sq.t|trztt|D]}t|tjr`|Sq`dS)N) isinstancetorchZTensorlisttuplemap get_a_vardictitems)objresultrD/tmp/pip-unpacked-wheel-gikjz4vx/torch/nn/parallel/parallel_apply.pyr s      r c szt|t|kst|dk r2t|t|ks@tnift|}|dk r^t|t|ksltndgt|}dd|D}dd|D}tittd fdd t|dkrfddtt|||||D}|D] }| q|D]}| qn&d |d |d |d |d |d g}t t|D],}|} t | t rh| || qH|S) a~Applies each `module` in :attr:`modules` in parallel on arguments contained in :attr:`inputs` (positional) and :attr:`kwargs_tup` (keyword) on each of :attr:`devices`. Args: modules (Module): modules to be parallelized inputs (tensor): inputs to the modules devices (list of int or torch.device): CUDA devices :attr:`modules`, :attr:`inputs`, :attr:`kwargs_tup` (if given), and :attr:`devices` (if given) should all have same length. Moreover, each element of :attr:`inputs` can either be a single object as the only argument to a module, or a collection of positional arguments. NcSsg|]}t|dqS)Tr.0xrrr /sz"parallel_apply..cSsg|]}tj|qSr)rcudacurrent_streamrrrrr0sc st|dkrt|}|dkr2tj|}ztj|Rtj|:td$t |t t fsp|f}|||}W5QRXW5QRXW5QRX||<W5QRXWn:t k rt d||d|<W5QRXYnXdS)N)Zenabledzin replica {} on device {})where)rZset_grad_enabledr Z get_devicerrdevicestreamrrrr Exceptionrformat)imoduleinputkwargsrroutput)autocast_enabled grad_enabledlockresultsrr_worker5s"   (( zparallel_apply.._workerc s4g|],\}\}}}}}tj||||||fdqS))targetargs) threadingThread)rrrrr rr)r&rrrIsr)NN)lenAssertionErrorr*LockrZis_grad_enabledZis_autocast_enabled enumeratezipstartjoinrangerrreraiseappend) modulesinputsZ kwargs_tupZdevicesZstreamsthreadsthreadoutputsrr!r)r&r"r#r$r%rparallel_applys:  & r;)NN) r*rZtorch.cuda._utilsrZtorch.cuda.amprZ torch._utilsrr r;rrrrs