U <ºc,ã@sDddlZddlmZGdd„dejjƒZdd„Zdd„Zd d „ZdS) éN)ÚReplicatedTensorc@s*eZdZdZeddd„ƒZedd„ƒZdS)ÚReplicatedTensorFunctionzv Autograd function to ensure gradients are replicated between the replicated tensor and the original one. NcCs| d¡t||ƒS)NF)Zset_materialize_gradsr)ÚctxÚinpÚ process_group©rúT/tmp/pip-unpacked-wheel-gikjz4vx/torch/nn/parallel/_replicated_tensor_ddp_interop.pyÚforward s z ReplicatedTensorFunction.forwardcCs|dfS©Nr)rZ grad_outputrrrÚbackwardsz!ReplicatedTensorFunction.backward)N)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú staticmethodr r rrrrrs  rcCst ||¡}|j|_|Sr )rÚapplyZgrad)ZtensorrZreplicated_tensorrrrÚ_make_replicated_tensors rc CsŽ| ¡}|j ¡D]0\}}|dk r6t||t||ƒƒqt|||ƒq|j ¡D]\}}t|||ƒqN|j ¡D]\}}t||t||ƒƒqn|Sr )Z_replicate_for_data_parallelÚ _parametersÚitemsÚsetattrrÚ_buffersZ_modulesÚ_replicate_module_recurse) ÚmodulerZreplicaÚ param_nameÚparamZ buffer_nameÚbufferÚ module_nameÚchildrrrrsrcCs&ddlm}||ƒstdƒ‚t||ƒS)Nr)Ú_replicatable_modulezKCannot replicate network where python modules are childrens of ScriptModule)Ztorch.nn.parallel.replicaterÚ RuntimeErrorr)ÚnetworkrrrrrÚ_replicate_module(s r!) ZtorchZ*torch.distributed._shard.replicated_tensorrZautogradZFunctionrrrr!rrrrÚs