U Jc @sddlZddlmZmZmZmZddlZddlmZ ddlm Z m Z dddgZ eej je dddZeeej jeeeej jffd d dZeeej jeeeej jffe dd dZdS) N)UnionIterableDictIterator) ProcessGroupgroupaverage_parametersget_params_to_average&average_parameters_or_parameter_groupsparams process_groupcCs|dk r |ntj}t|r dSt|\}}tdd|D}|t|}tj rdtj tj ||dd}|D]2}|||| |||_|| 7}qzdS)z Averages all the given parameters. For allreduce efficiency, all the parameters are flattened into a contiguous buffer. Thus, it requires extra memory of the same size as the given parameters. NcSsg|]}|jdqS))dataZreshape).0prV/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/algorithms/model_averaging/utils.py sz&average_parameters..)rr)rZWORLDdistZ_rank_not_in_group itertoolsteetorchcatZget_world_sizeZcudaZ is_availableZ synchronizeZ all_reduceZnumelZview_asZtype_asr)r r Z group_to_useZ params_it1Z params_it2Z flat_paramsoffsetrrrrrs   ")r cCs|g}|D]n}t|tjjr4|}|jdk rv||qt|trb|dD]}|jdk rF||qFqtdt|dq|S)z Returns a list of parameters that need to average, which filters out the parameters that do not contain any gradients. Args: params: The parameters of a model or parameter groups of an optimizer. Nr zParameter input of type z is not supported) isinstancernn ParameterZgradappenddictNotImplementedErrortype)r Zfiltered_paramsparamZ param_datarrrr ,s     cCsttt||dS)zM Averages parameters of a model or parameter groups of an optimizer. N)riterr r rrrr Cs)rtypingrrrrrZtorch.distributedZ distributedrrr__all__rrrstrr r rrrrs    .