U Jc @sUdZddlZddlZddlmZmZmZmZmZddl Z ddl m Z ddl m Z ddl mZgZeeed<ee jeefZee Zee efZe je jee jddfdd d Zedd d d Ze jeeee fee jeedddZe jeeee feee jeedddZdS)zPer-layer profilers.N)Any GeneratorListUnionSequence)Tensor)Batch__all__)moduledevicereturnccs0|D]&}t|}||||VqdS)zOCopies layers for ease to profile. It doesn't modify the given module. N)copydeepcopytoZtrain)r r layerZ layer_copyrT/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/pipeline/sync/_balance/profile.pylayerwise_sandboxs   r)batchr cCs*t|D]\}}||j||<qdS)zDetaches from autograd graph.N) enumeratedetachrequires_grad_ requires_grad)rixrrrr&sr)r sampletimeoutr r cs2tdd|Drtdt|}t|D]"\}}|||j||<q.dd|D}t }t ||kr|} tt ||D]\}} t| |j dkrt j |t } | | } tdd| D} | rt j| | |j dkrt j |t } ||| | qqhdfd d|DS) z!Profiles elapsed times per layer.css|]}|jdk VqdSN)Zgrad.0prrr .sz profile_times..z#some parameter already has gradientcSsg|]}gqSrr)r _rrr 5sz!profile_times..cudacss|]}|jr|VqdSr)r)r yrrrr"Fsi@Bcs"g|]}tfdd|DqS)c3s|]}t|VqdSr)int)r tusrrr"Qsz+profile_times...)sum)r bufr)rrr$Qs)any parameters ValueErrorr rrrrrtimertypetorchr%Z synchronizecalltupleZautogradZbackwardappend)r rrr Z_batchrrZ time_bufsZbegun_atrrZtickZbackward_tensorsZtockrr)r profile_times,s0     r6)r inputchunks param_scaler r cCs|jdkrtdt|}g}|dd|}t|D]*\}} | dd|| j||<q8t ||D]j} t|t j |} | | }t j |} | | } tdd| D}| |||}|t|qn|S)z%Profiles CUDA memory usage per layer.r%z'size profiler supports only CUDA devicerNcss|]}|VqdSr)Zstoragenbytesrrrrr"lsz profile_sizes..)r1r/r sizerrrrrrr2r%Zmemory_allocatedr3r+r.r5r')r r7r8r9r rZsizesZ latent_scalerrrZ memory_beforeZ memory_afterZ latent_sizeZ param_sizer<rrr profile_sizesTs" $   r=)__doc__rr0typingrrrrrr2rZtorch.nnnnZ microbatchr r str__annotations__r r'ZDeviceZTensorsZTensorOrTensorsZ SequentialModulerrfloatr6r=rrrrs*    " *)