U Kc]@sddlZddlZddlZddlmZmZmZmZddlZddl m Z m Z m Z ddl mZmZddlmZddlmZdd d d d d ddddddddddddddgZddZejddZdFeeefd d!dZd"d ZdGeeefdd#d$d Zdd%d&d ZdHeeefeeefd#d'd ZdIeeefeeefd#d(d Z dJeeefdd#d)dZ!dKeeefdd#d*dZ"dLeeefdd#d+dZ#dMeeefdd#d,dZ$dNeeefed#d-dZ%dOeeefed#d.dZ&dPeeefed#d/dZ'dQeeefed#d0dZ(dReeefed#d1dZ)dSeeefed#d2dZ*d3dZ+dTeeefe,ed5d6dZ-dUeeefed#d7dZ.dVeeefeeefd#d8dZ/dWe,eeefd9d:d;Z0dXeeefd d= 1MB allocations). - ``small_pool``: statistics for the small allocation pool (as of October 2019, for size < 1MB allocations). Metric type: - ``current``: current value of this metric. - ``peak``: maximum value of this metric. - ``allocated``: historical total increase in this metric. - ``freed``: historical total decrease in this metric. In addition to the core statistics, we also provide some simple event counters: - ``"num_alloc_retries"``: number of failed ``cudaMalloc`` calls that result in a cache flush and retry. - ``"num_ooms"``: number of out-of-memory errors thrown. The caching allocator can be configured via ENV to not split blocks larger than a defined size (see Memory Management section of the Cuda Semantics documentation). This helps avoid memory framentation but may have a performance penalty. Additional outputs to assist with tuning and evaluating impact: - ``"max_split_size"``: blocks above this size will not be split. - ``"oversize_allocations.{current,peak,allocated,freed}"``: number of over-size allocation requests received by the memory allocator. - ``"oversize_segments.{current,peak,allocated,freed}"``: number of over-size reserved segments from ``cudaMalloc()``. Args: device (torch.device or int, optional): selected device. Returns statistics for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. csRt|tr@t|dkr|d7}|D]\}}|||q&n||fdS)Nr.)r,dictlenitemsappend)prefixobjkv_recurse_add_to_resultresultr$r%r@s   z,memory_stats.._recurse_add_to_resultr()rsort collections OrderedDict)r)statsr$r?r%rs F  cCs"ts iSt|dd}tj|S)zNReturns the result of :func:`~torch.cuda.memory_stats` as a nested dictionary.Toptional)rrr#r Z_cuda_memoryStatsr(r$r$r%rs cCst|dd}tj|S)a~Resets the "accumulated" (historical) stats tracked by the CUDA memory allocator. See :func:`~torch.cuda.memory_stats` for details. Accumulated stats correspond to the `"allocated"` and `"freed"` keys in each individual stat dict, as well as `"num_alloc_retries"` and `"num_ooms"`. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. TrG)rr#r Z!_cuda_resetAccumulatedMemoryStatsr(r$r$r%rs cCst|dd}tj|S)aResets the "peak" stats tracked by the CUDA memory allocator. See :func:`~torch.cuda.memory_stats` for details. Peak stats correspond to the `"peak"` key in each individual stat dict. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. TrG)rr#r Z_cuda_resetPeakMemoryStatsr(r$r$r%rs cCstdtt|dS)aResets the starting point in tracking maximum GPU memory occupied by tensors for a given device. See :func:`~torch.cuda.max_memory_allocated` for details. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. warning:: This function now calls :func:`~torch.cuda.reset_peak_memory_stats`, which resets /all/ peak memory stats. .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. zytorch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.r(warningswarn FutureWarningrr(r$r$r%rs cCstdtt|dS)aResets the starting point in tracking maximum GPU memory managed by the caching allocator for a given device. See :func:`~torch.cuda.max_memory_cached` for details. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. warning:: This function now calls :func:`~torch.cuda.reset_peak_memory_stats`, which resets /all/ peak memory stats. .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. zvtorch.cuda.reset_max_memory_cached now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.r(rIr(r$r$r%r!s cCst|dddS)a`Returns the current GPU memory occupied by tensors in bytes for a given device. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: This is likely less than the amount shown in `nvidia-smi` since some unused memory can be held by the caching allocator and some context needs to be created on GPU. See :ref:`cuda-memory-management` for more details about GPU memory management. r(zallocated_bytes.all.currentrrgetr(r$r$r%r;scCst|dddS)aReturns the maximum GPU memory occupied by tensors in bytes for a given device. By default, this returns the peak allocated memory since the beginning of this program. :func:`~torch.cuda.reset_peak_memory_stats` can be used to reset the starting point in tracking this metric. For example, these two functions can measure the peak allocated memory usage of each iteration in a training loop. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. r(zallocated_bytes.all.peakrrMr(r$r$r%rMscCst|dddS)aReturns the current GPU memory managed by the caching allocator in bytes for a given device. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. r(zreserved_bytes.all.currentrrMr(r$r$r%rcs cCst|dddS)aReturns the maximum GPU memory managed by the caching allocator in bytes for a given device. By default, this returns the peak cached memory since the beginning of this program. :func:`~torch.cuda.reset_peak_memory_stats` can be used to reset the starting point in tracking this metric. For example, these two functions can measure the peak cached memory amount of each iteration in a training loop. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. r(zreserved_bytes.all.peakrrMr(r$r$r%rsscCstdtt|dS)z4Deprecated; see :func:`~torch.cuda.memory_reserved`.zGtorch.cuda.memory_cached has been renamed to torch.cuda.memory_reservedr()rJrKrLrr(r$r$r%rs cCstdtt|dS)z8Deprecated; see :func:`~torch.cuda.max_memory_reserved`.zOtorch.cuda.max_memory_cached has been renamed to torch.cuda.max_memory_reservedr()rJrKrLrr(r$r$r%rs cCs tjS)a)Returns a snapshot of the CUDA memory allocator state across all devices. Interpreting the output of this function requires familiarity with the memory allocator internals. .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. )r#r _cuda_memorySnapshotr$r$r$r%rs F)r) abbreviatedr1c Cstt|dd}t|d}dd}dd}dd |fd d |fd d |fdd|fdd|fdd|fdd|fdd|fg}g}|d|d|d|d|d|d|D]\}}} |dd|fg} |s| d| dd \} } } }| D]\}}|d!|d!}||d"}||d#}||d$}||d%}| d&krV|} |} |} |}|d'|| || | || | || | ||qqd(d)|fd*d+|fg}|D]~\}}} |d|d!}||d"}||d#}||d$}||d%}|d'|| ||| ||| ||| ||q|dd,|d-}|D]\}}|||d!d.<q>d/d0|jf|d1S)2aReturns a human-readable printout of the current memory allocator statistics for a given device. This can be useful to display periodically during training, or when handling out-of-memory exceptions. Args: device (torch.device or int, optional): selected device. Returns printout for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). abbreviated (bool, optional): whether to return an abbreviated summary (default: False). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. TrGr(cSsVddddddg}|d}|ddD]$}|d kr4qJ|}|d }|d }q$d ||S) NzB ZKBZMBZGBZTBZPBrri iz{:7d} {}r4)szZpref_szprefixesr; new_prefixr$r$r% _format_sizes z$memory_summary.._format_sizecSsPdddg}|d}|ddD]$}|dkr.qD|}|d}|d}qd||S) N KMrriq iz {:7d} {} rQ)ZcntZpref_cntrSr;rTr$r$r% _format_counts  z%memory_summary.._format_countZallocated_byteszAllocated memoryZ active_bytesz Active memoryZreserved_byteszGPU reserved memoryZinactive_split_byteszNon-releasable memoryZ allocationZ Allocationsactivez Active allocssegmentzGPU reserved segmentsZinactive_splitzNon-releasable allocszK===========================================================================z= {_:16} PyTorch CUDA memory summary, device ID {device:<17d} zK---------------------------------------------------------------------------zX {_:9} CUDA OOMs: {num_ooms:<12d} | {_:6} cudaMalloc retries: {num_alloc_retries:<8d} zK Metric | Cur Usage | Peak Usage | Tot Alloc | Tot Freed all)Z large_poolz from large pool)Z small_poolz from small pool)NNNNr6currentpeak allocatedfreedNz {:<21} | {} | {} | {} | {} Zoversize_allocationszOversize allocationsZoversize_segmentszOversize GPU segmentsrB)_r)-|z| |z| )rrr:r4r9replacejoin)r)rPrFrUrYZmetrics_to_displaylinesZ metric_keyZ metric_name formatterZ submetricsZcurrent_prefvalZ peak_prefvalZallocated_prefvalZ freed_prefvalZ submetric_keyZsubmetric_namer;r]r^r_r`Zfmt_dictr=r>r$r$r%rs                               cCsz ddl}Wntk r"YdSXddlm}z |Wn|k rRYdSXt|dd}||}||}g}|d|t|dkr|d |D],}|j d }|d |j d d |ddqd |S)aReturns a human-readable printout of the running processes and their GPU memory use for a given device. This can be useful to display periodically during training, or when handling out-of-memory exceptions. Args: device (torch.device or int, optional): selected device. Returns printout for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). rNz.pynvml module not found, please install pynvml)NVMLError_DriverNotLoadedz-cuda driver can't be loaded, is cuda enabled?TrGzGPU:zno processes are runningizprocess z>10dz uses z>12.3fz MB GPU memory ) pynvmlModuleNotFoundErrorrhZnvmlInitrZnvmlDeviceGetHandleByIndexZ$nvmlDeviceGetComputeRunningProcessesr:r8Z usedGpuMemorypidre)r)rjrhhandleZprocsrfpZmemr$r$r%r "s(           cCs*|dkrtj}t|}tj|S)aReturns the global free and total GPU memory occupied for a given device using cudaMemGetInfo. Args: device (torch.device or int, optional): selected device. Returns statistic for the current device, given by :func:`~torch.cuda.current_device`, if :attr:`device` is ``None`` (default). .. note:: See :ref:`cuda-memory-management` for more details about GPU memory management. N)r#r*r+rZcudartZcudaMemGetInfor(r$r$r%r!Ds  Zenabledr)c Cs&tj|t|W5QRXdSr")r#r*r)r Z_cuda_recordMemoryHistoryror$r$r%_record_memory_historyVsrpc Cs,tj|tW5QRSQRXdSr")r#r*r)r rOr(r$r$r% _snapshotZsrq output.svgc Cs6|dkrt}t|d}|t|W5QRXdSNw)ropenwrite _segmentsfilenameZsnapshotfr$r$r%_save_segment_usage^s r{c Cs6|dkrt}t|d}|t|W5QRXdSrs)rrurv_memoryrxr$r$r%_save_memory_usageds r}envcCs tj|Sr")r#r Z1_cuda_cudaCachingAllocator_set_allocator_settingsr~r$r$r%_set_allocator_settingsjsr)NN)N)N)N)N)N)N)N)N)N)N)N)N)N)NF)N)N)N)N)rrN)rrN)5rD contextlibrJtypingrrrrr#rBrrr Z _memory_vizr rwr r|Z torch.typesr r __all__r&contextmanagerr'r-rrrrstrrrrrrrrrrrrrrboolrr r!rprqr{r}rr$r$r$r%sn   ""X" |""