U &c;@s^dZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z ddl mZmZernddlmZerddlmZeeZd ad d ZGd d d e ZGddde ZGddde ZGddde ZGddde Ze eZ d!e e e!ee!fe e e!ee!fe!e e e"e dddZ#d"e e e$e edddZ%e"e!ddd Z&dS)#z Utilities for working with the local dataset cache. This file is adapted from the AllenNLP library at https://github.com/allenai/allennlp Copyright by the AllenNLP authors. N) defaultdict)IterableList NamedTupleOptionalUnion)is_tf_availableis_torch_available) empty_cache)contextFcCstSN)_is_memory_tracing_enabledrr@/tmp/pip-unpacked-wheel-ymerj3tt/transformers/benchmark_utils.pyis_memory_tracing_enabledsrc@s:eZdZUdZeed<eed<eed<eed<eed<dS)Framea `Frame` is a NamedTuple used to gather the current frame state. `Frame` has the following fields: - 'filename' (string): Name of the file currently executed - 'module' (string): Name of the module currently executed - 'line_number' (int): Number of the line currently executed - 'event' (string): Event that triggered the tracing (default will be "line") - 'line_text' (string): Text of the line in the python script filenamemodule line_numbereventZ line_textN)__name__ __module__ __qualname____doc__str__annotations__intrrrrr"s  rc@s*eZdZUdZeed<eed<eed<dS)UsedMemoryStatea `UsedMemoryState` are named tuples with the following fields: - 'frame': a `Frame` namedtuple (see below) storing information on the current tracing frame (current file, location in current file) - 'cpu_memory': CPU RSS memory state *before* executing the line - 'gpu_memory': GPU used memory *before* executing the line (sum for all GPUs or for only `gpus_to_trace` if provided) frameZ cpu_memoryZ gpu_memoryN)rrrrrrrrrrrr3s rc@s(eZdZUdZeed<edddZdS)Memoryz `Memory` NamedTuple have a single field `bytes` and you can get a human readable string of the number of bytes by calling `__repr__` - `byte` (integer): number of bytes, bytes)returncCs t|jSr )bytes_to_human_readabler!)selfrrr__repr__GszMemory.__repr__N)rrrrrrrr%rrrrr ?s r c@s2eZdZUdZeed<eed<eed<eed<dS) MemoryStatea `MemoryState` are namedtuples listing frame + CPU/GPU memory with the following fields: - `frame` (`Frame`): the current frame (see above) - `cpu`: CPU memory consumed at during the current frame as a `Memory` named tuple - `gpu`: GPU memory consumed at during the current frame as a `Memory` named tuple - `cpu_gpu`: CPU + GPU memory consumed at during the current frame as a `Memory` named tuple rcpugpucpu_gpuN)rrrrrrr rrrrr&Ks r&c@s2eZdZUdZeeed<eeed<eed<dS) MemorySummaryan `MemorySummary` namedtuple otherwise with the fields: - `sequential`: a list of `MemoryState` namedtuple (see below) computed from the provided `memory_trace` by substracting the memory after executing each line from the memory before executing said line. - `cumulative`: a list of `MemoryState` namedtuple (see below) with cumulative increase in memory for each line obtained by summing repeted memory increase for a line if it's executed several times. The list is sorted from the frame with the largest memory consumption to the frame with the smallest (can be negative if memory is released) - `total`: total memory increase during the full tracing as a `Memory` named tuple (see below). Line with memory release (negative consumption) are ignored if `ignore_released_memory` is `True` (default). sequential cumulativetotalN)rrrrrr&rr rrrrr*Ys   r*line)modules_to_tracemodules_not_to_traceevents_to_trace gpus_to_tracer"c sz ddl}Wn"tk r.tddYnX|tzsz8start_memory_tracing..traceit..c3s|]}|kVqdSr rr4r7rrr9s__file__z.pycz.pyor)r isinstancerlisttuple f_globalsallanyf_linenoendswith linecachegetlinerstriprZ memory_infoZrssr torch_empty_cacher tf_contextr Z _clear_cachesnvmlInitZnvmlDeviceGetHandleByIndexZnvmlDeviceGetMemoryInfoused nvmlShutdownrappend)rrargslinenorr.Z traced_statecpu_memZmemgpu_memihandleZmeminfoZ mem_state Zdevicesr1Zlog_gpu memory_tracer0r/processr3traceitr7rrVsZ  $$        z%start_memory_tracing..traceitT)psutil ImportErrorloggerwarningProcessosgetpidr3rIr=rangeZnvmlDeviceGetCountrKOSErrorZ NVMLErrorr r syssettracer)r/r0r1r2rWrrSrstart_memory_tracingls6'       F rbT)rTignore_released_memoryr"c CsDda|dk r@t|dkr@g}tdd}t|dd|ddD]\\}}}\}}} ||} | |} | | } |t|t| t| t| d||d| 7<||d| 7<||d | 7<qFtt| d dd d } td d| D} |rt dd|D}nt dd|D}t|}t || |dSdS)a Stop memory tracing cleanly and return a summary of the memory trace if a trace is given. Args: - `memory_trace` (optional output of start_memory_tracing, default: None): memory trace to convert in summary - `ignore_released_memory` (boolean, default: None): if True we only sum memory increase to compute total memory Return: - None if `memory_trace` is None - `MemorySummary` namedtuple otherwise with the fields: - `sequential`: a list of `MemoryState` namedtuple (see below) computed from the provided `memory_trace` by substracting the memory after executing each line from the memory before executing said line. - `cumulative`: a list of `MemoryState` namedtuple (see below) with cumulative increase in memory for each line obtained by summing repeted memory increase for a line if it's executed several times. The list is sorted from the frame with the largest memory consumption to the frame with the smallest (can be negative if memory is released) - `total`: total memory increase during the full tracing as a `Memory` named tuple (see below). Line with memory release (negative consumption) are ignored if `ignore_released_memory` is `True` (default). `Memory` named tuple have fields - `byte` (integer): number of bytes, - `string` (string): same as human readable string (ex: "3.5MB") `Frame` are namedtuple used to list the current frame state and have the following fields: - 'filename' (string): Name of the file currently executed - 'module' (string): Name of the module currently executed - 'line_number' (int): Number of the line currently executed - 'event' (string): Event that triggered the tracing (default will be "line") - 'line_text' (string): Text of the line in the python script `MemoryState` are namedtuples listing frame + CPU/GPU memory with the following fields: - `frame` (`Frame`): the current frame (see above) - `cpu`: CPU memory consumed at during the current frame as a `Memory` named tuple - `gpu`: GPU memory consumed at during the current frame as a `Memory` named tuple - `cpu_gpu`: CPU + GPU memory consumed at during the current frame as a `Memory` named tuple FNrcSs dddgS)Nrrrrrr*z%stop_memory_tracing..r;rr'r(r)rcSs |ddS)Nrrgr)xrrrrd;reT)keyreversecss4|],\}\}}}t|t|t|t|dVqdS)rfN)r&r )r5r cpu_mem_inc gpu_mem_inccpu_gpu_mem_incrrrr9=s z&stop_memory_tracing..css|]}td|jjVqdS)rN)maxr)r!r5Z step_tracerrrr9Escss|]}|jjVqdSr )r)r!rorrrr9Gs)r+r,r-) rlenrziprLr&r sortedr=itemssumr*)rTrcZmemory_diff_traceZcumulative_memory_dictrrOrPZ next_frameZ next_cpu_memZ next_gpu_memrkrlrmZcumulative_memoryZ total_memoryrrrstop_memory_tracingsH&   ru) memory_amountr"cCs<dD],}|dkr(|dkr(d||S|d}qd|S)zX Utility to convert a number of bytes (int) in a human readable string (with units) )BZKBZMBZGBgg@z{:.3f}{}z{:.3f}TB)format)rvunitrrrr#Ns  r#)NNr.N)NT)'rrDloggingr\r` collectionsrtypingrrrrrZ file_utilsr r Z torch.cudar rGZtensorflow.python.eagerr rH getLoggerrrYrrrrr r&r*Z MemoryTracerrrbboolrur#rrrrsP         N