U Vcn@sdZddlZddlZddlmZddlmZddlmZddlmZddlmZddlm Z dd lm Z dd l m Z dd l m Z dd l mZdd l mZddlmZeZejZeddddgZe jZe jZe jZe jZediZediZdZGdddeZddZdddZ ddZ!dS)zATakes a generator of values, and accumulates them for a frontend.N)directory_loader)directory_watcher)event_file_loader) io_wrapper)plugin_asset_util) reservoir) tag_types) config_pb2) event_pb2) graph_pb2)meta_graph_pb2) tb_logging TensorEvent wall_timestep tensor_protoi.c@seZdZdZd0ddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/ZdS)1EventAccumulatoraAn `EventAccumulator` takes an event generator, and accumulates the values. The `EventAccumulator` is intended to provide a convenient Python interface for loading Event data written during a TensorFlow run. TensorFlow writes out `Event` protobuf objects, which have a timestamp and step number, and often contain a `Summary`. Summaries can have different kinds of data stored as arbitrary tensors. The Summaries also have a tag, which we use to organize logically related data. The `EventAccumulator` supports retrieving the `Event` and `Summary` data by its tag. Calling `Tags()` gets a map from `tagType` (i.e., `tensors`) to the associated tags for those data types. Then, the functional endpoint (i.g., `Accumulator.Tensors(tag)`) allows for the retrieval of all data associated with that tag. The `Reload()` method synchronously loads all of the data written so far. Fields: most_recent_step: Step of last Event proto added. This should only be accessed from the thread that calls Reload. This is -1 if nothing has been loaded yet. most_recent_wall_time: Timestamp of last Event proto added. This is a float containing seconds from the UNIX epoch, or -1 if nothing has been loaded yet. This should only be accessed from the thread that calls Reload. path: A file path to a directory containing tf events files, or a single tf events file. The accumulator will load events from this path. tensors_by_tag: A dictionary mapping each tag name to a reservoir.Reservoir of tensor summaries. Each such reservoir will only use a single key, given by `_TENSOR_RESERVOIR_KEY`. @@Tensors NTc Cst|pt}i}tD]&}||kr.||||<qt|||<q||_t|pJi|_d|_d|_d|_d|_i|_i|_ i|_ t |_ tt|_t |_||_t||||_t |_||_d|_d|_d|_d|_dS)aConstruct the `EventAccumulator`. Args: path: A file path to a directory containing tf events files, or a single tf events file. The accumulator will load events from this path. size_guidance: Information on how much data the EventAccumulator should store in memory. The DEFAULT_SIZE_GUIDANCE tries not to store too much so as to avoid OOMing the client. The size_guidance should be a map from a `tagType` string to an integer representing the number of items to keep per tag for items of that `tagType`. If the size is 0, all events are stored. tensor_size_guidance: Like `size_guidance`, but allowing finer granularity for tensor summaries. Should be a map from the `plugin_name` field on the `PluginData` proto to an integer representing the number of items to keep per tag. Plugins for which there is no entry in this map will default to the value of `size_guidance[event_accumulator.TENSORS]`. Defaults to `{}`. purge_orphaned_data: Whether to discard any events that were "orphaned" by a TensorFlow restart. event_file_active_filter: Optional predicate for determining whether an event file latest load timestamp should be considered active. If passed, this will enable multifile directory loading. detect_file_replacement: Optional boolean; if True, event file loading will try to detect when a file has been replaced with a new version that contains additional data, by monitoring the file size. NF)dictDEFAULT_SIZE_GUIDANCE_size_guidance_tensor_size_guidance_first_event_timestamp_graph_graph_from_metagraph _meta_graph_tagged_metadatasummary_metadatatensors_by_tag threadingLock_tensors_by_tag_lock collections defaultdict_plugin_to_tag_to_content_plugin_tag_lockpath_GeneratorFromPath _generator_generator_mutexpurge_orphaned_data_seen_session_startmost_recent_stepmost_recent_wall_time file_version) selfr'Z size_guidanceZtensor_size_guidancer+event_file_active_filterdetect_file_replacementZsizeskeyr4a/tmp/pip-unpacked-wheel-g8kmtpbc/tensorboard/backend/event_processing/plugin_event_accumulator.py__init___s<#     zEventAccumulator.__init__c Cs0|j |jD]}||qW5QRX|S)zLoads all events added since the last call to `Reload`. If `Reload` was never called, loads all events in the file. Returns: The `EventAccumulator`. )r*r)Load _ProcessEventr0eventr4r4r5ReloadszEventAccumulator.ReloadcCst|j|S)aHReturn a list of all plugin assets for the given plugin. Args: plugin_name: The string name of a plugin to retrieve assets for. Returns: A list of string plugin asset names, or empty list if none are available. If the plugin was not registered, an empty list is returned. )rZ ListAssetsr'r0 plugin_namer4r4r5 PluginAssetss zEventAccumulator.PluginAssetscCst|j||S)a4Return the contents of a given plugin asset. Args: plugin_name: The string name of a plugin. asset_name: The string name of an asset. Returns: The string contents of the plugin asset. Raises: KeyError: If the asset is not available. )rZ RetrieveAssetr')r0r=Z asset_namer4r4r5RetrievePluginAssets z$EventAccumulator.RetrievePluginAssetc Csp|jdk r|jS|jPz,t|j}|||jWW5QRStk r`tdYnXW5QRXdS)aReturns the timestamp in seconds of the first event. If the first event has been loaded (either by this method or by `Reload`, this returns immediately. Otherwise, it will load in the first event. Note that this means that calling `Reload` will cause this to block until `Reload` has finished. Returns: The timestamp in seconds of the first event that was loaded. Raises: ValueError: If no events have been loaded and there were no events found on disk. Nz!No event timestamp could be found)rr*nextr)r7r8 StopIteration ValueErrorr9r4r4r5FirstEventTimestamps  z$EventAccumulator.FirstEventTimestampc CsB|j2||jkrtd|t|j|W5QRSQRXdS)aReturns a dict mapping tags to content specific to that plugin. Args: plugin_name: The name of the plugin for which to fetch plugin-specific content. Raises: KeyError: if the plugin name is not found. Returns: A dict mapping tag names to bytestrings of plugin-specific content-- by convention, in the form of binary serialized protos. zPlugin %r could not be found.N)r&r%KeyErrorrr<r4r4r5PluginTagToContents  z#EventAccumulator.PluginTagToContentc Cs(|jt|jW5QRSQRXdS)zReturn a set of plugins with summary data. Returns: The distinct union of `plugin_data.plugin_name` fields from all the `SummaryMetadata` protos stored in this accumulator. N)r& frozensetr%r0r4r4r5 ActivePluginsszEventAccumulator.ActivePluginscCs |j|S)zGiven a summary tag name, return the associated metadata object. Args: tag: The name of a tag, as a string. Raises: KeyError: If the tag is not found. Returns: A `SummaryMetadata` protobuf. )rr0tagr4r4r5SummaryMetadatas z EventAccumulator.SummaryMetadatacCs t|jS)zReturn summary metadata for all tags. Returns: A dict `d` such that `d[tag]` is a `SummaryMetadata` proto for the keyed tag. )rrrGr4r4r5AllSummaryMetadatasz#EventAccumulator.AllSummaryMetadatac Cs|jdkr|j|_|drPt|j}|jrJ|j|krJtd|j|||_|||dr|j dk rxtd|j |_ d|_ n|dr |j dk rtd|j |_ |j dks|j r t}||j |j r |j dk rtd d |_ |j |_ n|d rL|jj}||jkrXz)EventAccumulator._Purge..rN)rPr\rJrZ FilterItemsrmvalues_GetPurgeMessager-r.rrrUrV)r0r:rsZ _NotExpired num_expiredr\Z tag_reservoirZ purge_msgr4r}r5rvAs0     zEventAccumulator._Purge)NNTNN)__name__ __module__ __qualname____doc__r6r;r>r?rCrErHrKrLr8rgrirjrkrlrnrXrprqr_rwrvr4r4r4r5r:s8' O    l rcCsd|||||S)z=Return the string message associated with TensorBoard purges.zDetected out of order event.step likely caused by a TensorFlow restart. Purging {} expired tensor events from Tensorboard display between the previous step: {} (timestamp: {}) and current step: {} (timestamp: {}).)rW)r-r.Z event_stepZevent_wall_timerr4r4r5rrs rcsf|s tdt|r"t|S|rFfdd}tj||tj|dSfdd}t||tjSdS)zECreate an event generator for file or directory at given path string.zpath must be a valid stringcs t|Sry)rZTimestampedEventFileLoaderr'r2r4r5r~sz$_GeneratorFromPath..)Z path_filterZ active_filtercs t|Sry)rEventFileLoaderrrr4r5r~sN) rBrZIsSummaryEventsFilerrrZDirectoryLoaderrZDirectoryWatcher)r'r1r2Zloader_factoryr4rr5r(s&    r(cCs>|d}zt|dWStk r8tdYdSXdS)zConvert the string file_version in event.proto into a float. Args: file_version: String file_version from event.proto Returns: Version number as a float. z brain.Event:rzpInvalid event.proto file_version. Defaulting to use of out-of-order event.step logic for purging expired events.N)splitfloatrBrUrV)r/tokensr4r4r5rTs rT)NN)"rr#r Z$tensorboard.backend.event_processingrrrrrrrZtensorboard.compat.protor r r r Ztensorboard.utilr Z get_loggerrU namedtuplerrardrerfrZSTORE_EVERYTHING_SIZE_GUIDANCErmobjectrrr(rTr4r4r4r5sL            <