U Vºcã@sVdZddlmZddlmZddlmZddlmZe ¡Z e ƒZ Gdd„de ƒZ dS) z1Implementation for a multi-file directory loader.é)Údirectory_watcher)Ú io_wrapper)Útf)Ú tb_loggingc@s>eZdZdZdd„dd„fdd„Zdd„Zd d „Zd d „Zd S)ÚDirectoryLoaderaŒLoader for an entire directory, maintaining multiple active file loaders. This class takes a directory, a factory for loaders, and optionally a path filter and watches all the paths inside that directory for new data. Each file loader created by the factory must read a path and produce an iterator of (timestamp, value) pairs. Unlike DirectoryWatcher, this class does not assume that only one file receives new data at a time; there can be arbitrarily many active files. However, any file whose maximum load timestamp fails an "active" predicate will be marked as inactive and no longer checked for new data. cCsdS©NT©)ÚxrrúY/tmp/pip-unpacked-wheel-g8kmtpbc/tensorboard/backend/event_processing/directory_loader.pyÚ3ózDirectoryLoader.cCsdSrr)Ú timestamprrr r 4r cCsH|dkrtdƒ‚|dkr tdƒ‚||_||_||_||_i|_i|_dS)a²Constructs a new MultiFileDirectoryLoader. Args: directory: The directory to load files from. loader_factory: A factory for creating loaders. The factory should take a path and return an object that has a Load method returning an iterator yielding (unix timestamp as float, value) pairs for any new data path_filter: If specified, only paths matching this filter are loaded. active_filter: If specified, any loader whose maximum load timestamp does not pass this filter will be marked as inactive and no longer read. Raises: ValueError: If directory or loader_factory are None. NzA directory is requiredzA loader factory is required)Ú ValueErrorÚ _directoryÚ_loader_factoryÚ _path_filterÚ_active_filterÚ_loadersÚ_max_timestamps)ÚselfÚ directoryZloader_factoryZ path_filterZ active_filterrrr Ú__init__/szDirectoryLoader.__init__c #s¢zFt ˆj¡}t‡fdd„|Dƒƒ}|D]}ˆ |¡D] }|Vq6q(WnVtjjk rœ}z4tjj   ˆj¡s~t   dˆj¡‚nt  d|¡W5d}~XYnXdS)aLoads new values from all active files. Yields: All values that have not been yielded yet. Raises: DirectoryDeletedError: If the directory has been permanently deleted (as opposed to being temporarily unavailable). c3s|]}ˆ |¡r|VqdS)N)r)Ú.0Úp©rrr Ú [s z'DirectoryLoader.Load..z)Directory %s has been permanently deletedz&Ignoring error during file loading: %sN)rZListDirectoryAbsoluterÚsortedÚ _LoadPathrÚerrorsZOpErrorÚioZgfileÚexistsrZDirectoryDeletedErrorÚloggerÚinfo)rZ all_pathsÚpathsÚpathÚvalueÚerrr ÚLoadOs  ÿÿzDirectoryLoader.LoadccsÚ|j |d¡}|tks"| ||¡r2t d|¡dS|j |d¡}|dkrˆz| |¡}Wn&tj j k r|t d|¡YdSX||j|<t  d|¡|  ¡D]"\}}|dks´||kr¸|}|Vqœ| ||¡sÖ||j|<dS)zÍGenerator for values from a single path's loader. Args: path: the path to load from Yields: All values from this path's loader that have not been yielded yet. NzSkipping inactive path %szSkipping nonexistent path %szLoading data from path %s) rÚgetÚ _INACTIVEÚ_MarkIfInactiver!ÚdebugrrrrZ NotFoundErrorr"r')rr$Ú max_timestampÚloaderr r%rrr rhs,  ÿ     zDirectoryLoader._LoadPathcCs:t d||¡|dk r6| |¡s6t|j|<|j|=dSdS)zNIf max_timestamp is inactive, returns True and marks the path as such.z"Checking active status of %s at %sNTF)r!r+rr)rr)rr$r,rrr r*ˆs  zDirectoryLoader._MarkIfInactiveN)Ú__name__Ú __module__Ú __qualname__Ú__doc__rr'rr*rrrr r sû  rN) r1Z$tensorboard.backend.event_processingrrZtensorboard.compatrZtensorboard.utilrZ get_loggerr!Úobjectr)rrrrr Ús