U <cp@sddlZddlZddlmZmZmZmZddlZddlm Z m Z m Z dddddgZ e ee d d dZ e eeee d d d Ze eee dddZe edddZde eee dddZe ee dddZe edddZe eje dddZdS)N)AnyListOptionalSet)DataPipe DataPipeGraph traverse_dpsapply_random_seedapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipes)graphreturncCs t|tSN)_get_all_graph_pipes_helperset)rrC/tmp/pip-unpacked-wheel-gikjz4vx/torch/utils/data/graph_settings.pyr s)rid_cachercCsLg}|D]:\}\}}||kr"q |||||t||q |Sr)itemsaddappendextendr)rrresultsZdp_iddatapipeZ sub_graphrrrrs  r)rnum_of_instances instance_idrcCsft|}t|}d}|D]H}t|dr|rt|dr|dk rPtdd|d|||||}q|S)N is_shardabler zVThis implementation of sharding can be only applied once per instance of DataPipeline.zAlready applied tozwhile trying to apply to)rr hasattrr RuntimeErrorr )rrrr all_pipesZalready_applied_topiperrrr "s"   )rrcCs8t|drt|dsdSt|jr0t|js4dSdS)N set_shuffleset_seedFT)rinspectismethodr#r$rrrr_is_shuffle_datapipe2s r()rshufflercCsb|dkr |St|}t|}dd|D}|sJ|rJtd|}|g}|D]}||qN|S)a. Traverse the graph of ``DataPipes`` to find and set shuffle attribute to each `DataPipe` that has APIs of ``set_shuffle`` and ``set_seed``. Args: datapipe: DataPipe that needs to set shuffle attribute shuffle: Shuffle option (default: ``None`` and no-op to the graph) NcSsg|]}t|r|qSr)r().0r"rrr Hsz*apply_shuffle_settings..z`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.)rr warningswarnr)r#)rr)rr!Z shufflersZshufflerrrrr :s  )rrngrcCstdt||S)Nz`apply_shuffle_seed` is deprecated since 1.12 and will be removed in the future releases. Please use `apply_random_seed` instead.)r,r-r )rr.rrrr WscCst|drt|jrdSdS)Nr$TF)rr%r&r$r'rrr_is_random_datapipe_sr/cCst|}t|}t}g}|D]2}t||kr0qt|r|||t|q|D].}ttj dtj dj |d }| |qV|S)a Traverse the graph of ``DataPipes`` to find random ``DataPipe`` with an API of ``set_seed`` then set the random seed based on the provided RNG. Args: datapipe: DataPipe that needs to set randomness rng: Random number generator to generate random seeds r)Zdtype) generator)rr ridr/rrinttorchemptyZint64Zrandom_itemr$)rr.rr!cacheZrandom_datapipesr"Z random_seedrrrr es     )N)r%r,typingrrrrr3Ztorch.utils.data.graphrrr__all__r r2rr boolr(r r r/ Generatorr rrrrs$