U Jc@sddlZddlmZddlmZmZddlmZddlm Z m Z ddl m Z m Z mZmZedd Gd d d ZGd d d ejZGdddejZdS)N) dataclass)ListAny)Future)Metadata MetadataIndex)LoadPlanSavePlan SavePlanner LoadPlannerT)frozenc@s&eZdZUeed<eed<eed<dS) WriteResultindexZ size_in_bytesZ storage_dataN)__name__ __module__ __qualname__r__annotations__intrrrO/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/_shard/checkpoint/storage.pyrs rc@seZdZdZejeddddZejeedddZ eje ee ed d d Z ejee e e ed d dZejee e eddddZdS) StorageWritera Interface used by ``save_state_dict`` to write to storage. One StorageWriter instance acts as both the coordinator and the follower in a distributed checkpoint. As part of initialization, each instance is told its role. A subclass should expect the following sequence of calls. 1) (all ranks) init() 2) (all ranks) prepare_local_plan() 3) (coordinator) prepare_global_plan() 4) (all ranks) write_data() 5) (coordinator) finish() N)is_coordinatorreturncCsdS)z Initialize this instance. Args: is_coordinator (bool): Whether this instance is reponsible for coordinating the checkpoint. Nr)selfrrrrinit+s zStorageWriter.initplanrcCsdS)a Perform storage-specific local planning. While this method can produce a completely different plan, the recomended way is to store storage specific data in SavePlan::storage_data. Args: plan (SavePlan): The local plan from the ``SavePlanner`` in use. Returns: A transformed ``SavePlan`` after storage local planning Nrrrrrrprepare_local_plan6sz StorageWriter.prepare_local_planplansrcCsdS)a Perform centralized planning of storage. This method is only called on the coordinator instance. While this method can produce a completely different plan, the prefered way is to store storage specific data in SavePlan::storage_data. Args: plans: A list of ``SavePlan`` instances, one for each rank. Returns: A list of transformed ``SavePlan`` after storage global planning Nrrr!rrrprepare_global_planFsz!StorageWriter.prepare_global_planrplannerrcCsdS)a Write all items from ``plan`` using ``planner`` to resolve the data. A subclass should call ``SavePlanner::resolve_data`` on each item from the plan to get access to the underlying object to write. Subclasses should lazily call `resolve_data` as it can allocate memory. In case of tensors, make following assuptions: - They might be on any device, including not matching the one on ``WriteItem::tensor_data`` - They might be views or not contiguous. Only the projection needs to be saved. Args: plan (SavePlan): The save plan to execute. planner (SavePlanner): Planner object to be used to resolve items to data. Returns: A future that completes to a list of WriteResult Nrrrr%rrr write_dataXszStorageWriter.write_data)metadataresultsrcCsdS)a Writes the metadata and marks the current checkpoint as sucessful. The actual format/schema used for serializing `metadata` is an implemetation detail. The only requirement is that it's recoverable in to the same object graph. Args: metadata (Metadata): metadata for the new checkpoint results: A list of WriteResults from all ranks. Returns: None Nr)rr(r)rrrfinishsszStorageWriter.finish)rrr__doc__abcabstractmethodboolrr rrr#r rrr'rr*rrrrrs  rc@seZdZdZejedddZejeeddddZ eje e d d d Z eje e e e d d dZ eje eeddddZdS) StorageReadera Interface used by ``load_state_dict`` to read from storage. One StorageReader instance acts as both the coordinator and the follower in a distributed checkpoint. As part of initialization, each instance is told its role. A subclass should expected the following sequence of calls by ``load_state_dict``: 1) (all ranks) read_metadata() 2) (all ranks) init 3) (all ranks) prepare_local_plan 4) (coordinator) prepare_global_plan 5) (all ranks) read_data )rcCsdS)z Reads the checkpoint metadata. Returns: The metatada object associated with the checkpoint being loaded. Nr)rrrr read_metadatas zStorageReader.read_metadataN)r(rrcCsdS)z Initialize this instance. Args: metadata (Metadata): The metadata schema to use. is_coordinator (bool): Whether this instance is reponsible for coordinating the checkpoint. Nr)rr(rrrrrs zStorageReader.initrcCsdS)a Perform storage-specific local planning. While this method can produce a completely different plan, the recomended way is to store storage specific data in LoadPlan::storage_data. Args: plan (LoadPlan): The local plan from the ``LoadPlan`` in use. Returns: A transformed ``LoadPlan`` after storage local planning Nrrrrrrsz StorageReader.prepare_local_planr cCsdS)a Perform centralized planning of storage loading. This method is only called on the coordinator instance. While this method can produce a completely different plan, the prefered way is to store storage specific data in LoadPlan::storage_data. Args: plans: A list of ``LoadPlan`` instances, one for each rank. Returns: A list of transformed ``LoadPlan`` after storage global planning Nrr"rrrr#sz!StorageReader.prepare_global_planr$cCsdS)a Reads all items from ``plan`` using ``planner`` to resolve the data. A subclass should call ``LoadPlanner::load_bytes`` to deserialize a BytesIO object into the right place. A subclass should call ``LoadPlanner::resolve_tensor`` to get access to the tensors that in should load data into. It's the StorageLayer responsibility to properly schedule any cross device copies required. Args: plan (LoadPlan): The local plan to execute on planner (LoadPlanner): The planner object to use to resolve items. Returns: A future that completes once all reads are finished. Nrr&rrr read_dataszStorageReader.read_data)rrrr+r,r-rr0r.rr rrr#r rr1rrrrr/s  r/)r,Z dataclassesrtypingrrZ torch.futuresrr(rrr%r r r r rABCrr/rrrrs  k