U Jc@sTddlZddlZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z ddlmZmZmZmZmZmZmZddlmZmZmZmZmZmZddlmZmZm Z dd l!m"Z"Gd d d eZ#Gd d d eZ$ee%efeedddZ&eeeedddZ'ee%efe(edddZ)eeeeeefdddZ*eedddZ+dS)N)ListTupleDictAnyUnioncast)narrow_tensor_by_index) ShardedTensor) SavePlanner LoadPlannerSavePlanLoadPlanReadItem WriteItem WriteItemType)BytesStorageMetadataTensorStorageMetadata MetadataIndexMetadataSTATE_DICT_TYPE STORAGE_TYPES)_create_read_items_create_write_items"_create_default_metadata_only_plan)find_state_dict_objectc@seZdZeeefeddddZedddZ e ee e ee fdd d Z eed d d ZeeejejfdddZeedddZeedddZdS)DefaultSavePlannerN state_dictis_coordinatorreturncCs||_||_dSN)rr)selfrrr#W/tmp/pip-unpacked-wheel-gikjz4vx/torch/distributed/_shard/checkpoint/default_planner.pyinit)szDefaultSavePlanner.initr cCst|j|j|_|jSr!)create_default_local_save_planrrplanr"r#r#r$create_local_plan-sz$DefaultSavePlanner.create_local_plan all_plansr cCst|\|_|_|j|jfSr!)create_default_global_save_plan global_planmetadata)r"r,r#r#r$create_global_plan1sz%DefaultSavePlanner.create_global_plannew_planr cCs ||_|Sr!)r(r"r2r#r#r$ finish_plan5szDefaultSavePlanner.finish_plan) write_itemr cCs||j}|||Sr!) lookup_objectindextransform_object)r"r5objectr#r#r$ resolve_data9s zDefaultSavePlanner.resolve_datar7r cCs t|j|Szo This is an extension from the planner interface to make it easy to extend the default planner rrr"r7r#r#r$r6=sz DefaultSavePlanner.lookup_object)r5r9cCs(|jtjkr$t}t|||}|Sr<)typerBYTE_IOioBytesIOtorchsave)r"r5r9bytesr#r#r$r8Cs   z#DefaultSavePlanner.transform_object)__name__ __module__ __qualname__rstrrboolr%r r*rrrr0r4rrrCTensorrArBr:rr6r8r#r#r#r$r(s rc@seZdZeeeddddZedddZe ee edd d Z eed d d Z e e jddddZe dddZe ejddddZeejdddZe ejdddZdS)DefaultLoadPlannerN)rr/rr cCs||_||_||_dSr!)rr/r)r"rr/rr#r#r$r%OszDefaultLoadPlanner.initr&cCst|j|jSr!)create_default_local_load_planrr/r)r#r#r$r*Tsz$DefaultLoadPlanner.create_local_plan)r.r cCst|Sr!)create_default_global_load_plan)r"r.r#r#r$r0Wsz%DefaultLoadPlanner.create_global_planr1cCs|Sr!r#r3r#r#r$r4ZszDefaultLoadPlanner.finish_plan) read_itemvaluer cCst||j|jj<dSr!)rCloadr dest_indexfqn)r"rOrPr#r#r$ load_bytes]szDefaultLoadPlanner.load_bytes)rOcCs||j}|||Sr!) lookup_tensorrRtransform_tensorr"rOtensorr#r#r$resolve_tensor`s z!DefaultLoadPlanner.resolve_tensor)rOrXr cCsdSr!r#rWr#r#r$ commit_tensordsz DefaultLoadPlanner.commit_tensorr;cCs t|j|Sr<r=r>r#r#r$rUgsz DefaultLoadPlanner.lookup_tensor)rOrXcCst||j|jSr<)rZ dest_offsetslengthsrWr#r#r$rVmsz#DefaultLoadPlanner.transform_tensor)rFrGrHrrrJr%rr*rr0r4rrArBrTrYrCrKrZrrUrVr#r#r#r$rLNsrL)rr/r cCs8g}|D]"\}}|j|}|t|||7}q t|Sr!)itemsZstate_dict_metadatarr)rr/requestsrSobjmdr#r#r$rMts   rMr+cCs|S)z Create global load plan used by DefaultLoadPlanner. The default load behavior involved no global coordination and this function currently doesn't change the local plans. r#)r,r#r#r$rNsrNrcCs:g}|D]$\}}t|ts"|r |t||7}q t|S)a Create the ``SavePlan`` used by DefaultSavePlanner. On non-coordinator ranks, this function ignores tensors and non-tensor objects, only producing writes for ShardedTensor objects. On the coordinator rank, produce writes for all values. )r\ isinstancer rr )rrr]rSr^r#r#r$r's r'c Csi}g}|D]}g}|jD]}|jtjks:|jj|ks:t|jtjkr`t||jj<| |q|j dk sntt t | |jjt |j j|j jgd}tj|jt|jd}tj||d}| ||j jdk std|jj|j |j jq| tj||dq |t|fS)a Create the global plan and metadata used by DefaultSavePlanner. Metadata is produced by concatenating the metadata of all ``WriteItem`` from the supplied plans. The only global planning change is to update index hints in all ``MetadataIndex`` objects. N) propertiessizechunks)r7z1Cannot create MD for tensor without bounds. FQN: )r\)r\r?rZSHARDr7rSAssertionErrorr@rappendZ tensor_datarr setdefaultrarb dataclassesreplacelenrcchunkr) r,r_Z new_plansr(Z new_itemsitemZ tensor_mdZ new_indexZnew_itemr#r#r$r-s4      r-)rr cCst|}t|g\}}|S)z^ Return the ``Metadata`` if DefaultSavePlanner was used to checkpoint ``state_dict``. )rr-)rr(_r_r#r#r$_create_default_local_metadatasrm),rgrAtypingrrrrrrrCZtorch.distributed._shard._utilsrZ'torch.distributed._shard.sharded_tensorr Zplannerr r r rrrrr/rrrrrrZplanner_helpersrrrutilsrrrLrIrMrNrJr'r-rmr#r#r#r$s&   $  &'    &