U <ci@sdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z mZmZmZmZmZmZddlZddlZddlmmZddlmZddlmZmZmZddlm Z ddl!m"Z"dd l#m$Z$d Z%ee$ej&ej'fZ(e"j)d d Z*d dddZ+e"j)ddddZ,e"j)dUddddZ-e"j)ddZ.e"j)e%fdddddZ/e"j)ddd d d!d!d!d"d#d$d%Z0e"j)d&d'Z1e"j)d(d)Z2e"j)d*d+Z3e"j)d,d-Z4e"j)d"d.d/d0Z5Gd1d2d2Z6e"j)d3d4d5d6d7d8d9d:Z7e"j)d3d;dd?d@Z8e"j)d3d;ddAdBZ9e"j)dVd3d4dCd7dDdEdFZ:e"j)dddddej;jdS)WzFunctions to verify exported ONNX model is functionally equivalent to original PyTorch model. ONNX Runtime is required, and is used as the ONNX backend for export verification. ) annotationsN)AnyCallableDictMappingOptionalSequenceTupleUnion)_C) _constants _experimentalutils)GLOBALS) _beartype)Number)ZCPUExecutionProvidercCs6g}|D](}t|tr&|t|q||q|SN) isinstancetupleextend_flatten_tuplesappend)elem flattenedtr;/tmp/pip-unpacked-wheel-gikjz4vx/torch/onnx/verification.pyr!s   rzUnion[list, np.ndarray]returncCst|tjr0|jr"|S|Snlt|ttfrLdd|DSt|t t t frft |St|trg}|D]}|t|t||gqx|S|S)NcSsg|] }t|qSr _to_numpy).0inprrr 4sz_to_numpy..)rtorchTensorZ requires_graddetachcpunumpylistrboolintfloatnparraydictrr )rrkrrrr -s   r r)cCs0|D]&}t|ttfs ||nt||q|Sr)rr)rr_inline_flatten_list)inputsZres_listirrrr1?sr1TcCs0g}|D]}|tj||dqdd|DS)Ncast_onnx_acceptedcSsg|] }t|qSrr)r!vrrrr#Osz$_unpack_to_numpy..)rrZunpack_quantized_tensor)valuesr5Zvalue_unpackedvaluerrr_unpack_to_numpyHs  r9c Csi}|r*t|dtr*|d}|dd}tt|}i}|D]\}}t|||<qBt|}|}t|D]D\}}|t|ks||j |krt d|d|||||j <qp| d|}|S)Nz(got too many positional inputs. inputs: z . kw_inputs: ) rr/r9ritemsr get_inputs enumeratelenname ValueErrorrun) ort_sessionr2Z kw_inputs ort_inputsZ input_nameinputZort_session_inputsr3ort_outsrrr_run_ortRs$   rFzUnion[str, io.BytesIO]z Sequence[str])model ort_providerscCsjz ddl}Wntk r(tdYnX|dkr6t}|}d|_|jt|trV|n|||d}|S)Nrz0onnxruntime is required for export verification.)Z providers) onnxruntime ImportError_ORT_PROVIDERSZSessionOptionsZlog_severity_levelZInferenceSessionrstrgetvalue)rGrHrJZsession_optionsrBrrr _ort_sessionhs rOz'Union[Sequence[_NumericType], Sequence]zEOptional[Union[_NumericType, Sequence[_NumericType], Sequence, Dict]]r,r*zOptional[float]rEpt_outsrtolatol check_shape check_dtype ignore_noneacceptable_error_percentagec CsP|rtj|\}}n t|gg}t|dd} t|g}t|t| ksftdt|dt| d|r|dksz|dkrtdt|| D]\} } z0|st | | \} } tj j | | |||d d Wqtk rH} zd|r6d t t j| | ||d t | j} | |kr6td | d| d|dWYqW5d} ~ XYqXqdS)a Compare ONNX Runtime and PyTorch outputs. Args: ort_outs: outputs from ONNX Runtime. pt_outs: outputs from PyTorch. rtol: relative tolerance in comparison between ONNX and PyTorch outputs. atol: absolute tolerance in comparison between ONNX and PyTorch outputs. ignore_none: Whether to ignore None type in torch output, which is usually the case with tracing. Set this to False, if torch output should keep None type, which is usually the case with exporting ScriptModules. acceptable_error_percentage: acceptable percentage of element mismatches in comparison. It should be a float of value between 0.0 and 1.0. Raises: AssertionError: if outputs from ONNX model and PyTorch model are not equal up to specified precision. ValueError: if arguments provided are invalid. Fr4z(Number of outputs differ ONNX runtime: (z ) PyTorch: ()g?gzAIf set, acceptable_error_percentage should be between 0.0 and 1.0T)rRrSrUZ equal_nan)rRrSzSuppressed AssertionError: z. Error percentage z within acceptable range .N)r$jit_flattenr1r9r>AssertionErrorr@zipr-Zbroadcast_arraysZtestingZ assert_closesumiscloseprodshapewarningswarn)rErQrRrSrTrUrVrW_Z pt_outs_npZort_outZpt_outeZerror_percentagerrr_compare_ort_pytorch_outputssV      rgcCs<t|tjtfr|f}t|}|r0t|}ni}||fS)aPrepare input for PyTorch model execution. Any future changes/formatting to the input before dispatching to the PyTorch model should be made in this function. Args: args: positional arguments for PyTorch model forward method. kwargs: keyword arguments for PyTorch model forward method. Returns: args: positional arguments for PyTorch model forward method. kwargs: keyword arguments for PyTorch model forward method. )rr$r%r/copydeepcopy)argskwargsrrr_prepare_input_for_pytorchs  rlcCsDt||\}}|s,t|dtr,|if}n|r<||f}n|}|S)aPrepare input for ONNX model export. Any future changes/formatting to the input before dispatching to the :func:`torch.onnx.export` api should be made in this function. Args: args: positional arguments for PyTorch model forward method. kwargs: keyword arguments for PyTorch model forward method. Returns: onnx_inputs: positional arguments for ONNX model export, as `args` in :func:`torch.onnx.export`. r:)rlrr/)rjrk onnx_inputsrrr_prepare_input_for_exports  rncs^t|||r tj\}nr<dikr<dd|dk rVfdd|DSSdS)aPrepare input for ONNX model execution in ONNX Runtime. Any future changes/formatting to the input before dispatching to the ONNX Runtime InferenceSession run should be made in this function. Args: args: positional arguments for PyTorch model forward method. kwargs: keyword arguments for PyTorch model forward method. Returns: onnx_inputs: positional arguments for ONNX model execution in ONNX Runtime. r:Ncsg|] }|qSrr)r!r3rmrrr#sz*_prepare_input_for_ort..)rnr$r[r\)rjrkremained_onnx_input_idxflattenrerror_prepare_input_for_orts  rrcCs4z t|WStk r.td|YSXdS)zHUsed for preserving original model in case forward mutates model states.zHFailed to clone model. Model state might be mutated during verification.N)rhri Exceptionrcrd)rGrrr_try_clone_models rt)rWc  sJtj f dd} | |||rF|D]}| |iq6dS)aCompare outputs from ONNX model runs with outputs from PyTorch model runs. ONNX Runtime is used for model execution backend for ONNX model. Raises: AssertionError: if outputs from ONNX model and PyTorch model are not equal up to specified precision. c sTt||\}}t}|||}t||}t|}t|| ddS)NrP)rlrtrrrFrg) input_args input_kwargsZpt_argsZ pt_kwargs model_copyrQrCrE rWrSrUrTrqrVrGrBrprRrr$compare_ort_pytorch_model_with_input@s(  zH_compare_ort_pytorch_model..compare_ort_pytorch_model_with_inputN)rbeartype)rGrBrurvadditional_test_inputsrprqrVrRrSrTrUrWryZtest_input_argsrrxr_compare_ort_pytorch_model's   r|c@s^eZdZdZejdddddZejddZejddd d d Zejdd d dZ dS) _GraphDiffz7A class to represent the difference between two graphs._C.Graphgraph_agraph_bcCs||_||_dS)zConstruct a _GraphDiff object. Args: graph_a (_C.Graph): First graph to compare. graph_b (_C.Graph): Second graph to compare. Nr)selfrrrrr__init__asz_GraphDiff.__init__cCs|S)z!See function :func:`diff_report`.) diff_report)rrrr__str__lsz_GraphDiff.__str__rM)linesrcCsddd|DS)N cSsg|] }d|qS) r)r!linerrrr#ssz&_GraphDiff._indent..)join splitlines)rrrrr_indentqsz_GraphDiff._indentrc Cs4|j}|j}t|}t|}||kr(dSt|d|d}d|d|g}t | | D]\}}t|t|krh| dtt|dt|d} d|d| g} |r| nd} | r| d|t| g|r| nd} | r| d|t| g| | q*qhd |S) a)Return a string representation of the graph difference. The report shows the first pair of nodes that diverges. It also shows the source location of the pair of nodes. Returns: graph_diff_report (str): A string representation of the graph difference. Tz Graph diff:zFirst diverging operator:z node diff:NzFormer source location:zLatter source location:r)rrrMdifflibZndiffrrr itertools zip_longestZnodesrZ sourceRanger) rrrZ graph_a_strZ graph_b_strZ graph_diffgraph_diff_reportZnode_aZnode_bZ node_diffZsource_printoutZstack_aZstack_brrrrus@    z_GraphDiff.diff_reportN) __name__ __module__ __qualname____doc__rrzrrrrrrrrr}^s  r}z.Union[torch.nn.Module, torch.jit.ScriptModule]z3Sequence[Tuple[Tuple[Any, ...], Mapping[str, Any]]]z_experimental.ExportOptionszfCallable[[torch.nn.Module, Tuple[Any, ...], Mapping[str, Any], _experimental.ExportOptions], _C.Graph]rM)rGtest_input_groupsexport_optionsmodel_to_graph_funcrc Cs`t|dkrtdd}|D]>\}}|||||}|dkr@|}qt||}|r|SqdS)aCheck if graph produced by `model_to_graph_func` is the same across `test_input_groups`. Args: model: See :func:`check_export_model_diff`. test_input_groups: See :func:`check_export_model_diff`. export_options: See :func:`check_export_model_diff`. model_to_graph_func: A function to convert a PyTorch model to a JIT IR graph. Returns: graph_diff_report (str): A string representation of the graph difference. z3Need at least two groups of test inputs to compare.Nr)r>r@r}r) rGrrrZ ref_jit_graphrjrk jit_graphrrrr_check_graph_diffs   rzTuple[Any, ...]zMapping[str, Any]r~)rGrjrkrrc Cs`|j}|j}t|||<t||}t||}t||\}}}}|W5QRSQRXdS)azAs part of the ONNX export steps, create a traced JIT graph from a PyTorch model. Args: model: See :func:`check_export_model_diff`. args: See :func:`check_export_model_diff`. kwargs: See :func:`check_export_model_diff`. export_options: See :func:`check_export_model_diff`. Returns: jit_graph (_C.Graph): A traced JIT graph. N)trainingverboserexporter_contextrnZ_pre_trace_quant_modelZ_create_jit_graph) rGrjrkrrr export_inputsrrerrr_traced_graph_from_models  rc Cs|j}|j}|j}|j}|j}|j} |j} |j} |dkr>tj }t |||sft j r^t jj}nt jj}|t_|t_t |||vt |j||} | dkri} t | || | t||} t || } t j|| || | || || d \}}}|W5QRSQRXdS)ayAs part of the ONNX export steps, export an ONNX JIT graph from a PyTorch model. Args: model: See :func:`check_export_model_diff`. args: See :func:`check_export_model_diff`. kwargs: See :func:`check_export_model_diff`. export_options: See :func:`check_export_model_diff`. Returns: onnx_graph (_C.Graph): An ONNX JIT graph. N)r dynamic_axes) opset_versionoperator_export_typeexport_modules_as_functionsrrr input_names output_namesr ZONNX_DEFAULT_OPSETrZ_setup_trace_module_map_C_onnxZ_CAFFE2_ATEN_FALLBACKZOperatorExportTypesZONNX_ATEN_FALLBACKZONNXrZexport_onnx_opset_versionrZ_decide_constant_foldingdo_constant_foldingZ_validate_dynamic_axesrnZ_decide_input_formatZ_model_to_graph)rGrjrkrrrrrrrrrrrZ onnx_graphrerrr_onnx_graph_from_modelsP     rz%Optional[_experimental.ExportOptions])rGrrrcCs8|dkrtn|}t|||t}|r*|St|||tS)aVerify exported model discrepancy between different groups of inputs. A graph is exported for each group of inputs. The exported graphs are then compared to each other, and discrepancies of first pair of nodes are reported. This function first checks the jit graph. If no discrepancies were found, it then checks the onnx graph. Unless otherwise specified, the jit/ONNX graph is expected to be the same, regardless of the inputs used for exporting. A discrepancy implies the graph exported is not accurate when run on other groups of inputs, which will typically results in runtime errors or mismatching output. Args: model (torch.nn.Module or torch.jit.ScriptModule): The model to be exported. test_input_groups (Sequence[Tuple[Tuple[Any, ...], Mapping[str, Any]]]): A sequence of input groups to be used to export the model. Each input group is a pair of (args, kwargs). export_options (_experimental.ExportOptions, optional): An _experimental.ExportOptions object that controls the export behavior. Returns: str: A string containing the diff of the exported models. N)r Z ExportOptionsrrr)rGrrZjit_diff_reportrrrcheck_export_model_diff1s rFgMbP?gHz>z$Union[torch.Tensor, Tuple[Any, ...]]zOptional[Mapping[str, Any]]zMOptional[Mapping[str, Union[Mapping[int, str], Mapping[str, Sequence[int]]]]]zOptional[Sequence[str]]ztorch.onnx.TrainingModez Optional[int]z8Optional[Sequence[Union[torch.Tensor, Tuple[Any, ...]]]]zOptional[Sequence[int]])rGrurvrrrrrrkeep_initializers_as_inputsrfixed_batch_sizeuse_external_datar{rprqrVrTrUrHrRrSrWcKs|tjjjkr|n|tjjjkr.|tt }t }| rj| t }tj|d}t||}t|}tj|||||| |||| || d t||}t||||| ||||||||d W5QRXW5QRXdS)a Verify model export to ONNX with ONNX Runtime. Args: model (torch.nn.Module or torch.jit.ScriptModule): See :func:`torch.onnx.export`. input_args (tuple): See :func:`torch.onnx.export`. input_kwargs (dict): See :func:`torch.onnx.export`. do_constant_folding (bool, optional): See :func:`torch.onnx.export`. dynamic_axes (dict, optional): See :func:`torch.onnx.export`. input_names (list, optional): See :func:`torch.onnx.export`. output_names (list, optional): See :func:`torch.onnx.export`. training (torch.onnx.TrainingMode): See :func:`torch.onnx.export`. opset_version (int, optional): See :func:`torch.onnx.export`. keep_initializers_as_inputs (bool, optional): See :func:`torch.onnx.export`. verbose (bool, optional): See :func:`torch.onnx.export`. fixed_batch_size (bool, optional): Legacy argument, used only by rnn test cases. use_external_data (bool, optional): Explicitly specify whether to export the model with external data. additional_test_inputs (list, optional): List of tuples. Each tuple is a group of input arguments to test. Currently only *args are supported. remained_onnx_input_idx (list, optional): If provided, only the specified inputs will be passed to the ONNX model. Supply a list when there are unused inputs in the model. Since unused inputs will be removed in the exported ONNX model, supplying all inputs will cause an error on unexpected inputs. This parameter tells the verifier which inputs to pass into the ONNX model. flatten (bool, optional): Default True. If True, unpack nested list/tuple/dict inputs into a flattened list of Tensors for ONNX. Set this to False if nested structures are to be preserved for ONNX, which is usually the case with exporting ScriptModules. ignore_none (bool, optional): Whether to ignore None type in torch output, which is usually the case with tracing. Set this to False, if torch output should keep None type, which is usually the case with exporting ScriptModules. Default to True. check_shape (bool, optional): Whether to check the shapes between PyTorch and ONNX Runtime outputs are exactly the same. Set this to False to allow output shape broadcasting. Default to True. check_dtype (bool, optional): Whether to check the dtypes between PyTorch and ONNX Runtime outputs are consistent. Default to True. ort_providers (sequence, optional): ONNX Runtime providers to use. rtol (float, optional): relative tolerance in comparison between ONNX and PyTorch outputs. atol (float, optional): absolute tolerance in comparison between ONNX and PyTorch outputs. acceptable_error_percentage (float, optional): acceptable percentage of element mismatches in comparison. It should be a float of value between 0.0 and 1.0. Raises: AssertionError: if outputs from ONNX model and PyTorch model are not equal up to specified precision. ValueError: if arguments provided are invalid. z model.onnx) rrrrrrrrr) rGrBrurvr{rprqrVrRrSrTrUrWN)r$onnx TrainingModeZTRAININGZtrainEVALevalZno_grad contextlib ExitStackioBytesIO enter_contexttempfileTemporaryDirectoryospathrrnrtrZ_exportrOr|)rGrurvrrrrrrrrrrr{rprqrVrTrUrHrRrSrWrestackZmodel_fZ tmpdir_pathZinputs_for_exportrwrBrrrverify]sRO   r)T)N)?r __future__rrrhrrrrrrctypingrrrrrrr r r(r-r$Ztorch._C._onnxr Z_onnxrZ torch.onnxr r rZtorch.onnx._globalsrZtorch.onnx._internalrZ torch.typesrrLr%ZndarrayZ _NumericTyperzrr r1r9rFrOrgrlrnrrrtr|r}rrrrrrrrrrrrs (       L    6I)C+