U Kc,@sddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZmZmZddlmZdd lmZmZmZd d Zed d Z eej!e dddZ"ed*ddZ#ddZ$eej!e dddZ%eddZ&d+ddZ'ej(j)Z)e)j*e)j+e)j,e)j-e)j.e)j/e)j0e)j1e)j2e)j3e)j4e)j5e)j6e)j7j8e)j7j9e)j:e)j;e)je)j?e)j@hZAeeAZAeddZBd,e e ejCfe e eDddd ZEd!d"ZFdaGd#d$ZHd%d&ZId-d(d)ZJdS).N)contextmanager)partial)CallableOptionalTupleUnion)get_decompositions) aot_function aot_modulemake_boxed_compiler)strip_overloads)default_partition draw_graph#min_cut_rematerialization_partitioncCs6|jjD] }|jtjjjkrtjjj|_q||SN) graphnodestargettorchopsaten_to_copyto recompile)fx_gnodercCst|t|dS)N)static_argnums)r r>)r2r?rrrnnc_jit|sr@cCst|j|Sr)r5r6r<rrr print_compiles rA)fnr?cKsDtttt|d}||t|tjjr4t|f|St |f|SdS)a Wrapper function over :func:`aot_function` and :func:`aot_module` to perform memory efficient fusion. It uses the :func:`min_cut_rematerialization_partition` partitioner to perform efficient recomputation. It uses NVFuser to compile the generated forward and backward graphs. .. warning:: This API is experimental and likely to change. Args: fn (Union[Callable, nn.Module]): A Python function or a ``nn.Module`` that takes one ore more arguments. Must return one or more Tensors. static_argnums (Optional[Tuple[Int]]): An option tuple of ints to mark the arguments of the function as static. **kwargs: Any other overrides you want to make to the settings Returns: Returns a ``Callable`` or ``nn.Module`` that retains the eager behavior of the original :attr:`fn`, but whose forward and backward graphs have gone through recomputation optimizations, and the graphs have been compiled with nvfuser. ) fw_compiler bw_compiler partition_fndecompositionsr?N) r3rdefault_decompositionsupdater(rnnModuler r )rBr?r&configrrrmemory_efficient_fusions  rLcCsH|dtddd|Ddddlm}||t||S)NfooaQ ############################################################## # To minimize FX graph, copy and paste the below and run it # ############################################################## import torch import torch.fx as fx from functorch.compile import minifier, check_nvfuser_subprocess, check_nvfuser_correctness_subprocess inps = cSsg|]}|j|jfqSr)shaper#).0irrr sz!debug_compile..a? inps = [torch.ones(shape, dtype=dtype, device='cuda') for (shape, dtype) in inps] from foo import FxModule mod = FxModule().cuda() with torch.jit.fuser("fuser2"): # check_nvfuser_subprocess can be replaced with check_nvfuser_correctness_subprocess minifier(fx.symbolic_trace(mod), inps, check_nvfuser_subprocess) r)FxModule) to_folderr5rMrRZcudar3)rr/rRrrr debug_compiles   rTc Csg}t|d}t|}g}|D]}t|dkrD|}|t}nX|\}}}} } | tjtjtj tj tjtj tt hkrtj dd|| | d}ntj|| | d}||q"W5QRX|S)zZ Return a random input for the given inputs meta generated from _save_fx_default. rbr r)r#r))openpickleloadr$randomZrandrintZint32Zint64boolZuint8floatrandintappend) Zinput_data_pathinputsr2Z inputs_metametar*inputrNstrider#r)rrr get_inputss.    rcc s`ddlm}fddfddfdd}fd d }fd d }|||||td S)aO The forward, backward, and joint computation graph will be stored in {folder_name}/{current_name}/{current_name}_forward_{graph_index}, {folder_name}/{current_name}/{current_name}_backward_{graph_index}, and {folder_name}/{current_name}/{current_name}_joint_{graph_index} respectively. The input shape of the graphs will be stored in the .input files. These files can be loaded with pickle, and is a list of format (type, shape, stride, dtype, device). In the case of type = int or float, it is just (type,). For joint graph input, it is a nested list [[],[]] where the two inner lists have the same format. If dump_example_input is True, example_inputs will be stored in .pt file. Since each function might produce multiple graphs, the graph_index is used to distinguish difference graphs r)aot_module_simplifiedcsg}t|dkrBt|dtrB||d7}||d7}|S|D]P}t|tksbt|tkrt|t|fqF|t||j||j |j fqF|S)Nrr ) r$r(tupler*rZr\r^rNrbr#r))r% input_metaarg)get_input_metarrrhsz(_save_fx_default..get_input_metacs@t|jjdkr6ttjdd|dtddSt|}|j t j j | |}tjd}|std|ddd|dt t|tddd|dtdd|dtddr.graph_saver_helpercs||d|S)NZforwardr)rzZfw_argsr~rrgraph_saver_forwardPs z-_save_fx_default..graph_saver_forwardcs||dtd7a|S)NZbackwardr )ro)rzZbw_argsrrrgraph_saver_backwardTs z._save_fx_default..graph_saver_backwardcs||dt||S)NZjoint)r)rzZ joint_argsrrrgraph_saver_jointZs z+_save_fx_default..graph_saver_joint)rCrDrErF)Zfunctorch.compilerdrG) r{r}r|rzZexample_inputsrdrrrr)r{r|r}rhr~r_save_fx_default s  "   rFcCsdatt|||S)as Dump the forward, backward, and joint computation graph. Example Usage: save_fx_func = graph_dumper_aot(current_name, folder_name, dump_example_input = False) optimize_ctx = torchdynamo.optimize( save_fx_func ) with torch.enable_grad(): with optimize_ctx: result = forward_and_backward_pass(model, example_inputs) r)rorr)r{r}r|rrrgraph_dumper_aotgs r)T)N)N)F)KrprlrsrWrY contextlibr functoolsrtypingrrrrrZtorch.fxrrZtorch.nnrIZ torch._decomprZ aot_autogradr r r Z compile_utilsr Z partitionersrrrrr!Z GraphModuler3r9r;r=r>r@rrdetachZ gelu_backwardZleaky_relu_backwardZsigmoid_backwardZthreshold_backwardZhardtanh_backwardZhardsigmoid_backwardZhardswish_backwardZ tanh_backwardZ silu_backwardZ elu_backwardZcudnn_batch_normZcudnn_batch_norm_backwardZ masked_fillZScalarZTensorZeluZ leaky_reluZhardtanhZ hardswishZ hardsigmoidZ conj_physicalZ is_same_sizerGrArJrZrLrTrorcrrrrrrs        0       +Z