U KcU@sddlZddlmZddlmZddlmZmZddlmZm Z ddl Z ej ee ddddZ ej ee edd d d Zej ee ddd d ZddZddZej dddZej dddZej dddZd?ej ee eedfddddZej ee ddd Zd@ej ee eedfdd!d"d#ZdAej eeee d%d&d'Zej d(d)d*Zej d(d+d,ZGd-d.d.ej ZGd/d0d0ej ZdBej ed1d2d3ZdCej ed1d4d5Zd6d7Z d8d9Z!dDd;d<Z"dEd=d>Z#dS)FN)Tensor)ListTuple)_named_parameters_named_buffers)objnamesreturncCs<t|dkrt||dntt||d|dddS)z Deletes the attribute specified by the given list of names. For example, to delete the attribute obj.conv.weight, use _del_nested_attr(obj, ['conv', 'weight']) rrN)lendelattr_del_nested_attrgetattrrr rB/tmp/pip-unpacked-wheel-gikjz4vx/functorch/_src/make_functional.pyr s r )rr valuer cCs@t|dkrt||d|ntt||d|dd|dS)z Set the attribute specified by the given list of names to value. For example, to set the attribute obj.conv.weight, use _del_nested_attr(obj, ['conv', 'weight'], value) rrN)r setattr_set_nested_attrr)rr rrrrrs rcCs:t|dkrt||dStt||d|dddS)Nrr)r r_get_nested_attrrrrrr+s rcCs tddS)Na$make_functional(module): we don't yet support models that do parameter tying (also sometimes known as weight sharing). Please try to rewrite your model by replacing all instances of the tied parameter with another and/or comment your support in https://github.com/pytorch/functorch/issues/446) RuntimeErrorrrrrraise_parameter_tying_error2srcCsdd|D}dd|D}t|}t|}||sBti}|D]\}}|gf||<qN|D],\}}||kst||d|dqldd|D}|S)a[ named_params is a dictionary of tensors: {'A': A, 'B': B} tied_named_params is another dictionary of tensors {'A': A, 'B': B, 'B_tied': B} with potentially tied (or 'duplicated') tensors This function creates a mapping from the names in named_params to the names in tied_named_params: {'A': ['A'], 'B': ['B', 'B_tied']}. cSsi|]\}}||qSrr.0kvrrr Dsz$create_names_map..cSsi|]\}}||qSrrrrrrrEsr.cSsi|]\}}||qSrr)rkeyrrrrrQs)setkeysissubsetAssertionErroritemsappendsplitvalues)Z named_paramsZtied_named_paramsZtensors_dict_keysZtied_tensors_dict_keysZtensor_to_mappingrZtensorresultrrrcreate_names_map;s    r(modc Cst||dd}t|}t||}i}|D]@\}}||krR|tj|dd||<||} t||d| q,t|dkrd\} } n t|\} } | | |fS)NF)Zremove_duplicatemeta)devicerr)rr)tupler(torchZ empty_likerr%r zip) r*Z_named_membersZ named_memberssubclassZall_named_members names_mapmemonamep replacementr paramsrrr_extract_membersUs      r7cCst|t|jtjS)aZ This function removes all the Parameters from the model and return them as a tuple as well as their original attribute names. The weights must be re-loaded with `load_weights` before the model can be used again. Note that this function modifies the model in place and after this call, mod.parameters() will be empty. )r7rZnamed_parametersnn Parameterr)rrrextract_weightsis r:cCst|t|jddS)NcSs|SNr)xrrrvz!extract_buffers..)r7rZ named_buffersr)rrrextract_buffersusr?F.)r*r r6r cCsHt||D]8\}}|r t|}t||dt||d|q dS)a  Reload a set of weights so that `mod` can be used again to perform a forward pass. Note that the `params` are regular Tensors (that can have history) and so are left as Tensors. This means that mod.parameters() will still be empty after this call. rN)r/r8r9r r%r)r*r r6 as_paramsr3r4rrr load_weightsys  rA)r*r1c Csdg}t||D]L\\}}}t|D]6\}}|dkrF|t||t||t|||q&q|S)Nr)r/r# enumerater$rr r) r*r1elemsr'_Z attr_nameselemi attr_namerrr _swap_states rH)r*r buffersr cCs*t||D]\}}t||d|q dS)Nr)r/rr%)r*r rIr@r3r4rrr load_bufferssrJr)modelweights weight_namescCsPt|t|kstt|||t|dkrLt|t|ks@tt||||S)zload_state(model, weights, weight_names, buffers=(), buffer_names=()) -> model load_state takes `weights` and `buffers` and assigns them to the model. This is the inverse operation of `make_functional_deprecated_v1`. r)r r"rArJ)rKrLrMrI buffer_namesrrr load_states    rO)rKcsFt}t|dkr tdt\}}fdd}||fS)amake_functional_deprecated_v1(model) -> weights, func, weight_names Given an nn.Module, make_functional_deprecated_v1 extracts the state (weights) and returns a functional version of the model, `func`. This makes it so that it is possible use transforms over the parameters of `model`. `func` can be invoked as follows: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, func, _ = make_functional_deprecated_v1(model) func(weights, (x,)) ``` And here is an example of applying the grad transform: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, _, func = make_functional_deprecated_v1(model) grad_weights = grad(func)(weights, (x,)) ``` To put the state back into a model, use `load_state`. rzmake_functional_deprecated_v1(model): `model` has buffers. Please use make_functional_with_buffers_deprecated_v1(model) instead.cst}t||||Sr;)copydeepcopyrA)rLdata mutable_modelZ descriptorsrKrrfuns  z*make_functional_deprecated_v1..fun)listrIr rr:)rKrIrLrDrUrrTrmake_functional_deprecated_v1s   rWcs:t\}}t\}}fdd}|||fS)a`make_functional_with_buffers_deprecated_v1(model) -> weights, buffers, func, weight_names, buffer_names Given an nn.Module, make_functional_with_buffers_deprecated_v1 extracts the state (weights and buffers) and returns a functional version of the model, `func`. `func` can be invoked as follows: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, buffers, func, _, _ = make_functional_with_buffers_deprecated_v1(model) func(weights, buffers, (x,)) ``` And here is an example of applying the grad transform: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, buffers, func, _, _ = make_functional_with_buffers_deprecated_v1(model) func(weights, buffers, (x,)) grad_weights = grad(func)(weights, buffers, (x,)) ``` To put the state back into a model, use `load_state`. cs*t}t||t||||Sr;)rPrQrArJ)rLrIrRrSZbuf_descriptorsrKZweight_descriptorsrrrUs   z7make_functional_with_buffers_deprecated_v1..fun)r:r?)rKrLrDrIrUrrXr*make_functional_with_buffers_deprecated_v1srYcs6eZdZdZfddZed ddZddZZS) FunctionalModuleWithBufferszW This is the callable object returned by :func:`make_functional_with_buffers`. cs:tt|||_||_||_t||_|j|dSr;) superrZ__init__stateless_model param_namesrNdict all_names_mapupdate)selfr]r^rNparam_names_mapbuffer_names_map __class__rrr\s  z$FunctionalModuleWithBuffers.__init__Fc CsTt|}t|\}}}t|\}}}|r>|D]} | dq.t|||||||fSNF)rPrQr:r?requires_grad_rZ) rKdisable_autograd_tracking model_copyr6r^rcrIrNrdparamrrr _create_froms  z(FunctionalModuleWithBuffers._create_fromc OsDt|j|jt|t|}z|j||WSt|j|j|XdSr;)rHr]r`rV)rbr6rIargskwargs old_staterrrforwardsz#FunctionalModuleWithBuffers.forward)F __name__ __module__ __qualname____doc__r\ staticmethodrlrp __classcell__rrrerrZs   rZcs6eZdZdZfddZed ddZddZZS) FunctionalModulezJ This is the callable object returned by :func:`make_functional`. cs$tt|||_||_||_dSr;)r[rxr\r]r^r1)rbr]r^r1rerrr\%szFunctionalModule.__init__FcCs@t|}t|\}}}|r0|D]}|dq t||||fSrg)rPrQr:rhrx)rKrirjr6r^r1rkrrrrl+s   zFunctionalModule._create_fromc Os8t|j|j|}z|j||WSt|j|j|XdSr;)rHr]r1)rbr6rmrnrorrrrp5szFunctionalModule.forward)Frqrrrerrx s   rxrKricCs.t|}t|dkr tdtj||dS)amake_functional(model, disable_autograd_tracking=False) -> func, params Given a ``torch.nn.Module``, :func:`make_functional` extracts the state (params) and returns a functional version of the model, ``func``. This makes it so that it is possible use transforms over the parameters of ``model``. ``func`` can be invoked as follows: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional x = torch.randn(4, 3) model = nn.Linear(3, 3) func, params = make_functional(model) func(params, x) And here is an example of applying the grad transform over the parameters of a model. .. code-block:: python import torch import torch.nn as nn from functorch import make_functional, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) func, params = make_functional(model) def compute_loss(params, x, t): y = func(params, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(params, x, t) If the model has any buffers, please use :func:`make_functional_with_buffers` instead. Args: model (torch.nn.Module): Input model. disable_autograd_tracking (bool): Flag to disable gradients tracking for output parameters. The returned params are unrelated to the set of params from the original model. If False (default), the params will have ``requires_grad=True`` on them (aka they will be trackable with regular PyTorch autograd), matching the requires_grad-ness of the params from the original model. Otherwise, the returned params will have ``requires_grad=False``. Default, False. If you plan on using regular PyTorch autograd (e.g., if you want to call ``.backward()`` or ``torch.autograd.grad()``, then set ``disable_autograd_tracking=False``. Otherwise, if you're only planning on using functorch's gradient transforms, then please set ``disable_autograd_tracking=True`` to avoid unnecessarily tracking history with PyTorch autograd. rzdmake_functional(model): `model` has buffers. Please use make_functional_with_buffers(model) instead.ri)rVrIr rrxrl)rKrirIrrrmake_functional?s9  r{cCstj||dS)amake_functional_with_buffers(model, disable_autograd_tracking=False) -> func, params, buffers Given a ``torch.nn.Module``, make_functional_with_buffers extracts the state (params and buffers) and returns a functional version of the model ``func`` that can be invoked like a function. ``func`` can be invoked as follows: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional_with_buffers x = torch.randn(4, 3) model = nn.Linear(3, 3) func, params, buffers = make_functional_with_buffers(model) func(params, buffers, x) And here is an example of applying the grad transform over the parameters of a model: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional_with_buffers, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) func, params, buffers = make_functional_with_buffers(model) def compute_loss(params, buffers, x, t): y = func(params, buffers, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(params, buffers, x, t) Args: model (torch.nn.Module): Input model. disable_autograd_tracking (bool): Flag to disable gradients tracking for output parameters. The returned params are unrelated to the set of params from the original model. If False (default), the params will have ``requires_grad=True`` on them (aka they will be trackable with regular PyTorch autograd), matching the requires_grad-ness of the params from the original model. Otherwise, the returned params will have ``requires_grad=False``. Default, False. If you plan on using regular PyTorch autograd (e.g., if you want to call ``.backward()`` or ``torch.autograd.grad()``, then set ``disable_autograd_tracking=False``. Otherwise, if you're only planning on using functorch's gradient transforms, then please set ``disable_autograd_tracking=True`` to avoid unnecessarily tracking history with PyTorch autograd. rz)rZrlryrrrmake_functional_with_bufferss6r|cCs"tt|}tdd|D}|S)Ncss|]}t|VqdSr;r.stackdetachrZshardsrrr sz"transpose_stack..)r-r/)Ztuple_of_tuple_of_tensorsresultsrrrtranspose_stacks rcst|dkrtdtdd|Ds@tdd|Ds@tdt|dtfdd|Dsjtdtd d |D\}}}t|}t|}|d||fS) a(combine_state_for_ensemble(models) -> func, params, buffers Prepares a list of torch.nn.Modules for ensembling with :func:`vmap`. Given a list of ``M`` ``nn.Modules`` of the same class, stacks all of their parameters and buffers together to make ``params`` and ``buffers``. Each parameter and buffer in the result will have an additional dimension of size ``M``. :func:`combine_state_for_ensemble` also returns ``func``, a functional version of one of the models in :attr:`models`. One cannot directly run ``func(params, buffers, *args, **kwargs)`` directly, you probably want to use ``vmap(func, ...)(params, buffers, *args, **kwargs)`` Here's an example of how to ensemble over a very simple model: .. code-block:: python num_models = 5 batch_size = 64 in_features, out_features = 3, 3 models = [torch.nn.Linear(in_features, out_features) for i in range(num_models)] data = torch.randn(batch_size, 3) fmodel, params, buffers = combine_state_for_ensemble(models) output = vmap(fmodel, (0, 0, None))(params, buffers, data) assert output.shape == (num_models, batch_size, out_features) .. warning:: All of the modules being stacked together must be the same (except for the values of their parameters/buffers). For example, they should be in the same mode (training vs eval). This API is subject to change -- we're investigating better ways to create ensembles and would love your feedback how to improve this. rz?combine_state_for_ensemble: Expected at least one model, got 0.css|] }|jVqdSr;Ztrainingrmrrrrsz-combine_state_for_ensemble..css|]}|j VqdSr;rrrrrrszTcombine_state_for_ensemble: Expected all models to have the same training/eval mode.c3s|]}t|kVqdSr;)typerZ model0_typrrrszHcombine_state_for_ensemble: Expected all models to be of the same class.cSsg|] }t|qSr)r|rrKrrr sz.combine_state_for_ensemble..)r rallrr/r)modelsfuncsr6rIrrrcombine_state_for_ensembles& $ rcpucsfdd}|S)Nc stdkrtdtdkr8}t|Sd}|dkrXtd|dtfddt|D}t\}}}tdd|D}tt|}td d|D}|||fS) N,NYI: ensemble_shape with more than 1 elementr num_models  should be > 0c3s|]}VqdSr;torrDrmr,rn model_classrrrsz3functional_init..wrapped..css|]}t|dVqdS)rN)rWrrrrrscss|]}t|VqdSr;r}rrrrrs)r ValueErrorrrWr-ranger/) rmrnrK num_modelsrrDfnr rLr,ensemble_shaperrmrnrwrappeds    z functional_init..wrappedrrrr,rrrrfunctional_initsrcsfdd}|S)Nc stdkrtdtdkr8}t|Sd}|dkrXtd|dtfddt|D}t\}}}}}ttdd|D\} } tt| } td d| D} tt| } td d| D} | | |||fS) Nrrrrrc3s|]}VqdSr;rrrrrrsz@functional_init_with_buffers..wrapped..css|]}t|ddVqdS)Nr)rYrrrrrscss|]}t|VqdSr;r}rrrrrscss|]}t|VqdSr;r}rrrrrs)r rrrWr-rrYr/) rmrnrKrrrDrrMrNrLrIrrrr s*      z-functional_init_with_buffers..wrappedrrrrrfunctional_init_with_buffers sr)F)F)rr)F)F)rr)rr)$r.Ztorch.nnr8rtypingrrZnamed_members_polyfillrrrPModulestrr rrrr(r7r:r?rArHrJrOrWrYrZrxboolr{r|rrrrrrrrsB      $  $ (%,@96