U Kc U@sddlZddlZddlmZddlmZmZmZmZmZm Z ddl m Z m Z m Z mZddlmZddlmZddlmZmZmZmZeeefZeeeed ffZd d Ze eee ed d dZeeeed ffedddZeeegefedddZeeeeee ee eefdddZe ee eeedddZ eeeed ffeeeeedddZ!ddZ"eedd d!d"Z#ed#d$d%Z$d1eeeeed'd(d)Z%d2eeeeed'd+d,Z&d-d.Z'ed/d0Z(dS)3N)Tensor)AnyCallableOptionalTupleUnionList) tree_flattentree_unflatten_broadcast_to_and_flattenTreeSpec) tree_map_)partial)_add_batch_dim_remove_batch_dim_vmap_decrement_nesting_vmap_increment_nesting.cs dtfdd}|S)Nzcfunctorch transforms don't yet support saved tensor hooks. Please open an issue with your use case.c s0tjj||W5QRSQRXdSN)torchZautogradgraphZdisable_saved_tensors_hooks)argskwargsfmessage7/tmp/pip-unpacked-wheel-gikjz4vx/functorch/_src/vmap.pyfn sz.doesnt_support_saved_tensors_hooks..fn) functoolswraps)rrrrr"doesnt_support_saved_tensors_hookss r!) flat_in_dims flat_argsreturncsZddt||Dtdkr(tdrRtfddDrRtdddS) NcSs"g|]\}}|dk r||qSr)size.0in_dimargrrr +sz0_validate_and_get_batch_size..rz/vmap: Expected at least one Tensor to vmap overc3s|]}|dkVqdS)rNr)r'r%Z batch_sizesrr /sz/_validate_and_get_batch_size..zTvmap: Expected all tensors to have the same size in the mapped dimension, got sizes z for the mapped dimension)ziplen ValueErrorany)r"r#rr+r_validate_and_get_batch_size(s  r1)batched_outputsr$cCst|trt|SdS)Nr ) isinstancetupler.)r2rrr _num_outputs6s r5)value num_elementserror_message_lambdar$cCs.t|ts|f|St||kr*t||Sr)r3r4r.r/)r6r7r8rrr _as_tuple?s     r9)in_dimsrfuncr$c Cst|ts8t|ts8tdt|d|dt|dt|dkrXtdt|dt|\}}t||}|dkrtdt|d|dt|dd |d t t ||D]\}\}}t|ts|dk rtdt|d|d |d t|tr4t|t s4tdt|d|d |d t|d |dk r|| ks\|| krtdt|d|d |d| d| d| d |dk r|dkr|| ||<qt |||||fS)Nvmap(z , in_dims=zv, ...)(): expected `in_dims` to be int or a (potentially nested) tuple matching the structure of inputs, got: .rz)(): got no inputs. Maybe you forgot to add inputs, or you are trying to vmap over a function with no inputs. The latter is unsupported.zb, ...)(): in_dims is not compatible with the structure of `inputs`. in_dims has structure r z but inputs has structure z, ...)(): Got in_dim=zE for an input but in_dim must be either an integer dimension or None.z' for an input but the input is of type zT. We cannot vmap over non-Tensor arguments, please use None as the respective in_dimz> for some input, but that input is a Tensor of dimensionality z so expected in_dim to satisfy -z <= in_dim < )r3intr4r/ _get_nametyper.r r enumerater-rdimr1) r:rr;r# args_specr"ir)r(rrr_process_batched_inputsGs<   ($(8rE)r"r# vmap_levelr$cs"fddt||D}t||S)Ncs(g|] \}}|dkr|n t||qSr)rr&rFrrr*|s z*_create_batched_inputs..)r-r )r"r#rFrCbatched_inputsrrGr_create_batched_inputsys rI)r2out_dimsrF batch_sizer;r$c st|\}|D]:}t|tjr"qtdtdtdt|dqfdd}t|tjrttrzg}qttrt dkr}dq|nt }|dkr|fd d t ||D} t | S) Nr<z , ...): `z%` must only return Tensors, got type z as a return.c s2tdtddtddd dS)Nr<, ..., out_dims=z`)(): out_dims is not compatible with the structure of `outputs`. out_dims has structure r z but outputs has structure r=)r/r?r r)r;rJ output_specrrincompatible_errors(z+_unwrap_batched..incompatible_errorr rcsg|]\}}t||qSr)r)r'Zbatched_outputout_dim)rKrFrrr*sz#_unwrap_batched..) r r3rrr/r?r@r>r4r.r r-r ) r2rJrFrKr;Zflat_batched_outputsoutrN flat_out_dimsZ flat_outputsr)rKr;rJrMrFr_unwrap_batcheds(  *     rRcCs,t|trdStdt|d|ddS)Nr<rLz): `out_dims` must be an int or a python collection of ints representing where in the outputs the vmapped dimension should appear.)r3r>r/r?)xr;rJrrr _check_ints  rT)rJr;r$cCs&t|trdSttt||d|dS)N)r;rJ)r3r>rrrT)rJr;rrr$_check_out_dims_is_int_or_int_pytrees rUr;cCst|dr|jSt|S)N__name__)hasattrrWreprrVrrrr?s r?error)r;r:rJ randomnessr$cs(ttfdd}|S)a vmap is the vectorizing map; ``vmap(func)`` returns a new function that maps :attr:`func` over some dimension of the inputs. Semantically, vmap pushes the map into PyTorch operations called by :attr:`func`, effectively vectorizing those operations. vmap is useful for handling batch dimensions: one can write a function :attr:`func` that runs on examples and then lift it to a function that can take batches of examples with ``vmap(func)``. vmap can also be used to compute batched gradients when composed with autograd. Args: func (function): A Python function that takes one or more arguments. Must return one or more Tensors. in_dims (int or nested structure): Specifies which dimension of the inputs should be mapped over. :attr:`in_dims` should have a structure like the inputs. If the :attr:`in_dim` for a particular input is None, then that indicates there is no map dimension. Default: 0. out_dims (int or Tuple[int]): Specifies where the mapped dimension should appear in the outputs. If :attr:`out_dims` is a Tuple, then it should have one element per output. Default: 0. randomness (str): Specifies whether the randomness in this vmap should be the same or different across batches. If 'different', the randomness for each batch will be different. If 'same', the randomness will be the same across batches. If 'error', any calls to random functions will error. Default: 'error'. WARNING: this flag only applies to random PyTorch operations and does not apply to Python's random module or numpy randomness. Returns: Returns a new "batched" function. It takes the same inputs as :attr:`func`, except each input has an extra dimension at the index specified by :attr:`in_dims`. It takes returns the same outputs as :attr:`func`, except each output has an extra dimension at the index specified by :attr:`out_dims`. .. warning: :func:`vmap` works best with functional-style code. Please do not perform any side-effects in :attr:`func`, with the exception of in-place PyTorch operations. Examples of side-effects include mutating Python data structures and assigning values to variables not captured in :attr:`func`. One example of using :func:`vmap` is to compute batched dot products. PyTorch doesn't provide a batched ``torch.dot`` API; instead of unsuccessfully rummaging through docs, use :func:`vmap` to construct a new function. >>> torch.dot # [D], [D] -> [] >>> batched_dot = functorch.vmap(torch.dot) # [N, D], [N, D] -> [N] >>> x, y = torch.randn(2, 5), torch.randn(2, 5) >>> batched_dot(x, y) :func:`vmap` can be helpful in hiding batch dimensions, leading to a simpler model authoring experience. >>> batch_size, feature_size = 3, 5 >>> weights = torch.randn(feature_size, requires_grad=True) >>> >>> def model(feature_vec): >>> # Very simple linear model with activation >>> return feature_vec.dot(weights).relu() >>> >>> examples = torch.randn(batch_size, feature_size) >>> result = functorch.vmap(model)(examples) :func:`vmap` can also help vectorize computations that were previously difficult or impossible to batch. One example is higher-order gradient computation. The PyTorch autograd engine computes vjps (vector-Jacobian products). Computing a full Jacobian matrix for some function f: R^N -> R^N usually requires N calls to ``autograd.grad``, one per Jacobian row. Using :func:`vmap`, we can vectorize the whole computation, computing the Jacobian in a single call to ``autograd.grad``. >>> # Setup >>> N = 5 >>> f = lambda x: x ** 2 >>> x = torch.randn(N, requires_grad=True) >>> y = f(x) >>> I_N = torch.eye(N) >>> >>> # Sequential approach >>> jacobian_rows = [torch.autograd.grad(y, x, v, retain_graph=True)[0] >>> for v in I_N.unbind()] >>> jacobian = torch.stack(jacobian_rows) >>> >>> # vectorized gradient computation >>> def get_vjp(v): >>> return torch.autograd.grad(y, x, v) >>> jacobian = functorch.vmap(get_vjp)(I_N) :func:`vmap` can also be nested, producing an output with multiple batched dimensions >>> torch.dot # [D], [D] -> [] >>> batched_dot = functorch.vmap(functorch.vmap(torch.dot)) # [N1, N0, D], [N1, N0, D] -> [N1, N0] >>> x, y = torch.randn(2, 3, 5), torch.randn(2, 3, 5) >>> batched_dot(x, y) # tensor of size [2, 3] If the inputs are not batched along the first dimension, :attr:`in_dims` specifies the dimension that each inputs are batched along as >>> torch.dot # [N], [N] -> [] >>> batched_dot = functorch.vmap(torch.dot, in_dims=1) # [N, D], [N, D] -> [D] >>> x, y = torch.randn(2, 5), torch.randn(2, 5) >>> batched_dot(x, y) # output is [5] instead of [2] if batched along the 0th dimension If there are multiple inputs each of which is batched along different dimensions, :attr:`in_dims` must be a tuple with the batch dimension for each input as >>> torch.dot # [D], [D] -> [] >>> batched_dot = functorch.vmap(torch.dot, in_dims=(0, None)) # [N, D], [D] -> [N] >>> x, y = torch.randn(2, 5), torch.randn(5) >>> batched_dot(x, y) # second arg doesn't have a batch dim because in_dim[1] was None If the input is a Python struct, :attr:`in_dims` must be a tuple containing a struct matching the shape of the input: >>> f = lambda dict: torch.dot(dict['x'], dict['y']) >>> x, y = torch.randn(2, 5), torch.randn(5) >>> input = {'x': x, 'y': y} >>> batched_dot = functorch.vmap(f, in_dims=({'x': 0, 'y': None},)) >>> batched_dot(input) By default, the output is batched along the first dimension. However, it can be batched along any dimension by using :attr:`out_dims` >>> f = lambda x: x ** 2 >>> x = torch.randn(2, 5) >>> batched_pow = functorch.vmap(f, out_dims=1) >>> batched_pow(x) # [5, 2] For any function that uses kwargs, the returned function will not batch the kwargs but will accept kwargs >>> x = torch.randn([2, 5]) >>> def f(x, scale=4.): >>> return x * scale >>> >>> batched_pow = functorch.vmap(f) >>> assert torch.allclose(batched_pow(x), x * 4) >>> batched_pow(x, scale=x) # scale is not batched, output has shape [2, 2, 5] .. note:: vmap does not provide general autobatching or handle variable-length sequences out of the box. cs6tt|\}}}}t||||f|Sr)rUrE _flat_vmap)rrrKr"r#rCr;r:rJr[rrwrappedfs zvmap..wrapped)_check_randomness_argrr )r;r:rJr[r^rr]rvmaps r`csVtdkr tdSddddtfdd}|S) a chunk_vmap is the vectorizing map (vmap) using chunks of input data. It is a mix of vmap (which vectorizes everything) and map (which executes things sequentially). ``chunk_vmap`` vectorizes the input with number of chunks at a time. For more details about vectorizing map, see :func:`vmap`. Args: func (function): A Python function that takes one or more arguments. Must return one or more Tensors. in_dims (int or nested structure): Specifies which dimension of the inputs should be mapped over. :attr:`in_dims` should have a structure like the inputs. If the :attr:`in_dim` for a particular input is None, then that indicates there is no map dimension. Default: 0. out_dims (int or Tuple[int]): Specifies where the mapped dimension should appear in the outputs. If :attr:`out_dims` is a Tuple, then it should have one element per output. Default: 0. randomness (str): Specifies whether the randomness in this vmap should be the same or different across batches. If 'different', the randomness for each batch will be different. If 'same', the randomness will be the same across batches. If 'error', any calls to random functions will error. Default: 'error'. WARNING: this flag only applies to random PyTorch operations and does not apply to Python's random module or numpy randomness. chunks (int): Number of chunks to use to split the input data. Default is 2. If equals to 1 then :func:`vmap` is called. Returns: Returns a new "batched" function. It takes the same inputs as :attr:`func`, except each input has an extra dimension at the index specified by :attr:`in_dims`. It takes returns the same outputs as :attr:`func`, except each output has an extra dimension at the index specified by :attr:`out_dims`. r )r:rJr[cs(tfddt||D}t|}|S)Nc3s2|]*\}}|dk r |j|dn|gVqdS)NrB)chunk)r'tr(chunks_rrr,sz;chunk_vmap.._get_chunk_flat_args..)r4r-)Z flat_args_Z flat_in_dims_rfZflat_args_chunkschunks_flat_argsrrer_get_chunk_flat_argss  z(chunk_vmap.._get_chunk_flat_argscSsNg}g}|D]$}t|\}}||||q |d}tt|}||fS)Nr)r appendlistr-)Zchunks_output_Zflat_chunks_outputZ arg_spec_listoutput flat_outputZ arg_specsarg_specflat_output_chunksrrr_flatten_chunks_outputs    z*chunk_vmap.._flatten_chunks_outputc stt|\}}}}||}g}dkr>tnd}|D]>}t||} |dk rft||t| |||f|qF|\} } ~t| } t | t | kst g} | D]"}| tj | d|d| d=q~ t | | S)Nsamerrb) rUrErZ get_rng_stater1Z set_rng_staterir\r r.AssertionErrorcatr )rr_r"r#rCrgZ chunks_outputrsrKrnrmrQrlrOrorhchunksr;r:rJr[rrwrapped_with_chunkss@      z'chunk_vmap..wrapped_with_chunks)r_r`rr )r;r:rJr[rvrwrrur chunk_vmapqs' #rxcCs|dkrtd|dS)N)rZZ differentrpzLOnly allowed values for randomness are 'error', 'different', or 'same'. Got ) RuntimeError)r[rrrr_sr_c KsDt||}z,t||||} || |} t| ||||WStXdSr)rrrIrR) r;rKr"r#rCrJr[rrFrHr2rrrr\s   r\)rrrZ)rrrZra))rrrtypingrrrrrrZtorch.utils._pytreer r r r Z pytree_hacksrrZtorch._C._functorchrrrrr>Z in_dims_tZ out_dims_tr!r1r5strr9rErIrRrTrUr?r`rxr_r\rrrrsz          3  )  ' n