U &cY@sddlZddlZddlZddlmZmZddlZddlmZmZm Z m Z ddl m Z ddl m ZddlmZddlmZdd lmZmZmZmZmZmZmZeeZzdd l mZWn&ek rGd d d e j ZYnXGd ddZ!Gddde j e!Z"ee#e#e#ddddZ$ddZ%dde&d dfddZ'Gddde(Z)Gddde j Z*Gddde j Z+Gd d!d!e j Z,Gd"d#d#e j Z-Gd$d%d%e j Z.Gd&d'd'e j Z/d(d)Z0d4d*d+Z1d5d,d-Z2d6d.d/Z3e#e#ed0ejfejd1d2d3Z4dS)7N)CallableTuple)Tensordevicedtypenn)CrossEntropyLoss) functional)get_activation)PretrainedConfig) DUMMY_INPUTSTF2_WEIGHTS_NAMETF_WEIGHTS_NAME WEIGHTS_NAME cached_path hf_bucket_url is_remote_url)Identitycs(eZdZdZfddZddZZS)rzFA placeholder identity operator that is argument-insensitive. cstdSN)super__init__)selfargskwargs __class__?/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_utils.pyr3szIdentity.__init__cCs|Srr)rinputrrrforward6szIdentity.forward__name__ __module__ __qualname____doc__rr __classcell__rrrrr/s rc@seZdZdZdeedddZeddZedd Z d d Z d d Z e e dddZ e edddZeedddZeee dddZdddZddZdS) ModuleUtilsMixinzF A few utilities for torch.nn.Modules, to be used as a mixin. F)only_trainablereturncCs0|rtdd|n|}tdd|DS)zQ Get number of (optionally, trainable) parameters in the module. cSs|jSr) requires_gradxrrrCz1ModuleUtilsMixin.num_parameters..css|]}|VqdSr)Znumel).0prrr Dsz2ModuleUtilsMixin.num_parameters..)filter parameterssum)rr(paramsrrrnum_parameters?szModuleUtilsMixin.num_parameterscOsLz ddl}Wntk r(tdYnX|t}|}|j|_dS)NrFYou need to install psutil (pip install psutil) to use memory tracing.)psutil ImportErrorProcessosgetpid memory_inforssmem_rss_pre_forward)modulerrr8processmemrrr_hook_rss_memory_pre_forwardFs z-ModuleUtilsMixin._hook_rss_memory_pre_forwardcOsrz ddl}Wntk r(tdYnX|t}|}|j|_|j|j}|t |drf|j nd|_ dS)Nrr7 mem_rss_diff) r8r9r:r;r<r=r>mem_rss_post_forwardr?hasattrrD)r@rrr8rArBrDrrr_hook_rss_memory_post_forwardRs  z.ModuleUtilsMixin._hook_rss_memory_post_forwardcCs2|D]}||j||jq|dS)a Add a memory hook before and after each sub-module forward pass to record increase in memory consumption. Increase in memory consumption is stored in a `mem_rss_diff` attribute for each module and can be reset to zero with `model.reset_memory_hooks_state()` N)modulesZregister_forward_pre_hookrCZregister_forward_hookrGreset_memory_hooks_staterr@rrradd_memory_hooks`s  z!ModuleUtilsMixin.add_memory_hookscCs$|D]}d|_d|_d|_qdSNr)rHrDrEr?rJrrrrIis z)ModuleUtilsMixin.reset_memory_hooks_state)r)cCst|jSr)nextr3rrrrrroszModuleUtilsMixin.devicecCst|jSr)rMr3rrNrrrrsszModuleUtilsMixin.dtype)encoder_attention_maskr)cCsj|dkr(|dddddddf}|dkrL|ddddddf}|j|jd}d|d}|S)z"type: torch.Tensor -> torch.TensorNr?e)dimtor)rrOZencoder_extended_attention_maskrrrinvert_attention_maskws   z&ModuleUtilsMixin.invert_attention_mask)attention_mask input_shaperc Cs|dkr*|dddddddf}n|dkr|jjr|\}}tj||d}|ddddf||d|ddddfk}||j}|dddddddf|ddddddf}q|ddddddf}ntd ||j |j|jd}d|d }|S) aMakes broadcastable attention mask and causal mask so that future and maked tokens are ignored. Arguments: attention_mask: torch.Tensor with 1 indicating tokens to ATTEND to input_shape: tuple, shape of input_ids device: torch.Device, usually self.device Returns: torch.Tensor with dtype of attention_mask.dtype rPNrQ)rr zAWrong shape for input_ids (shape {}) or attention_mask (shape {})rRrSg) rUconfigZ is_decodertorcharangerepeatrVr ValueErrorformatshape) rrXrYrZextended_attention_mask batch_sizeZ seq_lengthZseq_idsZ causal_maskrrrget_extended_attention_masks&  . 6  z,ModuleUtilsMixin.get_extended_attention_maskcCs6|dk r(|||}|dkr2|d}n dg|}|S)a # Prepare head mask if needed # 1.0 in head_mask indicate we keep the head attention_probs has shape bsz x n_heads x N x N Arguments: head_mask: torch.Tensor or None: has shape [num_heads] or [num_hidden_layers x num_heads] num_hidden_layers: int Returns: Tensor of shape shape [num_hidden_layers x batch x num_heads x seq_length x seq_length] or list with [None] for each layer NT)_convert_head_mask_to_5d unsqueeze)r head_masknum_hidden_layersZis_attention_chunkedrrr get_head_masks    zModuleUtilsMixin.get_head_maskcCs|dkr<|dddd}||dddd}n"|dkr^|ddd}|dks|td||j|jd}|S)zD-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]r rrcrQzhead_mask.dim != 5, instead rR)rUreexpandAssertionErrorrVr)rrfrgrrrrds  z)ModuleUtilsMixin._convert_head_mask_to_5dN)F)F)r"r#r$r%boolintr6 staticmethodrCrGrKrIpropertyrrrrWtuplerbrhrdrrrrr':s    , r'cseZdZdZdZiZdZeddZfddZ edd Z d d Z d d Z ddZ ddZddZd3ddZddZd4ddZddZddZddZed d!Zd"d#Zd$d%Zd&d'Zd(d)Zed5d*d+Zd,d-Zd.d/Z e!e"e#e"e#d0d1d2Z$Z%S)6PreTrainedModela/ Base class for all models. :class:`~transformers.PreTrainedModel` takes care of storing the configuration of the models and handles methods for loading/downloading/saving models as well as a few methods common to all models to (i) resize the input embeddings and (ii) prune heads in the self-attention heads. Class attributes (overridden by derived classes): - ``config_class``: a class derived from :class:`~transformers.PretrainedConfig` to use as configuration class for this model architecture. - ``pretrained_model_archive_map``: a python ``dict`` of with `short-cut-names` (string) as keys and `url` (string) of associated pretrained weights as values. - ``load_tf_weights``: a python ``method`` for loading a TensorFlow checkpoint in a PyTorch model, taking as arguments: - ``model``: an instance of the relevant subclass of :class:`~transformers.PreTrainedModel`, - ``config``: an instance of the relevant subclass of :class:`~transformers.PretrainedConfig`, - ``path``: a path (string) to the TensorFlow checkpoint. - ``base_model_prefix``: a string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model. NcCsdttiS)zx Dummy inputs to do a forward pass in the network. Returns: torch.Tensor with dummy inputs input_ids)r[Ztensorr rNrrr dummy_inputsszPreTrainedModel.dummy_inputscs6tt|ts,td|jj|jj||_dS)NzParameter config in `{}(config)` should be an instance of class `PretrainedConfig`. To create a model from a pretrained model use `model = {}.from_pretrained(PRETRAINED_MODEL_NAME)`) rr isinstancer r^r_rr"rZ)rrZinputsrrrrrs  zPreTrainedModel.__init__cCst||j|Sr)getattrbase_model_prefixrNrrr base_modelszPreTrainedModel.base_modelcCs&t||j|}||k r|StdS)z Returns the model's input embeddings. Returns: :obj:`nn.Module`: A torch module mapping vocabulary to hidden states. N)rwrxget_input_embeddingsNotImplementedError)rryrrrrzsz$PreTrainedModel.get_input_embeddingscCs*t||j|}||k r"||ntdS)z Set model's input embeddings Args: value (:obj:`nn.Module`): A module mapping vocabulary to hidden states. N)rwrxset_input_embeddingsr{)rvalueryrrrr|s z$PreTrainedModel.set_input_embeddingscCsdS)z Returns the model's output embeddings. Returns: :obj:`nn.Module`: A torch module mapping hidden states to vocabulary. NrrNrrrget_output_embeddingssz%PreTrainedModel.get_output_embeddingscCs$|}|dk r |||dS)z Tie the weights between the input embeddings and the output embeddings. If the `torchscript` flag is set in the configuration, can't handle parameter sharing so we are cloning the weights instead. N)r~_tie_or_clone_weightsrz)routput_embeddingsrrr tie_weights(szPreTrainedModel.tie_weightscCs|jjrt|j|_n|j|_t|dddk rhtjj |j j d|jj d|j j dfdd|j _ t |drt |dr|j|_dS)zZ Tie or clone module weights depending of whether we are using TorchScript or not biasNrZconstant out_featuresnum_embeddings)rZZ torchscriptr Parameterweightclonerwr[r padrdatar`rFrr)rrZinput_embeddingsrrrr2sz%PreTrainedModel._tie_or_clone_weightscCs>t||j|}||}|dkr$|S||j_||_||S)a Resize input token embeddings matrix of the model if new_num_tokens != config.vocab_size. Take care of tying weights embeddings afterwards if the model class has a `tie_weights()` method. Arguments: new_num_tokens: (`optional`) int: New number of tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or None: does nothing and just returns a pointer to the input tokens ``torch.nn.Embeddings`` Module of the model. Return: ``torch.nn.Embeddings`` Pointer to the input tokens Embeddings Module of the model N)rwrx_resize_token_embeddingsrZ vocab_sizer)rnew_num_tokensryZ model_embedsrrrresize_token_embeddingsDs  z'PreTrainedModel.resize_token_embeddingscCs&|}|||}|||Sr)rz_get_resized_embeddingsr|)rrold_embeddingsnew_embeddingsrrrr_s  z(PreTrainedModel._resize_token_embeddingscCs|dkr |S|j\}}||kr&|St||}||jj||t||}|jjd|ddf|jjd|ddf<|S)a Build a resized Embedding Module from a provided token Embedding Module. Increasing the size will add newly initialized vectors at the end Reducing the size will remove vectors from the end Args: new_num_tokens: (`optional`) int New number of tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end Reducing the size will remove vectors from the end If not provided or None: return the provided token Embedding Module. Return: ``torch.nn.Embeddings`` Pointer to the resized Embedding Module or the old Embedding Module if new_num_tokens is None N) rsizerZ EmbeddingrVr _init_weightsminr)rrrZold_num_tokensZold_embedding_dimrZnum_tokens_to_copyrrrres   ,z'PreTrainedModel._get_resized_embeddingscCs.||j|jjr"||jj|dS)z* Initialize and prunes weights if needed. N)applyrrZ pruned_heads prune_headsrrNrrr init_weightss zPreTrainedModel.init_weightscCsN|D]4\}}t|jj|gt|B}t||jj|<q|j|dS)aZ Prunes heads of the base model. Arguments: heads_to_prune: dict with keys being selected layer indices (`int`) and associated values being the list of heads to prune in said layer (list of `int`). E.g. {1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on layer 1 and heads 2 and 3 on layer 2. N)itemssetrZrgetlistryZ _prune_heads)rZheads_to_prunelayerZheadsZ union_headsrrrrs zPreTrainedModel.prune_headscCstj|stdt|dr$|jn|}|jjg|j_ tj |t }t |jddrddl mm}|rx|j||||n|j|t||td|dS)a  Save a model and its configuration file to a directory, so that it can be re-loaded using the `:func:`~transformers.PreTrainedModel.from_pretrained`` class method. Arguments: save_directory: directory to which to save. zPSaving path should be a directory where the model and configuration can be savedr@ xla_deviceFrNzModel weights saved in {})r;pathisdirrkrFr@rr"rZZ architecturesjoinrrwtorch_xla.core.xla_modelcore xla_modelZis_master_ordinalsave_pretrainedsave state_dictr[loggerinfor_)rZsave_directoryZ model_to_saveZoutput_model_filexmrrrrs   zPreTrainedModel.save_pretrainedc s|dd}|dd|dd}|dd}|dd}|dd}|d d} |d d} |d d} |d d } t|ts|dk r|n|} jj| f||d ||| | d|\}}n|}|dk r|jkrj|}n tj|r|r(tj tj |t dr(tj |t d}np|rVtj tj |t rVtj |t }nBtj tj |t r~tj |t }ntdt t t dg|njtj |st|r|}nLtj |dr|std|d|d}nt||rt nt | d}zt|||| || d}WnXtk rt|jkrFd|}n"d|d j|t t t g}t|YnX||krtd|ntd||nd}|f||}dkr|sztj|ddWntk rtdYnXggg|rx|dr2|||dd}nBzddlm}|||d d}Wn"tk rrt d YnXng}g}D]P}d}d!|kr|!d!d"}d#|kr|!d#d$}|r|"||"|qt#||D]\}}||<qt$d%d%dk r"_&d6t'j(d'fd(d) d&}|}t)fd*d+D}t*|j+s|rj+d,}t*|j+r|st$|j+}||d-|j,j-|j,j-kr|.}fd.d/|.D}/||t0dkrtd0|j,j-t0dkr.loadc3s|]}|jVqdSr) startswithrx)r/sclsrrr1sz2PreTrainedModel.from_pretrained..r)rcs g|]}|jddqS)rrc)splitrx)r/keyrrr sz3PreTrainedModel.from_pretrained..z7Weights of {} not initialized from pretrained model: {}z0Weights from pretrained model not used in {}: {}z*Error(s) in loading state_dict for {}: {}z )rrrr)rr):poprur config_classfrom_pretrainedpretrained_model_archive_mapr;rrisfilerrrrEnvironmentErrorr_rrkrrkeysrrr[r ExceptionOSErrorendswithZload_tf_weightsZ transformersrr9errorreplaceappendziprwcopyrrModuleanyrFrxrr"rextendlen RuntimeErrorrevalrrrrZsend_cpu_data_to_devicerV) rZpretrained_model_name_or_pathZ model_argsrrZrrrrrrrrZ config_pathZ model_kwargsZ archive_fileZresolved_archive_filemsgmodelrZold_keysZnew_keysrZnew_keyZold_keyZ start_prefixZ model_to_loadZhas_prefix_moduleZbase_model_state_dictZ)head_model_state_dict_without_base_prefixZ loading_inforr)rrrrrrrrrsTG               "                           zPreTrainedModel.from_pretrainedcKsd|iS)Nrsr)rrsrrrrprepare_inputs_for_generationsz-PreTrainedModel.prepare_inputs_for_generationcKs|Srr)rlogitsrrrrprepare_logits_for_generationsz-PreTrainedModel.prepare_logits_for_generationcCs8t|dks|dkrdSt|jdr4|jjdkr4dSdS)zWDuring generation, decide whether to pass the `past` variable to the next forward pass.r Fmem_lenrT)rrFrZr)routputs use_cacherrr _use_caches zPreTrainedModel._use_cachecCsft||D]T}t||D]>}|||fdkrJ|||f|9<q |||f|<q q dS)zGrepetition penalty (from CTRL paper https://arxiv.org/abs/1909.05858). rN)rangertolist)rZlprobsra num_beamsZprev_output_tokensrepetition_penaltyiZprevious_tokenrrrenforce_repetition_penalty_s z+PreTrainedModel.enforce_repetition_penalty_c Ks(|dkrtd|dk r |n|jj}|dk r4|n|jj}|dk rH|n|jj}|dk r\|n|jj}|dk rp|n|jj}|dk r|n|jj}|dk r|n|jj }|dk r|n|jj }| dk r| n|jj } | dk r| n|jj } | dk r| n|jj } | dk r| n|jj} |dk r|n|jj}|dk r(|n|jj}|dk r>|n|jj}| dk rT| n|jj} |dk rj|n|jj}|dk r|n|jj}|dk r|jd}nd}t|tr|dkstdt|tr|dkstdt|tstdt|tstdt|tstd t|tr0|dks8td |dksJtd t|tr`|dkshtd d| krdksntd | dkstd|dk st| tr| dkstd| dkst| tr| dkstd|dkst|tr |dkstd|dks&tdt|tr<|dksDtdt|trZ|dksbtd| dkst| trt| dtstd|dkrt| tr| dkstdtj|df| tjt|j d}n|!dkstd|dkr,|dkr|dks,tdn||ks,td|dkrZ| dk rZ| |krZ|"| }n|dkrp|#|j}| dkr|dk rt$%d&||} t'|jdr|jj(}n2|jj)rt'|jd rt'|jj*dr|jj*j(}|r||}|}n|}d}|jj)rr|dkr| }|dk s(td!t'|d"sBtd#&|t+|j,s^td$&|j,|,}|||d%}|dks|dkr|jd&}|-d.||||}|-d.||||}|/0|||}|/0|||}|jj)rtj||df|tjt|j d}d}||djdksPtd'|d(|djdd)t1|0d&d2d||0d&3|j }|d4d|f|dd}nd}|jd&}|dkr|j5||||||||| | || | | |||||||||||d*}n2|j6|||||||| | || | | |||||||d+}|S),a Generates sequences for models with a LM head. The method currently supports greedy decoding, beam-search decoding, sampling with temperature, sampling with top-k or nucleus sampling. Adapted in part from `Facebook's XLM beam search code`_. .. _`Facebook's XLM beam search code`: https://github.com/facebookresearch/XLM/blob/9e6f6814d17be4fe5b15f2e6c43eb2b2d76daeb4/src/model/transformer.py#L529 Parameters: input_ids: (`optional`) `torch.LongTensor` of shape `(batch_size, sequence_length)` The sequence used as a prompt for the generation. If `None` the method initializes it as an empty `torch.LongTensor` of shape `(1,)`. max_length: (`optional`) int The max length of the sequence to be generated. Between `min_length` and infinity. Default to 20. min_length: (`optional`) int The min length of the sequence to be generated. Between 0 and infinity. Default to 0. do_sample: (`optional`) bool If set to `False` greedy decoding is used. Otherwise sampling is used. Defaults to `False` as defined in `configuration_utils.PretrainedConfig`. early_stopping: (`optional`) bool if set to `True` beam search is stopped when at least `num_beams` sentences finished per batch. Defaults to `False` as defined in `configuration_utils.PretrainedConfig`. num_beams: (`optional`) int Number of beams for beam search. Must be between 1 and infinity. 1 means no beam search. Default to 1. temperature: (`optional`) float The value used to module the next token probabilities. Must be strictly positive. Default to 1.0. top_k: (`optional`) int The number of highest probability vocabulary tokens to keep for top-k-filtering. Between 1 and infinity. Default to 50. top_p: (`optional`) float The cumulative probability of parameter highest probability vocabulary tokens to keep for nucleus sampling. Must be between 0 and 1. Default to 1. repetition_penalty: (`optional`) float The parameter for repetition penalty. Between 1.0 and infinity. 1.0 means no penalty. Default to 1.0. pad_token_id: (`optional`) int Padding token. Default to specicic model pad_token_id or None if it does not exist. bos_token_id: (`optional`) int BOS token. Defaults to `bos_token_id` as defined in the models config. eos_token_id: (`optional`) int EOS token. Defaults to `eos_token_id` as defined in the models config. length_penalty: (`optional`) float Exponential penalty to the length. Default to 1. no_repeat_ngram_size: (`optional`) int If set to int > 0, all ngrams of size `no_repeat_ngram_size` can only occur once. bad_words_ids: (`optional`) list of lists of int `bad_words_ids` contains tokens that are not allowed to be generated. In order to get the tokens of the words that should not appear in the generated text, use `tokenizer.encode(bad_word, add_prefix_space=True)`. num_return_sequences: (`optional`) int The number of independently computed returned sequences for each element in the batch. Default to 1. attention_mask (`optional`) obj: `torch.LongTensor` of same shape as `input_ids` Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. Defaults to `None`. `What are attention masks? <../glossary.html#attention-mask>`__ decoder_start_token_id=None: (`optional`) int If an encoder-decoder model starts decoding with a different token than BOS. Defaults to `None` and is changed to `BOS` later. use_cache: (`optional`) bool If `use_cache` is True, past key values are used to speed up decoding if applicable to model. Defaults to `True`. model_specific_kwargs: (`optional`) dict Additional model specific kwargs will be forwarded to the `forward` function of the model. Return: output: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)` sequence_length is either equal to max_length or shorter if all batches finished early due to the `eos_token_id` Examples:: tokenizer = AutoTokenizer.from_pretrained('distilgpt2') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('distilgpt2') # Download model and configuration from S3 and cache. outputs = model.generate(max_length=40) # do greedy decoding print('Generated: {}'.format(tokenizer.decode(outputs[0], skip_special_tokens=True))) tokenizer = AutoTokenizer.from_pretrained('openai-gpt') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('openai-gpt') # Download model and configuration from S3 and cache. input_context = 'The dog' input_ids = tokenizer.encode(input_context, return_tensors='pt') # encode input context outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3, temperature=1.5) # generate 3 independent sequences using beam search decoding (5 beams) with sampling from initial context 'The dog' for i in range(3): # 3 output sequences were generated print('Generated {}: {}'.format(i, tokenizer.decode(outputs[i], skip_special_tokens=True))) tokenizer = AutoTokenizer.from_pretrained('distilgpt2') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('distilgpt2') # Download model and configuration from S3 and cache. input_context = 'The dog' input_ids = tokenizer.encode(input_context, return_tensors='pt') # encode input context outputs = model.generate(input_ids=input_ids, max_length=40, temperature=0.7, num_return_sequences=3) # 3 generate sequences using by sampling for i in range(3): # 3 output sequences were generated print('Generated {}: {}'.format(i, tokenizer.decode(outputs[i], skip_special_tokens=True))) tokenizer = AutoTokenizer.from_pretrained('ctrl') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('ctrl') # Download model and configuration from S3 and cache. input_context = 'Legal My neighbor is' # "Legal" is one of the control codes for ctrl input_ids = tokenizer.encode(input_context, return_tensors='pt') # encode input context outputs = model.generate(input_ids=input_ids, max_length=50, temperature=0.7, repetition_penalty=1.2) # generate sequences print('Generated: {}'.format(tokenizer.decode(outputs[0], skip_special_tokens=True))) tokenizer = AutoTokenizer.from_pretrained('gpt2') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('gpt2') # Download model and configuration from S3 and cache. input_context = 'My cute dog' # "Legal" is one of the control codes for ctrl bad_words_ids = [tokenizer.encode(bad_word, add_prefix_space=True) for bad_word in ['idiot', 'stupid', 'shut up']] input_ids = tokenizer.encode(input_context, return_tensors='pt') # encode input context outputs = model.generate(input_ids=input_ids, max_length=100, do_sample=True, bad_words_ids=bad_words_ids) # generate sequences without allowing bad_words to be generated Na$You tried to generate sequences with a model that does not have a LM Head.Please use another model class (e.g. `OpenAIGPTLMHeadModel`, `XLNetLMHeadModel`, `GPT2LMHeadModel`, `CTRLLMHeadModel`, `T5WithLMHeadModel`, `TransfoXLLMHeadModel`, `XLMWithLMHeadModel`, `BartForConditionalGeneration` )rr z3`max_length` should be a strictly positive integer.z*`min_length` should be a positive integer.z `do_sample` should be a boolean.z%`early_stopping` should be a boolean.z `use_cache` should be a boolean.z2`num_beams` should be a strictly positive integer.z*`temperature` should be strictly positive.z%`top_k` should be a positive integer.z"`top_p` should be between 0 and 1.rSz$`repetition_penalty` should be >= 1.zIIf input_ids is not defined, `bos_token_id` should be a positive integer.z,`pad_token_id` should be a positive integer.z,`eos_token_id` should be a positive integer.z-`length_penalty` should be strictly positive.z4`no_repeat_ngram_size` should be a positive integer.z=`num_return_sequences` should be a strictly positive integer.zZ`bad_words_ids` is either `None` or a list of lists of tokens that should not be generatedzyou should either supply a context to complete as `input_ids` input or a `bos_token_id` (integer >= 0) as a first token to start the generation.rrrQz>Input prompt should be of shape (batch_size, sequence length).FzGreedy decoding will always produce the same output for num_beams == 1 and num_return_sequences > 1. Please set num_return_sequences = 1zxGreedy beam search decoding cannot return more sequences than it has beams. Please set num_beams >= num_return_sequenceszHSetting `pad_token_id` to {} (first `eos_token_id`) to generate sequencerdecoderzWdecoder_start_token_id or bos_token_id has to be defined for encoder-decoder generation get_encoderz/{} should have a 'get_encoder' function definedz{} should be a method)rXrcz5expected encoder_outputs[0] to have 1st dimension bs=z, got  )cur_len max_length min_length do_sampleearly_stopping temperaturetop_ktop_prno_repeat_ngram_size bad_words_ids bos_token_id pad_token_iddecoder_start_token_id eos_token_idranum_return_sequenceslength_penaltyrrencoder_outputsrXrmodel_specific_kwargs)rrrrrrrrrrrrrrrarrXrr)7r~AttributeErrorrZrrrrrrrrrrrrrrrrrrr`rurmrkrlrr[fulllongrMr3rrUnenew_onesrwarningr_rFris_encoder_decoderrcallablerrerj contiguousviewr\r]rV index_select_generate_beam_search_generate_no_beam_search) rrsrrrrrrrrrrrrrrrrrXrrrrarZeffective_batch_sizeZeffective_batch_multencoderrZ input_ids_lenrZexpanded_batch_idxsoutputrrrgenerates   "                          zPreTrainedModel.generatec%Cs||d}|||}|}||krZ|j|f|||d|}|f|}|ddddddf}|||r|d}| dkr|||d|| | dkrt||| |}t|D]}td ||||f<q| dk r t|| }t|D]}td ||||f<q|dk r6||kr6td |dd|f<|r|dkrN||}t |||d}t j |dd }t j |dd d}nt j|dd }|dk r||| d|}n|}t j||dgdd }|dk r||k} || }!||!|d|| |dkr qZ|jjd krPt j|||jddfgdd }|d}q$||kr| dk std |||| }"n|}"t|D]*\}#}$|$d||#|"|#d||#f<q|"S) z Generate sequences for each example without beam search (num_beams == 1). All returned sequence are generated independantly. r pastrXrrNrcrSinf)rrrUZ num_samplesFzB`Pad_token_id` has to be defined if batches have different lengths) newfill_rrrcalc_banned_ngram_tokensrfloatcalc_banned_bad_words_idstop_k_top_p_filteringFsoftmaxr[ multinomialsqueezeZargmaxcatremulrrlZ masked_fill_Zmul_maxrZrrr`ritemrk enumerate)%rrsrrrrrrrrrrrrrrrarrXrrZunfinished_sents sent_lengthsr model_inputsrnext_token_logits banned_tokens batch_idxprobsZ next_tokenZ tokens_to_addZ eos_in_sentsZ+is_sents_unfinished_and_token_to_add_is_eosdecodedhypo_idxhyporrrrsx           $z(PreTrainedModel._generate_no_beam_searchcE sfddt|D}tj|ftj|jd}|dkrPd|ddddf<|d}|}d dt|D}|krp|j|f|||d |}|f|}|d dddddf} |||r|d}| d kr|| ||| |d kr| |} |j j r|dkr|j | |d } t j | dd}!dk rR||krRtd |!ddf<| d kr|}"t||"| |}#t|#D]\}$}%td |!|$|%f<qz| dk rt|| }%t|%D]\}$}%td |!|$|%f<q|!j|fkstd|!j|f|r|!|dddf|!}&t|&|| dd}&|&|}&t j|&dd}'tj|'dd}(t|&d|(})tj|)ddd\})}*t|(d|*}(nF|!|dddf|!})|)|})tj|)ddddd\})}(|)|(kr|dfks ntg}+t|D]~},||,r|t||,ksFtddk rZ|dk sbtd|+d |d fgqg}-tt|(|,|)|,D]\}.\}/}0|/}1|/}2|,|1}3dk r|2 kr|.k}4|4rq||,!||3"|0 n|-#|0|2|3ft|-krq.q||,pT||,j$|)|,% |d||,<t|-ksptd|+|-t|+|,dkstqt&|rqpt|+|kst|'dd|+D}|'dd|+D}5|'dd|+D}6||6ddf}tj(||5)dgdd}|dk r6|*||6}|j j dkrftj(||+|jd dfgdd}|d}qpt|D]},||,rqxdk r t&fdd|(|,Dr t&|)|,df|||,ks td |)dddf|,|||,tD]6}1|,|1}3||3 }7||3}8||,!|8|7qqx|rX|n||}9|rjdn|}:|'|9};g}t|:D]4}?|:|$|?}@|>.d}At|A|;|@<|<#|Aqq|;/ |;% kr||dk s td$t/|;% d}B|'|9|B0|}Ct|<D]<\}$}D|D|C|$d|;|$f<|;|$kr<|C|$|;|$f<q.rFrTNr rccSsg|]}dqS)Frr0rrrr+srrrS)rrrrzShapes of scores: {} != {}rQ)rrmin_tokens_to_keeprT) descendingrU)rUlargestsortedz?Batch can only be done if at least {} beams have been generatedzMgenerated beams >= num_beams -> eos_token_id and pad_token have to be defined)rzBeam should always be fullcSsg|] }|dqS)rrr/r,rrrrscSsg|] }|dqS)r rr6rrrrscSsg|] }|dqS)rQrr6rrrrsc3s|]}|k VqdSr)r$)r/token_id)rrrrr1sz8PreTrainedModel._generate_beam_search..z\If batch_idx is not done, final next scores: {} have to equal to accumulated beam_scores: {}cSs|dSrLrr+rrrr-r.z7PreTrainedModel._generate_beam_search..)rz `Pad_token_id` has to be definedc3s|]}t|kVqdSrr)r/r.)rrrr1s)7rr[zerosrrr rrrrZrrrZ log_softmaxrr%rr`rkr_ expand_asrr rrgathersorttopkrrrrr$addrris_doner#allrr!re_reorder_cacherr5beamsrrrstacktyperrVrMr3)ErrsrrrrrrrrrrrrrrrrarrrrrrXrrZgenerated_hypsZ beam_scoresrdoner'rr(ZscoresZnum_batch_hypothesesZbanned_batch_tokensrr)Z_scoresr+Z next_tokensZ next_scoresZnext_scores_indicesZnext_batch_beamr*Znext_sent_beamZbeam_token_rankZ beam_token_idZbeam_token_scoreZbeam_idr7Zeffective_beam_idZ&is_beam_token_worse_than_top_num_beamsZ beam_tokensbeam_idxZ final_scoreZ final_tokensZoutput_batch_sizeZ%output_num_return_sequences_per_batchr&bestZ hypothesesZ sorted_hypsjZeffective_batch_idxZbest_hypZ sent_max_lenr,r.r)rrrrrrrr sj         .                     "z%PreTrainedModel._generate_beam_search)rrFr)cstfdd|DS)Nc3s|]}|dVqdS)r N)r )r/Z layer_pastrFrrr1sz1PreTrainedModel._reorder_cache..)rp)rrFrrIrrA szPreTrainedModel._reorder_cache)N)N)NNNNNNNNNNNNNNNNNNNN)&r"r#r$r%rrrxrortrryrzr|r~rrrrrrrr classmethodrrrrrr[Zno_gradrrr rnrrrAr&rrrrrqsv       " " %  yrq)prev_input_ids num_hyposrrr)c sdkrddt|DSddt|Dt|D]b}||}tfddtDD].}t|dd}||g|dg||<qjq8fdd fd dt|D}|S) z6Copied from fairseq for no_repeat_ngram in beam_searchr cSsg|]}gqSrrr0rrrrsz,calc_banned_ngram_tokens..cSsg|]}iqSrrr0rrrrscsg|]}|dqSrr)r/r) gen_tokensrrrsNrccs4d}t||f}||gSNr )rprr)r-Z start_idxZ ngram_idx)rgenerated_ngramsrrKrr_get_generated_ngramss z7calc_banned_ngram_tokens.._get_generated_ngramscsg|] }|qSrr)r/r-)rPrrr%s)rrrrpr) rKrLrridxZgenerated_ngramZngramZprev_ngram_tupler)r)rPrrMrOrrKrrs   rcszg}fdd}D]`}g}|D]H}t|dks>td||||dddkrZq ||dq ||q|S)NcsFt|dkrdSt|tkr$dS|t| d|kr>dSdSdS)NrTFr8)Z prev_tokenstokensrKrr _tokens_match,s z0calc_banned_bad_words_ids.._tokens_matchrz9Banned words token sequences {} cannot have an empty listrcF)rrkr_rr)rKrr)rTZprev_input_ids_sliceZbanned_tokens_sliceZbanned_token_seqrrSrr)s  rrSZInfc Cs|dkr>tt|||d}|t||ddk}|||<|dkrtj|dd\}}tjtj|dddd}||k} |dkrd| d d |f<| d d df | d dd f<d| d <| d|| }|||<|S) a Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (batch size, vocabulary size) if top_k > 0: keep only top k tokens with highest probability (top-k filtering). if top_p < 1.0: keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751) Make sure we keep at least min_tokens_to_keep per batch example in the output From: https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317 rrc).rcNrST)r3rr .N).r) rr#rr[r=r<cumsumrrrZscatter) rrrZ filter_valuer2Zindices_to_removeZ sorted_logitsZsorted_indicesZcumulative_probsZsorted_indices_to_removerrrrMs  rc@s.eZdZddZddZddZd dd ZdS) r/cCs,|d|_||_||_||_g|_d|_dS)z7 Initialize n-best list of hypotheses. r geAN)rrrrrB worst_score)rrrrrrrrrqs  zBeamHypotheses.__init__cCs t|jS)z3 Number of hypotheses in the list. )rrBrNrrr__len__|szBeamHypotheses.__len__cCs|t||j}t||jks*||jkr|j||ft||jkrtddt|jD}|j|dd=|dd|_nt||j|_dS)z3 Add a new hypothesis to the list. cSsg|]\}\}}||fqSrr)r/rQrr1rrrrs z&BeamHypotheses.add..rr N) rrrrVrBrr5r%r)rZhypZ sum_logprobsZscoreZ sorted_scoresrrrr>szBeamHypotheses.addNcCsJt||jkrdS|jrdS|dkr*|j}|||j}|j|k}|SdS)z If there are enough hypotheses and that none of the hypotheses being generated can become better than the worst one in the heap, then we are done with this sentence. FTN)rrrrrrV)rZbest_sum_logprobsrZ cur_scoreretrrrr?s zBeamHypotheses.is_done)N)r"r#r$rrWr>r?rrrrr/ps r/cs$eZdZfddZddZZS)Conv1DcsNt||_t||}tjj|ddt||_ tt ||_ dS)z Conv1D layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2) Basically works like a Linear layer but the weights are transposed g{Gz?)ZstdN) rrnfr[emptyrinitZnormal_rrr9r)rrZZnxwrrrrs    zConv1D.__init__c CsF|dd|jf}t|j|d|d|j}|j|}|S)Nrc)rrZr[Zaddmmrr r)rr,Zsize_outrrrr s  zConv1D.forward)r"r#r$rr r&rrrrrYs rYcs*eZdZdZfddZdddZZS)PoolerStartLogitsz9 Compute SQuAD start_logits from sequence hidden states. cstt|jd|_dSrN)rrrLinear hidden_sizedenserrZrrrrs zPoolerStartLogits.__init__NcCsZ||d}|dk rVt|jtjkrB|d|d|}n|d|d|}|S)z Args: **p_mask**: (`optional`) ``torch.FloatTensor`` of shape `(batch_size, seq_len)` invalid position mask such as query and special symbols (PAD, SEP, CLS) 1.0 means token should be masked. rcNr ꌠ9Y>)F)rar rMr3rr[float16)r hidden_statesp_maskr,rrrr s zPoolerStartLogits.forward)Nr!rrrrr^s r^cs*eZdZdZfddZdddZZS)PoolerEndLogitszX Compute SQuAD end_logits from sequence hidden states and start token hidden state. csRtt|jd|j|_t|_tj|j|j d|_t|jd|_ dS)NrQ)Zepsr ) rrrr_r`dense_0Tanh activation LayerNormZlayer_norm_epsdense_1rbrrrrs   zPoolerEndLogits.__init__NcCs|dk s|dk std|dk rh|jdd\}}|ddddfdd|}|d|}|d|d}|tj||gdd}||}||}| | d}|dk rt | j tjkr|d|d|}n|d|d|}|S) a Args: One of ``start_states``, ``start_positions`` should be not None. If both are set, ``start_positions`` overrides ``start_states``. **start_states**: ``torch.LongTensor`` of shape identical to hidden_states hidden states of the first tokens for the labeled span. **start_positions**: ``torch.LongTensor`` of shape ``(batch_size,)`` position of the first token for the labeled span: **p_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, seq_len)`` Mask of invalid position such as query and special symbols (PAD, SEP, CLS) 1.0 means token should be masked. N7One of start_states, start_positions should be not Nonercrr rcrd)rkr`rjr;rir[r!rkrlrmr rMr3rre)rrf start_statesstart_positionsrgslenhszr,rrrr s(   zPoolerEndLogits.forward)NNNr!rrrrrhs rhcs*eZdZdZfddZdddZZS)PoolerAnswerClasszT Compute SQuAD 2.0 answer class from classification and start tokens hidden states. csBtt|jd|j|_t|_tj|jddd|_dS)NrQr Fr) rrrr_r`rirjrkrmrbrrrrs  zPoolerAnswerClass.__init__NcCs|jd}|dk s"|dk s"td|dk rX|ddddfdd|}|d|d}|dk r|ddddfdd|}|d|d}n|dddddf}|tj||gdd}||}| |d}|S)a Args: One of ``start_states``, ``start_positions`` should be not None. If both are set, ``start_positions`` overrides ``start_states``. **start_states**: ``torch.LongTensor`` of shape identical to ``hidden_states``. hidden states of the first tokens for the labeled span. **start_positions**: ``torch.LongTensor`` of shape ``(batch_size,)`` position of the first token for the labeled span. **cls_index**: torch.LongTensor of shape ``(batch_size,)`` position of the CLS token. If None, take the last token. note(Original repo): no dependency on end_feature so that we can obtain one single `cls_logits` for each sample rcNrnror) r`rkrjr;r rir[r!rkrm)rrfrprq cls_indexrsZcls_token_stater,rrrr s$  zPoolerAnswerClass.forward)NNNr!rrrrrts rtcs*eZdZdZfddZdddZZS) SQuADHeada A SQuAD head inspired by XLNet. Parameters: config (:class:`~transformers.XLNetConfig`): Model configuration class with all the parameters of the model. Inputs: **hidden_states**: ``torch.FloatTensor`` of shape ``(batch_size, seq_len, hidden_size)`` hidden states of sequence tokens **start_positions**: ``torch.LongTensor`` of shape ``(batch_size,)`` position of the first token for the labeled span. **end_positions**: ``torch.LongTensor`` of shape ``(batch_size,)`` position of the last token for the labeled span. **cls_index**: torch.LongTensor of shape ``(batch_size,)`` position of the CLS token. If None, take the last token. **is_impossible**: ``torch.LongTensor`` of shape ``(batch_size,)`` Whether the question has a possible answer in the paragraph or not. **p_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, seq_len)`` Mask of invalid position such as query and special symbols (PAD, SEP, CLS) 1.0 means token should be masked. Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs: **loss**: (`optional`, returned if both ``start_positions`` and ``end_positions`` are provided) ``torch.FloatTensor`` of shape ``(1,)``: Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. **start_top_log_probs**: (`optional`, returned if ``start_positions`` or ``end_positions`` is not provided) ``torch.FloatTensor`` of shape ``(batch_size, config.start_n_top)`` Log probabilities for the top config.start_n_top start token possibilities (beam-search). **start_top_index**: (`optional`, returned if ``start_positions`` or ``end_positions`` is not provided) ``torch.LongTensor`` of shape ``(batch_size, config.start_n_top)`` Indices for the top config.start_n_top start token possibilities (beam-search). **end_top_log_probs**: (`optional`, returned if ``start_positions`` or ``end_positions`` is not provided) ``torch.FloatTensor`` of shape ``(batch_size, config.start_n_top * config.end_n_top)`` Log probabilities for the top ``config.start_n_top * config.end_n_top`` end token possibilities (beam-search). **end_top_index**: (`optional`, returned if ``start_positions`` or ``end_positions`` is not provided) ``torch.LongTensor`` of shape ``(batch_size, config.start_n_top * config.end_n_top)`` Indices for the top ``config.start_n_top * config.end_n_top`` end token possibilities (beam-search). **cls_logits**: (`optional`, returned if ``start_positions`` or ``end_positions`` is not provided) ``torch.FloatTensor`` of shape ``(batch_size,)`` Log probabilities for the ``is_impossible`` label of the answers. cs<t|j|_|j|_t||_t||_t||_ dSr) rr start_n_top end_n_topr^ start_logitsrh end_logitsrt answer_classrbrrrrTs    zSQuADHead.__init__NcCsd}|j||d}|dk r|dk r||||fD]"} | dk r.| dkr.| dq.|j|||d} t} | ||} | | |} | | d}|dk r|dk r|j|||d}t}|||}||d7}|f|}n|\}}}t j |dd }t j ||j dd \}}|ddd|}t |d |}|dd|dd}|d|}|dk rb|dnd}|j|||d } t j | dd }t j ||jdd \}}|d|j |j}|d|j |j}t d ||}|j|||d }|||||f|}|S)Nr)rgr rc)rqrgrQ)rqrvg?rro)rprgz blh,bl->bh)rprv)rzrUZsqueeze_r{rr|rZBCEWithLogitsLossrrrr[r=rxrerjr;r:ryr Zeinsum)rrfrqZ end_positionsrvZ is_impossiblergrrzr,r{Zloss_fctZ start_lossZend_lossZ total_lossZ cls_logitsZ loss_fct_clsZcls_lossZbszrrrsZstart_log_probsZstart_top_log_probsZstart_top_indexZstart_top_index_exprpZhidden_states_expandedZ end_log_probsZend_top_log_probsZ end_top_indexrrrr ]sX         zSQuADHead.forward)NNNNNr!rrrrrw+s( rwcs0eZdZdZedfdd ZdddZZS) SequenceSummarya- Compute a single vector summary of a sequence hidden states according to various possibilities: Args of the config class: summary_type: - 'last' => [default] take the last token hidden state (like XLNet) - 'first' => take the first token hidden state (like Bert) - 'mean' => take the mean of all tokens hidden states - 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2) - 'attn' => Not implemented now, use multi-head attention summary_use_proj: Add a projection after the vector extraction summary_proj_to_labels: If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False. summary_activation: 'tanh' or another string => add an activation to the output, Other => no activation. Default summary_first_dropout: Add a dropout before the projection and activation summary_last_dropout: Add a dropout after the projection and activation )rZcstt|dd|_|jdkr&tt|_t|drv|jrvt|dr`|j r`|j dkr`|j }n|j }t |j ||_t|dd}|rt|nt|_t|_t|dr|jdkrt |j|_t|_t|d r|jdkrt |j|_dS) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrZsummary_activationsummary_first_dropoutsummary_last_dropout)rrrwr~r{rsummaryrFrrZ num_labelsr`rr_r rk first_dropoutrZDropout last_dropoutr)rrZZ num_classesZactivation_stringrrrrs$   zSequenceSummary.__init__NcCs|jdkr|dddf}n|jdkr8|dddf}n|jdkrP|jdd}n|jd kr|dkrtj|d ddddf|jd dtjd }n2|dd}|d |d| df}| d | d }n|jdkrt | |}||}||}||}|S)a hidden_states: float Tensor in shape [bsz, ..., seq_len, hidden_size], the hidden-states of the last layer. cls_index: [optional] position of the classification token if summary_type == 'cls_index', shape (bsz,) or more generally (bsz, ...) where ... are optional leading dimensions of hidden_states. if summary_type == 'cls_index' and cls_index is None: we take the last token of the sequence as classification token rNrcfirstrmeanr rrv.rorR)rcr)r~rr[Z full_liker`rrerjrUrr;r r{rrrkr)rrfrvrrrrr s&    0"     zSequenceSummary.forward)N)r"r#r$r%r rr r&rrrrr}sr}cCs2||}tj|dd||}||S)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. :param torch.Tensor x: :return torch.Tensor: r r)rrmr[rUZtype_asr)rsZ padding_idxmaskZincremental_indicesrrr"create_position_ids_from_input_idss rcCs||jj}|j||}|jdk rX|dkrF|j}n|j|}t|j}t |||<t j |d|d|jdk d|jj}d|j_ |j |d|j_ |jdk rd|j_ |j |d|j_ |S)z Prune a linear layer (a model parameters) to keep only entries in index. Return the pruned layer as a new layer with requires_grad=True. Used to remove heads. Nr rruFT)rVrrr rdetachrrrrrr_r*copy_r rindexrUWbZnew_sizeZ new_layerrrrprune_linear_layers"  ( rcCs||jj}|j||}|dkr<|j}n|j|}t|j}t |||<t |d|d|jj}d|j_ |j | d|j_ d|j_ |j | d|j_ |S)a  Prune a Conv1D layer (a model parameters) to keep only entries in index. A Conv1D work as a Linear layer (see e.g. BERT) but the weights are transposed. Return the pruned layer as a new layer with requires_grad=True. Used to remove heads. rr FT)rVrrr rrrrrrrYr*rr rrrrprune_conv1d_layers rcCs^t|tjr&t|||dkrdn|dSt|trJt|||dkrBdn|dStd|jdS)z Prune a Conv1D or nn.Linear layer (a model parameters) to keep only entries in index. Return the pruned layer as a new layer with requires_grad=True. Used to remove heads. Nrrr zCan't prune layer of class {}) rurr_rrYrr^r_r)rrrUrrr prune_layer,s   r.) chunk_size chunk_dim forward_fnr)cst|dkstd||djtfdd|DsBtdttj}|t|ksrtd|t||dkr|dj|dkstd|d||dj|tfdd|D}tfd dt |D}t j |d S|S) a This function chunks the `input_tensors` into smaller input tensor parts of size `chunk_size` over the dimension `chunk_dim`. It then applies a layer `forward_fn` to each chunk independently to save memory. If the `forward_fn` is independent across the `chunk_dim` this function will yield the same result as not applying it. Args: chunk_size: int - the chunk size of a chunked tensor. `num_chunks` = `len(input_tensors[0]) / chunk_size` chunk_dim: int - the dimension over which the input_tensors should be chunked forward_fn: fn - the forward fn of the model input_tensors: tuple(torch.Tensor) - the input tensors of `forward_fn` which are chunked Returns: a Tensor with the same shape the foward_fn would have given if applied Examples:: # rename the usual forward() fn to forward_chunk() def forward_chunk(self, hidden_states): hidden_states = self.decoder(hidden_states) return hidden_states # implement a chunked forward function def forward(self, hidden_states): return apply_chunking_to_forward(self.chunk_size_lm_head, self.seq_len_dim, self.forward_chunk, hidden_states) rz${} has to be a tuple/list of tensorsc3s|]}|jkVqdSr)r`r/Z input_tensor) tensor_shaperrr1Ysz,apply_chunking_to_forward..z-All input tenors have to be of the same shapezJforward_chunk_fn expects {} arguments, but only {} input tensors are givenzHThe dimension to be chunked {} has to be a multiple of the chunk size {}c3s|]}|jdVqdS)rN)chunkr)r num_chunksrrr1osc3s|]}|VqdSrr)r/Zinput_tensors_chunk)rrrr1qsr) rrkr_r`r@inspect signaturer3rprr[r!)rrrZ input_tensorsZnum_args_in_forward_chunk_fnZinput_tensors_chunksZ output_chunksr)rrrrrapply_chunking_to_forward9s:    r)r)r )N)5rloggingr;typingrrr[rrrrZtorch.nnrr rZ activationsr Zconfiguration_utilsr Z file_utilsr rrrrrr getLoggerr"rrr9rr'rqrmrrrrobjectr/rYr^rhrtrwr}rrrrrrrrrs^    $  I$#2//uM