U &c@sdZddlZddlZddlZddlZddlZddlmmZ ddlmZddl m Z ddl m Z ddlmZmZmZmZddlmZmZeeZd d d d d dZddZGdddejZGdddejZGdddejZGdddejZGdddejZ GdddejZ!GdddejZ"Gdd d eZ#Gd!d"d"e#Z$d#Z%d$Z&ed%e%Gd&d'd'e#Z'ed(e%Gd)d*d*e#Z(dS)+z PyTorch T5 model. N)nn)CrossEntropyLoss)T5Config) DUMMY_INPUTS DUMMY_MASKadd_start_docstrings add_start_docstrings_to_callable)PreTrainedModelprune_linear_layerz5https://cdn.huggingface.co/t5-small-pytorch_model.binz4https://cdn.huggingface.co/t5-base-pytorch_model.binz5https://cdn.huggingface.co/t5-large-pytorch_model.binz2https://cdn.huggingface.co/t5-3b-pytorch_model.binz3https://cdn.huggingface.co/t5-11b-pytorch_model.bin)zt5-smallzt5-basezt5-largezt5-3bzt5-11bc Cszddl}ddl}ddl}Wn tk r<tdYnXtj|}t d ||j |}g}i} |D]:\} } t d | | |j || } || | | | <qr|D]} | d} tdd| Drt d d| | | dqd | d kr*t d d| | | dq|}| | } | D]}|d |rZ|d |}n|g}|ddkrzt|d}nFzt||d}Wn2tk rt d d| Yq:YnXt|dkr:t|d}||}q:|ddkrt|d}|ddkr*t d | j| || } z|j| jks>tWn<tk r|}z|j|j| jf7_W5d}~XYnXt d | t| |j|_ | | dqt d d| !|S)z- Load tf checkpoints in a pytorch model. rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}/css|]}|dkVqdS))Zadam_vZadam_mZAdamWeightDecayOptimizerZAdamWeightDecayOptimizer_1Z global_stepN).0nr r Qsz(load_tf_weights_in_t5..z Skipping {}Z_slot_z [A-Za-z]+_\d+z_(\d+))kernelZscale embeddingweightrrz+Transposing numpy weight of shape {} for {}zInitialize PyTorch weight {}z'Weights not copied to PyTorch model: {}z, )"reZnumpyZ tensorflow ImportErrorloggererrorospathabspathinfoformatZtrainZlist_variablesZ load_variableappendsplitanyjoinpop fullmatchgetattrAttributeErrorlenintshape transposeAssertionErrorargstorchZ from_numpyZastypeZfloat32datakeys)modelconfigZtf_checkpoint_pathrnptfZtf_pathZ init_varsnamesZ tf_weightsnamer*arrayZtxt_namepointerZm_nameZ scope_namesnumer r rload_tf_weights_in_t54s|                r;cs&eZdZdfdd ZddZZS) T5LayerNormư>cs&ttt||_||_dS)zg Construct a layernorm module in the T5 style No bias and no substraction of mean. N)super__init__r Parameterr.onesrvariance_epsilon)selfZ hidden_sizeeps __class__r rr?s zT5LayerNorm.__init__cCs2|djddd}|t||j}|j|S)NrrT)Zkeepdim)powmeanr.sqrtrBr)rCxZvariancer r rforwardszT5LayerNorm.forward)r=__name__ __module__ __qualname__r?rK __classcell__r r rErr<sr<cs$eZdZfddZddZZS)T5DenseReluDensecsHttj|j|jdd|_tj|j|jdd|_t|j |_ dSNFbias) r>r?rLineard_modeld_ffwiwoDropout dropout_ratedropoutrCr2rEr rr?s zT5DenseReluDense.__init__cCs,||}t|}||}||}|SN)rXFZrelur\rY)rC hidden_stateshr r rrKs     zT5DenseReluDense.forwardrLr r rErrQs rQcs$eZdZfddZddZZS) T5LayerFFcs8tt||_t|j|jd|_t |j |_ dS)NrD) r>r?rQDenseReluDenser<rVlayer_norm_epsilon layer_normrrZr[r\r]rEr rr?s  zT5LayerFF.__init__cCs&||}||}|||}|Sr^)rfrdr\)rCr`norm_xy layer_outputr r rrKs  zT5LayerFF.forwardrLr r rErrbs rbcsLeZdZdedfdd ZddZedd d Zd d ZdddZ Z S) T5AttentionFr2cst|j|_||_|j|_|j|_|j|_|j|_|j|_ |j |_ |j |j|_ t j|j|j dd|_t j|j|j dd|_t j|j|j dd|_t j|j |jdd|_|jrt |j|j |_t|_dSrR)r>r? is_decoderhas_relative_attention_biasoutput_attentionsrelative_attention_num_bucketsrVd_kv num_headsn_headsr[r\ inner_dimrrUqkvo Embeddingrelative_attention_biasset pruned_headsrCr2rmrEr rr?s" zT5Attention.__init__cst|dkrdSt|j|j}t||j}|D](tfdd|jD8d|<q2|d d}t t|| }t |j||_t |j||_t |j||_t |j|dd|_|jt||_|j|j|_|j||_dS)Nrc3s|]}|krdndVqdS)rrNr )rraheadr rrsz*T5Attention.prune_heads..rrdim)r(r.rArrrprzr{sumview contiguouseqarangelongr rtrurvrwrsunion)rCheadsmaskindexr r}r prune_headss   zT5Attention.prune_headsT c Csd}| }|r:|d}||dktj|7}t|}nt|t|}|d}||k}|t||t||||tj}t |t ||d}|t |||7}|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on. Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rrr) tor.rabsmaxZ zeros_likelogfloatmathminZ full_likewhere) relative_position bidirectional num_bucketsZ max_distanceretrZ max_exactZis_smallZ val_if_larger r r_relative_position_buckets" &z%T5Attention._relative_position_bucketcCstj|tjddddf}tj|tjddddf}||}|j||j |jd}||jjj }||}| dddg d}|S)z' Compute binned relative position bias )ZdtypeN)rrrrr) r.rrrrlrorryrdeviceZpermuteZ unsqueeze)rCqlenklenZcontext_positionZmemory_positionrZ rp_bucketvaluesr r r compute_biass zT5Attention.compute_biasNc sp|\} } |dk rfjdks(tdt|dksFtdt||dkr`| |djdn|} n| } |dkrx| } n |d} fdd } fd d }| |}|dkr| |}| |}n,|dkr|}}| |}| |}|dk rH|dkr@|\}}t j ||gdd }t j ||gdd }n|\}}jrf|dkrf||ff}nd }t d||}|dkrڈj st d| | }|dk r|ddddddddf}|dk r||}||7}tj|dd |}tj|jjd}|dk r ||}t ||}||}|}|f|}jrZ||f}j rl||f}|S)zd Self-attention (if kv is None) or attention over source sentence (provided by kv). NTz*Encoder cannot cache past key value statesrzSpast_key_value_state should have 2 past states: keys and values. Got {} past statesrrcs|djjddS)z projection rrr)rrrrpr+rJbsrCr rr*Hsz"T5Attention.forward..shapecs|dddjS)z compute context rrr)r+rrrsrrr runshapeLsz$T5Attention.forward..unshaperr^zbnqd,bnkd->bnqkzANo position_bias provided and no weights to compute position_biasr)ptraining)sizerlr,r(rr*rtrurvr.catZeinsumrm ValueErrorrr_ZsoftmaxrZtype_asr\rmatmulrwrn)rCinputrkv position_biaspast_key_value_state head_mask query_length use_cacherrZ real_qlenrr*rrtrurvZk_Zv_present_key_value_stateZscoresweightscontextoutputsr rrrK%sp                 zT5Attention.forward)F)Trr)NNNNNNF) rMrNrOrr?r staticmethodrrrKrPr r rErrjs /rjcs(eZdZdfdd ZdddZZS) T5LayerSelfAttentionFcs<tt||d|_t|j|jd|_t |j |_ dSNrmrc) r>r?rj SelfAttentionr<rVrerfrrZr[r\r|rEr rr?s zT5LayerSelfAttention.__init__Nc CsL||}|j||||||d}|d} ||| } | f|dd} | S)N)rrrrrrr)rfrr\) rCr`attention_maskrrrrrgattention_outputrhrirr r rrKs zT5LayerSelfAttention.forward)F)NNNNFrLr r rErrs rcs(eZdZdfdd ZdddZZS) T5LayerCrossAttentionFcs<tt||d|_t|j|jd|_t |j |_ dSr) r>r?rjEncDecAttentionr<rVrerfrrZr[r\r|rEr rr?s zT5LayerCrossAttention.__init__Nc  CsP||} |j| |||||||d} | d} ||| } | f| dd} | S)N)rrrrrrrrr)rfrr\)rCr`rrrrrrrrgrrhrirr r rrKs  zT5LayerCrossAttention.forward)F)NNNNFNrLr r rErrs rcs(eZdZdfdd ZdddZZS) T5BlockFcs^t|j|_t|_|jt||d|jrJ|jt||d|jt |dS)Nr) r>r?rlr ModuleListlayerr rrrbr|rEr rr?s  zT5Block.__init__Nc  CsP|dk rr|jstd|dkr"dnd} d| | dkr8dndt|} t|| ksXt| |dd} |dd} nd\} } |jd||||| | d }|dd\}}|dd}|jr*|dk r*|dk r|djd}nd}|jd |||||| || d }|d}|dk r||d }||dd}|jd |}|f}||f|}|S) Nz,Only decoder can use `past_key_value_states`rzcThere should be {} past states. 2 (past / key) for self attention.{} Got {} past key / value statesz"2 (past / key) for cross attention)NNr)rrrrrr)rrrrrrrr)rlr,rr(rr*)rCr`rrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrrrZ"expected_num_past_key_value_states error_messageZself_attn_past_key_value_stateZcross_attn_past_key_value_stateZself_attention_outputsrZattention_outputsrZcross_attention_outputsrr r rrKsX      zT5Block.forward)F)NNNNNNNFrLr r rErrs rc@s<eZdZdZeZeZeZ dZ e ddZ ddZ ddZd S) T5PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. Z transformercCs$tt}tt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r.Ztensorrr)rCrZ input_mask dummy_inputsr r rr.s  zT5PreTrainedModel.dummy_inputscCs|jj}t|tr(|jj|dnt|ttfrR|j jjj d|ddnXt|t r|j jjj d||jj ddt|j dr|j jdk r|j jj|jjjj d||jjddt|jdr|jjdk r|jjjnt|tr|jj }|jj}|jj}|jjjj d|||dd|jjjj d||dd|jjjj d||dd|jjjj d|||dd|jr|jjjj d||dddS)z Initialize the weights g?g)rHZstdrTN)r2Zinitializer_factor isinstancer<rr/Zfill_T5ModelT5ForConditionalGenerationsharedZnormal_rQrXrVhasattrrTZzero_rYrWrjrprqrtrurvrwrmry)rCmoduleZfactorrVrprrr r r _init_weights9s,       zT5PreTrainedModel._init_weightscCs|jj}|jj}|dk s td||j}|dddf|dddf<||d<|dk sdtd||dk|t |dk std |S) Nzself.model.config.decoder_start_token_id has to be defined. In T5 it is usually set to the pad_token_id. See T5 docs for more information.rr).rz1self.model.config.pad_token_id has to be defined.rz9Verify that `lm_labels` has only positive values and -100) r2decoder_start_token_id pad_token_idr,Z new_zerosr*cloneZ masked_fill_r.allitem)rCrrrZshifted_input_idsr r r _shift_rightYs  zT5PreTrainedModel._shift_rightN)rMrNrO__doc__rZ config_classT5_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapr;Zload_tf_weightsZbase_model_prefixpropertyrrrr r r rr$s  rcs@eZdZd fdd ZddZddZdd Zdd d ZZS)T5StackNcsvtj|_j|_||_j|_tfddtj D|_ t j j d|_tj|_|dS)Ncs g|]}tt|dkdqS)rr)rbool)rirkr r ysz$T5Stack.__init__..rc)r>r?rnoutput_hidden_states embed_tokensrlrrrange num_layersblockr<rVrefinal_layer_normrZr[r\ init_weights)rCr2rrErkrr?ps zT5Stack.__init__cCs|jSr^rrCr r rget_input_embeddingsszT5Stack.get_input_embeddingscCs|jSr^rrr r rget_output_embeddingsszT5Stack.get_output_embeddingscCs ||_dSr^rrCZnew_embeddingsr r rset_input_embeddingsszT5Stack.set_input_embeddingsFc  Cs|dk r|dk rtdnT|dk r<|} |d| d}n2|dk rV|dd} n|jrftdntd|dkr|jdk std||}| \} } |dk r| dkstd| | df|ddjd | } n| } |dkrt | |  |j }|jr0|dkr0|dk r0|jd} t | |  |j }|dkrJdgt |j }||| |j }|jrx|dk rx||}nd}|||jj}d }d }d }d}d}||}tt|j |D]\}\}}|jr||f}|||||||||||d }|dd \}}|dkrJ||jrd nd }|jrJ|dk rJ||jrDd nd }||f}|jr||d f}q||}||}|jr||f}|f}|dkr|jstd|||f}|jr||f}|jr||f}|S)NzDYou cannot specify both input_ids and inputs_embeds at the same timerzEYou have to specify either decoder_input_ids or decoder_inputs_embedsz5You have to specify either input_ids or inputs_embedsz;You have to intialize the model with valid token embeddingsrzCInput shape is {}, but should be {} when using past_key_value_satesrrr )rrrrrrrrrTz@`use_cache` can only be set to `True` if {} is used as a decoder)rrrrlrr,rr*r.rArrr(rZget_extended_attention_maskZinvert_attention_maskZ get_head_maskr2rr\ enumerateziprrnr)rCrrrr inputs_embedsrpast_key_value_statesrZ input_shapeZ batch_sizeZ seq_lengthZmask_seq_lengthZencoder_seq_lengthZextended_attention_maskZencoder_extended_attention_maskZpresent_key_value_statesZall_hidden_statesZall_attentionsrrr`rZ layer_modulerZ layer_outputsrrr r rrKs                   zT5Stack.forward)N)NNNNNNNF) rMrNrOr?rrrrKrPr r rErrosraf The T5 model was proposed in `Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer`_ by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. It's an encoder decoder transformer pre-trained in a text-to-text denoising generative setting. This model is a PyTorch `torch.nn.Module`_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. .. _`Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer`: https://arxiv.org/abs/1910.10683 .. _`torch.nn.Module`: https://pytorch.org/docs/stable/nn.html#module Parameters: config (:class:`~transformers.T5Config`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a] Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using :class:`transformers.T5Tokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details. To know more on how to prepare :obj:`input_ids` for pre-training take a look at `T5 Training <./t5.html#training>`_ . attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. encoder_outputs (:obj:`tuple(tuple(torch.FloatTensor)`, `optional`, defaults to :obj:`None`): Tuple consists of (`last_hidden_state`, `optional`: `hidden_states`, `optional`: `attentions`) `last_hidden_state` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. decoder_input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`, defaults to :obj:`None`): Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation. If `decoder_past_key_value_states` is used, optionally only the last `decoder_input_ids` have to be input (see `decoder_past_key_value_states`). To know more on how to prepare :obj:`decoder_input_ids` for pre-training take a look at `T5 Training <./t5.html#training>`_ . decoder_attention_mask (:obj:`torch.BoolTensor` of shape :obj:`(batch_size, tgt_seq_len)`, `optional`, defaults to :obj:`None`): Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default. decoder_past_key_value_states (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up decoding. If `decoder_past_key_value_states` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)` instead of all `decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`, defaults to :obj:`True`): If `use_cache` is True, `decoder_past_key_value_states` are returned and can be used to speed up decoding (see `decoder_past_key_value_states`). inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, target_sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`decoder_input_ids` you can choose to directly pass an embedded representation. If `decoder_past_key_value_states` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `decoder_past_key_value_states`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. head_mask: (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**. z[The bare T5 Model transformer outputting raw hidden-stateswithout any specific head on top.c sVeZdZfddZddZddZddZd d Zd d Ze e dddZ Z S)rcs`t|t|j|j|_t|}t ||j|_ t|}d|_ t ||j|_ | dS)NT)r>r?rrx vocab_sizerVrcopydeepcopyrencoderrldecoderrrCr2Zencoder_configZdecoder_configrEr rr?Hs   zT5Model.__init__cCs|jSr^rrr r rrUszT5Model.get_input_embeddingscCs"||_|j||j|dSr^rrrrrr r rrXs zT5Model.set_input_embeddingscCs|jSr^rrr r r get_encoder]szT5Model.get_encodercCs|jSr^rrr r r get_decoder`szT5Model.get_decodercCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrZ attentionr)rCZheads_to_prunerrr r r _prune_headscszT5Model._prune_headsNTc  Cs|dkr|j|||| d}|d} |dk rb|dk rF|ddddf}| dk rb| ddddf} |j||| || || |d} |dkr|| dff} | dd| | dd} | |S) a Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs. last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `decoder_past_key_value_states` is used only the last hidden-state of the sequences of shape :obj:`(batch_size, 1, hidden_size)` is output. decoder_past_key_value_states (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length, embed_size_per_head)`, `optional`, returned when ``use_cache=True``): Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up sequential decoding (see `decoder_past_key_value_states` input). Note that when using `decoder_past_key_value_states`, the model only outputs the last `hidden-state` of the sequence of shape :obj:`(batch_size, 1, config.vocab_size)`. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import T5Tokenizer, T5Model tokenizer = T5Tokenizer.from_pretrained('t5-small') model = T5Model.from_pretrained('t5-small') input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt") # Batch size 1 outputs = model(input_ids=input_ids, decoder_input_ids=input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NrrrrrrrrrrrrrrTrr)rr)rCrrencoder_outputsrrdecoder_past_key_value_statesrrdecoder_inputs_embedsrr`decoder_outputspastr r rrKks61 zT5Model.forward) NNNNNNTNNN) rMrNrOr?rrrrrr T5_INPUTS_DOCSTRINGrKrPr r rErrCs$ rz1T5 Model with a `language modeling` head on top. c sfeZdZfddZddZddZddZd d Zd d Ze e dddZ ddZ ddZ ZS)rcs~t||j|_t|j|j|_t |}t ||j|_ t |}d|_ t ||j|_ tj|j|jdd|_|dS)NTFrS)r>r?rV model_dimrrxrrrrrrrlrrUlm_headrrrEr rr?s   z#T5ForConditionalGeneration.__init__cCs|jSr^rrr r rrsz/T5ForConditionalGeneration.get_input_embeddingscCs"||_|j||j|dSr^rrr r rrs z/T5ForConditionalGeneration.set_input_embeddingscCs|jSr^)r rr r rrsz0T5ForConditionalGeneration.get_output_embeddingscCs|jSr^rrr r rrsz&T5ForConditionalGeneration.get_encodercCs|jSr^rrr r rrsz&T5ForConditionalGeneration.get_decoderNTc  CsV|dkr|j||| | d}|d} |dk rD|dkrD| dkrD||}|dk r|dks\td|dk rx|ddddf}| dk r| ddddf} |j||| || || |d} |dkr|| dff}| dd|| d d} | d}||jd }||}|f| dd} |dk rNtd d }||d|d|d}|f| } | |S) a lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[-100, 0, ..., config.vocab_size - 1]`. All labels set to ``-100`` are ignored (masked), the loss is only computed for labels in ``[0, ..., config.vocab_size]`` Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs. loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided): Classification loss (cross entropy). prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). If `past_key_value_states` is used only the last prediction_scores of the sequences of shape :obj:`(batch_size, 1, hidden_size)` is output. decoder_past_key_value_states (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length, embed_size_per_head)`, `optional`, returned when ``use_cache=True``): Contains pre-computed key and value hidden-states of the attention blocks. Can be used to speed up sequential decoding (see `decoder_past_key_value_states` input). Note that when using `decoder_past_key_value_states`, the model only outputs the last `prediction_score` of the sequence of shape :obj:`(batch_size, 1, config.vocab_size)`. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention. Examples:: from transformers import T5Tokenizer, T5ForConditionalGeneration tokenizer = T5Tokenizer.from_pretrained('t5-small') model = T5ForConditionalGeneration.from_pretrained('t5-small') input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt") # Batch size 1 outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, lm_labels=input_ids) loss, prediction_scores = outputs[:2] tokenizer = T5Tokenizer.from_pretrained('t5-small') model = T5ForConditionalGeneration.from_pretrained('t5-small') input_ids = tokenizer.encode("summarize: Hello, my dog is cute", return_tensors="pt") # Batch size 1 outputs = model.generate(input_ids) Nrrz=Decoder should not use cached key value states when training.rrTrrrr)Z ignore_index) rrr,rr r rrr)rCrrrrrrrZ lm_labelsrrrr`rrZsequence_outputZ lm_logitsZloss_fctZlossr r rrKsL;      z"T5ForConditionalGeneration.forwardcKsJ|dk stdt|dkr(|d}}n|d|d}}|||||dS)Nz*past has to be defined for encoder_outputsrrr)rrrrr)r,r()rCrrrrkwargsrrr r rprepare_inputs_for_generationSs  z8T5ForConditionalGeneration.prepare_inputs_for_generationcCst|dkrtd|S|d}|df}d}|D]Z}d}|D]}||d|f}q@|dj|djksptt|t|kst||f}q4||fS)NrzHYou might want to consider setting `use_cache=True` to speed up decodingrrr )r(rwarningZ index_selectr*r,)rCrZbeam_idxZ decoder_pastZreordered_decoder_pastZlayer_past_statesZreordered_layer_past_statesZlayer_past_stater r r_reorder_cacheds      z)T5ForConditionalGeneration._reorder_cache) NNNNNNTNNNN)rMrNrOr?rrrrrr r rKr rrPr r rErrs*  pr))rrloggingrrr.Ztorch.nn.functionalrZ functionalr_Ztorch.nnrZconfiguration_t5rZ file_utilsrrrr Zmodeling_utilsr r getLoggerrMrrr;Moduler<rQrbrjrrrrrZT5_START_DOCSTRINGr rrr r r rsN     ZS#VK1w