U &ºcó7ã@sÊdZddlZddlZddlmZddlmZmZddlmZddl m Z m Z ddl m Z dd lmZmZdd lmZmZmZmZmZe e¡Zd d d œZd1dd„Zdd„Ze ejjje dœZejZ Gdd„dej!ƒZ"Gdd„dej!ƒZ#Gdd„dej!ƒZ$Gdd„deƒZ%dZ&dZ'ede&ƒGdd„de%ƒƒZ(ed e&ƒGd!d"„d"e%ƒƒZ)ed#e&ƒGd$d%„d%e%ƒƒZ*ed&e&ƒGd'd(„d(e%ƒƒZ+ed)e&ƒGd*d+„d+e%ƒƒZ,ed,e&ƒGd-d.„d.e%ƒƒZ-ed,e&ƒGd/d0„d0e%ƒƒZ.dS)2z PyTorch XLNet model. éN)Únn)ÚCrossEntropyLossÚMSELoss)Ú functionalé)Úgelu_newÚswish)Ú XLNetConfig)Úadd_start_docstringsÚ add_start_docstrings_to_callable)ÚPoolerAnswerClassÚPoolerEndLogitsÚPoolerStartLogitsÚPreTrainedModelÚSequenceSummaryz=https://cdn.huggingface.co/xlnet-base-cased-pytorch_model.binz>https://cdn.huggingface.co/xlnet-large-cased-pytorch_model.bin)zxlnet-base-casedzxlnet-large-casedc Cs$i}t|dƒr¤t|dƒr$|jj|d<t|dƒrRd|krR|jjj|d<|jjj|d<t|dƒrž|jdk ržd  |j¡|krž|jj|d  |j¡<|jj|d  |j¡<|j }|  |j j|j d œ¡t |jƒD]Æ\}}d |}|  |d |jjj|d|jjj|d|jj|d|jj|d|jj|d|jj|d|jj|d|jjj|d|jjj|d|jjj|d|jjj|d|jjj|d|jjji ¡qÄ|jrìg}g}g} g} |jD]>}| |jj¡| |jj¡|  |jj¡|  |jj¡qªn |jg}|jg}|jg} |jg} |  ||| | dœ¡|S)z˜ A map of modules from TF to PyTorch. I use a map to keep the PyTorch model as identical to the original PyTorch model as possible. Ú transformerÚlm_losszmodel/lm_loss/biasÚsequence_summaryz%model/sequnece_summary/summary/kernelz#model/sequnece_summary/summary/biasÚ logits_projNz model/regression_{}/logit/kernelzmodel/regression_{}/logit/bias)z-model/transformer/word_embedding/lookup_tablez#model/transformer/mask_emb/mask_embzmodel/transformer/layer_%d/zrel_attn/LayerNorm/gammazrel_attn/LayerNorm/betazrel_attn/o/kernelzrel_attn/q/kernelzrel_attn/k/kernelzrel_attn/r/kernelzrel_attn/v/kernelzff/LayerNorm/gammazff/LayerNorm/betazff/layer_1/kernelzff/layer_1/biaszff/layer_2/kernelzff/layer_2/bias)zmodel/transformer/r_r_biaszmodel/transformer/r_w_biaszmodel/transformer/r_s_biaszmodel/transformer/seg_embed)ÚhasattrrÚbiasrÚsummaryÚweightZfinetuning_taskÚformatrrÚupdateÚword_embeddingÚmask_embÚ enumerateÚlayerÚrel_attnÚ layer_normÚoÚqÚkÚrÚvÚffÚlayer_1Úlayer_2Zuntie_rÚappendÚr_r_biasÚr_w_biasÚr_s_biasÚ seg_embed) ÚmodelÚconfigÚ tf_weightsÚ tf_to_pt_mapÚiÚbZ layer_strZr_r_listZr_w_listZr_s_listZseg_embed_list©r4ú?/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_xlnet.pyÚbuild_tf_xlnet_to_pytorch_map)s”   ÿþýþÿóÿ üÿr6c Csxzddl}ddl}Wn tk r4t d¡‚YnX|j |¡}i}|D]0\}}t d ||¡¡|j  ||¡} | ||<qJt |||ƒ} |   ¡D]Æ\}} t d |¡¡||krÄt d |¡¡q||} d|krd|ksðd |ksðd |krt d ¡|  | ¡} t | tƒr¼t| ƒ| jdks(t‚t| ƒD]ˆ\} } | | d f}z| j|jksXt‚Wn<tk r–}z|j| j|jf7_‚W5d}~XYnXt d  || ¡¡t |¡| _q0npz| j| jksÐt‚Wn<tk r}z|j| j| jf7_‚W5d}~XYnXt d |¡¡t | ¡| _| |d¡| |dd¡| |dd¡qt d d | ¡¡¡¡|S)z, Load tf checkpoints in a pytorch model rNzšLoading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z"Loading TF weight {} with shape {}z Importing {}z*{} not in tf pre-trained weights, skippingÚkernelr&rZlogitZ Transposing.z)Initialize PyTorch weight {} for layer {}zInitialize PyTorch weight {}z/Adamz/Adam_1z'Weights not copied to PyTorch model: {}z, )ZnumpyZ tensorflowÚ ImportErrorÚloggerÚerrorZtrainZlist_variablesÚinforZ load_variabler6ÚitemsÚ transposeÚ isinstanceÚlistÚlenÚshapeÚAssertionErrorrÚargsÚtorchZ from_numpyÚdataÚpopÚjoinÚkeys)r.r/Ztf_pathÚnpÚtfZ init_varsr0ÚnamerAÚarrayr1Úpointerr2Zp_iZarr_iÚer4r4r5Úload_tf_weights_in_xlnet|s^ ÿ    $      rO)ZgeluÚrelurcs^eZdZ‡fdd„Zdd„Zeddd„ƒZeddd „ƒZdd d „Zddd„Z ddd„Z ‡Z S)ÚXLNetRelativeAttentioncsptƒ ¡|j|_|j|jdkr6td|j|jfƒ‚|j|_|j|_|j|_d|jd|_t  t   |j|j|j¡¡|_ t  t   |j|j|j¡¡|_ t  t   |j|j|j¡¡|_t  t   |j|j|j¡¡|_t  t   |j|j|j¡¡|_t  t   |j|j¡¡|_t  t   |j|j¡¡|_t  t   |j|j¡¡|_t  t   d|j|j¡¡|_t|j|jd|_t |j¡|_dS)NrzLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rçà?é©Zeps)ÚsuperÚ__init__Úoutput_attentionsÚd_modelZn_headÚ ValueErrorZd_headÚscalerÚ ParameterrDÚ FloatTensorr"r#r%r!r$r*r,r+r-ÚXLNetLayerNormÚlayer_norm_epsr ÚDropoutÚdropout©Úselfr/©Ú __class__r4r5rVÁs.  ÿÿzXLNetRelativeAttention.__init__cCst‚dS©N©ÚNotImplementedError)rbZheadsr4r4r5Ú prune_headsÞsz"XLNetRelativeAttention.prune_headséÿÿÿÿc Cs||j}| |d|d|d|d¡}|dd…df}| |d|dd|d|d¡}t |dtj||jtjd¡}|S)zbnijrj)rsNrzibnd,snd->ibnszijbs,ibns->bnijiÜÿz ijbn->bnijgꌠ9Y>)F©Údimzbnij,jbnd->ibndz bnij->ijbn)rDÚeinsumr+r*rurAr,r-rZrmZfloat16ÚFÚsoftmaxr`rW)rbZq_headÚk_head_hÚv_head_hÚk_head_rÚseg_matÚ attn_maskÚ head_maskÚacZbdZefZ attn_scoreÚ attn_probÚattn_vecr4r4r5Ú rel_attn_coreýs(  z$XLNetRelativeAttention.rel_attn_coreTcCs4t d||j¡}| |¡}|r&||}| |¡}|S)zPost-attention processing.z ibnd,hnd->ibh)rDrxr!r`r )rbÚhrƒZresidualZattn_outÚoutputr4r4r5Úpost_attention's   z%XLNetRelativeAttention.post_attentionc  Cs|dk rL|dk r2| ¡dkr2tj||gdd} n|} t d| |j¡} t d| |j¡} t d||j¡} t d||j¡}|j|| | | ||| d}|j rœ|\}}|  ||¡}t d||j¡}|dk rt d||¡}|j|| | | ||| d}|j rö|\}}t d||¡}n(|j|| | | ||| d}|j r.|\}}|  ||¡}|j rô||f}n¨|dk rx| ¡dkrxtj||gdd} n|} t d||j¡}t d| |j¡} t d| |j¡} t d||j¡} |j|| | | ||| d}|j rä|\}}|  ||¡}d}||f}|j r||f}|S)Nrrrvz ibh,hnd->ibnd)r~rr€zmbnd,mlb->lbndzlbnd,mlb->mbnd) rwrDÚcatrxr#r%r$r"r„rWr‡)rbr…ÚgÚ attn_mask_hÚ attn_mask_gr$r~ÚmemsÚtarget_mappingr€rˆr{r|r}Zq_head_hZ attn_vec_hZ attn_prob_hÚoutput_hZq_head_gZ attn_vec_gZ attn_prob_gÚoutput_gr‚rƒÚoutputsr4r4r5Úforward3s’ ÿ  ÿÿ  ÿ  zXLNetRelativeAttention.forward)ri)ri)NNN)T)NNN) Ú__name__Ú __module__Ú __qualname__rVrhÚ staticmethodrtrur„r‡r‘Ú __classcell__r4r4rcr5rQÀs    * rQcs$eZdZ‡fdd„Zdd„Z‡ZS)ÚXLNetFeedForwardcsttƒ ¡t|j|jd|_t |j|j¡|_ t |j|j¡|_ t  |j ¡|_ t |jtƒrht|j|_n|j|_dS)NrT)rUrVr]rXr^r rÚLinearZd_innerr'r(r_r`r>Z ff_activationÚstrÚACT2FNÚactivation_functionrarcr4r5rV“s  zXLNetFeedForward.__init__cCsH|}| |¡}| |¡}| |¡}| |¡}| |¡}| ||¡}|Sre)r'r›r`r(r )rbÚinpr†r4r4r5r‘žs     zXLNetFeedForward.forward©r’r“r”rVr‘r–r4r4rcr5r—’s r—cs&eZdZ‡fdd„Zddd„Z‡ZS)Ú XLNetLayercs0tƒ ¡t|ƒ|_t|ƒ|_t |j¡|_dSre) rUrVrQrr—r&rr_r`rarcr4r5rVªs   zXLNetLayer.__init__Nc Cs`|j||||||||| d } | dd…\}}|dk r>| |¡}| |¡}||f| dd…} | S)N)rŒrr€rS)rr&) rbrŽrrŠr‹r$r~rŒrr€rr4r4r5r‘°s"÷   zXLNetLayer.forward)NNNrr4r4rcr5rž©s  ÿržc@s(eZdZdZeZeZeZ dZ dd„Z dS)ÚXLNetPreTrainedModelz† An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rc Csät|tjtjfƒrL|jjjd|jjdt|tjƒrà|j dk rà|j j  ¡n”t|t ƒrr|j j  ¡|jj  d¡nnt|t ƒrÀ|j|j|j|j|j|j|j|j|jf D]}|jjd|jjdq¤n t|tƒrà|jjjd|jjddS)z! Initialize the weights. g)ZmeanZstdNçð?)r>rr˜Ú EmbeddingrrEZnormal_r/Zinitializer_rangerZzero_r]Zfill_rQr"r#r%r!r$r*r,r+r-Ú XLNetModelr)rbÚmoduleÚparamr4r4r5Ú _init_weightsÒs*   ÷  z"XLNetPreTrainedModel._init_weightsN) r’r“r”Ú__doc__r Z config_classÚ"XLNET_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_maprOZload_tf_weightsZbase_model_prefixr¥r4r4r4r5rŸÈs rŸao This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.XLNetConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a' Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `mems` output below). Can be used to speed up sequential decoding. The token ids which have their mems given to this model should not be passed as input ids as they have already been computed. `use_cache` has to be set to `True` to make use of `mems`. perm_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to indicate the attention pattern for each input token with values selected in ``[0, 1]``: If ``perm_mask[k, i, j] = 0``, i attend to j in batch k; if ``perm_mask[k, i, j] = 1``, i does not attend to j in batch k. If None, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_predict, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to indicate the output tokens to use. If ``target_mapping[k, i, j] = 1``, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token. The classifier token should be represented by a ``2``. `What are token type IDs? <../glossary.html#token-type-ids>`_ input_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding. Kept for compatibility with the original code base. You can only uses one of `input_mask` and `attention_mask` Mask values selected in ``[0, 1]``: ``1`` for tokens that are MASKED, ``0`` for tokens that are NOT MASKED. head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (:obj:`bool`): If `use_cache` is True, `mems` are returned and can be used to speed up decoding (see `mems`). Defaults to `True`. z_The bare XLNet Model transformer outputting raw hidden-states without any specific head on top.c sneZdZ‡fdd„Zdd„Zdd„Zdd„Zd d „Zd d „Ze ddd„ƒZ ddd„Z e e ƒddd„ƒZ‡ZS)r¢csÀtƒ ˆ¡ˆj|_ˆj|_ˆj|_ˆj|_ˆj|_ˆj|_ˆj|_ˆj |_ ˆj |_ ˆj |_ t   ˆjˆj¡|_t  t ddˆj¡¡|_t  ‡fdd„tˆj ƒDƒ¡|_t  ˆj¡|_| ¡dS)Nrcsg|] }tˆƒ‘qSr4)rž)Ú.0Ú_©r/r4r5Ú Isz'XLNetModel.__init__..)rUrVrWÚoutput_hidden_statesÚmem_lenÚ reuse_lenrXÚ same_lengthÚ attn_typeÚbi_dataÚ clamp_lenÚn_layerrr¡Ú vocab_sizerr[rDr\rZ ModuleListÚrangerr_r`Ú init_weightsrarcrªr5rV9s   zXLNetModel.__init__cCs|jSre©r©rbr4r4r5Úget_input_embeddingsNszXLNetModel.get_input_embeddingscCs ||_dSrer·)rbZnew_embeddingsr4r4r5Úset_input_embeddingsQszXLNetModel.set_input_embeddingscCst‚dSrerf)rbZheads_to_pruner4r4r5Ú _prune_headsTszXLNetModel._prune_headscCsœt ||g¡}tj|dd}t ||g¡}tj||gdd}|jr†tj|dd}tj|dd…d|…f||dd…|d…fgdd}| t|  ¡ƒ¡}|S)aw Creates causal attention mask. Float mask where 1.0 indicates masked, 0.0 indicates not-masked. Args: qlen: Sequence length mlen: Mask length :: same_length=False: same_length=True: < qlen > < qlen > ^ [0 0 0 0 0 1 1 1 1] [0 0 0 0 0 1 1 1 1] [0 0 0 0 0 0 1 1 1] [1 0 0 0 0 0 1 1 1] qlen [0 0 0 0 0 0 0 1 1] [1 1 0 0 0 0 0 1 1] [0 0 0 0 0 0 0 0 1] [1 1 1 0 0 0 0 0 1] v [0 0 0 0 0 0 0 0 0] [1 1 1 1 0 0 0 0 0] r)ZdiagonalrvriN) rDZonesZtriuÚzerosrˆr¯ZtrilÚtoÚnextÚ parameters)rbÚqlenÚmlenrZmask_upZ attn_mask_padÚretZmask_lor4r4r5Ú create_maskWs6zXLNetModel.create_maskcCsb|jdk r"|jdkr"|d|j…}|dkr<||j d…}ntj||gdd|j d…}| ¡S)Nrrv)r®r­rDrˆÚdetach)rbZcurr_outZprev_memZnew_memr4r4r5Ú cache_memus zXLNetModel.cache_memNcCs\t d||¡}tjt |¡t |¡gdd}|dd…ddd…f}|dk rX| d|d¡}|S)Nzi,d->idrirv)rDrxrˆÚsinÚcosÚexpand)Zpos_seqÚinv_freqÚbszZ sinusoid_inpÚpos_embr4r4r5Úpositional_embeddings zXLNetModel.positional_embeddingc Csvtjd|jdtjd}dt d||j¡}|jdkrD|| }}n&|jdkrZ|d}}ntd  |j¡ƒ‚|jr&tj||d tjd}tj| | d tjd} |j dkrÌ|  |j |j ¡}|   |j |j ¡} |dk rú|  |||d ¡} |  | ||d ¡} n|  ||¡} |  | |¡} tj | | gdd } n:t ||d ¡}|j dkrR|  |j |j ¡}|  |||¡} |   t| ¡ƒ¡} | S)Nrg@©rmri'ÚbiÚunirizUnknown `attn_type` {}.gð¿r rSrv)rDrorXÚfloatÚpowr°rYrr±r²ÚclamprÌrˆr½r¾r¿) rbrÀrsrÊZfreq_seqrÉÚbegÚendZ fwd_pos_seqZ bwd_pos_seqZ fwd_pos_embZ bwd_pos_embrËr4r4r5Úrelative_positional_encodingŒs2       z'XLNetModel.relative_positional_encodingTc $ CsÆ|dk r| dk rtdƒ‚nh|dk rJ| dd¡ ¡}|jd|jd} } n8| dk rz|  dd¡ ¡} | jd| jd} } ntdƒ‚|dk rš| dd¡ ¡nd}|dk r¶| dd¡ ¡nd}|dk rÒ| dd¡ ¡nd}|dk rð| ddd¡ ¡nd}|dk r| ddd¡ ¡nd}|dk r:|ddk r:|djdnd} | | }t| ¡ƒj}t| ¡ƒj}|j dkr”|  | | ¡}|dd…dd…ddf}n"|j dkr¦d}ntd   |j ¡ƒ‚|dksÒ|dksÒt d ƒ‚|dkrî|dk rîd |}|dk r|dk r|d|}n<|dk r.|dkr.|d}n|dkrH|dk rH|}nd}|dk rÖ| dkrŽt  |jd| | g¡ |¡}t j||gdd }|dkr¶|dd…dd…dd…df}n ||dd…dd…dd…df7}|dk rî|dk |¡}|dk r^t  | ¡ |¡ }| dkr6t jt  | | g¡ |¡|gd d }||dd…dd…ddfdk |¡}nd}| dk rr| }n | |¡}| |¡}|dk r²|j |jd| d ¡}| |¡}nd}|dk r2| dkrôt j| | gt j|d}t j||gdd }n|}|dd…df|ddd…fk ¡}tj|dd |¡}nd}|j| || d}| |¡}|dk rÖ| ¡dkrš| d¡ d¡ d¡ d¡}| |jd d d d ¡}n$| ¡dkr¾| d¡ d¡ d¡}|jt| ¡ƒjd}n dg|j}d}|dkrdgt|jƒ}g}g}t|jƒD]®\} }!|j dk rR|j dkrR| dkrR|| !||| ¡f}|j"rv| #|dk rp||fn|¡|!|||||||| ||| d }"|"dd…\}}|j$r| #|"d¡q|j"ræ| #|dk rà||fn|¡| |dk rø|n|¡}#|# ddd¡ ¡f}"|j dk r>|j dkr>| dkr>|"|f}"|j"r€|dk rdt%dd„|Dƒƒ}nt%dd„|Dƒƒ}|"|f}"|j$rÂ|dk r¦t%dd„|Dƒƒ}nt%dd„|Dƒƒ}|"|f}"|"S)a« Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_predict, hidden_size)`): Sequence of hidden-states at the last layer of the model. `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLNetTokenizer, XLNetModel import torch tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') model = XLNetModel.from_pretrained('xlnet-large-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=False)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NzDYou cannot specify both input_ids and inputs_embeds at the same timerrz5You have to specify either input_ids or inputs_embedsrSrÏrÎzUnsupported attention type: {}z8You can only use one of input_mask (uses 1 for padding) r rvri©rmrl)Z num_classes)rÊrÍr4T)rŠr‹r$r~rŒrr€css*|]"}|D]}| ddd¡ ¡Vq qdS©rrrSN©ÚpermuteÚ contiguous)r¨Úhsr…r4r4r5Ú ‰sz%XLNetModel.forward..css |]}| ddd¡ ¡VqdSr×rØ)r¨rÛr4r4r5rÜ‹scss |]}tdd„|DƒƒVqdS)css"|]}| dddd¡ ¡VqdS©rSrjrrNrØ)r¨Z att_streamr4r4r5rÜ‘sz/XLNetModel.forward...N)Útuple©r¨Útr4r4r5rÜscss"|]}| dddd¡ ¡VqdSrÝrØrßr4r4r5rÜ”s)&rYr=rÚrArÙr¾r¿rmrlr°rÃrrBrDr¼r½rˆZeyerr`rrÈrpryZone_hotrÕrwÚ unsqueezer³r@rrr­rÅr¬r)rWrÞ)$rbÚ input_idsÚattention_maskrŒÚ perm_maskrÚtoken_type_idsÚ input_maskr€Ú inputs_embedsÚ use_cacherÀrÊrÁrsZ dtype_floatrlrZ data_maskZ mems_maskZ non_tgt_maskZ word_emb_krŽZ word_emb_qrZmem_padZcat_idsr~rËZnew_memsZ attentionsÚ hidden_statesr2Z layer_modulerr†r4r4r5r‘³sð5  *           "(       $   ÿ  "÷ "    ÿ  zXLNetModel.forward)N)N) NNNNNNNNNT)r’r“r”rVr¹rºr»rÃrÅr•rÌrÕr ÚXLNET_INPUTS_DOCSTRINGr‘r–r4r4rcr5r¢4s*   'õr¢zoXLNet Model with a language modeling head on top (linear layer with weights tied to the input embeddings). c s>eZdZ‡fdd„Zdd„Zdd„Zeeƒd d d „ƒZ‡Z S) ÚXLNetLMHeadModelcsHtƒ |¡|j|_|j|_t|ƒ|_tj|j|j dd|_ |  ¡dS)NT)r) rUrVr°r¯r¢rrr˜rXr´rr¶rarcr4r5rV s   zXLNetLMHeadModel.__init__cCs|jSre)rr¸r4r4r5Úget_output_embeddingsªsz&XLNetLMHeadModel.get_output_embeddingsc Ks²|jd}tj|dftj|jd}tj||gdd}|jd}tj|||ftj|jd}d|dd…dd…df<tj|d|ftj|jd}d|d<||||dd œ} |r®|| d <| S) NrrrÖrvr ri)rrrirè)rârärrèrŒ)rArDr¼rprlrˆrÐ) rbrâZpastÚkwargsZeffective_batch_sizeZ dummy_tokenZsequence_lengthrärÚinputsr4r4r5Úprepare_inputs_for_generation­s0  ÿÿüz.XLNetLMHeadModel.prepare_inputs_for_generationNTc  Csx|j||||||||| | d } | | d¡} | f| dd…}| dk rttƒ}||  d|  d¡¡|  d¡ƒ}|f|}|S)aÈ labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, num_predict)`, `optional`, defaults to :obj:`None`): Labels for masked language modeling. `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. The labels should correspond to the masked input words that should be predicted and depends on `target_mapping`. Note in order to perform standard auto-regressive language modeling a `` token has to be added to the `input_ids` (see `prepare_inputs_for_generation` fn and examples below) Indices are selected in ``[-100, 0, ..., config.vocab_size]`` All labels set to ``-100`` are ignored, the loss is only computed for labels in ``[0, ..., config.vocab_size]`` Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape `(1,)`, `optional`, returned when ``labels`` is provided) Language modeling loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_predict, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLNetTokenizer, XLNetLMHeadModel import torch tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') model = XLNetLMHeadModel.from_pretrained('xlnet-large-cased') # We show how to setup inputs to predict a next token using a bi-directional context. input_ids = torch.tensor(tokenizer.encode("Hello, my dog is very ", add_special_tokens=False)).unsqueeze(0) # We will predict the masked token perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float) perm_mask[:, :, -1] = 1.0 # Previous tokens don't see last token target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float) # Shape [1, 1, seq_length] => let's predict one token target_mapping[0, 0, -1] = 1.0 # Our first (and only) prediction will be the last token of the sequence (the masked token) outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping) next_token_logits = outputs[0] # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size] # The same way can the XLNetLMHeadModel be used to be trained by standard auto-regressive language modeling. input_ids = torch.tensor(tokenizer.encode("Hello, my dog is very ", add_special_tokens=False)).unsqueeze(0) # We will predict the masked token labels = torch.tensor(tokenizer.encode("cute", add_special_tokens=False)).unsqueeze(0) assert labels.shape[0] == 1, 'only one word will be predicted' perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float) perm_mask[:, :, -1] = 1.0 # Previous tokens don't see last token as is done in standard auto-regressive lm training target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float) # Shape [1, 1, seq_length] => let's predict one token target_mapping[0, 0, -1] = 1.0 # Our first (and only) prediction will be the last token of the sequence (the masked token) outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping, labels=labels) loss, next_token_logits = outputs[:2] # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size] © rãrŒrärrårær€rçrèrrNri)rrrÚviewÚsize)rbrârãrŒrärrårær€rçrèÚlabelsÚtransformer_outputsÚlogitsrÚloss_fctÚlossr4r4r5r‘Îs&Nö  zXLNetLMHeadModel.forward) NNNNNNNNNTN) r’r“r”rVrìrïr rêr‘r–r4r4rcr5rëšs  !ôrëzˆXLNet Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c s.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚXLNetForSequenceClassificationcsFtƒ |¡|j|_t|ƒ|_t|ƒ|_t |j |j¡|_ |  ¡dSre) rUrVÚ num_labelsr¢rrrrr˜rXrr¶rarcr4r5rV<s    z'XLNetForSequenceClassification.__init__NTc  Csª|j||||||||| | d } | d} | | ¡} | | ¡}|f| dd…}| dk r¦|jdkr|tƒ}|| d¡|  d¡ƒ}n tƒ}|| d|j¡|  d¡ƒ}|f|}|S)a¹ labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`) Labels for computing the sequence classification/regression loss. Indices should be in ``[0, ..., config.num_labels - 1]``. If ``config.num_labels == 1`` a regression loss is computed (Mean-Square loss), If ``config.num_labels > 1`` a classification loss is computed (Cross-Entropy). Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (:obj:`torch.FloatTensor` of shape :obj:(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLNetTokenizer, XLNetForSequenceClassification import torch tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') model = XLNetForSequenceClassification.from_pretrained('xlnet-large-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 labels = torch.tensor([1]).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, logits = outputs[:2] rðrrNri)rrrrùrrñr)rbrârãrŒrärrårær€rçrèrórôr†rõrrör÷r4r4r5r‘Fs0:ö     z&XLNetForSequenceClassification.forward) NNNNNNNNNTN©r’r“r”rVr rêr‘r–r4r4rcr5rø6s ôrøz›XLNet Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. c s.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚXLNetForTokenClassificationcs<tƒ |¡|j|_t|ƒ|_t |j|j¡|_|  ¡dSre) rUrVrùr¢rrr˜Ú hidden_sizeÚ classifierr¶rarcr4r5rV¦s   z$XLNetForTokenClassification.__init__NTc  CsÊ|j||||||||| | d } | d} | | ¡}|f| dd…} | dk rÆtƒ}|dk r¢| d¡dk}| d|j¡}t ||  d¡t |j¡  | ¡¡}|||ƒ}n|| d|j¡|  d¡ƒ}|f| } | S)a labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the multiple choice classification loss. Indices should be in ``[0, ..., num_choices]`` where `num_choices` is the size of the second dimension of the input tensors. (see `input_ids` above) Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided): Classification loss. logits (:obj:`torch.FloatTensor` of shape :obj:(batch_size, config.num_labels)`): Classification scores (before SoftMax). mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLNetTokenizer, XLNetForTokenClassification import torch tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') model = XLNetForTokenClassification.from_pretrained('xlnet-large-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) # Batch size 1 labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) scores = outputs[0] rðrrNri) rrýrrñrùrDÚwhereZtensorÚ ignore_indexZtype_as)rbrârãrŒrärrårær€rçrèrórÚsequence_outputrõröZ active_lossZ active_logitsZ active_labelsr÷r4r4r5r‘¯s:;ö  ÿ  z#XLNetForTokenClassification.forward) NNNNNNNNNTNrúr4r4rcr5rû s ôrûz—XLNet Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RACE/SWAG tasks. c s.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚXLNetForMultipleChoicecs<tƒ |¡t|ƒ|_t|ƒ|_t |jd¡|_ |  ¡dS)Nr) rUrVr¢rrrrr˜rXrr¶rarcr4r5rVs    zXLNetForMultipleChoice.__init__NTc  Csú|jd} | d| d¡¡} |dk r6| d| d¡¡nd}|dk rT| d| d¡¡nd}|dk rr| d| d¡¡nd}|j| |||||||| | d }|d}| |¡}| |¡}| d| ¡}|f|dd…}| dk rötƒ}|||  d¡ƒ}|f|}|S)a² labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the multiple choice classification loss. Indices should be in ``[0, ..., num_choices]`` where `num_choices` is the size of the second dimension of the input tensors. (see `input_ids` above) Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: loss (:obj:`torch.FloatTensor`` of shape ``(1,)`, `optional`, returned when :obj:`labels` is provided): Classification loss. classification_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_choices)`): `num_choices` is the second dimension of the input tensors. (see `input_ids` above). Classification scores (before SoftMax). mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLNetTokenizer, XLNetForMultipleChoice import torch tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased') model = XLNetForMultipleChoice.from_pretrained('xlnet-base-cased') choices = ["Hello, my dog is cute", "Hello, my cat is amazing"] input_ids = torch.tensor([tokenizer.encode(s) for s in choices]).unsqueeze(0) # Batch size 1, 2 choices labels = torch.tensor(1).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, classification_scores = outputs[:2] rriN) rårærãrŒrärr€rçrèr)rArñròrrrr)rbrârårærãrŒrärr€rçrèróZ num_choicesZflat_input_idsZflat_token_type_idsZflat_attention_maskZflat_input_maskrôr†rõZreshaped_logitsrrör÷r4r4r5r‘s:= ö    ÿ  zXLNetForMultipleChoice.forward) NNNNNNNNNTNrúr4r4rcr5r s ôrzÕXLNet Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c s.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚXLNetForQuestionAnsweringSimplecs<tƒ |¡|j|_t|ƒ|_t |j|j¡|_|  ¡dSre) rUrVrùr¢rrr˜rüÚ qa_outputsr¶rarcr4r5rV„s   z(XLNetForQuestionAnsweringSimple.__init__NTc  Cs |j||||||||| | d } | d}| |¡}|jddd\}}| d¡}| d¡}||f| dd…} | dk r| dk rt|  ¡ƒdkr˜|  d¡} t|  ¡ƒdkr²|  d¡} | d¡}|  d|¡|  d|¡t|d}||| ƒ}||| ƒ}||d}|f| } | S) a start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLNetTokenizer, XLNetForQuestionAnsweringSimple import torch tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased') model = XLNetForQuestionAnsweringSimple.from_pretrained('xlnet-base-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 start_positions = torch.tensor([1]) end_positions = torch.tensor([3]) outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) loss = outputs[0] rðrrrirvrSN)rÿ)rrÚsplitZsqueezer@ròZclamp_r)rbrârãrŒrärrårær€rçrèÚstart_positionsÚ end_positionsrrrõÚ start_logitsÚ end_logitsZ ignored_indexröÚ start_lossÚend_lossÚ total_lossr4r4r5r‘s@Cö              z'XLNetForQuestionAnsweringSimple.forward) NNNNNNNNNTNNrúr4r4rcr5r~s órcs.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚXLNetForQuestionAnsweringcsPtƒ |¡|j|_|j|_t|ƒ|_t|ƒ|_t|ƒ|_ t |ƒ|_ |  ¡dSre) rUrVÚ start_n_topÚ end_n_topr¢rrrr rr Ú answer_classr¶rarcr4r5rVÿs     z"XLNetForQuestionAnswering.__init__NTc) Cs(|j||||||||| | d }|d}|j||d}|dd…}| dk r| dk r| | || fD]"}|dk r`| ¡dkr`| d¡q`|j|| |d}tƒ}||| ƒ}||| ƒ}||d}|dk rø| dk rø|j|| |d }t ¡}||| ƒ}||d 7}|f|}n|  ¡\}}}t j |dd } t j | |jdd \}!}"|" d¡ dd|¡}#t  |d |#¡}$|$ d¡ d|dd¡}$| d¡ |$¡}%|dk r”| d¡nd}|j|%|$|d }t j |dd }&t j |&|jdd \}'}(|' d|j|j¡}'|( d|j|j¡}(t  d|| ¡}$|j||$|d}|!|"|'|(|f|}|S)aV start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. is_impossible (``torch.LongTensor`` of shape ``(batch_size,)``, `optional`, defaults to :obj:`None`): Labels whether a question has an answer or no answer (SQuAD 2.0) cls_index (``torch.LongTensor`` of shape ``(batch_size,)``, `optional`, defaults to :obj:`None`): Labels for position (index) of the classification token to use as input for computing plausibility of the answer. p_mask (``torch.FloatTensor`` of shape ``(batch_size, sequence_length)``, `optional`, defaults to :obj:`None`): Optional mask of tokens which can't be in answers (e.g. [CLS], [PAD], ...). 1.0 means token should be masked. 0.0 mean token is not masked. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned if both :obj:`start_positions` and :obj:`end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (``torch.FloatTensor`` of shape ``(batch_size, config.start_n_top)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (``torch.LongTensor`` of shape ``(batch_size, config.start_n_top)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (``torch.FloatTensor`` of shape ``(batch_size, config.start_n_top * config.end_n_top)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Log probabilities for the top ``config.start_n_top * config.end_n_top`` end token possibilities (beam-search). end_top_index (``torch.LongTensor`` of shape ``(batch_size, config.start_n_top * config.end_n_top)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Indices for the top ``config.start_n_top * config.end_n_top`` end token possibilities (beam-search). cls_logits (``torch.FloatTensor`` of shape ``(batch_size,)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Log probabilities for the ``is_impossible`` label of the answers. mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLNetTokenizer, XLNetForQuestionAnswering import torch tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased') model = XLNetForQuestionAnswering.from_pretrained('xlnet-base-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 start_positions = torch.tensor([1]) end_positions = torch.tensor([3]) outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) loss = outputs[0] rðr)Úp_maskrNri)rrrS)rÚ cls_indexrRrvéþÿÿÿ)Ú start_statesrz blh,bl->bh)rr)rrrwZsqueeze_rrrrZBCEWithLogitsLossròryrzrDZtopkr rárÈZgatherZ expand_asrrñrx))rbrârãrŒrärrårær€rçrèrrZ is_impossiblerrrôrérrrrrrör r r Z cls_logitsZ loss_fct_clsZcls_lossrÊÚslenZhszZstart_log_probsZstart_top_log_probsZstart_top_indexZstart_top_index_exprZhidden_states_expandedZ end_log_probsZend_top_log_probsZ end_top_indexr4r4r5r‘ s‚Qö        ÿ  ÿÿ ÿÿz!XLNetForQuestionAnswering.forward)NNNNNNNNNTNNNNNrúr4r4rcr5r ùs$ ðr )N)/r¦ÚloggingrDrZtorch.nnrrrryZ activationsrrZconfiguration_xlnetr Z file_utilsr r Zmodeling_utilsr r rrrÚ getLoggerr’r9r§r6rOrPršZ LayerNormr]ÚModulerQr—ržrŸZXLNET_START_DOCSTRINGrêr¢rërørûrrr r4r4r4r5Ús|    þ S>S' 9þdýýeýhýlývý