U &cX @sdZddlZddlZddlZddlZddlZddlmZddlm Z m Z ddlm Z ddl mZddlmZdd lmZmZdd lmZmZmZmZeeZd d d dddddddd ZddZd6ddZGdddejZ GdddejZ!GdddeZ"d Z#d!Z$ed"e#Gd#d$d$e"Z%Gd%d&d&ejZ&ed'e#Gd(d)d)e"Z'ed*e#Gd+d,d,e"Z(ed-e#Gd.d/d/e"Z)ed0e#Gd1d2d2e"Z*ed3e#Gd4d5d5e"Z+dS)7z PyTorch XLM model. N)nn)CrossEntropyLossMSELoss) functional)gelu) XLMConfig)add_start_docstrings add_start_docstrings_to_callable)PreTrainedModelSequenceSummary SQuADHeadprune_linear_layerzhttps://cdn.huggingface.co/xlm-mlm-ende-1024-pytorch_model.binz>https://cdn.huggingface.co/xlm-mlm-enfr-1024-pytorch_model.binz>https://cdn.huggingface.co/xlm-mlm-enro-1024-pytorch_model.binzDhttps://cdn.huggingface.co/xlm-mlm-tlm-xnli15-1024-pytorch_model.binz@https://cdn.huggingface.co/xlm-mlm-xnli15-1024-pytorch_model.binz>https://cdn.huggingface.co/xlm-clm-enfr-1024-pytorch_model.binz>https://cdn.huggingface.co/xlm-clm-ende-1024-pytorch_model.binz4sz;create_sinusoidal_embeddings...)range)rr)rrr4sz0create_sinusoidal_embeddings..rrrF) rarrayrtorchZ FloatTensorsincosZdetach_Z requires_grad)Zn_posroutZ position_encrrrcreate_sinusoidal_embeddings3s 44r cCstj|tj|jd}|dk r"|}n(||ks6t||dddfk}|d}|r|ddddf||d|ddddfk}n|}|||fkst|dks||||fkst||fS)zH Generate hidden states mask, and optionally an attention mask. dtypedeviceNrrF) rarangelongr#maxitemAssertionErrorsizerepeat)slenlengthscausal padding_maskalenmaskbs attn_maskrrr get_masks;s 0r3cs6eZdZeZfddZddZdddZZ S) MultiHeadAttentioncstttj|_|j|_||_||_|j |_ |j|jdksFt t |||_t |||_t |||_t |||_t|_dS)Nr)super__init__nextr4NEW_IDlayer_idoutput_attentionsrn_headsattention_dropoutdropoutr(rLinearq_link_linv_linout_linset pruned_heads)selfr;rconfig __class__rrr6Xs  zMultiHeadAttention.__init__cs|j|j}t|dkrdSt|j|}t||j}|D](tfdd|jD8d|<q<|d d}t t|| }t |j||_t |j||_t |j||_t |j|dd|_|jt||_||j|_|j||_dS)Nrc3s|]}|krdndVqdS)rrNr)rhheadrr nsz1MultiHeadAttention.prune_heads..rr)rr;lenrZonesrCrDsumview contiguouseqr$r%rr?r@rArBunion)rEheadsZattention_head_sizer0indexrrJr prune_headsgs"    zMultiHeadAttention.prune_headsNcs$|\}}|dkr0|dkr"|n |d|}n |d}j} j| |dkrbd||fn dd|f} fdd} fdd} | |} |dkr| |}| |}n6|dksԈj|kr|}}| |}| |}|dk rhj|krZ|dkrL|j\}}tj||gd d }tj||gd d }n|j\}}||f|j<| t } t | | d d}|d k | |}||td  tj|d d |}tj|jjd}|dk r||}t ||}| |}|f}jr ||f}|S)zd Self-attention (if kv is None) or attention over source sentence (provided by kv). Nr+rcs|djddS)z projection rMrr)rPr; transposexr1Z dim_per_headrErrshapesz)MultiHeadAttention.forward..shapecs |dddjS)z compute context rrrM)rXrQrPr;rYr[rrunshapesz+MultiHeadAttention.forward..unshaperrrinfrMptraining)r)r;rr?r@rAr9rcatmathsqrtmatmulrXrP expand_asZ masked_fill_floatFZsoftmaxtype_asr=rarBr:)rEinputr0kvcache head_maskqlenrZklenr;Z mask_reshaper\r]qkvZk_Zv_scoresweightscontextoutputsrr[rforward|sN  $       zMultiHeadAttention.forward)NNN) __name__ __module__ __qualname__ itertoolscountr8r6rVrv __classcell__rrrGrr4Ts r4cs$eZdZfddZddZZS)TransformerFFNcsDt|j|_t|||_t|||_|jr8tnt j |_ dSN) r5r6r=rr>lin1lin2Zgelu_activationrrhZreluact)rEZin_dimZ dim_hiddenZout_dimrFrGrrr6s  zTransformerFFN.__init__cCs6||}||}||}tj||j|jd}|S)Nr_)rrrrhr=ra)rErjrZrrrrvs    zTransformerFFN.forward)rwrxryr6rvr|rrrGrr}s r}csDeZdZdZeZeZdZdZ fddZ e ddZ dd Z ZS) XLMPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. N transformercstj||dSr~)r5r6)rEinputskwargsrGrrr6szXLMPreTrainedModel.__init__c Cstdddddgdddddgdddddgg}tdddddgdddddgdddddgg}|jjr|jjdkrtdddddgdddddgdddddgg}nd}|||d S) NrrrrW) input_idsattention_masklangs)rtensorrF use_lang_embn_langs)rEZ inputs_listZ attns_listZ langs_listrrr dummy_inputss ..0zXLMPreTrainedModel.dummy_inputscCst|tjr:|jdk r:|jjdk r:tjj|jd|jjdt|tjr|jdk r|jj dk rtjj|jd|jj dt |dr|j dk rtj |j dt|tj r|j j|jjddS)z Initialize the weights. Nr)ZmeanZstdbiasgg?) isinstancer EmbeddingrFZembed_init_stdinitZnormal_weightr>Zinit_stdhasattrrZ constant_ LayerNormdataZzero_Zfill_)rEmodulerrr _init_weightss    z XLMPreTrainedModel._init_weights)rwrxry__doc__rZ config_class XLM_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZload_tf_weightsZbase_model_prefixr6propertyrrr|rrrGrrs  ram This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.XLMConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a5 Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ langs (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): A parallel sequence of tokens to be used to indicate the language of each token in the input. Indices are languages ids which can be obtained from the language names by using two conversion mappings provided in the configuration of the model (only provided for multilingual models). More precisely, the `language name -> language id` mapping is in `model.config.lang2id` (dict str -> int) and the `language id -> language name` mapping is `model.config.id2lang` (dict int -> str). See usage examples detailed in the `multilingual documentation `__. token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ lengths (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Length of each sentence that can be used to avoid performing attention on padding token indices. You can also use `attention_mask` for the same result (see above), kept here for compatbility. Indices selected in ``[0, ..., input_ids.size(-1)]``: cache (:obj:`Dict[str, torch.FloatTensor]`, `optional`, defaults to :obj:`None`): dictionary with ``torch.FloatTensor`` that contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `cache` output below). Can be used to speed up sequential decoding. The dictionary object will be modified in-place during the forward pass to add newly computed hidden-states. head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. z]The bare XLM Model transformer outputting raw hidden-states without any specific head on top.c sFeZdZfddZddZddZddZeed d d Z Z S) XLMModelc sNt||j|_|j|_|j|_|j |_|jr\}}|j"t.|j|jkr|/t.|t0t1t.|iq|2dS) Nz,Currently XLM can only be used as an encoderrrz-transformer dim must be a multiple of n_heads)rr)Z padding_idx)Zeps)rFrD)3r5r6r:output_hidden_statesZ is_encoderZ is_decoderNotImplementedErrorr-rrn_wordsZ eos_index pad_indexemb_dimrZ hidden_dimr;n_layersr=r<r(rrZmax_position_embeddingsposition_embeddingsZsinusoidal_embeddingsr rlang_embeddings embeddingsrZlayer_norm_epslayer_norm_embZ ModuleList attentions layer_norm1ffns layer_norm2rappendr4r}rrDcopyitemsintrVlistmap init_weights)rErF_rDlayerrTrGrrr6:sV          zXLMModel.__init__cCs|jSr~rrErrrget_input_embeddingsszXLMModel.get_input_embeddingscCs ||_dSr~r)rEZnew_embeddingsrrrset_input_embeddingsszXLMModel.set_input_embeddingscCs&|D]\}}|j||qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)rrrV)rEZheads_to_prunerrTrrr _prune_headsszXLMModel._prune_headsNc Cs|dk r|\} } n| dd\} } |dkrb|dk rR||jkjdd}nt| g| }|d| kstt|| kstt | ||j |d\} } |dk r|j n| j }|dkrtj | tj|d}| d| | f}n|| | fkst|dk r|| | fkst|||jj}|dk r|dk r| |d}|dd| df}|dd| df}|dk r|dd| df}| dd| df} | dd| df} | dkr||} | ||| }|dk r|jr|jdkr|||}|dk r*|||}||}tj||j|jd }|| d|j9}d }d }t|jD]}|jr||f}|j ||| |||d }|d}|j!r||df}tj||j|jd }||}|j"||}||j#||}|j$||}|| d|j9}qp|jr4||f}|dk rT|d|d7<|f}|jrl||f}|j!r~||f}|S) a Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLMTokenizer, XLMModel import torch tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = XLMModel.from_pretrained('xlm-mlm-en-2048') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NrMrrr)r.r!r+r_r)rlrm)%r)rrOr%rZ LongTensorr(r&r'r3r-r#r$Z unsqueezeexpandZ get_head_maskrFrrrrfrrrrrhr=rator"rrrr:rrr)rErrrtoken_type_ids position_idsr,rlrm inputs_embedsr1r+r0r2r#Z_slenrZ hidden_statesriZ attn_outputsZattnrurrrrvsz*             zXLMModel.forward) NNNNNNNNN) rwrxryr6rrrr XLM_INPUTS_DOCSTRINGrvr|rrrGrr5s Frcs*eZdZdZfddZdddZZS) XLMPredLayerz? Prediction layer (cross_entropy or adaptive_softmax). csht|j|_|j|_|j|_|j}|jdkrHtj||jdd|_ntj ||j|j |j dd|_dS)NFT)r)Z in_featuresZ n_classesZcutoffsZ div_valueZ head_bias) r5r6asmrrrrr>projZAdaptiveLogSoftmaxWithLossZ asm_cutoffsZ asm_div_value)rErFrrGrrr6/s  zXLMPredLayer.__init__NcCsd}|jdkrV||}|f|}|dk rtj|d|j|ddd}|f|}n8|j|}|f|}|dk r|||\}}|f|}|S)z6 Compute the loss, and optionally the scores. rFNrMZelementwise_mean)Z reduction)rrrhZ cross_entropyrPrZlog_prob)rErZyrurrlossrrrrrvAs        zXLMPredLayer.forward)N)rwrxryrr6rvr|rrrGrr*s rz}The XLM Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). c s>eZdZfddZddZddZeed dd ZZ S) XLMWithLMHeadModelcs,t|t||_t||_|dSr~)r5r6rrr pred_layerrrErFrGrrr6[s   zXLMWithLMHeadModel.__init__cCs|jjSr~)rrrrrrget_output_embeddingsbsz(XLMWithLMHeadModel.get_output_embeddingscKsj|jj}|jj}|jd}tj|df|tj|jd}tj||gdd}|dk r\t ||}nd}||dS)Nrrr!r)rr) rF mask_token_idlang_idr\rfullr%r#rbZ full_like)rErrrrZeffective_batch_sizeZ mask_tokenrrrrprepare_inputs_for_generationes z0XLMWithLMHeadModel.prepare_inputs_for_generationNc  CsD|j||||||||| d } | d} || | } | | dd} | S)aH labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids`` Indices are selected in ``[-100, 0, ..., config.vocab_size]`` All labels set to ``-100`` are ignored (masked), the loss is only computed for labels in ``[0, ..., config.vocab_size]`` Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape `(1,)`, `optional`, returned when ``labels`` is provided) Language modeling loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLMTokenizer, XLMWithLMHeadModel import torch tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = XLMWithLMHeadModel.from_pretrained('xlm-mlm-en-2048') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple rrrrr,rlrmrrrN)rr)rErrrrrr,rlrmrlabelstransformer_outputsoutputrurrrrvrs4  zXLMWithLMHeadModel.forward) NNNNNNNNNN) rwrxryr6rrr rrvr|rrrGrrUs  rzXLM Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c s.eZdZfddZeedddZZS)XLMForSequenceClassificationcs4t||j|_t||_t||_|dSr~)r5r6 num_labelsrrr sequence_summaryrrrGrrr6s    z%XLMForSequenceClassification.__init__Nc  Cs|j||||||||| d } | d} || } | f| dd}| dk r|jdkrpt}|| d| d}n t}|| d|j| d}|f|}|S)a5 labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ..., config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss), If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`label` is provided): Classification (or regression if config.num_labels==1) loss. logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLMTokenizer, XLMForSequenceClassification import torch tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = XLMForSequenceClassification.from_pretrained('xlm-mlm-en-2048') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 labels = torch.tensor([1]).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, logits = outputs[:2] rrrNrM)rrrrrPr)rErrrrrr,rlrmrrrrlogitsruloss_fctrrrrrvs,4    z$XLMForSequenceClassification.forward) NNNNNNNNNNrwrxryr6r rrvr|rrrGrrs rzXLM Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c s.eZdZfddZeedddZZS)XLMForQuestionAnsweringSimplecs4t|t||_t|j|j|_| dSr~) r5r6rrrr> hidden_sizer qa_outputsrrrGrrr6 s  z&XLMForQuestionAnsweringSimple.__init__Nc  Cs |j||||||||| d } | d} || }|jddd\}}|d}|d}||f}| dk r| dk rt| dkr| d} t| dkr| d} |d}| d|| d|t|d}||| }||| }||d}|f|}|| dd}|S) at start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLMTokenizer, XLMForQuestionAnsweringSimple import torch tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = XLMForQuestionAnsweringSimple.from_pretrained('xlm-mlm-en-2048') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 start_positions = torch.tensor([1]) end_positions = torch.tensor([3]) outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) loss = outputs[0] rrrrMrN) ignore_indexr)rrsplitZsqueezerNr)Zclamp_r)rErrrrrr,rlrmrstart_positions end_positionsrsequence_outputrZ start_logitsZ end_logitsruZ ignored_indexrZ start_lossZend_lossZ total_lossrrrrv(sD;              z%XLMForQuestionAnsweringSimple.forward) NNNNNNNNNNNrrrrGrrs rzXLM Model with a beam-search span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs.eZdZfddZeedddZZS)XLMForQuestionAnsweringcs,t|t||_t||_|dSr~)r5r6rrr rrrrGrrr6s   z XLMForQuestionAnswering.__init__Nc CsN|j||||||||| d }|d}|j|| | | | |d}||dd}|S)a start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. is_impossible (``torch.LongTensor`` of shape ``(batch_size,)``, `optional`, defaults to :obj:`None`): Labels whether a question has an answer or no answer (SQuAD 2.0) cls_index (``torch.LongTensor`` of shape ``(batch_size,)``, `optional`, defaults to :obj:`None`): Labels for position (index) of the classification token to use as input for computing plausibility of the answer. p_mask (``torch.FloatTensor`` of shape ``(batch_size, sequence_length)``, `optional`, defaults to :obj:`None`): Optional mask of tokens which can't be in answers (e.g. [CLS], [PAD], ...). 1.0 means token should be masked. 0.0 mean token is not masked. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned if both :obj:`start_positions` and :obj:`end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (``torch.FloatTensor`` of shape ``(batch_size, config.start_n_top)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (``torch.LongTensor`` of shape ``(batch_size, config.start_n_top)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (``torch.FloatTensor`` of shape ``(batch_size, config.start_n_top * config.end_n_top)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Log probabilities for the top ``config.start_n_top * config.end_n_top`` end token possibilities (beam-search). end_top_index (``torch.LongTensor`` of shape ``(batch_size, config.start_n_top * config.end_n_top)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Indices for the top ``config.start_n_top * config.end_n_top`` end token possibilities (beam-search). cls_logits (``torch.FloatTensor`` of shape ``(batch_size,)``, `optional`, returned if ``start_positions`` or ``end_positions`` is not provided): Log probabilities for the ``is_impossible`` label of the answers. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLMTokenizer, XLMForQuestionAnswering import torch tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = XLMForQuestionAnswering.from_pretrained('xlm-mlm-en-2048') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 start_positions = torch.tensor([1]) end_positions = torch.tensor([3]) outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) loss = outputs[0] rr)rr cls_index is_impossiblep_maskrN)rr)rErrrrrr,rlrmrrrrrrrrrurrrrvs,K  zXLMForQuestionAnswering.forward)NNNNNNNNNNNNNNrrrrGrrs" rzXLM Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cs.eZdZfddZeedddZZS)XLMForTokenClassificationcsJt||j|_t||_t|j|_t|j |j|_ | dSr~) r5r6rrrrZDropoutr=r>r classifierrrrGrrr6 s   z"XLMForTokenClassification.__init__NcCs|j||||||d}|d} || } || } | f|dd}|dk rt} |dk r|ddk} | d|j} t| |dt| j  |}| | |}n| | d|j|d}|f|}|S)a, labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels - 1]``. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when ``labels`` is provided) : Classification loss. scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.num_labels)`) Classification scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import XLMTokenizer, XLMForTokenClassification import torch tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-100-1280') model = XLMForTokenClassification.from_pretrained('xlm-mlm-100-1280') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) # Batch size 1 labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, scores = outputs[:2] )rrrrrmrrNrMr) rr=rrrPrrwhererrri)rErrrrrrmrrurrrZ active_lossZ active_logitsZ active_labelsrrrrrvs4/     z!XLMForTokenClassification.forward)NNNNNNNrrrrGrrs r)N),rrzloggingrcZnumpyrrrZtorch.nnrrrrhZ activationsrZconfiguration_xlmrZ file_utilsr r Zmodeling_utilsr r r r getLoggerrwloggerrr r3Moduler4r}rZXLM_START_DOCSTRINGrrrrrrrrrrrrs~      h& 7r+_\qp