U &cݙ @sdZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z ddl mZmZmZmZmZddlmZeeZdd d d d d ddddd ZddZddZdejfddZGdddejjjZ GdddejjjZ!GdddejjjZ"Gdd d eZ#d!Z$d"Z%e d#e$Gd$d%d%e#Z&Gd&d'd'ejjjZ'e d(e$Gd)d*d*e#Z(e d+e$Gd,d-d-e#Z)e d.e$Gd/d0d0e#Z*dS)1z TF 2.0 XLM model. N) XLMConfig)add_start_docstrings add_start_docstrings_to_callable)TFPreTrainedModelTFSequenceSummaryTFSharedEmbeddingsget_initializer shape_list) BatchEncodingz6https://cdn.huggingface.co/xlm-mlm-en-2048-tf_model.h5z8https://cdn.huggingface.co/xlm-mlm-ende-1024-tf_model.h5z8https://cdn.huggingface.co/xlm-mlm-enfr-1024-tf_model.h5z8https://cdn.huggingface.co/xlm-mlm-enro-1024-tf_model.h5z>https://cdn.huggingface.co/xlm-mlm-tlm-xnli15-1024-tf_model.h5z:https://cdn.huggingface.co/xlm-mlm-xnli15-1024-tf_model.h5z8https://cdn.huggingface.co/xlm-clm-enfr-1024-tf_model.h5z8https://cdn.huggingface.co/xlm-clm-ende-1024-tf_model.h5z6https://cdn.huggingface.co/xlm-mlm-17-1280-tf_model.h5z7https://cdn.huggingface.co/xlm-mlm-100-1280-tf_model.h5) zxlm-mlm-en-2048zxlm-mlm-ende-1024zxlm-mlm-enfr-1024zxlm-mlm-enro-1024zxlm-mlm-tlm-xnli15-1024zxlm-mlm-xnli15-1024zxlm-clm-enfr-1024zxlm-clm-ende-1024zxlm-mlm-17-1280zxlm-mlm-100-1280c stfddt|D}tt|dddddf|dddddf<tt|dddddf|dddddf<dS)Ncs$g|]fddtDqS)c s(g|] }tdd|dqS)i')nppower).0j)dimpos@/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_tf_xlm.py 1sz;create_sinusoidal_embeddings...)range)rr)rrr1sz0create_sinusoidal_embeddings..rr r)r arrayrtfconstantsincos)Zn_posroutZ position_encrrrcreate_sinusoidal_embeddings0s4rcCs(ddtj|tjd}||S)a Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415 g??g@)rmatherfsqrt)xZcdfrrrgelu6s r$c Cst|d}|dk r|}n&t|}tj||ddtjf}|rtt|tjtjddf||df|tjddtjf}n|}tj t|||g|dkst||||gkst tj ||d}tj ||d}||fS)zH Generate hidden states mask, and optionally an attention mask. rNrFdtype) r rrr ZlessnewaxisZ less_equalZtile debugging assert_equalAssertionErrorcast) slenlengthscausal padding_maskr&bsmaskalen attn_maskrrr get_masksAs   "r4cs6eZdZeZfddZddZdddZZ S) TFMultiHeadAttentionc stjf|ttj|_|j|_||_||_|j|jdksBt t j j j |t|jdd|_t j j j |t|jdd|_t j j j |t|jdd|_t j j j |t|jdd|_t j j |j|_t|_dS)Nrq_linZkernel_initializernamek_linv_linout_lin)super__init__nextr5NEW_IDlayer_idoutput_attentionsrn_headsr*rkeraslayersDenser init_stdr6r9r:r;Dropoutattention_dropoutdropoutset pruned_heads)selfrBrconfigkwargs __class__rrr=ds zTFMultiHeadAttention.__init__cCstdSNNotImplementedError)rLheadsrrr prune_headsssz TFMultiHeadAttention.prune_headsFcs|\}}}}}t|\}} |dkr>|dkr0|n |d|} n t|d} j} j| tt|dkrvd|| fn dd| f} fdd} fdd}| |}|dkr| |}| |}n8|dksj|kr|}}| |}| |}|dk r~j|krp|dkrb|j\}}tj ||gd d }tj ||gd d }n|j\}}||f|j<|t }tj ||d d }t || }|d d|}tjj|dd }j||d}|dk r||}t ||}||}|f}jr||f}|S)zd Self-attention (if kv is None) or attention over source sentence (provided by kv). Nr,rcs tjt|djfddS)z projection rr rrVperm)r transposereshaperBr#r0Z dim_per_headrLrrshapesz(TFMultiHeadAttention.call..shapecs"ttj|dddjfS)z compute context rXrYrW)rr\r[rBr]r^rrunshapesz*TFMultiHeadAttention.call..unshaper ZaxisT)Z transpose_bgꌠ9Y>)FrrWtraining)r rBrlenr6r9r:r@rconcatr r"matmulr\nnZsoftmaxrIr;rA)rLinputsrcinputr1kvcache head_maskqlenrZklenrBZ mask_reshaper_r`qkvZk_Zv_Zscoresweightscontextoutputsrr^rcallvsP  (        zTFMultiHeadAttention.call)F) __name__ __module__ __qualname__ itertoolscountr?r=rUrt __classcell__rrrOrr5`s r5cs&eZdZfddZdddZZS)TFTransformerFFNc s|tjf|tjjj|t|jdd|_tjjj|t|jdd|_ |j rZtjj t ntjj j|_tjj|j|_dS)Nlin1r7lin2)r<r=rrCrDrEr rFr|r}Zgelu_activationZ Activationr$Z activationsZreluactrGrI)rLZin_dimZ dim_hiddenZout_dimrMrNrOrrr=s  zTFTransformerFFN.__init__FcCs0||}||}||}|j||d}|S)Nrb)r|r~r}rI)rLrircr#rrrrts    zTFTransformerFFN.call)F)rurvrwr=rtrzrrrOrr{s r{c s>eZdZfddZddZddZddZd d d ZZS)TFXLMMainLayerc s|tjf||j|_|j|_|j|_|j |_|jr>td|j|_|j|_|j |_ |j |_ |j |_ |j |_ |j |_|jd|_|j|_|j|_|j|jdkstdtjj|j|_tjj|j|_tjjj|j|jt|jdd|_|jrt|jdkr0|j r0tjjj|j|jt|jdd|_t |j |j|jd d |_!tjjj"|j#d d |_$g|_%g|_&g|_'g|_(t)|jD]}|j%*t+|j|j|d ,|d|j&*tjjj"|j#d,|d |j'*t-|j|j|j|d,|d|j(*tjjj"|j#d,|d qt.|drx|j/01}i|_/|D]>\}}|j%t2|j|jkr8|3t2|t4t5t2|iq8dS)Nz,Currently XLM can only be used as an encoderrz-transformer dim must be a multiple of n_headsposition_embeddings)Zembeddings_initializerr8rlang_embeddings embeddingsZinitializer_ranger8layer_norm_emb)epsilonr8zattentions_._{})rMr8zlayer_norm1_._{}z ffns_._{}zlayer_norm2_._{}rK)6r<r=rAoutput_hidden_statesZ is_encoderZ is_decoderrSr.n_langs use_lang_embn_wordsZ eos_index pad_indexZemb_dimrZ hidden_dimrBn_layersr*rrCrDrGrIrHZ EmbeddingZmax_position_embeddingsr Zembed_init_stdrZsinusoidal_embeddingsrrrZLayerNormalizationZlayer_norm_epsr attentions layer_norm1ffns layer_norm2rappendr5formatr{hasattrrKcopyitemsintrUlistmap)rLrMrNirKZlayerrTrOrrr=s    zTFXLMMainLayer.__init__cCs|jSrQ)rrLrrrget_input_embeddings#sz#TFXLMMainLayer.get_input_embeddingscCstdSrQrR)rLZnew_num_tokensrrr_resize_token_embeddings&sz'TFXLMMainLayer._resize_token_embeddingscCstdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel NrR)rLZheads_to_prunerrr _prune_heads)szTFXLMMainLayer._prune_headsNFc Cst|ttfr|d} t|dkr*|dn|}t|dkrB|dn|}t|dkrZ|dn|}t|dkrr|dn|}t|dkr|dn|}t|dkr|dn|}t|dkr|dn|}t|d kr|d n| } t|d kstd nt|ttfr~|d } |d |}|d|}|d|}|d|}|d|}|d|}|d|}|d| } t|d kstd n|} | dk r| dk rtdn@| dk rt | \} } n(| dk rt | dd\} } ntd|dkr.| dk rt j t j t | |jt jddd}nt | g| t j}t jt |d| t| ||j|d\}}|dkrzt jt | dd}nt jt || | g|dk rt jt || | g|dk rtn dg|j}|dk rd| dk rd| |d}| dd| df} |dd| df}|dk r8|dd| df}|dd| df}|dd| df}| dkrx|| } | ||}|dk r|jr|jdkr|||}|dk r|||}||}|j|| d}||dt j f}d}d}t|jD]}|j!r||f}|j"|||d|||g| d}|d}|j#rZ||df}|j|| d}||}|j$||}||j%||}|j&||}||dt j f}q|j!r||f}|dk r|d|'d7<|f}|j!r||f}|j#r||f}|S)Nrrr rVr zToo many inputs. input_idsattention_masklangstoken_type_ids position_idsr-rkrl inputs_embedszDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsr%ra)r/r,rb.r)( isinstancetuplerrdr*dictr get ValueErrorr rZ reduce_sumr+ not_equalrint32Zconvert_to_tensorr(r)r4r.Z expand_dimsrrSrrrrrrrrIr'rrrArrrsize)rLrhrrrrr-rkrlrrcrr0r,r1r3Z_slenZtensor hidden_statesrrZ attn_outputsZattnrsrrrrt0s               &                  zTFXLMMainLayer.call) NNNNNNNNF) rurvrwr=rrrrtrzrrrOrrs Z rc@s(eZdZdZeZeZdZe ddZ dS)TFXLMPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerc Cstdddddgdddddgdddddgg}tdddddgdddddgdddddgg}|jjr|jjdkrtdddddgdddddgdddddgg}nd}|||d S) Nrrrrr rVrr)rrr)rrrMrr)rLZ inputs_listZ attns_listZ langs_listrrr dummy_inputss ..0z!TFXLMPreTrainedModel.dummy_inputsN) rurvrw__doc__rZ config_class#TF_XLM_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZbase_model_prefixpropertyrrrrrrs ra .. note:: TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` Parameters: config (:class:`~transformers.XLMConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ langs (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): A parallel sequence of tokens to be used to indicate the language of each token in the input. Indices are languages ids which can be obtained from the language names by using two conversion mappings provided in the configuration of the model (only provided for multilingual models). More precisely, the `language name -> language id` mapping is in `model.config.lang2id` (dict str -> int) and the `language id -> language name` mapping is `model.config.id2lang` (dict int -> str). See usage examples detailed in the `multilingual documentation `__. token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ lengths (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Length of each sentence that can be used to avoid performing attention on padding token indices. You can also use `attention_mask` for the same result (see above), kept here for compatbility. Indices selected in ``[0, ..., input_ids.size(-1)]``: cache (:obj:`Dict[str, tf.Tensor]`, `optional`, defaults to :obj:`None`): dictionary with ``tf.Tensor`` that contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `cache` output below). Can be used to speed up sequential decoding. The dictionary object will be modified in-place during the forward pass to add newly computed hidden-states. head_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. z\The bare XLM Model transformer outputing raw hidden-states without any specific head on top.cs,eZdZfddZeeddZZS) TFXLMModelcs&tj|f||t|dd|_dS)Nrr8)r<r=rrrLrMrhrNrOrrr=?szTFXLMModel.__init__cKs|j|f|}|S)a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: last_hidden_state (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import XLMTokenizer, TFXLMModel tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = TFXLMModel.from_pretrained('xlm-mlm-en-2048') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple )r)rLrhrNrsrrrrtCszTFXLMModel.callrurvrwr=rXLM_INPUTS_DOCSTRINGrtrzrrrOrr:s rcs4eZdZdZfddZfddZddZZS)TFXLMPredLayerz? Prediction layer (cross_entropy or adaptive_softmax). c s@tjf||j|_|j|_|j|_|jdkr8||_ntdS)NF)r<r=Zasmrrinput_embeddingsrS)rLrMrrNrOrrr=ks zTFXLMPredLayer.__init__cs(|j|jfdddd|_t|dS)NzerosTbias)r_Z initializerZ trainabler8)Z add_weightrrr<build)rLZ input_shaperOrrr|szTFXLMPredLayer.buildcCs|j|dd}||j}|S)NZlinear)mode)rr)rLrrrrrts zTFXLMPredLayer.call)rurvrwrr=rrtrzrrrOrrfs  rz}The XLM Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). cs<eZdZfddZddZddZeeddZZ S) TFXLMWithLMHeadModelcs:tj|f||t|dd|_t||jjdd|_dS)Nrrzpred_layer_._proj)r<r=rrrr pred_layerrrOrrr=szTFXLMWithLMHeadModel.__init__cCs|jjSrQ)rrrrrrget_output_embeddingssz*TFXLMWithLMHeadModel.get_output_embeddingscKsj|jj}|jj}|jd}tj|dftjd|}tj||gdd}|dk r\t||}nd}||dS)Nrrr%ra)rhr) rM mask_token_idlang_idr_rZonesrreZ ones_like)rLrhrNrrZeffective_batch_sizeZ mask_tokenrrrrprepare_inputs_for_generations z2TFXLMWithLMHeadModel.prepare_inputs_for_generationcKs6|j|f|}|d}||}|f|dd}|S)a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: prediction_scores (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import XLMTokenizer, TFXLMWithLMHeadModel tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = TFXLMWithLMHeadModel.from_pretrained('xlm-mlm-en-2048') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple rrN)rr)rLrhrNtransformer_outputsoutputrsrrrrts  zTFXLMWithLMHeadModel.call) rurvrwr=rrrrrtrzrrrOrrs  rzXLM Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs,eZdZfddZeeddZZS)TFXLMForSequenceClassificationcs@tj|f|||j|_t|dd|_t||jdd|_dS)Nrrsequence_summaryr)r<r= num_labelsrrrrFrrrOrrr=sz'TFXLMForSequenceClassification.__init__cKs6|j|f|}|d}||}|f|dd}|S)a Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: logits (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import XLMTokenizer, TFXLMForSequenceClassification tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = TFXLMForSequenceClassification.from_pretrained('xlm-mlm-en-2048') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 labels = tf.constant([1])[None, :] # Batch size 1 outputs = model(input_ids) logits = outputs[0] rrN)rr)rLrhrNrrlogitsrsrrrrts  z#TFXLMForSequenceClassification.callrrrrOrrs rzXLM Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs,eZdZfddZeeddZZS)TFXLMForQuestionAnsweringSimplecsDtj|f||t|dd|_tjjj|jt |j dd|_ dS)Nrr qa_outputsr7) r<r=rrrrCrDrErr rFrrrOrrr=sz(TFXLMForQuestionAnsweringSimple.__init__c Ksh|j|f|}|d}||}tj|ddd\}}tj|dd}tj|dd}||f|dd}|S)a Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: start_scores (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import XLMTokenizer, TFXLMForQuestionAnsweringSimple tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048') model = TFXLMForQuestionAnsweringSimple.from_pretrained('xlm-mlm-en-2048') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) start_scores, end_scores = outputs[:2] rr rWrarN)rrrsplitZsqueeze) rLrhrNrZsequence_outputrZ start_logitsZ end_logitsrsrrrrts!  z$TFXLMForQuestionAnsweringSimple.callrrrrOrrs r)+rrxloggingr Znumpyr Z tensorflowrZconfiguration_xlmrZ file_utilsrrZmodeling_tf_utilsrrrr r Ztokenization_utilsr getLoggerruloggerrrr$Zfloat32r4rCrDZLayerr5r{rrZXLM_START_DOCSTRINGrrrrrrrrrrsj    X 7(!?1