U &c@sdZddlZddlZddlZddlmZddlm Z m Z ddl m Z m Z mZmZmZmZddlmZeeZdd d Zd d Zd dZejjeejjjejjedZGdddejjj Z!Gdddejjj Z"Gdddejjj Z#Gdddejjj Z$eGdddejjj Z%Gddde Z&dZ'dZ(e de'Gdd d e&Z)e d!e'Gd"d#d#e&Z*e d$e'Gd%d&d&e&Z+e d'e'Gd(d)d)e&Z,e d*e'Gd+d,d,e&Z-dS)-z TF 2.0 XLNet model. N) XLNetConfig)add_start_docstrings add_start_docstrings_to_callable)TFPreTrainedModelTFSequenceSummaryTFSharedEmbeddingsget_initializerkeras_serializable shape_list) BatchEncodingz7https://cdn.huggingface.co/xlnet-base-cased-tf_model.h5z8https://cdn.huggingface.co/xlnet-large-cased-tf_model.h5)zxlnet-base-casedzxlnet-large-casedc Cs:ddttdtj|dt|d}||S)z Implementation of the gelu activation function. XLNet is using OpenAI GPT's gelu Also see https://arxiv.org/abs/1606.08415 ??gHm?)tftanhnpsqrtpipow)xZcdfrB/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_tf_xlnet.pygelu.s2rcCs|t|SN)rZsigmoid)rrrrswish7sr)rrelurcsXeZdZfddZfddZddZddd Zdd d ZdddZdddZ Z S)TFXLNetRelativeAttentionc stjf||j|_|j|jdkr:td|j|jf|j|_|j|_|j|_d|jd|_|j|_t j j j |j dd|_t j j |j|_dS)NrzLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rr layer_normepsilonname)super__init__output_attentionsd_modeln_head ValueErrord_headscaleinitializer_rangerkeraslayersLayerNormalizationlayer_norm_epsrDropoutdropoutselfconfigkwargs __class__rrr$Cs z!TFXLNetRelativeAttention.__init__cs,t|j}|j|j|j|jf|ddd|_|j|j|j|jf|ddd|_|j|j|j|jf|ddd|_|j|j|j|jf|ddd|_ |j|j|j|jf|ddd|_ |j|j|jfddd d|_ |j|j|jfddd d|_ |j|j|jfddd d|_ |jd |j|jf|dd d|_t|dS)NTqshape initializerZ trainabler"kvorzerosr_r_biasr_s_biasr_w_biasr seg_embed)r r+ add_weightr&r'r)r8r<r=r>r?rArBrCrDr#buildr3 input_shaper;r6rrrFVsp     zTFXLNetRelativeAttention.buildcCstdSrNotImplementedError)r3Zheadsrrr prune_headsusz$TFXLNetRelativeAttention.prune_headscCst|}t||d|d|d|df}|dddf}t||d|dd|d|df}|ddd|ddddf}|S)zijbnr)rMNrzibnd,snd->ibnszijbs,ibns->ijbnigꌠ9Y>)FZaxistrainingzijbn,jbnd->ibnd)reinsumrCrArNr rBrDr*dtypeZfloat16nnZsoftmaxr1r%)r3inputsrQZq_headk_head_hv_head_hk_head_rseg_mat attn_mask head_maskacZbdZefZ attn_score attn_probattn_vecrrr rel_attn_cores*  z&TFXLNetRelativeAttention.rel_attn_coreTcCs@|\}}td||j}|j||d}|r2||}||}|S)zPost-attention processing.z ibnd,hnd->ibhrP)rrRr>r1r)r3rUZresidualrQhr^Zattn_outoutputrrrpost_attentions z'TFXLNetRelativeAttention.post_attentionc CsT|\ }}}}}}} } } |dk r| dk rLtt| dkrLtj| |gdd} n|} td| |j} td| |j}td||j}td||j}|j || ||||| g|d}|j r|\}}|j ||g|d}td||j}| dk r0td|| }|j || ||||| g|d}|j r |\}}td|| }n,|j || ||||| g|d}|j r\|\}}|j ||g|d}|j r6||f}n| dk rtt| dkrtj| |gdd} n|} td||j}td| |j} td| |j}td||j}|j || ||||| g|d}|j r |\}}|j ||g|d}d}||f}|j rP||f}|S)NrrrOz ibh,hnd->ibndrPzmbnd,mlb->lbndzlbnd,mlb->mbnd) lenr rconcatrRr<r=r?r8r_r%rb)r3rUrQr`gZ attn_mask_hZ attn_mask_gr?rYmemstarget_mappingr[catrVrWrXZq_head_hZ attn_vec_hZ attn_prob_houtput_hZq_head_gZ attn_vec_gZ attn_prob_goutput_gr]r^outputsrrrcallsl    zTFXLNetRelativeAttention.call)rL)F)TF)F) __name__ __module__ __qualname__r$rFrKrNr_rbrl __classcell__rrr6rrBs   - rcs&eZdZfddZdddZZS)TFXLNetFeedForwardc stjf|tjjj|jdd|_tjjj|j t |j dd|_ tjjj|j t |j dd|_tjj|j|_t|jtrt|j|_n|j|_dS)Nrr layer_1Zkernel_initializerr"layer_2)r#r$rr,r-r.r/rDenseZd_innerr r+rrr&rtr0r1 isinstanceZ ff_activationstrACT2FNactivation_functionr2r6rrr$"s  zTFXLNetFeedForward.__init__FcCsP|}||}||}|j||d}||}|j||d}|||}|S)NrP)rrryr1rtr)r3inprQrarrrrl1s   zTFXLNetFeedForward.call)Frmrnror$rlrprrr6rrq!s rqcs&eZdZfddZdddZZS) TFXLNetLayerc s@tjf|t|dd|_t|dd|_tjj |j |_ dS)Nrel_attnr"ff) r#r$rr}rqrrr,r-r0r1r2r6rrr$=szTFXLNetLayer.__init__FcCsZ|j||d}|dd\}}|dk r4|j||d}|j||d}||f|dd}|S)NrPr)r}r)r3rUrQrkrirjrrrrlCszTFXLNetLayer.call)Fr{rrr6rr|<s r|cs0eZdZfddZfddZddZZS) TFXLNetLMHeadc s tjf||j|_||_dSr)r#r$ vocab_sizeinput_embeddings)r3r4rr5r6rrr$PszTFXLNetLMHead.__init__cs(|j|jfdddd|_t|dS)Nr@Tbiasr9)rErrr#rF)r3rHr6rrrFWszTFXLNetLMHead.buildcCs|j|dd}||j}|S)NZlinear)mode)rr)r3 hidden_statesrrrrl[s zTFXLNetLMHead.call)rmrnror$rFrlrprrr6rrOs  rc sxeZdZeZfddZddZddZddZd d Z e j fd d Z d dZ edddZdddZdddZZS)TFXLNetMainLayerc stjf|j|_j|_j|_j|_j|_j|_j|_j |_ j |_ j |_ j |_ j |_ tjjj dd|_fddtj D|_tjjj|_dS)Nword_embeddingr+r"csg|]}td|dqS)z layer_._{}r~)r|format).0ir4rr xsz-TFXLNetMainLayer.__init__..)r#r$r%output_hidden_statesmem_len reuse_lenr& same_length attn_typebi_data clamp_lenn_layer use_bfloat16r+rrrrangelayerrr,r-r0r1r2r6rrr$es*zTFXLNetMainLayer.__init__cCs|jSr)rr3rrrget_input_embeddings{sz%TFXLNetMainLayer.get_input_embeddingscCs*t|j}|jdd|jf|ddd|_dS)NrTmask_embr9)r r+rEr&rrGrrrrF~s  zTFXLNetMainLayer.buildcCstdSrrI)r3Znew_num_tokensrrr_resize_token_embeddingssz)TFXLNetMainLayer._resize_token_embeddingscCstdSrrI)r3Zheads_to_prunerrr _prune_headsszTFXLNetMainLayer._prune_headsc Cstj||g|d}t|dd}t|dd}tj||g|d}t|||gd}|jrt|dd} t|ddd|f| ||dd|dfgd}|S)a Creates causal attention mask. Float mask where 1.0 indicates masked, 0.0 indicates not-masked. Args: qlen: TODO Lysandre didn't fill mlen: TODO Lysandre didn't fill :: same_length=False: same_length=True: < qlen > < qlen > ^ [0 0 0 0 0 1 1 1 1] [0 0 0 0 0 1 1 1 1] [0 0 0 0 0 0 1 1 1] [1 0 0 0 0 0 1 1 1] qlen [0 0 0 0 0 0 0 1 1] [1 1 0 0 0 0 0 1 1] [0 0 0 0 0 0 0 0 1] [1 1 1 0 0 0 0 0 1] v [0 0 0 0 0 0 0 0 0] [1 1 1 1 0 0 0 0 0] rSrrLrN)ronesZmatrix_band_partr@rdr) r3qlenmlenrSrZZmask_uZmask_diaZ attn_mask_padretZmask_lrrr create_masks8zTFXLNetMainLayer.create_maskcCsb|jdk r"|jdkr"|d|j}|dkr<||j d}nt||gd|j d}t|S)z cache hidden states into memory.Nr)rrrrdZ stop_gradient)r3Zcurr_outZprev_memZnew_memrrr cache_mems zTFXLNetMainLayer.cache_memNcCs`td||}tjt|t|gdd}|dddddf}|dk r\t|d|dg}|S)Nzi,d->idrLrOr)rrRrdsincostile)Zpos_seqinv_freqbszZ sinusoid_inppos_embrrrpositional_embeddings z%TFXLNetMainLayer.positional_embeddingcCstd|jd}|dk r0|tjkr0tj||d}dd||j}|jdkrZ|| }}n&|jdkrp|d }}ntd |j|jrxt||d } t| | d } |dk r|tjkrtj| |d} tj| |d} |j dkr t | |j |j } t | |j |j } |dk rL|d dks&t | | ||d } | | ||d } n| | |} | | |} tj | | gdd} n`t||d } |dk r|tjkrtj| |d} |j dkrt | |j |j } | | ||} | S)z$create relative positional encoding.rg@Nrri'biunirLzUnknown `attn_type` {}.grrrO)rrr&float32castrr(rrrZ clip_by_valueAssertionErrorrrd)r3rrMrrSZfreq_seqrbegendZ fwd_pos_seqZ bwd_pos_seqZ fwd_pos_embZ bwd_pos_embrrrrrelative_positional_encodings@        z-TFXLNetMainLayer.relative_positional_encodingTFc % CsPt|ttfr|d} t|dkr,|dn|}t|dkrD|dn|}t|dkr\|dn|}t|dkrt|dn|}t|dkr|dn|}t|dkr|dn|}t|dkr|dn|}t|d kr|d n| } t|d kr|d n| } t|d kstd nt|ttfr|d } |d|}|d|}|d|}|d|}|d|}|d|}|d|}|d| } |d| } t|d kstd n|} | dk r| dk rtdnd| dk rt j | dd} t | dd\} }n6| dk r$t j | dd} t | dd\} }ntd|dk rDt j |ddnd}|dk r`t j |ddnd}|dk r|t j |ddnd}|dk rt j |ddnd}|dk rt j |ddnd}|dk r|ddk rt |ddnd}|| }|j rt j nt j}|jdkr2|| |}|ddddddf}n"|jdkrDd}ntd|j|dksp|dksptd |dkr|dk rd!t j||d"}|dk r|dk r|d|}n<|dk r|dkr|d}n|dkr|dk r|}nd}|dk rtt jt |d||g|d"}t j||gdd#}|dkrT|dddddddf}n ||dddddddf7}|dk rt j|dk|d"}|dk rt j| |d" }t jt j| |g|d"|gd$d#}t j||ddddddfdk|d"}nd}| dk r | }n || }|j|| d%}|dk rXt |jt |d|dg}|j|| d%}nd}|dk rt j||gt jd"}t ||gd}t t t |dddf|dddft j}t j|d|d"}nd}|j| |||d&}|j|| d%}|dk rz|dkr>| d d d d}|!|j"d$d$d$d$}n$|dkrb| d d d}|j#t$|%j&d"}n dg|j"}d'}|dkrdgt|j'}g}g} t(|j'D]\}!}"|j)dk r|j)dkr| d(kr||*|||!f}|j+r| ,|dk r||fn||"||||||||!|||!g | d%}#|#dd\}}|j-r|,|#dq|j+r| ,|dk r||fn||j|dk r|n|| d%}$t j |$ddf}#|j)dk r|j)dkr| d(kr|#|f}#|j+r(|dk r td)d*| D} ntd+d*| D} |#| f}#|j-rLtd,d*|D}|#|f}#|#S)-Nrrrr zToo many inputs. input_idsattention_maskrf perm_maskrgtoken_type_ids input_maskr[ inputs_embeds use_cachezDYou cannot specify both input_ids and inputs_embeds at the same time)rrpermrrrz5You have to specify either input_ids or inputs_embeds)rrrrrzUnsupported attention type: {}zYou can only use one of input_mask (uses 1 for padding) or attention_mask (uses 0 for padding, added for compatbility with BERT). Please choose one.rrrOrLrP)rrSrTcss&|]}|D]}tj|ddVq qdSrrNr transpose)rhsr`rrr sz(TFXLNetMainLayer.call..css|]}tj|ddVqdSrr)rrrrrrscss|]}tj|ddVqdS))rrrrrNr)rtrrrrs).rvtuplelistrcrdictr getr(rrr rZbfloat16rrrrrr@rdZeyerr1rrint32Z logical_notequalZone_hotrZdimZ unsqueezeexpandrtonext parametersrSr enumeraterrrappendr%)%r3rUrrfrrgrrr[rrrQrrrrrMZ dtype_floatrZZ data_maskZ mems_maskZ non_tgt_maskZ word_emb_kriZ word_emb_qrjZmem_padZcat_idsrYrZnew_memsZ attentionsrrZ layer_modulerkrarrrrls             ,          ,    4    ""    zTFXLNetMainLayer.call)N)NN) NNNNNNNNTF)rmrnror config_classr$rrFrrrrrr staticmethodrrrlrprrr6rras,   3rc@seZdZdZeZeZdZdS)TFXLNetPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerN) rmrnro__doc__rr%TF_XLNET_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZbase_model_prefixrrrrrsra .. note:: TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` Parameters: config (:class:`~transformers.XLNetConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.XLNetTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ mems (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `mems` output below). Can be used to speed up sequential decoding. The token ids which have their mems given to this model should not be passed as input ids as they have already been computed. perm_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to indicate the attention pattern for each input token with values selected in ``[0, 1]``: If ``perm_mask[k, i, j] = 0``, i attend to j in batch k; if ``perm_mask[k, i, j] = 1``, i does not attend to j in batch k. If None, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, num_predict, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to indicate the output tokens to use. If ``target_mapping[k, i, j] = 1``, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ input_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding. Kept for compatibility with the original code base. You can only uses one of `input_mask` and `attention_mask` Mask values selected in ``[0, 1]``: ``1`` for tokens that are MASKED, ``0`` for tokens that are NOT MASKED. head_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (:obj:`bool`): If `use_cache` is True, `mems` are returned and can be used to speed up decoding (see `mems`). Defaults to `True`. z^The bare XLNet Model transformer outputing raw hidden-states without any specific head on top.cs,eZdZfddZeeddZZS) TFXLNetModelcs&tj|f||t|dd|_dS)Nrr~)r#r$rrr3r4rUr5r6rrr$szTFXLNetModel.__init__cKs|j|f|}|S)a( Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: last_hidden_state (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the last layer of the model. mems (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import XLNetTokenizer, TFXLNetModel tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') model = TFXLNetModel.from_pretrained('xlnet-large-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple )r)r3rUr5rkrrrrl"s#zTFXLNetModel.callrmrnror$rXLNET_INPUTS_DOCSTRINGrlrprrr6rrs rzoXLNet Model with a language modeling head on top (linear layer with weights tied to the input embeddings). cs<eZdZfddZddZddZeeddZZ S) TFXLNetLMHeadModelcs:tj|f||t|dd|_t||jjdd|_dS)Nrr~lm_loss)r#r$rrrrrrr6rrr$OszTFXLNetLMHeadModel.__init__cCs|jjSr)rrrrrrget_output_embeddingsTsz(TFXLNetLMHeadModel.get_output_embeddingsc Ks|jd}tj|dftjd}tj||gdd}|jd}tj|||dftjd}tj||dftjd}tj||gdd}tj|d|dftjd} tj|ddftjd} tj| | gdd} ||| |dd}|r||d<|S) NrrrrOrLr)rUrrgrrf)r:rr@rrdrr) r3rUZpastr5Zeffective_batch_sizeZ dummy_tokenZsequence_lengthrZperm_mask_seq_endrgZtarget_mapping_seq_endrrrprepare_inputs_for_generationWs$  z0TFXLNetLMHeadModel.prepare_inputs_for_generationcKs6|j|f|}|d}||}|f|dd}|S)a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: prediction_scores (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). mems (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf import numpy as np from transformers import XLNetTokenizer, TFXLNetLMHeadModel tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') model = TFXLNetLMHeadModel.from_pretrained('xlnet-large-cased') # We show how to setup inputs to predict a next token using a bi-directional context. input_ids = tf.constant(tokenizer.encode("Hello, my dog is very ", add_special_tokens=True))[None, :] # We will predict the masked token perm_mask = np.zeros((1, input_ids.shape[1], input_ids.shape[1])) perm_mask[:, :, -1] = 1.0 # Previous tokens don't see last token target_mapping = np.zeros((1, 1, input_ids.shape[1])) # Shape [1, 1, seq_length] => let's predict one token target_mapping[0, 0, -1] = 1.0 # Our first (and only) prediction will be the last token of the sequence (the masked token) outputs = model(input_ids, perm_mask=tf.constant(perm_mask, dtype=tf.float32), target_mapping=tf.constant(target_mapping, dtype=tf.float32)) next_token_logits = outputs[0] # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size] rrN)rr)r3rUr5transformer_outputsZ hidden_statelogitsrkrrrrlvs + zTFXLNetLMHeadModel.call) rmrnror$rrrrrlrprrr6rrIs  rzXLNet Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs,eZdZfddZeeddZZS) TFXLNetForSequenceClassificationcs^tj|f|||j|_t|dd|_t||jdd|_tj j j |jt |jdd|_ dS)Nrr~sequence_summaryr logits_projrs)r#r$ num_labelsrrrr+rrr,r-rur rrr6rrr$sz)TFXLNetForSequenceClassification.__init__cKs@|j|f|}|d}||}||}|f|dd}|S)a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: logits (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). mems (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import XLNetTokenizer, TFXLNetForSequenceClassification tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') model = TFXLNetForSequenceClassification.from_pretrained('xlnet-large-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) logits = outputs[0] rrN)rrrr3rUr5rrarrkrrrrls #  z%TFXLNetForSequenceClassification.callrrrr6rrs rzXLNet Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cs$eZdZfddZddZZS)TFXLNetForTokenClassificationcsLtj|f|||j|_t|dd|_tjjj|jt |j dd|_ dS)Nrr~ classifierrs) r#r$rrrrr,r-rur r+rrr6rrr$sz&TFXLNetForTokenClassification.__init__cKs6|j|f|}|d}||}|f|dd}|S)a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: logits (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:(batch_size, config.num_labels)`): Classification scores (before SoftMax). mems (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import XLNetTokenizer, TFXLNetForTokenClassification tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') model = TFXLNetForTokenClassification.from_pretrained('xlnet-large-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :] # Batch size 1 outputs = model(input_ids) scores = outputs[0] rrN)rrrrrrrls " z"TFXLNetForTokenClassification.callr{rrr6rrs rzXLNet Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs,eZdZfddZeeddZZS)!TFXLNetForQuestionAnsweringSimplecsDtj|f||t|dd|_tjjj|jt |j dd|_ dS)Nrr~ qa_outputsrs) r#r$rrrr,r-rurr r+rrr6rrr$+sz*TFXLNetForQuestionAnsweringSimple.__init__c Ksh|j|f|}|d}||}tj|ddd\}}tj|dd}tj|dd}||f|dd}|S)ap Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.XLNetConfig`) and inputs: loss (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_scores (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). mems (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` or :obj:`Numpy array` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import XLNetTokenizer, TFXLNetForQuestionAnsweringSimple tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased') model = TFXLNetForQuestionAnsweringSimple.from_pretrained('xlnet-base-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) start_scores, end_scores = outputs[:2] rrrLrOrN)rrrsplitZsqueeze) r3rUr5rZsequence_outputrZ start_logitsZ end_logitsrkrrrrl2s'  z&TFXLNetForQuestionAnsweringSimple.callrrrr6rr%s r).rloggingZnumpyrZ tensorflowrZconfiguration_xlnetrZ file_utilsrrZmodeling_tf_utilsrrrr r r Ztokenization_utilsr getLoggerrmloggerrrrr,r-Z ActivationZ activationsrrxZLayerrrqr|rrrZXLNET_START_DOCSTRINGrrrrrrrrrrsl       `\ 8,\;6