U &c@sdZddlZddlZddlZddlmZddlm Z m Z m Z ddl m Z mZmZmZddlmZeeZdd d d d d dddddddddddddddZddZdd Zd!d"Zejjeejjjejjeejjed#ZGd$d%d%ejjj Z!Gd&d'd'ejjj Z"Gd(d)d)ejjj Z#Gd*d+d+ejjj Z$Gd,d-d-ejjj Z%Gd.d/d/ejjj Z&Gd0d1d1ejjj Z'Gd2d3d3ejjj Z(Gd4d5d5ejjj Z)Gd6d7d7ejjj Z*Gd8d9d9ejjj Z+Gd:d;d;ejjj Z,Gdd?d?ejjj Z.Gd@dAdAe Z/dBZ0dCZ1e dDe0GdEdFdFe/Z2e dGe0GdHdIdIe/Z3e dJe0GdKdLdLe/Z4e dMe0GdNdOdOe/Z5e dPe0GdQdRdRe/Z6e dSe0GdTdUdUe/Z7e dVe0GdWdXdXe/Z8e dYe0GdZd[d[e/Z9dS)\z TF 2.0 BERT model. N) BertConfig)MULTIPLE_CHOICE_DUMMY_INPUTSadd_start_docstrings add_start_docstrings_to_callable)TFPreTrainedModelget_initializerkeras_serializable shape_list) BatchEncodingz8https://cdn.huggingface.co/bert-base-uncased-tf_model.h5z9https://cdn.huggingface.co/bert-large-uncased-tf_model.h5z6https://cdn.huggingface.co/bert-base-cased-tf_model.h5z7https://cdn.huggingface.co/bert-large-cased-tf_model.h5zEhttps://cdn.huggingface.co/bert-base-multilingual-uncased-tf_model.h5zChttps://cdn.huggingface.co/bert-base-multilingual-cased-tf_model.h5z8https://cdn.huggingface.co/bert-base-chinese-tf_model.h5z=https://cdn.huggingface.co/bert-base-german-cased-tf_model.h5zLhttps://cdn.huggingface.co/bert-large-uncased-whole-word-masking-tf_model.h5zJhttps://cdn.huggingface.co/bert-large-cased-whole-word-masking-tf_model.h5z\https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-tf_model.h5zZhttps://cdn.huggingface.co/bert-large-cased-whole-word-masking-finetuned-squad-tf_model.h5zEhttps://cdn.huggingface.co/bert-base-cased-finetuned-mrpc-tf_model.h5zChttps://cdn.huggingface.co/cl-tohoku/bert-base-japanese/tf_model.h5zVhttps://cdn.huggingface.co/cl-tohoku/bert-base-japanese-whole-word-masking/tf_model.h5zHhttps://cdn.huggingface.co/cl-tohoku/bert-base-japanese-char/tf_model.h5z[https://cdn.huggingface.co/cl-tohoku/bert-base-japanese-char-whole-word-masking/tf_model.h5zJhttps://cdn.huggingface.co/TurkuNLP/bert-base-finnish-cased-v1/tf_model.h5zLhttps://cdn.huggingface.co/TurkuNLP/bert-base-finnish-uncased-v1/tf_model.h5zEhttps://cdn.huggingface.co/wietsedv/bert-base-dutch-cased/tf_model.h5)zbert-base-uncasedzbert-large-uncasedzbert-base-casedzbert-large-casedzbert-base-multilingual-uncasedzbert-base-multilingual-casedzbert-base-chinesezbert-base-german-casedz%bert-large-uncased-whole-word-maskingz#bert-large-cased-whole-word-maskingz5bert-large-uncased-whole-word-masking-finetuned-squadz3bert-large-cased-whole-word-masking-finetuned-squadzbert-base-cased-finetuned-mrpczbert-base-japanesez%bert-base-japanese-whole-word-maskingzbert-base-japanese-charz*bert-base-japanese-char-whole-word-maskingzbert-base-finnish-cased-v1zbert-base-finnish-uncased-v1zbert-base-dutch-casedcCs(ddtj|tjd}||S)a Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415 ??g@)tfmatherfsqrtxZcdfrA/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_tf_bert.pygelu9s rc Cs:ddttdtj|dt|d}||S)zGaussian Error Linear Unit. This is a smoother version of the RELU. Original paper: https://arxiv.org/abs/1606.08415 Args: x: float Tensor to perform activation. Returns: `x` with the GELU activation applied. r r gHm?)rtanhnprpipowrrrrgelu_newDs 2rcCs|t|SN)rZsigmoid)rrrrswishQsr)rrelurrcsHeZdZdZfddZfddZddd Zdd d Zd d ZZ S)TFBertEmbeddingszLConstruct the embeddings from word, position and token_type embeddings. c stjf||j|_|j|_|j|_tjjj|j |jt |jdd|_ tjjj|j |jt |jdd|_ tjjj|jdd|_tjj|j|_dS)Nposition_embeddings)Zembeddings_initializernametoken_type_embeddings LayerNormepsilonr#)super__init__ vocab_size hidden_sizeinitializer_rangerkeraslayersZ EmbeddingZmax_position_embeddingsrr"Ztype_vocab_sizer$LayerNormalizationlayer_norm_epsr%Dropouthidden_dropout_probdropoutselfconfigkwargs __class__rrr)as$ zTFBertEmbeddings.__init__c sFtd&|jd|j|jgt|jd|_W5QRXt |dS)z"Build shared word embedding layer word_embeddingsZweight)shape initializerN) rZ name_scope add_weightr*r+rr,r:r(buildr5 input_shaper8rrr>ys  zTFBertEmbeddings.build embeddingFcCs:|dkr|j||dS|dkr(||Std|dS)a Get token embeddings of inputs. Args: inputs: list of three int64 tensors with shape [batch_size, length]: (input_ids, position_ids, token_type_ids) mode: string, a valid value is one of "embedding" and "linear". Returns: outputs: (1) If mode == "embedding", output embedding tensor, float32 with shape [batch_size, length, embedding_size]; (2) mode == "linear", output linear tensor, float32 with shape [batch_size, length, vocab_size]. Raises: ValueError: if mode is not valid. Shared weights logic adapted from https://github.com/tensorflow/models/blob/a009f4fb9d2fc4949e32192a944688925ef78659/official/transformer/v2/embedding_layer.py#L24 rAtraininglinearzmode {} is not valid.N) _embedding_linear ValueErrorformat)r5inputsmoderCrrrcalls  zTFBertEmbeddings.callc Cs|\}}}}|dk rt|}nt|dd}|d}|dkr\tj|tjdtjddf}|dkrpt|d}|dkrt|j|}||} | |} || | } | | } |j | |d} | S)z)Applies embedding based on inputs tensor.Nr)ZdtyperrB) r rrangeZint32newaxisfillZgatherr:r"r$r%r3) r5rIrC input_ids position_idstoken_type_ids inputs_embedsr@ seq_lengthr"r$ embeddingsrrrrEs"       zTFBertEmbeddings._embeddingcCsPt|d}t|d}t|d|jg}tj||jdd}t||||jgS)a Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_size] Returns: float32 tensor with shape [batch_size, length, vocab_size]. rrrLTZ transpose_b)r rreshaper+matmulr:r*)r5rI batch_sizelengthrlogitsrrrrFs   zTFBertEmbeddings._linear)rAF)F) __name__ __module__ __qualname____doc__r)r>rKrErF __classcell__rrr8rr!]s    r!cs.eZdZfddZddZdddZZS) TFBertSelfAttentionc stjf||j|jdkr2td|j|jf|j|_|j|_|j|jdksVtt|j|j|_|j|j|_ t j j j |j t|jdd|_t j j j |j t|jdd|_t j j j |j t|jdd|_t j j |j|_dS)NrzLThe hidden size (%d) is not a multiple of the number of attention heads (%d)querykernel_initializerr#keyvalue)r(r)r+num_attention_headsrGoutput_attentionsAssertionErrorintattention_head_size all_head_sizerr-r.Denserr,rbrerfr1Zattention_probs_dropout_probr3r4r8rrr)s8 zTFBertSelfAttention.__init__cCs.t||d|j|jf}tj|ddddgdS)NrLrrrrperm)rrWrgrk transpose)r5rrYrrrtranspose_for_scoressz(TFBertSelfAttention.transpose_for_scoresFcCs|\}}}t|d}||}||}||} |||} |||} || |} tj| | dd} tt| dtj}| tj |} |dk r| |} tj j | dd}|j ||d}|dk r||}t|| }tj|dddd gd }t||d|jf}|jr||fn|f}|S) NrTrVrLZaxisrBrrrrn)r rbrerfrqrrXcastfloat32rrnnZsoftmaxr3rprWrlrh)r5rIrC hidden_statesattention_mask head_maskrYZmixed_query_layerZmixed_key_layerZmixed_value_layerZ query_layerZ key_layerZ value_layerZattention_scoresZdkZattention_probsZ context_layeroutputsrrrrKs:          zTFBertSelfAttention.call)F)r\r]r^r)rqrKr`rrr8rras racs&eZdZfddZdddZZS)TFBertSelfOutputc sXtjf|tjjj|jt|jdd|_ tjjj |j dd|_ tjj |j|_dSNdensercr%r&r(r)rr-r.rmr+rr,r|r/r0r%r1r2r3r4r8rrr)szTFBertSelfOutput.__init__FcCs2|\}}||}|j||d}|||}|SNrBr|r3r%r5rIrCrv input_tensorrrrrKs  zTFBertSelfOutput.call)Fr\r]r^r)rKr`rrr8rrzs rzcs.eZdZfddZddZdddZZS) TFBertAttentionc s.tjf|t|dd|_t|dd|_dS)Nr5r#output)r(r)raself_attentionrz dense_outputr4r8rrr)#szTFBertAttention.__init__cCstdSrNotImplementedError)r5Zheadsrrr prune_heads(szTFBertAttention.prune_headsFc CsJ|\}}}|j|||g|d}|j|d|g|d}|f|dd}|SNrBrr)rr) r5rIrCrrwrxZ self_outputsattention_outputryrrrrK+s  zTFBertAttention.call)F)r\r]r^r)rrKr`rrr8rr"s rcs$eZdZfddZddZZS)TFBertIntermediatec sRtjf|tjjj|jt|jdd|_ t |j t rFt |j |_n|j |_dS)Nr|rc)r(r)rr-r.rmZintermediate_sizerr,r| isinstance hidden_actstrACT2FNintermediate_act_fnr4r8rrr)5s zTFBertIntermediate.__init__cCs||}||}|Sr)r|rr5rvrrrrK?s  zTFBertIntermediate.callrrrr8rr4s rcs&eZdZfddZdddZZS) TFBertOutputc sXtjf|tjjj|jt|jdd|_ tjjj |j dd|_ tjj |j|_dSr{r}r4r8rrr)FszTFBertOutput.__init__FcCs2|\}}||}|j||d}|||}|Sr~rrrrrrKNs  zTFBertOutput.call)Frrrr8rrEs rcs&eZdZfddZdddZZS) TFBertLayerc s<tjf|t|dd|_t|dd|_t|dd|_dS)N attentionr intermediater)r(r)rrrrr bert_outputr4r8rrr)XszTFBertLayer.__init__Fc CsX|\}}}|j|||g|d}|d}||}|j||g|d} | f|dd} | Sr)rrr) r5rIrCrvrwrxZattention_outputsrZintermediate_outputZ layer_outputryrrrrK^s  zTFBertLayer.call)Frrrr8rrWs rcs&eZdZfddZdddZZS) TFBertEncoderc s<tjf|j|_j|_fddtjD|_dS)Ncsg|]}td|dqS)z layer_._{}r)rrH).0ir6rr nsz*TFBertEncoder.__init__..)r(r)rhoutput_hidden_statesrMnum_hidden_layerslayerr4r8rrr)jszTFBertEncoder.__init__Fc Cs|\}}}d}d}t|jD]J\}} |jr4||f}| ||||g|d} | d}|jr|| df}q|jrx||f}|f} |jr| |f} |jr| |f} | S)NrrBrr) enumeraterrrh) r5rIrCrvrwrxZall_hidden_statesZall_attentionsrZ layer_moduleZ layer_outputsryrrrrKps$     zTFBertEncoder.call)Frrrr8rris rcs$eZdZfddZddZZS) TFBertPoolerc s2tjf|tjjj|jt|jddd|_ dS)Nrr|)rdZ activationr#) r(r)rr-r.rmr+rr,r|r4r8rrr)szTFBertPooler.__init__cCs|dddf}||}|S)Nr)r|)r5rvZfirst_token_tensor pooled_outputrrrrKs zTFBertPooler.callrrrr8rrs rcs$eZdZfddZddZZS)TFBertPredictionHeadTransformc shtjf|tjjj|jt|jdd|_ t |j t rFt |j |_n|j |_tjjj|jdd|_dSr{)r(r)rr-r.rmr+rr,r|rrrrtransform_act_fnr/r0r%r4r8rrr)s z&TFBertPredictionHeadTransform.__init__cCs"||}||}||}|Sr)r|rr%rrrrrKs   z"TFBertPredictionHeadTransform.callrrrr8rrs rcs0eZdZfddZfddZddZZS)TFBertLMPredictionHeadc s.tjf||j|_t|dd|_||_dS)N transformr)r(r)r*rrinput_embeddingsr5r6rr7r8rrr)szTFBertLMPredictionHead.__init__cs(|j|jfdddd|_t|dS)NzerosTbias)r;r<Z trainabler#)r=r*rr(r>r?r8rrr>szTFBertLMPredictionHead.buildcCs&||}|j|dd}||j}|S)NrD)rJ)rrrrrrrrKs  zTFBertLMPredictionHead.call)r\r]r^r)r>rKr`rrr8rrs rcs$eZdZfddZddZZS) TFBertMLMHeadc s"tjf|t||dd|_dS)N predictionsr)r(r)rrrr8rrr)szTFBertMLMHead.__init__cCs||}|Sr)r)r5sequence_outputprediction_scoresrrrrKs zTFBertMLMHead.callrrrr8rrs rcs$eZdZfddZddZZS) TFBertNSPHeadc s.tjf|tjjjdt|jdd|_dS)Nrseq_relationshiprc) r(r)rr-r.rmrr,rr4r8rrr)s zTFBertNSPHead.__init__cCs||}|Sr)r)r5rseq_relationship_scorerrrrKs zTFBertNSPHead.callrrrr8rrs rcsBeZdZeZfddZddZddZddZd d d Z Z S)TFBertMainLayerc sDtjf||j|_t|dd|_t|dd|_t|dd|_dS)NrUrencoderpooler) r(r)rr!rUrrrrr4r8rrr)s zTFBertMainLayer.__init__cCs|jSr)rUr5rrrget_input_embeddingssz$TFBertMainLayer.get_input_embeddingscCstdSrr)r5Znew_num_tokensrrr_resize_token_embeddingssz(TFBertMainLayer._resize_token_embeddingscCstdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel Nr)r5Zheads_to_prunerrr _prune_headsszTFBertMainLayer._prune_headsNFcCs>t|ttfr|d}t|dkr*|dn|}t|dkrB|dn|}t|dkrZ|dn|}t|dkrr|dn|}t|dkr|dn|}t|dkstdnrt|ttfr|d }|d |}|d |}|d |}|d |}|d|}t|dkstdn|}|dk r4|dk r4tdn8|dk rHt |} n$|dk rdt |dd} ntd|dkrt | d}|dkrt | d}|ddt j t j ddf} t | t j} d| d} |dk rtn dg|j}|j||||g|d} |j| | |g|d} | d} || }| |f| dd}|S)NrrrrToo many inputs.rPrwrRrQrxrSzDYou cannot specify both input_ids and inputs_embeds at the same timerLz5You have to specify either input_ids or inputs_embedsr grB)rtuplelistlenridictr getrGr rrOrNrsrtrrrUrr)r5rIrwrRrQrxrSrCrPr@Zextended_attention_maskZembedding_outputZencoder_outputsrrryrrrrKsT                    zTFBertMainLayer.call)NNNNNF) r\r]r^r config_classr)rrrrKr`rrr8rrs  rc@seZdZdZeZeZdZdS)TFBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. bertN) r\r]r^r_rr$TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZbase_model_prefixrrrrrFsra This model is a `tf.keras.Model `__ sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. .. note:: TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` Parameters: config (:class:`~transformers.BertConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`__ position_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`__ head_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. training (:obj:`boolean`, `optional`, defaults to :obj:`False`): Whether to activate dropout modules (if set to :obj:`True`) during training or to de-activate them (if set to :obj:`False`) for evaluation. z]The bare Bert Model transformer outputing raw hidden-states without any specific head on top.cs,eZdZfddZeeddZZS) TFBertModelcs&tj|f||t|dd|_dS)Nrr)r(r)rrr5r6rIr7r8rrr)szTFBertModel.__init__cKs|j|f|}|S)a Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (:obj:`tf.Tensor` of shape :obj:`(batch_size, hidden_size)`): Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during Bert pretraining. This output is usually *not* a good summary of the semantic content of the input, you're often better with averaging or pooling the sequence of hidden-states for the whole input sequence. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import BertTokenizer, TFBertModel tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = TFBertModel.from_pretrained('bert-base-uncased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple )r)r5rIr7ryrrrrKs%zTFBertModel.callr\r]r^r)rBERT_INPUTS_DOCSTRINGrKr`rrr8rrs rzBert Model with two heads on top as done during the pre-training: a `masked language modeling` head and a `next sentence prediction (classification)` head. cs4eZdZfddZddZeeddZZS)TFBertForPreTrainingcsHtj|f||t|dd|_t|dd|_t||jjdd|_dS)Nrr nsp___cls mlm___cls) r(r)rrrnsprrUmlmrr8rrr)szTFBertForPreTraining.__init__cCs|jjSrrrUrrrrget_output_embeddingssz*TFBertForPreTraining.get_output_embeddingscKsV|j|f|}|dd\}}|j||ddd}||}||f|dd}|S)ae Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import BertTokenizer, TFBertForPreTraining tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = TFBertForPreTraining.from_pretrained('bert-base-uncased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) prediction_scores, seq_relationship_scores = outputs[:2] NrrCFrB)rrrr)r5rIr7ryrrrrrrrrKs   zTFBertForPreTraining.call r\r]r^r)rrrrKr`rrr8rrs rz3Bert Model with a `language modeling` head on top. cs4eZdZfddZddZeeddZZS)TFBertForMaskedLMcs:tj|f||t|dd|_t||jjdd|_dS)Nrrr)r(r)rrrrUrrr8rrr)szTFBertForMaskedLM.__init__cCs|jjSrrrrrrrsz'TFBertForMaskedLM.get_output_embeddingscKsB|j|f|}|d}|j||ddd}|f|dd}|S)an Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: prediction_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import BertTokenizer, TFBertForMaskedLM tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = TFBertForMaskedLM.from_pretrained('bert-base-uncased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) prediction_scores = outputs[0] rrCFrBrN)rrr)r5rIr7ryrrrrrrKs zTFBertForMaskedLM.callrrrr8rrs rzKBert Model with a `next sentence prediction (classification)` head on top. cs,eZdZfddZeeddZZS)TFBertForNextSentencePredictioncs4tj|f||t|dd|_t|dd|_dS)Nrrr)r(r)rrrrrr8rrr)=sz(TFBertForNextSentencePrediction.__init__cKs6|j|f|}|d}||}|f|dd}|S)a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: seq_relationship_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, 2)`) Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import BertTokenizer, TFBertForNextSentencePrediction tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = TFBertForNextSentencePrediction.from_pretrained('bert-base-uncased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) seq_relationship_scores = outputs[0] rrN)rr)r5rIr7ryrrrrrrKCs  z$TFBertForNextSentencePrediction.callrrrr8rr9s rzBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs,eZdZfddZeeddZZS)TFBertForSequenceClassificationcs^tj|f|||j|_t|dd|_tjj|j |_ tjjj |jt |j dd|_dSNrr classifierrcr(r) num_labelsrrrr-r.r1r2r3rmrr,rrr8rrr)qsz(TFBertForSequenceClassification.__init__cKsL|j|f|}|d}|j||ddd}||}|f|dd}|S)aP Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: logits (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import BertTokenizer, TFBertForSequenceClassification tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) logits = outputs[0] rrCFrBrNrr3rr)r5rIr7ryrr[rrrrK{s  z$TFBertForSequenceClassification.callrrrr8rrks rzBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cs:eZdZfddZeddZeed ddZZ S) TFBertForMultipleChoicecsTtj|f||t|dd|_tjj|j|_ tjjj dt |j dd|_ dS)Nrrrrrc)r(r)rrrr-r.r1r2r3rmrr,rrr8rrr)sz TFBertForMultipleChoice.__init__cCsdttiS)zf Dummy inputs to build the network. Returns: tf.Tensor with dummy inputs rP)rZconstantrrrrr dummy_inputssz$TFBertForMultipleChoice.dummy_inputsNFcCs*t|ttfr|d}t|dkr*|dn|}t|dkrB|dn|}t|dkrZ|dn|}t|dkrr|dn|}t|dkr|dn|}t|dkstdnnt|tr|d }|d |}|d |}|d |}|d |}|d|}t|dkstdn|}|dk r6t|d} t|d} nt|d} t|d} |dk rht |d| fnd} |dk rt |d| fnd} |dk rt |d| fnd} |dk rt |d| fnd}| | | |||g}|j ||d}|d}|j ||d}| |}t |d| f}|f|dd}|S)a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: classification_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, num_choices)`: `num_choices` is the size of the second dimension of the input tensors. (see `input_ids` above). Classification scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import BertTokenizer, TFBertForMultipleChoice tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = TFBertForMultipleChoice.from_pretrained('bert-base-uncased') choices = ["Hello, my dog is cute", "Hello, my cat is amazing"] input_ids = tf.constant([tokenizer.encode(s) for s in choices])[None, :] # Batch size 1, 2 choices outputs = model(input_ids) classification_scores = outputs[0] rrrrrrrrrPrwrRrQrxrSNrLrB) rrrrrirrr rrWrr3r)r5rIrwrRrQrxrSrCrPZ num_choicesrTZflat_input_idsZflat_attention_maskZflat_token_type_idsZflat_position_idsZ flat_inputsryrr[Zreshaped_logitsrrrrKsP*             zTFBertForMultipleChoice.call)NNNNNF) r\r]r^r)propertyrrrrKr`rrr8rrs  rzBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cs,eZdZfddZeeddZZS)TFBertForTokenClassificationcs^tj|f|||j|_t|dd|_tjj|j |_ tjjj |jt |j dd|_dSrrrr8rrr)"sz%TFBertForTokenClassification.__init__cKsL|j|f|}|d}|j||ddd}||}|f|dd}|S)a3 Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import BertTokenizer, TFBertForTokenClassification tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = TFBertForTokenClassification.from_pretrained('bert-base-uncased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) scores = outputs[0] rrCFrBrNr)r5rIr7ryrr[rrrrK,s  z!TFBertForTokenClassification.callrrrr8rrs rzBert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs,eZdZfddZeeddZZS)TFBertForQuestionAnsweringcsLtj|f|||j|_t|dd|_tjjj|jt |j dd|_ dS)Nrr qa_outputsrc) r(r)rrrrr-r.rmrr,rrr8rrr)\sz#TFBertForQuestionAnswering.__init__cKsh|j|f|}|d}||}tj|ddd\}}tj|dd}tj|dd}||f|dd}|S)ad Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: start_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import BertTokenizer, TFBertForQuestionAnswering tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = TFBertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad') question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" encoding = tokenizer.encode_plus(question, text) input_ids, token_type_ids = encoding["input_ids"], encoding["token_type_ids"] start_scores, end_scores = model(tf.constant(input_ids)[None, :], token_type_ids=tf.constant(token_type_ids)[None, :]) all_tokens = tokenizer.convert_ids_to_tokens(input_ids) answer = ' '.join(all_tokens[tf.math.argmax(tf.squeeze(start_scores)) : tf.math.argmax(tf.squeeze(end_scores))+1]) assert answer == "a nice puppet" rrrLrrN)rrrsplitZsqueeze)r5rIr7ryrr[Z start_logitsZ end_logitsrrrrKes& zTFBertForQuestionAnswering.callrrrr8rrVs r):r_loggingZnumpyrZ tensorflowrZconfiguration_bertrZ file_utilsrrrZmodeling_tf_utilsrrr r Ztokenization_utilsr getLoggerr\loggerrrrrr-r.Z ActivationZ activationsr rZLayerr!rarzrrrrrrrrrrrrZBERT_START_DOCSTRINGrrrrrrrrrrrrrs        gL"  j ).82/5r5