U &cT@sdZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z mZmZmZddlmZeeZdd d d d d dZddZddZGdddejjjZGdddejjjZGdddejjjZGdddejjjZGdddejjjZ GdddejjjZ!Gdd d e Z"d!Z#d"Z$e d#e#Gd$d%d%e"Z%Gd&d'd'ejjjZ&e d(e#Gd)d*d*e"Z'e d+e#Gd,d-d-e"Z(e d.e#Gd/d0d0e"Z)e d1e#Gd2d3d3e"Z*dS)4z TF 2.0 DistilBERT model N)DistilBertConfig)add_start_docstrings add_start_docstrings_to_callable)TFPreTrainedModelTFSharedEmbeddingsget_initializer shape_list) BatchEncodingz>https://cdn.huggingface.co/distilbert-base-uncased-tf_model.h5zNhttps://cdn.huggingface.co/distilbert-base-uncased-distilled-squad-tf_model.h5z.shapecs"ttj|dddjfS)z group heads rhrirN)r rPrkr_rlrmrrunshapesz.TFMultiHeadSelfAttention.call..unshapeTrOgꌠ9Y>)Fr rNZaxisrGN)r r)r_rYr\r]rrr rQrPnnZsoftmaxr.r^r`)r0rCr=querykeyvaluemask head_maskZq_lengthr)Zk_lengthZ mask_reshaper5roqkvZscoresweightscontextrrmrrEs0      zTFMultiHeadSelfAttention.call)F)rTrUrVr'rgrErWrrr3rrXs rXcs&eZdZfddZdddZZS)TFFFNc stjf|tjj|j|_tjjj|jt |j dd|_ tjjj|j t |j dd|_ |jdksvtd|j|jdkrtjjtntjjj|_dS)Nlin1rZlin2)relurz+activation ({}) must be in ['relu', 'gelu']r)r&r'r r*r+r-r.rb hidden_dimrrr}r)r~ activationrarB ActivationrZ activationsrr/r3rrr's" zTFFFN.__init__FcCs0||}||}||}|j||d}|S)NrG)r}rr~r.)r0inputr=rrrrrEs    z TFFFN.call)FrTrUrVr'rErWrrr3rr|s r|cs&eZdZfddZdddZZS)TFTransformerBlockc stjf||j|_|j|_|j|_tjj|j |_ |j |_ |j |_ |j|jdks\t t |dd|_tjjjddd|_t|dd|_tjjjddd|_dS) Nr attentionr r" sa_layer_normr$ffnoutput_layer_norm)r&r'r_r)rr r*r+r-r.rr`rarXrr,rr|rrr/r3rrr'szTFTransformerBlock.__init__Fc Cs~|\}}}|j|||||g|d}|jr2|\}}n|d}|||}|j||d}|||}|f} |jrz|f| } | S)a Parameters ---------- x: tf.Tensor(bs, seq_length, dim) attn_mask: tf.Tensor(bs, seq_length) Outputs ------- sa_weights: tf.Tensor(bs, n_heads, seq_length, seq_length) The attention weights ffn_output: tf.Tensor(bs, seq_length, dim) The output of the transformer block contextualization. rGr)rr`rrr) r0rCr=r attn_maskrvZ sa_outputZ sa_weightsZ ffn_outputoutputrrrrE1s   zTFTransformerBlock.call)Frrrr3rrs rcs&eZdZfddZdddZZS) TFTransformerc sDtjf|j|_j|_j|_fddtjD|_dS)Ncsg|]}td|dqS)z layer_._{}r)rrB).0ir1rr [sz*TFTransformer.__init__..)r&r'Zn_layersr`output_hidden_statesrKlayerr/r3rrr'Us zTFTransformer.__init__FcCs|\}}}d}d}|}t|jD]p\} } |jr8||f}| |||| g|d} | d}|jrt| dkslt| d} || f}q t| dks tq |jr||f}|f} |jr| |f} |jr| |f} | S)a Parameters ---------- x: tf.Tensor(bs, seq_length, dim) Input sequence embedded. attn_mask: tf.Tensor(bs, seq_length) Attention mask on the sequence. Outputs ------- hidden_state: tf.Tensor(bs, seq_length, dim) Sequence of hiddens states in the last (top) layer all_hidden_states: Tuple[tf.Tensor(bs, seq_length, dim)] Tuple of length n_layers with the hidden states from each layer. Optional: only if output_hidden_states=True all_attentions: Tuple[tf.Tensor(bs, n_heads, seq_length, seq_length)] Tuple of length n_layers with the attention weights from each layer Optional: only if output_attentions=True rrGrNrrr) enumeraterrr`lenra)r0rCr=rrrvZall_hidden_statesZall_attentions hidden_staterZ layer_moduleZ layer_outputsZ attentionsoutputsrrrrE]s,      zTFTransformer.call)Frrrr3rrTs rcs>eZdZfddZddZddZddZd d d ZZS)TFDistilBertMainLayerc s6tjf||j|_t|dd|_t|dd|_dS)NrMr transformer)r&r'num_hidden_layersrrMrrr/r3rrr'szTFDistilBertMainLayer.__init__cCs|jSrd)rMr0rrrget_input_embeddingssz*TFDistilBertMainLayer.get_input_embeddingscCstdSrdre)r0Znew_num_tokensrrr_resize_token_embeddingssz.TFDistilBertMainLayer._resize_token_embeddingscCstdSrdre)r0Zheads_to_prunerrr _prune_headssz"TFDistilBertMainLayer._prune_headsNFc Cst|ttfrt|d}t|dkr*|dn|}t|dkrB|dn|}t|dkrZ|dn|}t|dkstdnVt|ttfr|d}|d|}|d |}|d |}t|dkstdn|}|dk r|dk rtd n6|dk rt |}n$|dk rt |dd }ntd |dkr.t |}t j |t j d}|dk rNtn dg|j}|j||d}|j|||g|d} | S)NrrrrzToo many inputs.rLattention_maskrvr<zDYou cannot specify both input_ids and inputs_embeds at the same timerNz5You have to specify either input_ids or inputs_embedsrF)r<rG)rHrIrJrradictr getrAr r ZonescastZfloat32rfrrMr) r0rCrrvr<r=rLr:Zembedding_outputZ tfmr_outputrrrrEs:           zTFDistilBertMainLayer.call)NNNF) rTrUrVr'rrrrErWrrr3rrs  rc@seZdZdZeZeZdZdS)TFDistilBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. distilbertN) rTrUrV__doc__rZ config_class*TF_DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZbase_model_prefixrrrrrsra This model is a `tf.keras.Model `__ sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. .. note:: TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` Parameters: config (:class:`~transformers.DistilBertConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. training (:obj:`boolean`, `optional`, defaults to :obj:`False`): Whether to activate dropout modules (if set to :obj:`True`) during training or to de-activate them (if set to :obj:`False`) for evaluation. zeThe bare DistilBERT encoder/transformer outputing raw hidden-states without any specific head on top.cs,eZdZfddZeeddZZS)TFDistilBertModelcs&tj|f||t|dd|_dS)Nrr)r&r'rrr0r1rCr2r3rrr'szTFDistilBertModel.__init__cKs|j|f|}|S)a{ Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers,DistilBertConfig`) and inputs: last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import DistilBertTokenizer, TFDistilBertModel tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = TFDistilBertModel.from_pretrained('distilbert-base-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :] # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple )r)r0rCr2rrrrrE szTFDistilBertModel.callrTrUrVr'rDISTILBERT_INPUTS_DOCSTRINGrErWrrr3rrs rcs0eZdZfddZfddZddZZS)TFDistilBertLMHeadc s tjf||j|_||_dSrd)r&r'r(input_embeddings)r0r1rr2r3rrr'BszTFDistilBertLMHead.__init__cs(|j|jfdddd|_t|dS)NzerosTbias)r5r6Z trainabler )r7r(rr&r8r9r3rrr8JszTFDistilBertLMHead.buildcCs|j|dd}||j}|S)Nr>)rD)rr)r0 hidden_statesrrrrENs zTFDistilBertLMHead.call)rTrUrVr'r8rErWrrr3rrAs  rz@DistilBert Model with a `masked language modeling` head on top. cs4eZdZfddZddZeeddZZS)TFDistilBertForMaskedLMcstj|f|||j|_|j|_|j|_t|dd|_tjj j |j t |j dd|_tjj t|_tjj jddd|_t||jjdd|_dS) Nrrvocab_transformrZr"vocab_layer_normr$vocab_projector)r&r'r`rr(rrr r*r+rbr)rrrrractr,rrrMrrr3rrr'Xsz TFDistilBertForMaskedLM.__init__cCs|jjSrd)rrrrrrget_output_embeddingsfsz-TFDistilBertForMaskedLM.get_output_embeddingscKsT|j|f|}|d}||}||}||}||}|f|dd}|S)a} Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers,DistilBertConfig`) and inputs: prediction_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import DistilBertTokenizer, TFDistilBertForMaskedLM tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = TFDistilBertForMaskedLM.from_pretrained('distilbert-base-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :] # Batch size 1 outputs = model(input_ids) prediction_scores = outputs[0] rrN)rrrrr)r0rCr2distilbert_outputrZprediction_logitsrrrrrEis    zTFDistilBertForMaskedLM.call) rTrUrVr'rrrrErWrrr3rrTs rzDistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs,eZdZfddZeeddZZS)%TFDistilBertForSequenceClassificationcs~tj|f|||j|_t|dd|_tjjj|j t |j ddd|_ tjjj|jt |j dd|_ tjj|j|_dS)Nrrrpre_classifier)r[rr classifierrZ)r&r' num_labelsrrr r*r+rbr)rrrrr-Zseq_classif_dropoutr.rr3rrr'sz.TFDistilBertForSequenceClassification.__init__cKsf|j|f|}|d}|dddf}||}|j||ddd}||}|f|dd}|S)a^ Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers,DistilBertConfig`) and inputs: logits (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import DistilBertTokenizer, TFDistilBertForSequenceClassification tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :] # Batch size 1 outputs = model(input_ids) logits = outputs[0] rNr=FrGr)rrr.rr)r0rCr2rrZ pooled_outputrSrrrrrEs  z*TFDistilBertForSequenceClassification.callrrrr3rrs rzDistilBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cs,eZdZfddZeeddZZS)"TFDistilBertForTokenClassificationcs^tj|f|||j|_t|dd|_tjj|j |_ tjjj |jt |j dd|_ dS)NrrrrZ)r&r'rrrr r*r+r-r.rbrrrrr3rrr'sz+TFDistilBertForTokenClassification.__init__cKsL|j|f|}|d}|j||ddd}||}|f|dd}|S)a@ Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers,DistilBertConfig`) and inputs: scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import DistilBertTokenizer, TFDistilBertForTokenClassification tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = TFDistilBertForTokenClassification.from_pretrained('distilbert-base-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :] # Batch size 1 outputs = model(input_ids) scores = outputs[0] rr=FrGrN)rr.rr)r0rCr2rZsequence_outputrSrrrrEs  z'TFDistilBertForTokenClassification.callrrrr3rrs rzDistilBert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs,eZdZfddZeeddZZS) TFDistilBertForQuestionAnsweringcsdtj|f||t|dd|_tjjj|jt |j dd|_ |jdksNt tjj |j|_dS)Nrr qa_outputsrZr)r&r'rrr r*r+rbrrrrrar-Z qa_dropoutr.rr3rrr'sz)TFDistilBertForQuestionAnswering.__init__c Ks~|j|f|}|d}|j||ddd}||}tj|ddd\}}tj|dd}tj|dd}||f|dd }|S) a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers,DistilBertConfig`) and inputs: start_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import DistilBertTokenizer, TFDistilBertForQuestionAnswering tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = TFDistilBertForQuestionAnswering.from_pretrained('distilbert-base-cased') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :] # Batch size 1 outputs = model(input_ids) start_scores, end_scores = outputs[:2] rr=FrGrrNrprN)rr.rrr splitZsqueeze) r0rCr2rrrSZ start_logitsZ end_logitsrrrrrEs  z%TFDistilBertForQuestionAnswering.callrrrr3rr s r)+rloggingrZnumpyrZ tensorflowr Zconfiguration_distilbertrZ file_utilsrrZmodeling_tf_utilsrrrr Ztokenization_utilsr getLoggerrTloggerrrrr*r+ZLayerrrXr|rrrrZDISTILBERT_START_DOCSTRINGrrrrrrrrrrrsj      fZ6=? &=;4