U &c[b@sZdZddlZddlZddlmZddlmZmZddl m Z m Z m Z ddl mZmZmZeeZdd d d d ZGd dde ZGddde ZGdddeZdZdZedeGdddeZGdddejjjZedeGdddeZGdddejjjZ edeGd d!d!eZ!ed"eGd#d$d$eZ"ed%eGd&d'd'eZ#dS)(z TF 2.0 RoBERTa model. N) RobertaConfig)add_start_docstrings add_start_docstrings_to_callable)TFBertEmbeddingsTFBertMainLayergelu)TFPreTrainedModelget_initializer shape_listz3https://cdn.huggingface.co/roberta-base-tf_model.h5z4https://cdn.huggingface.co/roberta-large-tf_model.h5z9https://cdn.huggingface.co/roberta-large-mnli-tf_model.h5z9https://cdn.huggingface.co/distilroberta-base-tf_model.h5)z roberta-basez roberta-largezroberta-large-mnlizdistilroberta-basecs>eZdZdZfddZddZddZd fd d ZZS) TFRobertaEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. c stj|f|d|_dS)Nr)super__init__ padding_idxselfconfigkwargs __class__D/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_tf_roberta.pyr,szTFRobertaEmbeddings.__init__cCs:tjtj||jtjd}tjj|dd|}||jS)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. :param tf.Tensor x: :return tf.Tensor: ZdtyperZaxis)tfcastmath not_equalrint32Zcumsum)rxmaskZincremental_indiciesrrr"create_position_ids_from_input_ids0sz6TFRobertaEmbeddings.create_position_ids_from_input_idscCs@t|d}tj|jd||jdtjdtjddf}|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. :param tf.Tensor inputs_embeds: :return tf.Tensor: rrN)r rrangerrZnewaxis)r inputs_embedsZ seq_length position_idsrrr&create_position_ids_from_inputs_embeds;s 0z:TFRobertaEmbeddings.create_position_ids_from_inputs_embedsFcsJ|\}}}}|dkr2|dk r(||}n ||}tj||||g|dS)z)Applies embedding based on inputs tensor.Ntraining)r!r%r _embedding)rinputsr'Z input_idsr$Ztoken_type_idsr#rrrr(Fs    zTFRobertaEmbeddings._embedding)F) __name__ __module__ __qualname____doc__rr!r%r( __classcell__rrrrr 's    r cs(eZdZdZfddZddZZS)TFRobertaMainLayerz? Same as TFBertMainLayer but uses TFRobertaEmbeddings. c s"tj|f|t|dd|_dS)N embeddingsname)r rr r0rrrrrYszTFRobertaMainLayer.__init__cCs|jSN)r0rrrrget_input_embeddings]sz'TFRobertaMainLayer.get_input_embeddings)r*r+r,r-rr5r.rrrrr/Ts r/c@seZdZdZeZeZdZdS)TFRobertaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. robertaN) r*r+r,r-rZ config_class'TF_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZbase_model_prefixrrrrr6asr6a This model is a `tf.keras.Model `__ sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. .. note:: TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` Parameters: config (:class:`~transformers.RobertaConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.RobertaTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`__ position_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`__ head_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. inputs_embeds (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, embedding_dim)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. training (:obj:`boolean`, `optional`, defaults to :obj:`False`): Whether to activate dropout modules (if set to :obj:`True`) during training or to de-activate them (if set to :obj:`False`) for evaluation. z`The bare RoBERTa Model transformer outputing raw hidden-states without any specific head on top.cs,eZdZfddZeeddZZS)TFRobertaModelcs&tj|f||t|dd|_dS)Nr7r1)r rr/r7rrr)rrrrrszTFRobertaModel.__init__cKs|j|f|}|S)a Returns: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (:obj:`tf.Tensor` of shape :obj:`(batch_size, hidden_size)`): Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during Bert pretraining. This output is usually *not* a good summary of the semantic content of the input, you're often better with averaging or pooling the sequence of hidden-states for the whole input sequence. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import RobertaTokenizer, TFRobertaModel tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = TFRobertaModel.from_pretrained('roberta-base') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple )r7)rr)routputsrrrcalls%zTFRobertaModel.callr*r+r,rrROBERTA_INPUTS_DOCSTRINGr<r.rrrrr9s r9cs4eZdZdZfddZfddZddZZS)TFRobertaLMHeadz*Roberta Head for masked language modeling.c sdtjf||j|_tjjj|jt|j dd|_ tjjj |j dd|_ tjjt|_||_dS)Ndensekernel_initializerr2 layer_norm)epsilonr2)r r vocab_sizerkeraslayersDense hidden_sizer initializer_ranger@ZLayerNormalizationZlayer_norm_epsrCZ Activationractdecoder)rrZinput_embeddingsrrrrrszTFRobertaLMHead.__init__cs(|j|jfdddd|_t|dS)NzerosTbias)shapeZ initializerZ trainabler2)Z add_weightrErNr build)rZ input_shaperrrrPszTFRobertaLMHead.buildcCs6||}||}||}|j|dd|j}|S)NZlinear)mode)r@rKrCrLrN)rfeaturesrrrrr<s    zTFRobertaLMHead.call)r*r+r,r-rrPr<r.rrrrr?s r?z6RoBERTa Model with a `language modeling` head on top. cs4eZdZfddZddZeeddZZS)TFRobertaForMaskedLMcs:tj|f||t|dd|_t||jjdd|_dS)Nr7r1lm_head)r rr/r7r?r0rTr:rrrrszTFRobertaForMaskedLM.__init__cCs|jjSr3)rTrLr4rrrget_output_embeddings sz*TFRobertaForMaskedLM.get_output_embeddingscKs6|j|f|}|d}||}|f|dd}|S)as Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: prediction_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import RobertaTokenizer, TFRobertaForMaskedLM tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = TFRobertaForMaskedLM.from_pretrained('roberta-base') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) prediction_scores = outputs[0] rN)r7rT)rr)rr;sequence_outputZprediction_scoresrrrr<s  zTFRobertaForMaskedLM.call) r*r+r,rrUrr>r<r.rrrrrSs rScs*eZdZdZfddZdddZZS)TFRobertaClassificationHeadz-Head for sentence-level classification tasks.c sdtj|f|tjjj|jt|jddd|_ tjj |j |_ tjjj|j t|jdd|_dS)Ntanhr@)rBZ activationr2out_projrA)r rrrFrGrHrIr rJr@Dropouthidden_dropout_probdropout num_labelsrZrrrrr9sz$TFRobertaClassificationHead.__init__FcCsJ|dddddf}|j||d}||}|j||d}||}|S)Nrr&)r]r@rZ)rrRr'rrrrr<Fs   z TFRobertaClassificationHead.call)F)r*r+r,r-rr<r.rrrrrX6s rXzRoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs,eZdZfddZeeddZZS)"TFRobertaForSequenceClassificationcs<tj|f|||j|_t|dd|_t|dd|_dS)Nr7r1 classifier)r rr^r/r7rXr`r:rrrrUsz+TFRobertaForSequenceClassification.__init__cKsB|j|f|}|d}|j||ddd}|f|dd}|S)a Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: logits (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import RobertaTokenizer, TFRobertaForSequenceClassification tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = TFRobertaForSequenceClassification.from_pretrained('roberta-base') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 labels = tf.constant([1])[None, :] # Batch size 1 outputs = model(input_ids) logits = outputs[0] rr'Fr&rVN)r7r`getrr)rr;rWlogitsrrrr<\s z'TFRobertaForSequenceClassification.callr=rrrrr_Os r_zRoBERTa Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cs,eZdZfddZeeddZZS)TFRobertaForTokenClassificationcs^tj|f|||j|_t|dd|_tjj|j |_ tjjj |jt |j dd|_dS)Nr7r1r`rA)r rr^r/r7rrFrGr[r\r]rHr rJr`r:rrrrsz(TFRobertaForTokenClassification.__init__cKsL|j|f|}|d}|j||ddd}||}|f|dd}|S)a8 Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import RobertaTokenizer, TFRobertaForTokenClassification tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = TFRobertaForTokenClassification.from_pretrained('roberta-base') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) scores = outputs[0] rr'Fr&rVN)r7r]rar`rbrrrr<s  z$TFRobertaForTokenClassification.callr=rrrrrds rdzRoBERTa Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs,eZdZfddZeeddZZS)TFRobertaForQuestionAnsweringcsLtj|f|||j|_t|dd|_tjjj|jt |j dd|_ dS)Nr7r1 qa_outputsrA) r rr^r/r7rrFrGrHr rJrfr:rrrrsz&TFRobertaForQuestionAnswering.__init__cKsh|j|f|}|d}||}tj|ddd\}}tj|dd}tj|dd}||f|dd}|S)a@ Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: start_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: # The checkpoint roberta-base is not fine-tuned for question answering. Please see the # examples/question-answering/run_squad.py example to see how to fine-tune a model to a question answering task. import tensorflow as tf from transformers import RobertaTokenizer, TFRobertaForQuestionAnswering tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = TFRobertaForQuestionAnswering.from_pretrained('roberta-base') input_ids = tokenizer.encode("Who was Jim Henson?", "Jim Henson was a nice puppet") start_scores, end_scores = model(tf.constant(input_ids)[None, :]) # Batch size 1 all_tokens = tokenizer.convert_ids_to_tokens(input_ids) answer = ' '.join(all_tokens[tf.math.argmax(start_scores, 1)[0] : tf.math.argmax(end_scores, 1)[0]+1]) rrVrN)r7rfrsplitZsqueeze)rr)rr;rWrcZ start_logitsZ end_logitsrrrr<s% z"TFRobertaForQuestionAnswering.callr=rrrrres re)$r-loggingZ tensorflowrZconfiguration_robertarZ file_utilsrrZmodeling_tf_bertrrrZmodeling_tf_utilsr r r getLoggerr*loggerr8r r/r6ZROBERTA_START_DOCSTRINGr>r9rFrGZLayerr?rSrXr_rdrerrrrsR  -  ).215