U &c3}@sldZddlZddlZddlmZddlmZmZddlmZddl m Z m Z ddl m Z mZmZmZmZddlmZeeZd d d d d ddZGddde ZdZdZe deGdddeZe deGdddeZGdddejZe deGdddeZe deGd d!d!eZ e d"eGd#d$d$eZ!Gd%d&d&ejZ"e d'eGd(d)d)eZ#dS)*zPyTorch RoBERTa model. N)CrossEntropyLossMSELoss) RobertaConfig)add_start_docstrings add_start_docstrings_to_callable)BertEmbeddings BertLayerNorm BertModelBertPreTrainedModelgelu)"create_position_ids_from_input_idsz9https://cdn.huggingface.co/roberta-base-pytorch_model.binz:https://cdn.huggingface.co/roberta-large-pytorch_model.binz?https://cdn.huggingface.co/roberta-large-mnli-pytorch_model.binz?https://cdn.huggingface.co/distilroberta-base-pytorch_model.binzIhttps://cdn.huggingface.co/roberta-base-openai-detector-pytorch_model.binzJhttps://cdn.huggingface.co/roberta-large-openai-detector-pytorch_model.bin)z roberta-basez roberta-largezroberta-large-mnlizdistilroberta-basezroberta-base-openai-detectorzroberta-large-openai-detectorcs6eZdZdZfddZd fdd ZddZZS) RobertaEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csHt||j|_tj|j|j|jd|_tj|j |j|jd|_ dS)N) padding_idx) super__init__Z pad_token_idrnnZ Embedding vocab_size hidden_sizeword_embeddingsZmax_position_embeddingsZposition_embeddingsselfconfig __class__A/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_roberta.pyr0s zRobertaEmbeddings.__init__NcsD|dkr0|dk r&t||j|j}n ||}tj||||dS)N)token_type_ids position_ids inputs_embeds)r rtodevice&create_position_ids_from_inputs_embedsrforward)r input_idsrrrrrrr#8s zRobertaEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. :param torch.Tensor inputs_embeds: :return torch.Tensor: Nr)Zdtyper!r)sizetorchZarangerlongr!Z unsqueezeexpand)rrZ input_shapeZsequence_lengthrrrrr"Ds z8RobertaEmbeddings.create_position_ids_from_inputs_embeds)NNNN)__name__ __module__ __qualname____doc__rr#r" __classcell__rrrrr+s  raq This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.RobertaConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.RobertaTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. zaThe bare RoBERTa Model transformer outputting raw hidden-states without any specific head on top.cs<eZdZdZeZeZdZfddZ ddZ ddZ Z S) RobertaModelz This class overrides :class:`~transformers.BertModel`. Please check the superclass for the appropriate documentation alongside usage examples. robertacs"t|t||_|dSN)rrr embeddings init_weightsrrrrrs  zRobertaModel.__init__cCs|jjSr1r2rrrrrget_input_embeddingssz!RobertaModel.get_input_embeddingscCs ||j_dSr1r4)rvaluerrrset_input_embeddingssz!RobertaModel.set_input_embeddings) r*r+r,r-r config_class$ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAPpretrained_model_archive_mapbase_model_prefixrr6r8r.rrrrr/s r/z6RoBERTa Model with a `language modeling` head on top. csBeZdZeZeZdZfddZddZ e e d ddZ Z S) RobertaForMaskedLMr0cs,t|t||_t||_|dSr1)rrr/r0 RobertaLMHeadlm_headr3rrrrrs   zRobertaForMaskedLM.__init__cCs|jjSr1)r?decoderr5rrrget_output_embeddingssz(RobertaForMaskedLM.get_output_embeddingsNc Csr|j||||||d}|d} || } | f|dd}|dk rnt} | | d|jj|d} | f|}|S)a2 masked_lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: masked_lm_loss (`optional`, returned when ``masked_lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Masked language modeling loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import RobertaTokenizer, RobertaForMaskedLM import torch tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = RobertaForMaskedLM.from_pretrained('roberta-base') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, masked_lm_labels=input_ids) loss, prediction_scores = outputs[:2] attention_maskrr head_maskrrNr%)r0r?rviewrr) rr$rCrrrDrZmasked_lm_labelsoutputssequence_outputZprediction_scoresloss_fctZmasked_lm_lossrrrr#s 0  zRobertaForMaskedLM.forward)NNNNNNN)r*r+r,rr9r:r;r<rrArROBERTA_INPUTS_DOCSTRINGr#r.rrrrr=s r=cs(eZdZdZfddZddZZS)r>z*Roberta Head for masked language modeling.csftt|j|j|_t|j|jd|_tj|j|j dd|_ t t |j |_|j|j _dS)N)ZepsF)bias)rrrLinearrdenser Zlayer_norm_eps layer_normrr@ Parameterr'zerosrKrrrrrs  zRobertaLMHead.__init__cKs*||}t|}||}||}|Sr1)rMr rNr@rfeatureskwargsxrrrr#s    zRobertaLMHead.forwardr*r+r,r-rr#r.rrrrr>s r>zRoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs:eZdZeZeZdZfddZe e dddZ Z S) RobertaForSequenceClassificationr0cs,t||j|_t||_t||_dSr1)rr num_labelsr/r0RobertaClassificationHead classifierrrrrrs  z)RobertaForSequenceClassification.__init__Nc Cs|j||||||d}|d} || } | f|dd}|dk r|jdkrjt} | | d|d} n t} | | d|j|d} | f|}|S)aC labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ..., config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss), If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`label` is provided): Classification (or regression if config.num_labels==1) loss. logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import RobertaTokenizer, RobertaForSequenceClassification import torch tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = RobertaForSequenceClassification.from_pretrained('roberta-base') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 labels = torch.tensor([1]).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, logits = outputs[:2] rBrrENrr%)r0rYrWrrFr) rr$rCrrrDrlabelsrGrHlogitsrIlossrrrr#!s&1   z(RobertaForSequenceClassification.forward)NNNNNNN r*r+r,rr9r:r;r<rrrJr#r.rrrrrVs rVzRoberta Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cs:eZdZeZeZdZfddZe e dddZ Z S)RobertaForMultipleChoicer0cs@t|t||_t|j|_t|j d|_ | dS)Nr) rrr/r0rDropouthidden_dropout_probdropoutrLrrYr3rrrrrus   z!RobertaForMultipleChoice.__init__NcCs|jd}|d|d} |dk r6|d|dnd} |dk rT|d|dnd} |dk rr|d|dnd} |j| | | | |d} | d}||}||}|d|}|f| dd} |dk rt}|||}|f| } | S)aK labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the multiple choice classification loss. Indices should be in ``[0, ..., num_choices]`` where `num_choices` is the size of the second dimension of the input tensors. (see `input_ids` above) Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: loss (:obj:`torch.FloatTensor`` of shape ``(1,)`, `optional`, returned when :obj:`labels` is provided): Classification loss. classification_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_choices)`): `num_choices` is the second dimension of the input tensors. (see `input_ids` above). Classification scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import RobertaTokenizer, RobertaForMultipleChoice import torch tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = RobertaForMultipleChoice.from_pretrained('roberta-base') choices = ["Hello, my dog is cute", "Hello, my cat is amazing"] input_ids = torch.tensor([tokenizer.encode(s, add_special_tokens=True) for s in choices]).unsqueeze(0) # Batch size 1, 2 choices labels = torch.tensor(1).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, classification_scores = outputs[:2] rr%N)rrrCrDrE)shaperFr&r0rarYr)rr$rrCrZrrDrZ num_choicesZflat_input_idsZflat_position_idsZflat_token_type_idsZflat_attention_maskrGZ pooled_outputr[Zreshaped_logitsrIr\rrrr#~s,3      z RobertaForMultipleChoice.forward)NNNNNNNr]rrrrr^ks r^zRoberta Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cs:eZdZeZeZdZfddZe e dddZ Z S)RobertaForTokenClassificationr0csJt||j|_t||_t|j|_t |j |j|_ | dSr1) rrrWr/r0rr_r`rarLrrYr3rrrrrs   z&RobertaForTokenClassification.__init__NcCs|j||||||d}|d} || } || } | f|dd}|dk rt} |dk r|ddk} | d|j} t| |dt| j  |}| | |}n| | d|j|d}|f|}|S)aQ labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels - 1]``. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when ``labels`` is provided) : Classification loss. scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.num_labels)`) Classification scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import RobertaTokenizer, RobertaForTokenClassification import torch tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = RobertaForTokenClassification.from_pretrained('roberta-base') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, scores = outputs[:2] rBrrENr%r) r0rarYrrFrWr'whereZtensor ignore_indexZtype_as)rr$rCrrrDrrZrGrHr[rIZ active_lossZ active_logitsZ active_labelsr\rrrr#s40     z%RobertaForTokenClassification.forward)NNNNNNNr]rrrrrcs rccs(eZdZdZfddZddZZS)rXz-Head for sentence-level classification tasks.cs@tt|j|j|_t|j|_t|j|j |_ dSr1) rrrrLrrMr_r`rarWout_projrrrrr6s z"RobertaClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|S)Nr)rarMr'tanhrfrQrrrr#<s     z!RobertaClassificationHead.forwardrUrrrrrX3s rXzRoberta Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs:eZdZeZeZdZfddZe e dddZ Z S)RobertaForQuestionAnsweringr0cs<t||j|_t||_t|j|j|_| dSr1) rrrWr/r0rrLr qa_outputsr3rrrrrPs   z$RobertaForQuestionAnswering.__init__Nc Cs|j||||||d} | d} || } | jddd\} } | d} | d} | | f| dd} |dk r|dk rt|dkr|d}t|dkr|d}| d}|d||d|t|d}|| |}|| |}||d}|f| } | S) af start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.RobertaConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: # The checkpoint roberta-large is not fine-tuned for question answering. Please see the # examples/question-answering/run_squad.py example to see how to fine-tune a model to a question answering task. from transformers import RobertaTokenizer, RobertaForQuestionAnswering import torch tokenizer = RobertaTokenizer.from_pretrained('roberta-base') model = RobertaForQuestionAnswering.from_pretrained('roberta-base') question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" input_ids = tokenizer.encode(question, text) start_scores, end_scores = model(torch.tensor([input_ids])) all_tokens = tokenizer.convert_ids_to_tokens(input_ids) answer = ' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]) rBrrr%)ZdimrEN)re)r0risplitZsqueezelenr&Zclamp_r)rr$rCrrrDrZstart_positionsZ end_positionsrGrHr[Z start_logitsZ end_logitsZ ignored_indexrIZ start_lossZend_lossZ total_lossrrrr#Ys8>              z#RobertaForQuestionAnswering.forward)NNNNNNNr]rrrrrhFs rh)$r-loggingr'Ztorch.nnrrrZconfiguration_robertarZ file_utilsrrZ modeling_bertrr r r r Zmodeling_utilsr getLoggerr*loggerr:rZROBERTA_START_DOCSTRINGrJr/r=Moduler>rVr^rcrXrhrrrrs`     ) &UV^`