U &cޗ@sdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZmZddlmZmZeeZd d d d d dddZddZGdddejZGdddejZGdddejZGdddejZGdddejZGdddeZdZ d Z!ed!e Gd"d#d#eZ"ed$e Gd%d&d&eZ#ed'e Gd(d)d)eZ$ed*e Gd+d,d,eZ%ed-e Gd.d/d/eZ&dS)0z PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) N)CrossEntropyLoss)gelu)DistilBertConfig)add_start_docstrings add_start_docstrings_to_callable)PreTrainedModelprune_linear_layerzDhttps://cdn.huggingface.co/distilbert-base-uncased-pytorch_model.binzThttps://cdn.huggingface.co/distilbert-base-uncased-distilled-squad-pytorch_model.binzBhttps://cdn.huggingface.co/distilbert-base-cased-pytorch_model.binzRhttps://cdn.huggingface.co/distilbert-base-cased-distilled-squad-pytorch_model.binzIhttps://cdn.huggingface.co/distilbert-base-german-cased-pytorch_model.binzOhttps://cdn.huggingface.co/distilbert-base-multilingual-cased-pytorch_model.binz\https://cdn.huggingface.co/distilbert-base-uncased-finetuned-sst-2-english-pytorch_model.bin)zdistilbert-base-uncasedz'distilbert-base-uncased-distilled-squadzdistilbert-base-casedz%distilbert-base-cased-distilled-squadzdistilbert-base-german-casedz"distilbert-base-multilingual-casedz/distilbert-base-uncased-finetuned-sst-2-englishc stfddt|D}tt|dddddf|dddddf<tt|dddddf|dddddf<|d|_dS)Ncs$g|]fddtDqS)c s(g|] }tdd|dqS)i')nppower).0j)dimposD/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_distilbert.py 6sz;create_sinusoidal_embeddings...)range)r r)rrr6sz0create_sinusoidal_embeddings..rr rF) r arrayrtorchZ FloatTensorsincosZdetach_ requires_grad)n_posroutZ position_encrrrcreate_sinusoidal_embeddings5s 44rcs$eZdZfddZddZZS) Embeddingscstttj|j|j|jd|_t|j|j|_ |j rPt |j|j|j j dtj |jdd|_ t|j|_dS)N)Z padding_idx)rrr-q=eps)super__init__nn Embedding vocab_sizerZ pad_token_idword_embeddingsZmax_position_embeddingsposition_embeddingsZsinusoidal_pos_embdsrweight LayerNormDropoutdropoutselfconfig __class__rrr#>s zEmbeddings.__init__cCsb|d}tj|tj|jd}|d|}||}||}||}| |}| |}|S)a7 Parameters ---------- input_ids: torch.tensor(bs, max_seq_length) The token ids to embed. Outputs ------- embeddings: torch.tensor(bs, max_seq_length, dim) The embedded tokens (plus position embeddings, no token_type embeddings) r)Zdtypedevicer) sizerarangelongr2Z unsqueeze expand_asr'r(r*r,)r. input_idsZ seq_lengthZ position_idsr'r( embeddingsrrrforwardJs     zEmbeddings.forward__name__ __module__ __qualname__r#r9 __classcell__rrr0rr=s rcs.eZdZfddZddZdddZZS) MultiHeadSelfAttentioncst|j|_|j|_tj|jd|_|j|_|j|jdksFt tj |j|jd|_ tj |j|jd|_ tj |j|jd|_ tj |j|jd|_t|_dS)NprZ in_featuresZ out_features)r"r#n_headsrr$r+Zattention_dropoutr,output_attentionsAssertionErrorLinearq_link_linv_linout_linset pruned_headsr-r0rrr#ds zMultiHeadSelfAttention.__init__cs|j|j}t|dkrdSt|j|}t||j}|D](tfdd|jD8d|<q<|d d}t t|| }t |j||_t |j||_t |j||_t |j|dd|_|jt||_||j|_|j||_dS)Nrc3s|]}|krdndVqdS)rrNr)r hheadrr |sz5MultiHeadSelfAttention.prune_heads..rr)rrClenronesrKrLsumview contiguouseqr4r5r rGrHrIrJunion)r.headsZattention_head_sizemaskindexrrNr prune_headsus"    z"MultiHeadSelfAttention.prune_headsNcs$|\}}|d}jjdd|f} fdd} fdd} | |} | |} | |}| t} t | | dd}|dk |  |}| |td  tjd d |}|}|d k r||}t ||}| |}|}jr||fS|fSd S) a Parameters ---------- query: torch.tensor(bs, seq_length, dim) key: torch.tensor(bs, seq_length, dim) value: torch.tensor(bs, seq_length, dim) mask: torch.tensor(bs, seq_length) Outputs ------- weights: torch.tensor(bs, n_heads, seq_length, seq_length) Attention weights context: torch.tensor(bs, seq_length, dim) Contextualized layer. Optional: only if `output_attentions=True` rcs|djddS)z separate heads rQrr )rUrC transposexbsZ dim_per_headr.rrshapesz-MultiHeadSelfAttention.forward..shapecs |dddjS)z group heads rr rQ)r]rVrUrCr^r`rrunshapesz/MultiHeadSelfAttention.forward..unshaper rinfrQrN)r3rrCrGrHrImathsqrtrmatmulr]rUr6Z masked_fill_floatr$ZSoftmaxr,rJrD)r.querykeyvaluerZ head_maskZq_lengthrZk_lengthZ mask_reshprbrcqkvZscoresweightscontextrr`rr9s.      zMultiHeadSelfAttention.forward)N)r;r<r=r#r\r9r>rrr0rr?cs r?cs$eZdZfddZddZZS)FFNcsxttj|jd|_tj|j|jd|_tj|j|jd|_ |j dks\t d |j |j dkrjt nt|_ dS)Nr@rB)Zrelurz+activation ({}) must be in ['relu', 'gelu']r)r"r#r$r+r,rFrZ hidden_dimlin1lin2 activationrEformatrReLUr-r0rrr#s z FFN.__init__cCs,||}||}||}||}|SN)rtrvrur,)r.inputr_rrrr9s     z FFN.forwardr:rrr0rrss rscs&eZdZfddZdddZZS)TransformerBlockcsbt|j|_|j|jdks&tt||_tj |jdd|_ t ||_ tj |jdd|_ dS)Nrr)Znormalized_shaper!)r"r#rDrrCrEr? attentionr$r* sa_layer_normrsffnoutput_layer_normr-r0rrr#s   zTransformerBlock.__init__NcCs||j|||||d}|jr$|\}}nt|tks4t|d}|||}||}|||}|f}|jrx|f|}|S)a Parameters ---------- x: torch.tensor(bs, seq_length, dim) attn_mask: torch.tensor(bs, seq_length) Outputs ------- sa_weights: torch.tensor(bs, n_heads, seq_length, seq_length) The attention weights ffn_output: torch.tensor(bs, seq_length, dim) The output of the transformer block contextualization. )rjrkrlrZrmr)r|rDtypetuplerEr}r~r)r.r_ attn_maskrmZ sa_outputZ sa_weightsZ ffn_outputoutputrrrr9s   zTransformerBlock.forward)NNr:rrr0rr{s r{cs&eZdZfddZdddZZS) TransformercsNt|j|_|j|_|j|_t|tfddt|jD|_ dS)Ncsg|]}tqSr)copydeepcopy)r _layerrrrsz(Transformer.__init__..) r"r#Zn_layersrDoutput_hidden_statesr{r$Z ModuleListrrr-r0rrr#s  zTransformer.__init__Nc Csd}d}|}t|jD]l\}}|jr.||f}|||||d} | d}|jrrt| dks^t| d} || f}qt| dkstq|jr||f}|f} |jr| |f} |jr| |f} | S)a Parameters ---------- x: torch.tensor(bs, seq_length, dim) Input sequence embedded. attn_mask: torch.tensor(bs, seq_length) Attention mask on the sequence. Outputs ------- hidden_state: torch.tensor(bs, seq_length, dim) Sequence of hiddens states in the last (top) layer all_hidden_states: Tuple[torch.tensor(bs, seq_length, dim)] Tuple of length n_layers with the hidden states from each layer. Optional: only if output_hidden_states=True all_attentions: Tuple[torch.tensor(bs, n_heads, seq_length, seq_length)] Tuple of length n_layers with the attention weights from each layer Optional: only if output_attentions=True rr_rrmrQr rr) enumeraterrrDrRrE) r.r_rrmZall_hidden_statesZall_attentions hidden_stateiZ layer_moduleZ layer_outputsZ attentionsoutputsrrrr9s*     zTransformer.forward)NNr:rrr0rrs rc@s(eZdZdZeZeZdZdZ ddZ dS)DistilBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. N distilbertcCst|tjr*|jjr*|jjjd|jjdt|tj rN|jjjd|jjdn&t|tj rt|j j |jj dt|tj r|j dk r|j j dS)z! Initialize the weights. g)ZmeanZstdg?N) isinstancer$r%r)rdataZnormal_r/Zinitializer_rangerFr*ZbiasZzero_Zfill_)r.modulerrr _init_weightsNs    z'DistilBertPreTrainedModel._init_weights) r;r<r=__doc__rZ config_class'DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZload_tf_weightsZbase_model_prefixrrrrrrDs rat This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.DistilBertConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.DistilBertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. zfThe bare DistilBERT encoder/transformer outputting raw hidden-states without any specific head on top.csFeZdZfddZddZddZddZeed d d Z Z S) DistilBertModelcs,t|t||_t||_|dSry)r"r#rr8r transformer init_weightsr-r0rrr#s   zDistilBertModel.__init__cCs|jjSryr8r'r.rrrget_input_embeddingssz$DistilBertModel.get_input_embeddingscCs ||j_dSryr)r.Znew_embeddingsrrrset_input_embeddingssz$DistilBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr|r\)r.Zheads_to_prunerrYrrr _prune_headsszDistilBertModel._prune_headsNc Cs|dk r|dk rtdn4|dk r,|}n"|dk rF|dd}ntd|dk r\|jn|j}|dkrxtj||d}|||jj}|dkr||}|j |||d}|d}|f|dd} | S) a Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.DistilBertConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import DistilBertTokenizer, DistilBertModel import torch tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = DistilBertModel.from_pretrained('distilbert-base-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NzDYou cannot specify both input_ids and inputs_embeds at the same timerQz5You have to specify either input_ids or inputs_embeds)r2rrr) ValueErrorr3r2rrSZ get_head_maskr/Znum_hidden_layersr8r) r.r7attention_maskrm inputs_embedsZ input_shaper2Z tfmr_outputrrrrrr9s"!   zDistilBertModel.forward)NNNN) r;r<r=r#rrrrDISTILBERT_INPUTS_DOCSTRINGr9r>rrr0rrs  rz@DistilBert Model with a `masked language modeling` head on top. cs6eZdZfddZddZeedddZZS) DistilBertForMaskedLMcsrt||j|_|j|_t||_t|j|j|_ tj |jdd|_ t|j|j |_ |t|_dS)Nrr )r"r#rDrrrr$rFrvocab_transformr*vocab_layer_normr&vocab_projectorrr mlm_loss_fctr-r0rrr#s  zDistilBertForMaskedLM.__init__cCs|jSry)rrrrrget_output_embeddingssz+DistilBertForMaskedLM.get_output_embeddingsNc Cs|j||||d}|d}||}t|}||}||}|f|dd} |dk r||d|d|d} | f| } | S)aI masked_lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.DistilBertConfig`) and inputs: loss (`optional`, returned when ``masked_lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Masked language modeling loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import DistilBertTokenizer, DistilBertForMaskedLM import torch tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = DistilBertForMaskedLM.from_pretrained('distilbert-base-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, masked_lm_labels=input_ids) loss, prediction_scores = outputs[:2] r7rrmrrrNrQ)rrrrrrrUr3) r.r7rrmrZmasked_lm_labelsZ dlbrt_output hidden_statesZprediction_logitsrZmlm_lossrrrr9s&'    zDistilBertForMaskedLM.forward)NNNNN) r;r<r=r#rrrr9r>rrr0rrs rzDistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs.eZdZfddZeedddZZS)#DistilBertForSequenceClassificationcs\t||j|_t||_t|j|j|_t|j|j|_ t |j |_ | dSry)r"r# num_labelsrrr$rFrpre_classifier classifierr+Zseq_classif_dropoutr,rr-r0rrr#/s  z,DistilBertForSequenceClassification.__init__Nc Cs|j||||d}|d}|dddf}||}t|}||}||} | f|dd} |dk r|jdkrt} | | d|d} n"t } | | d|j|d} | f| } | S)ad labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ..., config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss), If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.DistilBertConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`label` is provided): Classification (or regression if config.num_labels==1) loss. logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import DistilBertTokenizer, DistilBertForSequenceClassification import torch tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 labels = torch.tensor([1]).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, logits = outputs[:2] rrNrrQ) rrr$rxr,rrZMSELossrUr) r.r7rrmrlabelsdistilbert_outputrZ pooled_outputlogitsrloss_fctlossrrrr9:s*(      z+DistilBertForSequenceClassification.forward)NNNNNr;r<r=r#rrr9r>rrr0rr)s rzDistilBert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs.eZdZfddZeedddZZS)DistilBertForQuestionAnsweringcsPt|t||_t|j|j|_|jdks6t t |j |_ | dS)Nr )r"r#rrr$rFrr qa_outputsrEr+Z qa_dropoutr,rr-r0rrr#s   z'DistilBertForQuestionAnswering.__init__NcCs |j||||d}|d}||}||} | jddd\} } | d} | d} | | f|dd} |dk r|dk rt|dkr|d}t|dkr|d}| d} |d| |d| tj | d}|| |}|| |}||d}|f| } | S) a start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.DistilBertConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length,)`): Span-end scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import DistilBertTokenizer, DistilBertForQuestionAnswering import torch tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 start_positions = torch.tensor([1]) end_positions = torch.tensor([3]) outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) loss, start_scores, end_scores = outputs[:3] rrrrQrN) ignore_indexr ) rr,rsplitZsqueezerRr3Zclamp_r$r)r.r7rrmrZstart_positionsZ end_positionsrrrZ start_logitsZ end_logitsrZ ignored_indexrZ start_lossZend_lossZ total_lossrrrr9s66              z&DistilBertForQuestionAnswering.forward)NNNNNNrrrr0rrys rzDistilBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cs.eZdZfddZeedddZZS) DistilBertForTokenClassificationcsJt||j|_t||_t|j|_t|j |j|_ | dSry) r"r#rrrr$r+r,rFZ hidden_sizerrr-r0rrr#s   z)DistilBertForTokenClassification.__init__NcCs|j||||d}|d}||}||}|f|dd}|dk rt} |dk r|ddk} |d|j} t| |dt| j  |} | | | } n| |d|j|d} | f|}|S)aY labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels - 1]``. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.DistilBertConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when ``labels`` is provided) : Classification loss. scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.num_labels)`) Classification scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import DistilBertTokenizer, DistilBertForTokenClassification import torch tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased') model = DistilBertForTokenClassification.from_pretrained('distilbert-base-cased') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) # Batch size 1 labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, scores = outputs[:2] )rrmrrr NrQr) rr,rrrUrrwhereZtensorrZtype_as)r.r7rrmrrrZsequence_outputrrZ active_lossZ active_logitsZ active_labelsrrrrr9s0'    z(DistilBertForTokenClassification.forward)NNNNNrrrr0rrs r)'rrloggingrfZnumpyr rZtorch.nnr$rZ activationsrZconfiguration_distilbertrZ file_utilsrrZmodeling_utilsrr getLoggerr;loggerrrModulerr?rsr{rrZDISTILBERT_START_DOCSTRINGrrrrrrrrrrsl     &b/= RLKa