U &ºcoã@sBddlZddlZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z mZmZmZe e¡Zddd d d d d œZd'dd„ZGdd„de ƒZGdd„dejƒZGdd„dejƒZGdd„deƒZdZdZe deƒGdd„deƒƒZe deƒGdd „d eƒƒZe d!eƒGd"d#„d#eƒƒZe d$eƒGd%d&„d&eƒƒZ dS)(éNé)Úget_activation)Ú ElectraConfig)Úadd_start_docstringsÚ add_start_docstrings_to_callable)ÚBertEmbeddingsÚ BertEncoderÚ BertLayerNormÚBertPreTrainedModelzKhttps://cdn.huggingface.co/google/electra-small-generator/pytorch_model.binzJhttps://cdn.huggingface.co/google/electra-base-generator/pytorch_model.binzKhttps://cdn.huggingface.co/google/electra-large-generator/pytorch_model.binzOhttps://cdn.huggingface.co/google/electra-small-discriminator/pytorch_model.binzNhttps://cdn.huggingface.co/google/electra-base-discriminator/pytorch_model.binzOhttps://cdn.huggingface.co/google/electra-large-discriminator/pytorch_model.bin)zgoogle/electra-small-generatorzgoogle/electra-base-generatorzgoogle/electra-large-generatorz"google/electra-small-discriminatorz!google/electra-base-discriminatorz"google/electra-large-discriminatorÚ discriminatorc Cszddl}ddl}ddl}Wn tk r<t d¡‚YnXtj |¡}t  d  |¡¡|j   |¡}g} g} |D]<\} } t  d  | | ¡¡|j   || ¡} |  | ¡|  | ¡qrt| | ƒD]Z\} } | }z t|tƒrâ|  dd¡} |dkr|  d d ¡} |  d d ¡} |  d d ¡} |  dd¡} |  d¡} tdd„| DƒƒrNt  d  |¡¡Wqº|}| D]Ü}| d|¡rv| d|¡}n|g}|ddks˜|ddkr¤t|dƒ}nj|ddksÀ|ddkrÌt|dƒ}nB|ddkræt|dƒ}n(|ddkrt|dƒ}nt||dƒ}t|ƒdkrVt|d ƒ}||}qV| d!¡rLt|dƒ}n|dkr`| | ¡} z|j| jksxt|ƒ‚Wn<tk r¶}z|j|j| jf7_‚W5d}~XYnXtd"  | ¡|ƒt  | ¡|_!Wqºt"k r}ztd  |¡| |ƒWY¢qºW5d}~XYqºXqº|S)#z- Load tf checkpoints in a pytorch model. rNz™Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}zelectra/embeddings/zgenerator/embeddings/Ú generatorzelectra/zdiscriminator/z generator/Zdense_1Údense_predictionz!generator_predictions/output_biaszgenerator_lm_head/biasú/css|]}|dkVqdS))Z global_stepZ temperatureN©)Ú.0ÚnrrúA/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_electra.pyÚ Dsz-load_tf_weights_in_electra..z Skipping {}z [A-Za-z]+_\d+z_(\d+)ÚkernelÚgammaZweightZ output_biasÚbetaZbiasZoutput_weightsZsquadÚ classifierérZ _embeddingszInitialize PyTorch weight {})#ÚreZnumpyZ tensorflowÚ ImportErrorÚloggerÚerrorÚosÚpathÚabspathÚinfoÚformatZtrainZlist_variablesZ load_variableÚappendÚzipÚ isinstanceÚElectraForMaskedLMÚreplaceÚsplitÚanyÚ fullmatchÚgetattrÚlenÚintÚendswithZ transposeÚshapeÚAssertionErrorÚargsÚprintÚtorchZ from_numpyÚdataÚAttributeError)ÚmodelÚconfigZtf_checkpoint_pathZdiscriminator_or_generatorrÚnpÚtfZtf_pathZ init_varsÚnamesZarraysÚnamer.ÚarrayÚ original_nameÚpointerZm_nameZ scope_namesÚnumÚerrrÚload_tf_weights_in_electras‚ ÿ                       r@cs eZdZdZ‡fdd„Z‡ZS)ÚElectraEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cs^tƒ |¡tj|j|j|jd|_t |j|j¡|_ t |j |j¡|_ t |j|j d|_dS)N)Z padding_idx)Zeps)ÚsuperÚ__init__ÚnnZ EmbeddingÚ vocab_sizeÚembedding_sizeZ pad_token_idÚword_embeddingsZmax_position_embeddingsZposition_embeddingsZtype_vocab_sizeZtoken_type_embeddingsr Zlayer_norm_epsÚ LayerNorm©Úselfr6©Ú __class__rrrCns  zElectraEmbeddings.__init__)Ú__name__Ú __module__Ú __qualname__Ú__doc__rCÚ __classcell__rrrKrrAksrAcs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚElectraDiscriminatorPredictionszEPrediction module for the discriminator, made up of two dense layers.cs6tƒ ¡t |j|j¡|_t |jd¡|_||_dS)Nr)rBrCrDÚLinearÚ hidden_sizeÚdenser r6rIrKrrrC|s z(ElectraDiscriminatorPredictions.__init__cCs,| |¡}t|jjƒ|ƒ}| |¡ ¡}|S©N)rUrr6Z hidden_actr Zsqueeze)rJÚdiscriminator_hidden_statesÚattention_maskÚ hidden_statesÚlogitsrrrÚforwardƒs z'ElectraDiscriminatorPredictions.forward©rMrNrOrPrCr[rQrrrKrrRys rRcs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚElectraGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.cs,tƒ ¡t|jƒ|_t |j|j¡|_dSrV) rBrCr rFrHrDrSrTrUrIrKrrrCŽs  z$ElectraGeneratorPredictions.__init__cCs$| |¡}tdƒ|ƒ}| |¡}|S)NZgelu)rUrrH)rJÚgenerator_hidden_statesrYrrrr[”s   z#ElectraGeneratorPredictions.forwardr\rrrKrr]‹s r]c@s eZdZdZeZeZeZ dZ dS)ÚElectraPreTrainedModelz† An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ÚelectraN) rMrNrOrPrÚ config_classÚ$ELECTRA_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapr@Zload_tf_weightsZbase_model_prefixrrrrr_œs r_ap This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.ElectraConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. aÊ Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.ElectraTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. a\The bare Electra Model transformer outputting raw hidden-states without any specific head on top. Identical to the BERT model except that it uses an additional linear layer between the embedding layer and the encoder if the hidden size and embedding size are different.Both the generator and discriminator checkpoints may be loaded into this model.csJeZdZeZ‡fdd„Zdd„Zdd„Zdd„Ze e ƒd d d „ƒZ ‡Z S) Ú ElectraModelcsPtƒ |¡t|ƒ|_|j|jkr4t |j|j¡|_t |ƒ|_ ||_ |  ¡dSrV) rBrCrAÚ embeddingsrFrTrDrSÚembeddings_projectrÚencoderr6Ú init_weightsrIrKrrrCìs    zElectraModel.__init__cCs|jjSrV©rdrG©rJrrrÚget_input_embeddings÷sz!ElectraModel.get_input_embeddingscCs ||j_dSrVrh)rJÚvaluerrrÚset_input_embeddingsúsz!ElectraModel.set_input_embeddingscCs*| ¡D]\}}|jj|j |¡qdS)z¥ Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)ÚitemsrfÚlayerZ attentionZ prune_heads)rJZheads_to_prunernZheadsrrrÚ _prune_headsýszElectraModel._prune_headsNc Csê|dk r|dk rtdƒ‚n4|dk r,| ¡}n"|dk rF| ¡dd…}ntdƒ‚|dk r\|jn|j}|dkrxtj||d}|dkr’tj|tj|d}| |||¡} | ||j j ¡}|j ||||d} t |dƒrÖ|  | ¡} |j| | |d } | S) aÐ Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.ElectraConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import ElectraModel, ElectraTokenizer import torch tokenizer = ElectraTokenizer.from_pretrained('google/electra-small-discriminator') model = ElectraModel.from_pretrained('google/electra-small-discriminator') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NzDYou cannot specify both input_ids and inputs_embeds at the same timeéÿÿÿÿz5You have to specify either input_ids or inputs_embeds)Údevice)Zdtyperq)Ú input_idsÚ position_idsÚtoken_type_idsÚ inputs_embedsre)rXÚ head_mask)Ú ValueErrorÚsizerqr2ZonesÚzerosÚlongZget_extended_attention_maskZ get_head_maskr6Znum_hidden_layersrdÚhasattrrerf) rJrrrXrtrsrvruZ input_shaperqZextended_attention_maskrYrrrr[s0)  ÿ  zElectraModel.forward)NNNNNN) rMrNrOrrarCrjrlrorÚELECTRA_INPUTS_DOCSTRINGr[rQrrrKrrcàs  ùrczÊ Electra model with a binary classification head on top as used during pre-training for identifying generated tokens. It is recommended to load the discriminator checkpoint into that model.cs.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚElectraForPreTrainingcs,tƒ |¡t|ƒ|_t|ƒ|_| ¡dSrV)rBrCrcr`rRÚdiscriminator_predictionsrgrIrKrrrCVs   zElectraForPreTraining.__init__NcCsÄ| ||||||¡}|d} | | |¡} | f} |dk r°t ¡} |dk rŠ| d| jd¡dk} |  d| jd¡| }|| }| || ¡ƒ}n| |  d| jd¡| ¡ƒ}|f| } | |dd…7} | S)aË labels (``torch.LongTensor`` of shape ``(batch_size, sequence_length)``, `optional`, defaults to :obj:`None`): Labels for computing the ELECTRA loss. Input should be a sequence of tokens (see :obj:`input_ids` docstring) Indices should be in ``[0, 1]``. ``0`` indicates the token is an original token, ``1`` indicates the token was replaced. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.ElectraConfig`) and inputs: loss (`optional`, returned when ``labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Total loss of the ELECTRA objective. scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`) Prediction scores of the head (scores for each token before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import ElectraTokenizer, ElectraForPreTraining import torch tokenizer = ElectraTokenizer.from_pretrained('google/electra-small-discriminator') model = ElectraForPreTraining.from_pretrained('google/electra-small-discriminator') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) prediction_scores, seq_relationship_scores = outputs[:2] rNrpr)r`r~rDZBCEWithLogitsLossÚviewr.Úfloat©rJrrrXrtrsrvruÚlabelsrWZdiscriminator_sequence_outputrZÚoutputÚloss_fctZ active_lossZ active_logitsZ active_labelsÚlossrrrr[]s,4ÿ  zElectraForPreTraining.forward)NNNNNNN©rMrNrOrCrr|r[rQrrrKrr}Ms ør}zø Electra model with a language modeling head on top. Even though both the discriminator and generator may be loaded into this model, the generator is the only model of the two to have been trained for the masked language modeling task.cs6eZdZ‡fdd„Zdd„Zeeƒddd„ƒZ‡ZS) r%cs>tƒ |¡t|ƒ|_t|ƒ|_t |j|j ¡|_ |  ¡dSrV) rBrCrcr`r]Úgenerator_predictionsrDrSrFrEÚgenerator_lm_headrgrIrKrrrC´s    zElectraForMaskedLM.__init__cCs|jSrV)rˆrirrrÚget_output_embeddings½sz(ElectraForMaskedLM.get_output_embeddingsNcCs€| ||||||¡}|d} | | ¡} | | ¡} | f} |dk rlt ¡} | |  d|jj¡| d¡ƒ} | f| } | |dd…7} | S)ax masked_lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.ElectraConfig`) and inputs: masked_lm_loss (`optional`, returned when ``masked_lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Masked language modeling loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import ElectraTokenizer, ElectraForMaskedLM import torch tokenizer = ElectraTokenizer.from_pretrained('google/electra-small-generator') model = ElectraForMaskedLM.from_pretrained('google/electra-small-generator') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, masked_lm_labels=input_ids) loss, prediction_scores = outputs[:2] rNrpr)r`r‡rˆrDÚCrossEntropyLossrr6rE)rJrrrXrtrsrvruZmasked_lm_labelsr^Zgenerator_sequence_outputZprediction_scoresrƒr„r…rrrr[Às$3ÿ   zElectraForMaskedLM.forward)NNNNNNN) rMrNrOrCr‰rr|r[rQrrrKrr%«s ør%z„ Electra model with a token classification head on top. Both the discriminator and generator may be loaded into this model.cs.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚElectraForTokenClassificationcsBtƒ |¡t|ƒ|_t |j¡|_t |j |j ¡|_ |  ¡dSrV) rBrCrcr`rDZDropoutZhidden_dropout_probÚdropoutrSrTÚ num_labelsrrgrIrKrrrCs   z&ElectraForTokenClassification.__init__NcCsÄ| ||||||¡}|d} | | ¡} | | ¡} | f} |dk r°t ¡} |dk rŠ| d¡dk} |  d|jj¡| }| d¡| }| ||ƒ}n| |  d|jj¡| d¡ƒ}|f| } | |dd…7} | S)a labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels - 1]``. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.ElectraConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when ``labels`` is provided) : Classification loss. scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.num_labels)`) Classification scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import ElectraTokenizer, ElectraForTokenClassification import torch tokenizer = ElectraTokenizer.from_pretrained('google/electra-small-discriminator') model = ElectraForTokenClassification.from_pretrained('google/electra-small-discriminator') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, scores = outputs[:2] rNrpr)r`rŒrrDrŠrr6rrrrrr[s.2ÿ    z%ElectraForTokenClassification.forward)NNNNNNNr†rrrKrr‹s ør‹)r )!Úloggingrr2Ztorch.nnrDZ activationsrZconfiguration_electrarZ file_utilsrrZ modeling_bertrrr r Ú getLoggerrMrrbr@rAÚModulerRr]r_ZELECTRA_START_DOCSTRINGr|rcr}r%r‹rrrrÚsT    ú Q  .úeúVúUû