U &ºc^Åã @sÚdZddlZddlZddlZddlZddlmZddlmZmZddl m Z ddl m Z m Z ddlmZmZmZmZddlmZe e¡Zd d d d d ddddœZdd„ZGdd„deƒZGdd„deƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„deƒZ d Z!d!Z"e d"e!ƒGd#d$„d$e ƒƒZ#e d%e!ƒGd&d'„d'e ƒƒZ$Gd(d)„d)ejƒZ%Gd*d+„d+ejƒZ&e d,e!ƒGd-d.„d.e ƒƒZ'e d/e!ƒGd0d1„d1e ƒƒZ(e d2e!ƒGd3d4„d4e ƒƒZ)e d5e!ƒGd6d7„d7e ƒƒZ*dS)8zPyTorch ALBERT model. éN)ÚCrossEntropyLossÚMSELossé)Ú AlbertConfig)Úadd_start_docstringsÚ add_start_docstrings_to_callable)ÚACT2FNÚBertEmbeddingsÚBertSelfAttentionÚprune_linear_layer)ÚPreTrainedModelz;https://cdn.huggingface.co/albert-base-v1-pytorch_model.binzhttps://cdn.huggingface.co/albert-xxlarge-v1-pytorch_model.binz;https://cdn.huggingface.co/albert-base-v2-pytorch_model.binzhttps://cdn.huggingface.co/albert-xxlarge-v2-pytorch_model.bin)zalbert-base-v1zalbert-large-v1zalbert-xlarge-v1zalbert-xxlarge-v1zalbert-base-v2zalbert-large-v2zalbert-xlarge-v2zalbert-xxlarge-v2c Cs*zddl}ddl}ddl}Wn tk r<t d¡‚YnXtj |¡}t  d  |¡¡|j   |¡}g}g} |D]<\} } t  d  | | ¡¡|j   || ¡} | | ¡|  | ¡qrt|| ƒD]\} } t| ƒqºt|| ƒD]L\} } | } |  dd¡} |  dd ¡} |  d d ¡} |  d d ¡} |  dd¡} |  dd¡} |  dd¡} |  dd¡} |  dd¡} |  dd¡} |  dd¡} |  dd¡} |  dd¡} |  dd¡} |  dd¡} |  dd ¡} |  d!d"¡} |  d#d$¡} t|  d¡ƒd%krìd&| ksäd'| krìd(| } d)| kr|  d*d+¡} |  d,d-¡} |  d¡} d.| ksJd/| ksJd0| ksJd1| ksJd2| krbt  d3  d | ¡¡¡qÖ|}| D]}| d4|¡rŒ| d5|¡}n|g}|dd6ks®|dd7krºt|d-ƒ}n¢|dd&ksÖ|dd8krât|d9ƒ}nz|dd'krüt|d-ƒ}n`|dd:krt|d;ƒ}nFzt||dƒ}Wn2tk rZt  d3  d | ¡¡¡YqjYnXt|ƒdkr t|d-ƒ}n|d6kr´| | ¡} z|j| jksÈt‚Wn<tk r}z|j|j| jf7_‚W5d}~XYnXtd?  | | ¡ƒt | ¡|_qÖ|S)@z( Load tf checkpoints in a pytorch model.rNz™Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}zmodule/ÚZffn_1Úffnzbert/zalbert/Z attention_1Ú attentionz transform/Z LayerNorm_1Úfull_layer_layer_normÚ LayerNormzattention/LayerNormz transformer/zintermediate/dense/zffn/intermediate/output/dense/z ffn_output/z/output/ú/z/self/z pooler/denseÚpoolerzcls/predictionsÚ predictionszpredictions/attentionzembeddings/attentionÚ embeddingsZ inner_group_zalbert_layers/Zgroup_zalbert_layer_groups/rZ output_biasZoutput_weightsz classifier/Zseq_relationshipzseq_relationship/output_zsop_classifier/classifier/ÚweightsÚweightZadam_mZadam_vZAdamWeightDecayOptimizerZAdamWeightDecayOptimizer_1Z global_stepz Skipping {}z [A-Za-z]+_\d+z_(\d+)ÚkernelÚgammaÚbetaÚbiasZsquadÚ classifieréiõÿÿÿZ _embeddingsz$Initialize PyTorch weight {} from {}) ÚreZnumpyZ tensorflowÚ ImportErrorÚloggerÚerrorÚosÚpathÚabspathÚinfoÚformatZtrainZlist_variablesZ load_variableÚappendÚzipÚprintÚreplaceÚlenÚsplitÚjoinÚ fullmatchÚgetattrÚAttributeErrorÚintÚ transposeÚshapeÚAssertionErrorÚargsÚtorchZ from_numpyÚdata)ÚmodelÚconfigZtf_checkpoint_pathrÚnpÚtfZtf_pathZ init_varsÚnamesZarraysÚnamer3ÚarrayÚ original_nameÚpointerZm_nameZ scope_namesÚnumÚe©rCú@/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_albert.pyÚload_tf_weights_in_albert.s¸ ÿ                        (    ÿþýüû           rEcs eZdZdZ‡fdd„Z‡ZS)ÚAlbertEmbeddingszQ Construct the embeddings from word, position and token_type embeddings. csbtƒ |¡tj|j|j|jd|_t |j|j¡|_ t |j |j¡|_ t jj |j|jd|_ dS)N)Z padding_idx©Zeps)ÚsuperÚ__init__ÚnnÚ EmbeddingÚ vocab_sizeÚembedding_sizeZ pad_token_idÚword_embeddingsZmax_position_embeddingsZposition_embeddingsZtype_vocab_sizeZtoken_type_embeddingsr6rÚlayer_norm_eps©Úselfr9©Ú __class__rCrDrI¯s  zAlbertEmbeddings.__init__)Ú__name__Ú __module__Ú __qualname__Ú__doc__rIÚ __classcell__rCrCrRrDrFªsrFcs.eZdZ‡fdd„Zdd„Zddd„Z‡ZS) ÚAlbertAttentioncsrtƒ |¡|j|_|j|_|j|_|j|j|_t |j¡|_ t  |j|j¡|_ tj |j|j d|_ tƒ|_dS©NrG)rHrIÚoutput_attentionsÚnum_attention_headsÚ hidden_sizeÚattention_head_sizerJÚDropoutZattention_probs_dropout_probÚdropoutÚLinearÚdenserrOÚsetÚ pruned_headsrPrRrCrDrI¹s zAlbertAttention.__init__csòt|ƒdkrdSt |j|j¡}t|ƒ|j}|D](‰ˆt‡fdd„|jDƒƒ‰d|ˆ<q2| d¡  ¡  d¡}t  t|ƒ¡|  ¡}t |j|ƒ|_t |j|ƒ|_t |j|ƒ|_t |j|dd|_|jt|ƒ|_|j|j|_|j |¡|_dS)Nrc3s|]}|ˆkrdndVqdS)rrNrC)Ú.0Úh©ÚheadrCrDÚ Ìsz.AlbertAttention.prune_heads..éÿÿÿÿr©Zdim)r+r6Úonesr\r^rcrdÚsumÚviewÚ contiguousÚeqZarangeÚlongr ÚqueryÚkeyÚvaluerbZ all_head_sizeÚunion)rQÚheadsÚmaskÚindexrCrgrDÚ prune_headsÅs   zAlbertAttention.prune_headsNcCs.| |¡}| |¡}| |¡}| |¡}| |¡}| |¡} t || dd¡¡} | t |j ¡} |dk rp| |} t j dd| ƒ} |  | ¡} |dk rš| |} t | | ¡} |   dddd¡ ¡} |jj ¡ |j|j |j¡ | j¡} |jj | j¡}t d| | ¡|}|  |¡}| ||¡}|jr(|| fS|fS) Nrjéþÿÿÿrkrrréz bfnd,ndh->bfh)rrrsrtZtranspose_for_scoresr6Úmatmulr2ÚmathÚsqrtr^rJZSoftmaxr`ZpermuterorbrÚtrnr\r]ÚtoÚdtyperZeinsumrr[)rQÚ input_idsÚattention_maskÚ head_maskZmixed_query_layerZmixed_key_layerZmixed_value_layerZ query_layerZ key_layerZ value_layerZattention_scoresZattention_probsZ context_layerÚwÚbZprojected_context_layerZprojected_context_layer_dropoutZlayernormed_context_layerrCrCrDÚforwardÜs:         ÿþÿ zAlbertAttention.forward)NN)rTrUrVrIryr‡rXrCrCrRrDrY¸s rYcs&eZdZ‡fdd„Zddd„Z‡ZS)Ú AlbertLayercsbtƒ ¡||_tj|j|jd|_t|ƒ|_ t  |j|j ¡|_ t  |j |j¡|_ t|j|_dSrZ)rHrIr9rJrr]rOrrYrraZintermediate_sizerÚ ffn_outputrÚ hidden_actÚ activationrPrRrCrDrI s  zAlbertLayer.__init__NcCsT| |||¡}| |d¡}| |¡}| |¡}| ||d¡}|f|dd…S)Nrr)rrr‹r‰r)rQÚ hidden_statesrƒr„Zattention_outputr‰rCrCrDr‡s   zAlbertLayer.forward)NN©rTrUrVrIr‡rXrCrCrRrDrˆ s rˆcs&eZdZ‡fdd„Zddd„Z‡ZS)ÚAlbertLayerGroupcs>tƒ ¡ˆj|_ˆj|_t ‡fdd„tˆjƒDƒ¡|_dS)Ncsg|] }tˆƒ‘qSrC)rˆ©reÚ_©r9rCrDÚ $sz-AlbertLayerGroup.__init__..) rHrIr[Úoutput_hidden_statesrJÚ ModuleListÚrangeÚinner_group_numÚ albert_layersrPrRr‘rDrIs zAlbertLayerGroup.__init__Nc Cs‚d}d}t|jƒD]D\}}|||||ƒ}|d}|jrF||df}|jr||f}q|f} |jrn| |f} |jr~| |f} | S)NrCrr)Ú enumerater—r[r“) rQrŒrƒr„Zlayer_hidden_statesZlayer_attentionsZ layer_indexZ albert_layerZ layer_outputÚoutputsrCrCrDr‡&s   zAlbertLayerGroup.forward)NNrrCrCrRrDrŽs rŽcs&eZdZ‡fdd„Zddd„Z‡ZS)ÚAlbertTransformercsVtƒ ¡ˆ|_ˆj|_ˆj|_t ˆjˆj¡|_ t  ‡fdd„t ˆj ƒDƒ¡|_ dS)Ncsg|] }tˆƒ‘qSrC)rŽrr‘rCrDr’Dsz.AlbertTransformer.__init__..)rHrIr9r[r“rJrarMr]Úembedding_hidden_mapping_inr”r•Únum_hidden_groupsÚalbert_layer_groupsrPrRr‘rDrI=s  zAlbertTransformer.__init__Nc CsÒ| |¡}d}|jr|f}t|jjƒD]€}t|jj|jjƒ}t||jj|jjƒ}|j|||||||d|…ƒ} | d}|jr–|| d}|jr&||f}q&|f} |jr¾| |f} |jrÎ| |f} | S)NrCrrrj) r›r“r•r9Únum_hidden_layersr1rœrr[) rQrŒrƒr„Zall_attentionsZall_hidden_statesÚiZlayers_per_groupÚ group_idxZlayer_group_outputr™rCrCrDr‡Fs. ý    zAlbertTransformer.forward)NNrrCrCrRrDrš<s ršc@s$eZdZdZeZeZdZdd„Z dS)ÚAlbertPreTrainedModelz† An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ÚalbertcCsvt|tjtjfƒrL|jjjd|jjdt|tjƒrr|j dk rr|j j  ¡n&t|tj ƒrr|j j  ¡|jj  d¡dS)z! Initialize the weights. g)ZmeanZstdNçð?) Ú isinstancerJrarKrr7Znormal_r9Zinitializer_rangerZzero_rZfill_)rQÚmodulerCrCrDÚ _init_weightsss  z#AlbertPreTrainedModel._init_weightsN) rTrUrVrWrÚ config_classÚ#ALBERT_PRETRAINED_MODEL_ARCHIVE_MAPÚpretrained_model_archive_mapÚbase_model_prefixr¦rCrCrCrDr¡js r¡aj This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Args: config (:class:`~transformers.AlbertConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a½ Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.AlbertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. z`The bare ALBERT Model transformer outputting raw hidden-states without any specific head on top.cs^eZdZeZeZeZdZ ‡fdd„Z dd„Z dd„Z dd „Z d d „Zeeƒdd d„ƒZ‡ZS)Ú AlbertModelr¢csNtƒ |¡||_t|ƒ|_t|ƒ|_t |j |j ¡|_ t  ¡|_ |  ¡dS©N)rHrIr9rFrršÚencoderrJrar]rZTanhÚpooler_activationÚ init_weightsrPrRrCrDrI¾s    zAlbertModel.__init__cCs|jjSr¬©rrN©rQrCrCrDÚget_input_embeddingsÉsz AlbertModel.get_input_embeddingscCs ||j_dSr¬r°)rQrtrCrCrDÚset_input_embeddingsÌsz AlbertModel.set_input_embeddingscCs$|jj}| ||¡}||j_|jjSr¬)rrNZ_get_resized_embeddings)rQZnew_num_tokensZold_embeddingsZnew_embeddingsrCrCrDÚ_resize_token_embeddingsÏs z$AlbertModel._resize_token_embeddingscCsT| ¡D]F\}}t||jjƒ}t|||jjƒ}|jj|j|j |¡qdS)aþ Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} ALBERT has a different architecture in that its layers are shared across groups, which then has inner groups. If an ALBERT model has 12 hidden layers and 2 hidden groups, with two inner groups, there is a total of 4 different layers. These layers are flattened: the indices [0,1] correspond to the two inner groups of the first hidden layer, while [2,3] correspond to the two inner groups of the second hidden layer. Any layer with in index other than [0,1,2,3] will result in an error. See base class PreTrainedModel for more information about head pruning N) Úitemsr1r9r–r­rr—rry)rQZheads_to_pruneZlayerrvr Zinner_group_idxrCrCrDÚ _prune_headsÕs zAlbertModel._prune_headsNcCs2|dk r|dk rtdƒ‚n4|dk r,| ¡}n"|dk rF| ¡dd…}ntdƒ‚|dk r\|jn|j}|dkrxtj||d}|dkr’tj|tj|d}| d¡ d¡} | jt |  ¡ƒj d } d | d } |  ||j j¡}|j||||d } |j| | |d } | d} | | | dd…df¡¡} | | f| dd…}|S)a Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.AlbertConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (:obj:`torch.FloatTensor`: of shape :obj:`(batch_size, hidden_size)`): Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pre-training. This output is usually *not* a good summary of the semantic content of the input, you're often better with averaging or pooling the sequence of hidden-states for the whole input sequence. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Example:: from transformers import AlbertModel, AlbertTokenizer import torch tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') model = AlbertModel.from_pretrained('albert-base-v2') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NzDYou cannot specify both input_ids and inputs_embeds at the same timerjz5You have to specify either input_ids or inputs_embeds)Údevice)rr·rr)rr£gˆÃÀ)Ú position_idsÚtoken_type_idsÚ inputs_embeds)r„r)Ú ValueErrorÚsizer·r6rlÚzerosrqZ unsqueezer€ÚnextÚ parametersrZ get_head_maskr9ržrr­r®r)rQr‚rƒr¹r¸r„rºZ input_shaper·Zextended_attention_maskZembedding_outputZencoder_outputsÚsequence_outputÚ pooled_outputr™rCrCrDr‡çs:1   ÿÿ zAlbertModel.forward)NNNNNN)rTrUrVrr§r¨r©rEZload_tf_weightsrªrIr²r³r´r¶rÚALBERT_INPUTS_DOCSTRINGr‡rXrCrCrRrDr«³s" ùr«z£Albert Model with two heads on top as done during the pre-training: a `masked language modeling` head and a `sentence order prediction (classification)` head. c s>eZdZ‡fdd„Zdd„Zdd„Zeeƒd dd „ƒZ‡Z S) ÚAlbertForPreTrainingcs>tƒ |¡t|ƒ|_t|ƒ|_t|ƒ|_| ¡|  ¡dSr¬) rHrIr«r¢Ú AlbertMLMHeadrÚ AlbertSOPHeadÚsop_classifierr¯Ú tie_weightsrPrRrCrDrIBs     zAlbertForPreTraining.__init__cCs| |jj|jjj¡dSr¬©Z_tie_or_clone_weightsrÚdecoderr¢rrNr±rCrCrDrÇLsz AlbertForPreTraining.tie_weightscCs|jjSr¬©rrÉr±rCrCrDÚget_output_embeddingsOsz*AlbertForPreTraining.get_output_embeddingsNc Cs®|j||||||d} | dd…\} } | | ¡} | | ¡} | | f| dd…} |dk rª|dk rªtƒ}||  d|jj¡| d¡ƒ}||  dd¡| d¡ƒ}||}|f| } | S)a masked_lm_labels (``torch.LongTensor`` of shape ``(batch_size, sequence_length)``, `optional`, defaults to :obj:`None`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` sentence_order_label (``torch.LongTensor`` of shape ``(batch_size,)``, `optional`, defaults to :obj:`None`): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see :obj:`input_ids` docstring) Indices should be in ``[0, 1]``. ``0`` indicates original order (sequence A, then sequence B), ``1`` indicates switched order (sequence B, then sequence A). Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: loss (`optional`, returned when ``masked_lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). sop_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import AlbertTokenizer, AlbertForPreTraining import torch tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') model = AlbertForPreTraining.from_pretrained('albert-base-v2') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) prediction_scores, sop_scores = outputs[:2] ©rƒr¹r¸r„rºNrrj)r¢rrÆrrnr9rL)rQr‚rƒr¹r¸r„rºÚmasked_lm_labelsZsentence_order_labelr™rÀrÁÚprediction_scoresZ sop_scoresÚloss_fctÚmasked_lm_lossZsentence_order_lossÚ total_lossrCrCrDr‡Rs&=ú    zAlbertForPreTraining.forward)NNNNNNNN© rTrUrVrIrÇrËrrÂr‡rXrCrCrRrDrÃ<s ÷rÃcs$eZdZ‡fdd„Zdd„Z‡ZS)rÄcsjtƒ ¡t |j¡|_t t |j¡¡|_ t  |j |j¡|_ t  |j|j¡|_ t|j|_|j |j _ dSr¬)rHrIrJrrMÚ Parameterr6r½rLrrar]rbrÉrrŠr‹rPrRrCrDrIªs  zAlbertMLMHead.__init__cCs0| |¡}| |¡}| |¡}| |¡}|}|Sr¬)rbr‹rrÉ)rQrŒrÎrCrCrDr‡¶s     zAlbertMLMHead.forwardrrCrCrRrDrÄ©s rÄcs$eZdZ‡fdd„Zdd„Z‡ZS)rÅcs.tƒ ¡t |j¡|_t |j|j¡|_ dSr¬) rHrIrJr_Úclassifier_dropout_probr`rar]Ú num_labelsrrPrRrCrDrIÂs zAlbertSOPHead.__init__cCs| |¡}| |¡}|Sr¬)r`r)rQrÁZdropout_pooled_outputÚlogitsrCrCrDr‡Ès  zAlbertSOPHead.forwardrrCrCrRrDrÅÁs rÅz4Albert Model with a `language modeling` head on top.cs>eZdZ‡fdd„Zdd„Zdd„Zeeƒd dd „ƒZ‡Z S) ÚAlbertForMaskedLMcs4tƒ |¡t|ƒ|_t|ƒ|_| ¡| ¡dSr¬)rHrIr«r¢rÄrr¯rÇrPrRrCrDrIÒs    zAlbertForMaskedLM.__init__cCs| |jj|jjj¡dSr¬rÈr±rCrCrDrÇÛszAlbertForMaskedLM.tie_weightscCs|jjSr¬rÊr±rCrCrDrËÞsz'AlbertForMaskedLM.get_output_embeddingsNc Csr|j||||||d}|d} | | ¡} | f|dd…}|dk rntƒ} | |  d|jj¡| d¡ƒ} | f|}|S)a& masked_lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.AlbertConfig`) and inputs: loss (`optional`, returned when ``masked_lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Masked language modeling loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Example:: from transformers import AlbertTokenizer, AlbertForMaskedLM import torch tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') model = AlbertForMaskedLM.from_pretrained('albert-base-v2') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, masked_lm_labels=input_ids) loss, prediction_scores = outputs[:2] ©r‚rƒr¹r¸r„rºrrNrj)r¢rrrnr9rL) rQr‚rƒr¹r¸r„rºrÍr™Zsequence_outputsrÎrÏrÐrCrCrDr‡ás 0ú  zAlbertForMaskedLM.forward)NNNNNNNrÒrCrCrRrDr×Îs ør×z•Albert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚAlbertForSequenceClassificationcsLtƒ |¡|j|_t|ƒ|_t |j¡|_t  |j |j j¡|_ |  ¡dSr¬)rHrIrÕr«r¢rJr_rÔr`rar]r9rr¯rPrRrCrDrI,s   z(AlbertForSequenceClassification.__init__Nc Cs¢|j||||||d}|d} | | ¡} | | ¡} | f|dd…}|dk rž|jdkrttƒ} | |  d¡| d¡ƒ} n tƒ} | |  d|j¡| d¡ƒ} | f|}|S)a3 labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the sequence classification/regression loss. Indices should be in ``[0, ..., config.num_labels - 1]``. If ``config.num_labels == 1`` a regression loss is computed (Mean-Square loss), If ``config.num_labels > 1`` a classification loss is computed (Cross-Entropy). Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.AlbertConfig`) and inputs: loss: (`optional`, returned when ``labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Classification (or regression if config.num_labels==1) loss. logits ``torch.FloatTensor`` of shape ``(batch_size, config.num_labels)`` Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import AlbertTokenizer, AlbertForSequenceClassification import torch tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') model = AlbertForSequenceClassification.from_pretrained('albert-base-v2') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) # Batch size 1 labels = torch.tensor([1]).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, logits = outputs[:2] rØrrNrj)r¢r`rrÕrrnr) rQr‚rƒr¹r¸r„rºÚlabelsr™rÁrÖrÏÚlossrCrCrDr‡6s(2ú     z'AlbertForSequenceClassification.forward)NNNNNNN©rTrUrVrIrrÂr‡rXrCrCrRrDrÙ&s ørÙzœAlbert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cs.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚAlbertForTokenClassificationcsLtƒ |¡|j|_t|ƒ|_t |j¡|_t  |j |j j¡|_ |  ¡dSr¬)rHrIrÕr«r¢rJr_Zhidden_dropout_probr`rar]r9rr¯rPrRrCrDrI‹s   z%AlbertForTokenClassification.__init__NcCs¼|j||||||d}|d} | | ¡} | | ¡} | f|dd…}|dk r¸tƒ} |dk r”| d¡dk} |  d|j¡| } | d¡| }| | |ƒ}n| |  d|j¡| d¡ƒ}|f|}|S)aR labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels - 1]``. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.AlbertConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when ``labels`` is provided) : Classification loss. scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.num_labels)`) Classification scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import AlbertTokenizer, AlbertForTokenClassification import torch tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') model = AlbertForTokenClassification.from_pretrained('albert-base-v2') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=labels) loss, scores = outputs[:2] rÌrrNrjr)r¢r`rrrnrÕ)rQr‚rƒr¹r¸r„rºrÚr™rÀrÖrÏZ active_lossZ active_logitsZ active_labelsrÛrCrCrDr‡•s,2ú     z$AlbertForTokenClassification.forward)NNNNNNNrÜrCrCrRrDrÝ…s ørÝzÖAlbert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c s.eZdZ‡fdd„Zeeƒddd„ƒZ‡ZS)ÚAlbertForQuestionAnsweringcs<tƒ |¡|j|_t|ƒ|_t |j|j¡|_|  ¡dSr¬) rHrIrÕr«r¢rJrar]Ú qa_outputsr¯rPrRrCrDrIìs   z#AlbertForQuestionAnswering.__init__Nc Cs|j||||||d} | d} | | ¡} | jddd\} } |  d¡} |  d¡} | | f| dd…} |dk rü|dk rüt| ¡ƒdkrŒ| d¡}t| ¡ƒdkr¦| d¡}|  d¡}| d|¡| d|¡t|d}|| |ƒ}|| |ƒ}||d}|f| } | S) a¹ start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.AlbertConfig`) and inputs: loss: (`optional`, returned when ``labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_scores ``torch.FloatTensor`` of shape ``(batch_size, sequence_length,)`` Span-start scores (before SoftMax). end_scores: ``torch.FloatTensor`` of shape ``(batch_size, sequence_length,)`` Span-end scores (before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: # The checkpoint albert-base-v2 is not fine-tuned for question answering. Please see the # examples/question-answering/run_squad.py example to see how to fine-tune a model to a question answering task. from transformers import AlbertTokenizer, AlbertForQuestionAnswering import torch tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2') model = AlbertForQuestionAnswering.from_pretrained('albert-base-v2') question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" input_dict = tokenizer.encode_plus(question, text, return_tensors='pt') start_scores, end_scores = model(**input_dict) rØrrrjrkrN)Z ignore_index)r¢rßr,Zsqueezer+r¼Zclamp_r)rQr‚rƒr¹r¸r„rºZstart_positionsZ end_positionsr™rÀrÖZ start_logitsZ end_logitsZ ignored_indexrÏZ start_lossZend_lossrÑrCrCrDr‡õs8:ú              z"AlbertForQuestionAnswering.forward)NNNNNNNNrÜrCrCrRrDrÞæs ÷rÞ)+rWÚloggingr}r"r6Ztorch.nnrJrrZconfiguration_albertrZ file_utilsrrZ modeling_bertrr r r Zmodeling_utilsr Ú getLoggerrTr r¨rErFrYÚModulerˆrŽršr¡ZALBERT_START_DOCSTRINGrÂr«rÃrÄrÅr×rÙrÝrÞrCrCrCrDÚs|    ø |Q. &þýh ÿUýZý\ý