U &c{@sPdZddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZddlmZmZmZmZeeZd d iZd d Zeje e d ZGdddejZGdddejZGdddejZ GdddeZ!dZ"dZ#ede"Gddde!Z$ede"Gddde!Z%ede"Gdd d e!Z&dS)!zPyTorch OpenAI GPT model.N)CrossEntropyLoss)gelu_newswish)OpenAIGPTConfig)add_start_docstrings add_start_docstrings_to_callable)Conv1DPreTrainedModelSequenceSummaryprune_conv1d_layerz openai-gptz7https://cdn.huggingface.co/openai-gpt-pytorch_model.binc sddl}ddldkr$tjtdtdddd}t |}W5QRXtd ddd}t |}W5QRX fd d |D}fd d t d D} | d|dd} dd t| |D} dd | D} z8|jjj| djkst|jjj| djks(tWnbtk r} zB| j|jjj| djf7_| j|jjj| djf7_W5d} ~ XYnXt| d|jj_t| d|jj_|d| d| dt|| D]\} } | dd} | dddks t| dd} | d} |} | D]}|d|rH| d|}n|g}|ddkrht| d} nB|ddkrt| d} n(|ddkrt| d} nt| |d} t|dkr(t|d}| |} q(z| j| jkstWn<tk r"} z| j| j| jf7_W5d} ~ XYnXz| j| jks8tWn<tk rv} z| j| j| jf7_W5d} ~ XYnXtd| t| | _q|S)zM Load tf pre-trained weights in a pytorch model (from NumPy arrays here) rNz.ckptzLoading weights from {}z/parameters_names.jsonrzutf-8)encodingz/params_shapes.jsoncsg|]}|qS)prod).0shape)npr@/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_openai.py 6sz1load_tf_weights_in_openai_gpt..cs g|]}d|qS)z/params_{}.npy)loadformat)rnropenai_checkpoint_folder_pathrrr7s cSsg|]\}}||qSr)Zreshape)rparamrrrrr9scSsg|] }|qSr)squeeze)rZarrrrrr>srz:0/z [A-Za-z]+\d+z(\d+)gweightbbiaswzInitialize PyTorch weight {})reZnumpyospathdirnameloggerinforopenjsonrZcumsumrangesplitZ concatenatezip tokens_embedr#rAssertionErrorpositions_embedargstorchZ from_numpydatapop fullmatchgetattrlenint)modelconfigrr(Z names_handlenamesZ shapes_handleZshapesoffsetsZ init_paramsenamearraypointerZm_nameZ scope_namesnumrrrload_tf_weights_in_openai_gpt'sv            rG)ZrelurZgelucsLeZdZdfdd ZddZdddZd d Zdd d Zdd dZZ S) AttentionFc st|}||jdks t|dtt||dd|||j|_||_ ||_ |j |_ t |d||_ t |||_t|j|_t|j|_t|_dS)Nrr%r)super__init__n_headr4Zregister_bufferr7Ztrilonesview split_sizescaleoutput_attentionsr c_attnc_projnnDropoutZ attn_pdrop attn_dropout resid_pdrop resid_dropoutset pruned_heads)selfnxn_ctxr?rPn_state __class__rrrKxs & zAttention.__init__cst|dkrdSt|j|j|j}t||j}|D](tfdd|jD8d|<q8|d d}t t|| }t |||j|d|jg}t|j|dd|_t|j|dd|_|j|j|jt||_|jt||_|j||_dS)Nrc3s|]}|krdndVqdS)rrNr)rhheadrr sz(Attention.prune_heads..rrr'Zdim)r<r7rMrLrOrYrZsumrN contiguouseqarangelongcatr rRrSunion)r[headsmaskindexZ index_attnrrbr prune_headss   zAttention.prune_headsNc Cst||}|jr&|t|d}|jddddd|dd|df}||dd|}|dk rx||}tjdd|}| |}|dk r||}t||g}|j r| ||S)Nrr rre) r7matmulrPmathsqrtsizer%rTZSoftmaxrVrQappend) r[qkvattention_mask head_maskr&r$outputsrrr_attns .  zAttention._attncCsD|dddd}|dd|d|df}|j|S)Nrr'rrIr r)permutergrurN)r[x new_x_shaperrr merge_headss&zAttention.merge_headscCsX|dd|j|d|jf}|j|}|rD|ddddS|ddddSdS)Nrrr'rIr)rurLrNr~)r[rrxrrrr split_headss & zAttention.split_headsc Cs||}|j|jdd\}}}||}|j|dd}||}||||||}|d}||}||}||}|g|dd} | S)Nr'reT)rxrr)rRr1rOrr}rrSrX) r[rrzr{querykeyvalue attn_outputsar|rrrforwards      zAttention.forward)F)NN)F)NN) __name__ __module__ __qualname__rKrpr}rrr __classcell__rrr_rrHws   rHcs$eZdZfddZddZZS)MLPcsFt|j}t|||_t|||_t|j|_t |j |_ dSN) rJrKn_embdr c_fcrSACT_FNSZafnactrTrUrWdropout)r[r^r?r\r_rrrKs     z MLP.__init__cCs$|||}||}||Sr)rrrSr)r[rrah2rrrrs z MLP.forwardrrrrKrrrrr_rrs rcs(eZdZdfdd ZdddZZS) BlockFcsXt|j}t|||||_tj||jd|_t d|||_ tj||jd|_ dS)N)Zeps) rJrKrrHattnrT LayerNormZlayer_norm_epsilonln_1rmlpln_2)r[r]r?rPr\r_rrrKs  zBlock.__init__Nc CsT|j|||d}|d}|||}||}|||}|g|dd} | S)N)rzr{rr)rrrr) r[rrzr{rrrmrar|rrrrs z Block.forward)F)NNrrrr_rrsrc@s(eZdZdZeZeZeZ dZ ddZ dS)OpenAIGPTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformercCs|t|tjtjtfrR|jjjd|jj dt|tjtfrx|j dk rx|j j n&t|tj rx|j j |jj ddS)z! Initialize the weights. g)ZmeanZstdN?) isinstancerTLinear Embeddingr r#r8Znormal_r?Zinitializer_ranger%Zzero_rZfill_)r[modulerrr _init_weightss  z&OpenAIGPTPreTrainedModel._init_weightsN) rrr__doc__rZ config_class'OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_maprGZload_tf_weightsZbase_model_prefixrrrrrrs ras This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.OpenAIGPTConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.OpenAIGPTTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. zdThe bare OpenAI GPT transformer model outputting raw hidden-states without any specific head on top.csFeZdZfddZddZddZddZeed d d Z Z S) OpenAIGPTModelcsztj|_j|_tjj|_tj j|_ t j |_ tfddtjD|_|dS)Ncsg|]}tjddqS)T)rP)rr])r_r?rrrPsz+OpenAIGPTModel.__init__..)rJrKrQoutput_hidden_statesrTr vocab_sizerr3Z n_positionsr5rUZ embd_pdropdropZ ModuleListr0n_layerra init_weightsr[r?r_rrrKHs  zOpenAIGPTModel.__init__cCs|jSrr3r[rrrget_input_embeddingsTsz#OpenAIGPTModel.get_input_embeddingscCs ||_dSrr)r[Znew_embeddingsrrrset_input_embeddingsWsz#OpenAIGPTModel.set_input_embeddingscCs(|D]\}}|j|j|qdS)zz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrarrp)r[Zheads_to_pruneZlayerrmrrr _prune_headsZszOpenAIGPTModel._prune_headsNcCs|dk r|dk rtdnD|dk r<|}|d|d}n"|dk rV|dd}ntd|dkr|dk rt|jn|j}tj|dtj|d}|dd|d}|dk r|dd}|jt | j d }d |d }| ||j j}|dkr||}||} |dk r6|d|d}||} nd} || | } || } || df} d } d }t|jD]P\}}|jr|| j| f}|| |||}|d} |jrr| |df} qr|jr|| j| f}| j| f}|jr||f}|jr || f}|S) a Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.OpenAIGPTConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the last layer of the model. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import OpenAIGPTTokenizer, OpenAIGPTModel import torch tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt') model = OpenAIGPTModel.from_pretrained('openai-gpt') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)dtypedevicerrr')rrrqr) ValueErrorrurNrr7rirjZ unsqueezetonext parametersrZ get_head_maskr?rr3r5r enumeraterarrQ)r[ input_idsrztoken_type_ids position_idsr{ inputs_embedsZ input_shaperZposition_embedsZtoken_type_embeds hidden_statesZ output_shapeZall_attentionsZall_hidden_statesiblockr|rrrrasX'            zOpenAIGPTModel.forward)NNNNNN) rrrrKrrrrOPENAI_GPT_INPUTS_DOCSTRINGrrrrr_rrCs rzOpenAI GPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). cs6eZdZfddZddZeedddZZS) OpenAIGPTLMHeadModelcs8t|t||_tj|j|jdd|_| dS)NFr%) rJrKrrrTrrrlm_headrrr_rrrKs  zOpenAIGPTLMHeadModel.__init__cCs|jSrrrrrrget_output_embeddingssz*OpenAIGPTLMHeadModel.get_output_embeddingsNcCs|j||||||d}|d} || } | f|dd} |dk r| dddddf} |dddf} t}|| d| d| d}|f| } | S)a labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids`` Indices are selected in ``[-100, 0, ..., config.vocab_size]`` All labels set to ``-100`` are ignored (masked), the loss is only computed for labels in ``[0, ..., config.vocab_size]`` Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.OpenAIGPTConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape `(1,)`, `optional`, returned when ``labels`` is provided) Language modeling loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers` with each tensor of shape :obj:`(2, batch_size, num_heads, sequence_length, embed_size_per_head)`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import OpenAIGPTTokenizer, OpenAIGPTLMHeadModel import torch tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt') model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=input_ids) loss, logits = outputs[:2] rzrrr{rrrN.r)rrrgrrNru)r[rrzrrr{rlabelstransformer_outputsr lm_logitsr| shift_logits shift_labelsloss_fctlossrrrrs$5  zOpenAIGPTLMHeadModel.forward)NNNNNNN rrrrKrrrrrrrr_rrs ratOpenAI GPT Model transformer with a language modeling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks. The two heads are two linear layers. The language modeling head has its weights tied to the input embeddings, the classification head takes as input the input of a specified classification token index in the input sequence). c s6eZdZfddZddZeedddZZS) OpenAIGPTDoubleHeadsModelcsHt|d|_t||_tj|j|jdd|_ t ||_ | dS)NrFr) rJrKZ num_labelsrrrTrrrrr multiple_choice_headrrr_rrrK6s    z"OpenAIGPTDoubleHeadsModel.__init__cCs|jSrrrrrrr@sz/OpenAIGPTDoubleHeadsModel.get_output_embeddingsNc Cs|j||||||d} | d} || } || |d} | | f| dd}| dk rt}|| d| d| d}|f|}|dk r| dddddf}|dddf}t}||d|d|d}|f|}|S)a mc_token_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, num_choices)`, `optional`, default to index of the last token of the input) Index of the classification token in each input sequence. Selected in the range ``[0, input_ids.size(-1) - 1[``. lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`) Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids`` Indices are selected in ``[-1, 0, ..., config.vocab_size]`` All labels set to ``-100`` are ignored (masked), the loss is only computed for labels in ``[0, ..., config.vocab_size]`` mc_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size)`, `optional`, defaults to :obj:`None`) Labels for computing the multiple choice classification loss. Indices should be in ``[0, ..., num_choices]`` where `num_choices` is the size of the second dimension of the input tensors. (see `input_ids` above) Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.OpenAIGPTConfig`) and inputs: lm_loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when ``lm_labels`` is provided): Language modeling loss. mc_loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`multiple_choice_labels` is provided): Multiple choice classification loss. lm_prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_choices, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). mc_prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_choices)`): Prediction scores of the multiple choice classification head (scores for each choice before SoftMax). past (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers` with each tensor of shape :obj:`(2, batch_size, num_heads, sequence_length, embed_size_per_head)`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import OpenAIGPTTokenizer, OpenAIGPTDoubleHeadsModel import torch tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt') model = OpenAIGPTDoubleHeadsModel.from_pretrained('openai-gpt') tokenizer.add_special_tokens({'cls_token': '[CLS]'}) # Add a [CLS] to the vocabulary (we should train it also!) model.resize_token_embeddings(len(tokenizer)) choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"] input_ids = torch.tensor([tokenizer.encode(s) for s in choices]).unsqueeze(0) # Batch size 1, 2 choices mc_token_ids = torch.tensor([input_ids.size(-1)-1, input_ids.size(-1)-1]).unsqueeze(0) # Batch size 1 outputs = model(input_ids, mc_token_ids=mc_token_ids) lm_prediction_scores, mc_prediction_scores = outputs[:2] rrrrN.)rrrrrrNrurg)r[rrzrrr{rZ mc_token_idsZ lm_labelsZ mc_labelsrrrZ mc_logitsr|rrrrrrrrCs.H   z!OpenAIGPTDoubleHeadsModel.forward) NNNNNNNNNrrrr_rr-s r)'rr/loggingrsr)r7Ztorch.nnrTrZ activationsrrZconfiguration_openairZ file_utilsrrZmodeling_utilsr r r r getLoggerrr,rrGZReLUrModulerHrrrZOPENAI_GPT_START_DOCSTRINGrrrrrrrrsJ    M^ & X