U &c@s6dZddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z ddlmZmZmZmZeeZd d d d d dZddZGdddejZGdddejZGdddejZGdddeZdZdZe deGdddeZe deGdd d eZ e d!eGd"d#d#eZ!dS)$zPyTorch OpenAI GPT-2 model.N)CrossEntropyLoss)ACT2FN) GPT2Config)add_start_docstrings add_start_docstrings_to_callable)Conv1DPreTrainedModelSequenceSummaryprune_conv1d_layerz1https://cdn.huggingface.co/gpt2-pytorch_model.binz8https://cdn.huggingface.co/gpt2-medium-pytorch_model.binz7https://cdn.huggingface.co/gpt2-large-pytorch_model.binz4https://cdn.huggingface.co/gpt2-xl-pytorch_model.binz7https://cdn.huggingface.co/distilgpt2-pytorch_model.bin)Zgpt2z gpt2-mediumz gpt2-largezgpt2-xlZ distilgpt2c Cs zddl}ddl}Wn tk r4tdYnXtj|}td ||j |}g}g}|D]@\} } td | | |j || } | | | | qjt||D]b\} } | dd} | d} |} | D]} |d| r|d | }n| g}|dd ks|dd kr*t| d } n^|dd krDt| d} nD|ddks`|ddkrzt| |d} t| d } nt| |d} t|dkrt|d}| |} qz| j| jkstWn<tk r}z|j| j| jf7_W5d}~XYnXtd | t| | _q|S)z, Load tf checkpoints in a pytorch model rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}/z [A-Za-z]+\d+z(\d+)wgweightbbiaswpewterzInitialize PyTorch weight {})reZ tensorflow ImportErrorloggererrorospathabspathinfoformatZtrainZlist_variablesZ load_variableappendsqueezezipsplit fullmatchgetattrlenintshapeAssertionErrorargstorchZ from_numpydata)modelconfigZgpt2_checkpoint_pathrtfZtf_pathZ init_varsnamesZarraysnamer'arraypointerZm_nameZ scope_namesnumer5>/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_gpt2.pyload_tf_weights_in_gpt2+sZ              r7csLeZdZdfdd ZddZdddZd d Zdd d Zdd dZZ S) AttentionFc st|j|_|}||jdks(t|dttj||ftj d dd|||dt d|j|_||_ ||_ t|d||_t|||_t|j|_t|j|_t|_dS)Nrrdtyper masked_bias)super__init__output_attentionsn_headr(Zregister_bufferr*ZtrilonesZuint8viewZtensor split_sizescalerc_attnc_projnnDropoutZ attn_pdrop attn_dropout resid_pdrop resid_dropoutset pruned_heads)selfnxn_ctxr-rEn_state __class__r5r6r?cs" $ zAttention.__init__cst|dkrdSt|j|j|j}t||j}|D](tfdd|jDd|<q8|d d}t t|| }t |||j|d|jg}t|j|dd|_t|j|dd|_|j|j|jt||_|jt||_|j||_dS)Nrc3s|]}|krdndVqdS)rrNr5).0hheadr5r6 sz(Attention.prune_heads..rrZdim)r%r*rBrArDrMrNsumrC contiguouseqarangelongcatr rFrGunion)rOheadsmaskindexZ index_attnr5rWr6 prune_headsxs   zAttention.prune_headsNc Cst||}|jr(|t|dd}|d|d}}|jdddd|||d|f} t| ||j |j }|dk r||}t j dd|}| |}|dk r||}t||g} |jr| || S)NrZg?r[)r*matmulrEfloatsizerwhereboolr;tor:rHZSoftmaxrJr@r) rOqkvattention_mask head_maskrZndnsrdoutputsr5r5r6_attns  &  zAttention._attncCsD|dddd}|dd|d|df}|j|S)Nrrrr=rgrZ)permuter]rjrC)rOx new_x_shaper5r5r6 merge_headss&zAttention.merge_headscCsX|dd|j|d|jf}|j|}|rD|ddddS|ddddSdS)NrZrrr=r)rjrArCrv)rOrwrorxr5r5r6 split_headss & zAttention.split_headscCs||}|j|jdd\}}}||}|j|dd}||}|dk r|ddd|d} } tj| |fdd}tj| |fdd}|dkrt|dd|f} nd } ||||||} | d} | | } | | } | | } | | g| dd}|S) Nrr[T)rorrgrZrN) rFr"rDrzZ transposer*rastackruryrGrL)rOrw layer_pastrqrr use_cachequerykeyvalueZpast_keyZ past_valuepresentZ attn_outputsartr5r5r6forwards&      zAttention.forward)F)NN)F)NNNF) __name__ __module__ __qualname__r?rfruryrzr __classcell__r5r5rSr6r8bs   r8cs$eZdZfddZddZZS)MLPcsFt|j}t|||_t|||_t|j|_t |j |_ dSr{) r>r?n_embdrc_fcrGrZactivation_functionactrHrIrKdropout)rOrRr-rPrSr5r6r?s     z MLP.__init__cCs$|||}||}||Sr{)rrrGr)rOrwrVh2r5r5r6rs z MLP.forwardrrrr?rrr5r5rSr6rs rcs(eZdZdfdd ZdddZZS) BlockFcsXt|j}tj||jd|_t|||||_tj||jd|_ t d|||_ dS)NZeps) r>r?rrH LayerNormlayer_norm_epsilonln_1r8attnln_2rmlp)rOrQr-rErPrSr5r6r?s  zBlock.__init__Nc CsX|j||||||d}|d}||}|||}||}|g|dd} | S)Nr}rqrrr~rr)rrrr) rOrwr}rqrrr~Z output_attnrmrtr5r5r6rsz Block.forward)F)NNNFrr5r5rSr6rsrcs8eZdZdZeZeZeZ dZ fddZ ddZ Z S)GPT2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformercstj||dSr{)r>r?)rOinputskwargsrSr5r6r?szGPT2PreTrainedModel.__init__cCs|t|tjtjtfrR|jjjd|jj dt|tjtfrx|j dk rx|j j n&t|tj rx|j j |jj ddS)z! Initialize the weights. g)ZmeanZstdN?) isinstancerHLinear Embeddingrrr+Znormal_r-Zinitializer_rangerZzero_rZfill_)rOmoduler5r5r6 _init_weightss  z!GPT2PreTrainedModel._init_weights)rrr__doc__rZ config_class!GPT2_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapr7Zload_tf_weightsZbase_model_prefixr?rrr5r5rSr6rs ran This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.GPT2Config`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. If `past` is used, optionally only the last `input_ids` have to be input (see `past`). Indices can be obtained using :class:`transformers.GPT2Tokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ past (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `past` output below). Can be used to speed up sequential decoding. If `past` is used, the user can optionally input only the last `input_ids` (those that don't have their past given to this model) of shape :obj:`(batch_size, 1)` instead of all `input_ids` of shape :obj:`(batch_size, sequence_length)`. attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, input_ids_length)`, `optional`, defaults to :obj:`None`): `input_ids_length` = `sequence_length if `past` is None else 1 Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token If `past` is used, optionally only the last `token_type_ids` have to be input (see `past`). `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `past` is used, optionally only the last `input_embeds` have to be input (see `past`). use_cache (:obj:`bool`): If `use_cache` is True, `past` key value states are returned and can be used to speed up decoding (see `past`). Defaults to `True`. z^The bare GPT2 Model transformer outputting raw hidden-states without any specific head on top.c sFeZdZfddZddZddZddZeed d d Z Z S) GPT2Modelcstj|_j|_tjj|_tj j|_ t j |_ tfddtjD|_tjjjd|_|dS)Ncsg|]}tjddqS)T)rE)rrQ)rU_r-r5r6 ]sz&GPT2Model.__init__..r)r>r?output_hidden_statesr@rHr vocab_sizerrZ n_positionsrrIZ embd_pdropdropZ ModuleListrangen_layerrVrrln_f init_weightsrOr-rSrr6r?Us  zGPT2Model.__init__cCs|jSr{rrOr5r5r6get_input_embeddingsbszGPT2Model.get_input_embeddingscCs ||_dSr{r)rOZnew_embeddingsr5r5r6set_input_embeddingseszGPT2Model.set_input_embeddingscCs(|D]\}}|j|j|qdS)zz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrVrrf)rOZheads_to_pruneZlayerrcr5r5r6 _prune_headshszGPT2Model._prune_headsNTc s~|dk r\|dk r$|ddddf}|dk r@|ddddf}|dk r\|ddddf}|dk rv|dk rvtdnX|dk r|} |d| d}|jd} n,|dk r|dd} |jd} ntd|dk r|d| d}|dk r|d| d}|dkrd} dgt|j}n|ddd} |dkr|dk rJ|jn|j} tj| | d| tj | d}| dd| d}|dk r| dkst d|| d}| d  d }|j t |jd }d |d }|||jj}|dkr||}||} |dk r ||}nd}|| |}||}| |df}d}g}d}tt|j|D]v\}\}}|jr||j|f}|||||||d}|dd \}}|dkr||f}|jrf||d qf||}|j|}|jr||f}|f}|dkr||f}|jr0||f}|jrz| ddd|djddtfdd|D}||f}|S)aO Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.GPT2Config`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the last layer of the model. If `past` is used only the last hidden-state of the sequences of shape :obj:`(batch_size, 1, hidden_size)` is output. past (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers` with each tensor of shape :obj:`(2, batch_size, num_heads, sequence_length, embed_size_per_head)`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import GPT2Tokenizer, GPT2Model import torch tokenizer = GPT2Tokenizer.from_pretrained('gpt2') model = GPT2Model.from_pretrained('gpt2') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NrZzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrg)r:devicez$batch_size has to be defined and > 0rrr9rr<r5rT)rZc3s|]}|jVqdSr{)rC)rUtZattention_output_shaper5r6rY sz$GPT2Model.forward..) ValueErrorrjrCr'r%rVrr*r_r` unsqueezer(rmnext parametersr:Z get_head_maskr-rrrr enumerater!rr@rrtuple)rO input_idspastrqtoken_type_ids position_idsrr inputs_embedsr~Z input_shapeZ batch_sizeZ past_lengthrZposition_embedsZtoken_type_embeds hidden_statesZ output_shapeZpresentsZall_attentionsZall_hidden_statesiblockr}rtrr5rr6ros0                       " zGPT2Model.forward)NNNNNNNT) rrrr?rrrrGPT2_INPUTS_DOCSTRINGrrr5r5rSr6rPs rz~The GPT2 Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). c s>eZdZfddZddZddZeed d d ZZ S) GPT2LMHeadModelcs8t|t||_tj|j|jdd|_| dS)NFr) r>r?rrrHrrrlm_headrrrSr5r6r?s  zGPT2LMHeadModel.__init__cCs|jSr{rrr5r5r6get_output_embeddingssz%GPT2LMHeadModel.get_output_embeddingscKs*|r|dddfd}|||ddS)NrZr~)rrr~)r)rOrrrr5r5r6prepare_inputs_for_generation!sz-GPT2LMHeadModel.prepare_inputs_for_generationNTc  Cs|j|||||||| d} | d} || } | f| dd} |dk r| dddddf}|dddf}t}||d|d|d}|f| } | S)aS labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids`` Indices are selected in ``[-100, 0, ..., config.vocab_size]`` All labels set to ``-100`` are ignored (masked), the loss is only computed for labels in ``[0, ..., config.vocab_size]`` Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.GPT2Config`) and inputs: loss (:obj:`torch.FloatTensor` of shape `(1,)`, `optional`, returned when ``labels`` is provided) Language modeling loss. prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers` with each tensor of shape :obj:`(2, batch_size, num_heads, sequence_length, embed_size_per_head)`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import torch from transformers import GPT2Tokenizer, GPT2LMHeadModel tokenizer = GPT2Tokenizer.from_pretrained('gpt2') model = GPT2LMHeadModel.from_pretrained('gpt2') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=input_ids) loss, logits = outputs[:2] rrqrrrrrr~rrN.rZ)rrr]rrCrj)rOrrrqrrrrrlabelsr~transformer_outputsr lm_logitsrt shift_logits shift_labelsloss_fctlossr5r5r6r(s(7   zGPT2LMHeadModel.forward) NNNNNNNNT) rrrr?rrrrrrr5r5rSr6rs rarThe GPT2 Model transformer with a language modeling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks. The two heads are two linear layers. The language modeling head has its weights tied to the input embeddings, the classification head takes as input the input of a specified classification token index in the input sequence). c s6eZdZfddZddZeed ddZZS) GPT2DoubleHeadsModelcsHt|d|_t||_tj|j|jdd|_ t ||_ | dS)NrFr) r>r?Z num_labelsrrrHrrrrr multiple_choice_headrrrSr5r6r?s    zGPT2DoubleHeadsModel.__init__cCs|jSr{rrr5r5r6rsz*GPT2DoubleHeadsModel.get_output_embeddingsNTc  Cs|j|||||||| d} | d} || }|| |d}||f| dd}| dk rt}||d|d| d}|f|}| dk r|dddddf}| dddf}t}||d|d|d}|f|}|S)a2 mc_token_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, num_choices)`, `optional`, default to index of the last token of the input) Index of the classification token in each input sequence. Selected in the range ``[0, input_ids.size(-1) - 1[``. lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`) Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids`` Indices are selected in ``[-1, 0, ..., config.vocab_size]`` All labels set to ``-100`` are ignored (masked), the loss is only computed for labels in ``[0, ..., config.vocab_size]`` mc_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size)`, `optional`, defaults to :obj:`None`) Labels for computing the multiple choice classification loss. Indices should be in ``[0, ..., num_choices]`` where `num_choices` is the size of the second dimension of the input tensors. (see `input_ids` above) Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.GPT2Config`) and inputs: lm_loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when ``lm_labels`` is provided): Language modeling loss. mc_loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`multiple_choice_labels` is provided): Multiple choice classification loss. lm_prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_choices, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). mc_prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_choices)`): Prediction scores of the multiple choice classification head (scores for each choice before SoftMax). past (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers` with each tensor of shape :obj:`(2, batch_size, num_heads, sequence_length, embed_size_per_head)`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import torch from transformers import GPT2Tokenizer, GPT2DoubleHeadsModel tokenizer = GPT2Tokenizer.from_pretrained('gpt2') model = GPT2DoubleHeadsModel.from_pretrained('gpt2') # Add a [CLS] to the vocabulary (we should train it also!) tokenizer.add_special_tokens({'cls_token': '[CLS]'}) model.resize_token_embeddings(len(tokenizer)) # Update the model embeddings with the new vocabulary size print(tokenizer.cls_token_id, len(tokenizer)) # The newly token the last token of the vocabulary choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"] encoded_choices = [tokenizer.encode(s) for s in choices] cls_token_location = [tokens.index(tokenizer.cls_token_id) for tokens in encoded_choices] input_ids = torch.tensor(encoded_choices).unsqueeze(0) # Batch size: 1, number of choices: 2 mc_token_ids = torch.tensor([cls_token_location]) # Batch size: 1 outputs = model(input_ids, mc_token_ids=mc_token_ids) lm_prediction_scores, mc_prediction_scores = outputs[:2] rrrZrN.)rrrr rrCrjr])rOrrrqrrrrrZ mc_token_idsZ lm_labelsZ mc_labelsr~rrrZ mc_logitsrtrrrrr5r5r6rs2O    zGPT2DoubleHeadsModel.forward) NNNNNNNNNNT) rrrr?rrrrrr5r5rSr6rzs r)"rloggingrr*Ztorch.nnrHrZ activationsrZconfiguration_gpt2rZ file_utilsrrZmodeling_utilsrr r r getLoggerrrrr7Moduler8rrrZGPT2_START_DOCSTRINGrrrrr5r5r5r6sN      7l 1>d