U &cw@sndZddlZddlZddlZddlmZddlm Z m Z ddl m Z m Z mZmZmZmZddlmZeeZdd iZd d Zd d ZejjeejjjejjedZGdddejjj Z!Gdddejjj Z"Gdddejjj Z#Gdddejjj Z$Gddde Z%dZ&dZ'e de&Gddde%Z(e de&Gdd d e%Z)e d!e&Gd"d#d#e%Z*dS)$z TF 2.0 OpenAI GPT model.N)OpenAIGPTConfig)add_start_docstrings add_start_docstrings_to_callable)TFConv1DTFPreTrainedModelTFSequenceSummaryTFSharedEmbeddingsget_initializer shape_list) BatchEncodingz openai-gptz1https://cdn.huggingface.co/openai-gpt-tf_model.h5c Cs:ddttdtj|dt|d}||S)zGaussian Error Linear Unit. This is a smoother version of the RELU. Original paper: https://arxiv.org/abs/1606.08415 Args: x: float Tensor to perform activation. Returns: `x` with the GELU activation applied. g??gHm?)tftanhnpsqrtpipow)xZcdfrC/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_tf_openai.pygelu*s 2rcCs|tj|SN)rmathZsigmoid)rrrrswish7sr)rrelurcsVeZdZdfdd ZddZeddZddd Zd d Zd d Z dddZ Z S) TFAttentionFc stjf||j|_|}||jdks,t||_|j|_||_||_t|d||j dd|_ t|||j dd|_ t j j|j|_t j j|j|_t|_dS)Nrrc_attninitializer_rangenamec_proj)super__init__output_attentionsn_headAssertionErrorn_ctxZ split_sizescalerr!rr#rkeraslayersDropoutZ attn_pdrop attn_dropout resid_pdrop resid_dropoutsetZ pruned_heads)selfnxr)configr*kwargsn_state __class__rrr%CszTFAttention.__init__cCsdSrr)r2Zheadsrrr prune_headsUszTFAttention.prune_headscCs<t|dddf}t|}||||k}t||S)z1's in the lower triangle, counting from the lower right corner. Same as tf.matrix_band_part(tf.ones([nd, ns]), -1, ns-nd), but doesn't produce garbage on TPUs. N)rrangecast)ndnsdtypeijmrrrcausal_attention_maskXs z!TFAttention.causal_attention_maskcCs|\}}}}}tj||dd}|jrJtt|dtj} |tj| }t|\} } } } |j| | |j d} t | dd| | g} || dd| }|dk r||}tj j |dd}|j ||d}|dk r||}t||g}|jr|||S) NT)Z transpose_br>rg@Zaxistraining)rmatmulr*r;r float32rrrBr>reshapennZsoftmaxr.r&append)r2inputsrGqkvattention_mask head_maskwZdk_r<r=boutputsrrr_attnbs& zTFAttention._attncCsFt|ddddg}t|}|dd|d|dg}t||S)NrrrrrC)r transposer rJr2rZx_shapeZ new_x_shaperrr merge_headsszTFAttention.merge_headscCs@t|}|dd|j|d|jg}t||}t|dS)NrC)rrrr)r r'rrJrYrZrrr split_headss  zTFAttention.split_headsc Cs|\}}}||}tj|ddd\}}}||}||}||}|j|||||g|d} | d} || } || } |j| |d} | g| dd} | S)NrrrErFrr)rrsplitr\rWr[r#r0) r2rMrGrrQrRquerykeyvalueZ attn_outputsarVrrrcalls       zTFAttention.call)F)F)F) __name__ __module__ __qualname__r%r9 staticmethodrBrWr[r\rb __classcell__rrr7rrBs rcs&eZdZfddZdddZZS)TFMLPc sXtjf||j}t|||jdd|_t|||jdd|_t|_t j j |j |_dS)Nc_fcr r#)r$r%n_embdrr!rir#ractrr+r,r-r/dropout)r2r6r4r5r3r7rrr%s zTFMLP.__init__FcCs,|||}||}|j||d}|S)NrF)rkrir#rl)r2rrGhh2rrrrbs z TFMLP.call)Frcrdrer%rbrgrrr7rrhs rhcs(eZdZdfdd ZdddZZS)TFBlockFc sltjf||j}t||||dd|_tjjj|j dd|_ t d||dd|_ tjjj|j dd|_ dS)Nattnr"ln_1)epsilonr"mlpln_2)r$r%rjrrqrr+r,ZLayerNormalizationZlayer_norm_epsilonrsrhrvrw)r2r)r4r*r5r3r7rrr%s zTFBlock.__init__c Csf|\}}}|j|||g|d}|d}|||}|j||d} ||| } | g|dd} | S)NrFrr)rqrsrvrw) r2rMrGrrQrRZ output_attnranrArmrVrrrrbs z TFBlock.call)F)Frorrr7rrpsrpcs>eZdZfddZddZddZddZd d d ZZS)TFOpenAIGPTMainLayercstj||j|_j|_j|_j|_j|_tjjj dd|_ t j j jjjtj dd|_t j j j|_fddtjD|_dS)N tokens_embedr positions_embed)Zembeddings_initializerr"c s$g|]}tjdd|dqS)Tzh_._{})r*r")rpr)format).0r?r4rr sz1TFOpenAIGPTMainLayer.__init__..)r$r%output_hidden_statesr&Zn_layernum_hidden_layersZ vocab_sizerjr r!rzrr+r,Z EmbeddingZ n_positionsr r{r-Z embd_pdropdropr:rmr2r4rMr5r7r~rr%s(zTFOpenAIGPTMainLayer.__init__cCs|jSr)rzr2rrrget_input_embeddingssz)TFOpenAIGPTMainLayer.get_input_embeddingscCstdSrNotImplementedError)r2Znew_num_tokensrrr_resize_token_embeddingssz-TFOpenAIGPTMainLayer._resize_token_embeddingscCstdS)zz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} Nr)r2Zheads_to_prunerrr _prune_headssz!TFOpenAIGPTMainLayer._prune_headsNFcst|ttfr|d}t|dkr*|dn|}t|dkrB|dn|}t|dkrZ|dn|}t|dkrr|dn|}t|dkr|dn|}t|dkstdnrt|ttfr|d }|d |}|d |}|d |}|d |}|d|}t|dkstdn|}|dk r4|dk r4tdnL|dk r\t |} t |d| dg}n$|dk rxt |dd} ntd|dkrt j | dt j dt jddf}|dk r|ddt jt jddf}t |t j}d|d}nd}|dk rtn dg|j}t |dt |dg}|dkr>|j|dd}||} |dk rzt |dt |dg}|j|dd} nd} || | } |j| |d} | t | dg} g}d}t|jD]X\}}|jr|t | | f}|| |||g|d}|d} |jr||dqt | | } |jr4|| f}| f}|jrL||f}|jr| dddgt |dddtfdd|D}||f}|S)NrrrrruToo many inputs. input_idsrQtoken_type_ids position_idsrR inputs_embedszDYou cannot specify both input_ids and inputs_embeds at the same timerCz5You have to specify either input_ids or inputs_embedsrDr gZ embeddingmoderFrrXc3s|]}t|VqdSr)rrJ)r}tZattention_output_shaperr Usz,TFOpenAIGPTMainLayer.call..) isinstancetuplelistlenr(dictr get ValueErrorr rrJr:Zint32Znewaxisr;rIrrrzr{r enumeratermrr&rL)r2rMrQrrrRrrGrZ input_shapeZposition_embedsZtoken_type_embeds hidden_statesZ output_shapeZall_attentionsZall_hidden_statesr?blockrVrrrrbs           "          & zTFOpenAIGPTMainLayer.call)NNNNNF) rcrdrer%rrrrbrgrrr7rrys  ryc@seZdZdZeZeZdZdS)TFOpenAIGPTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerN) rcrdre__doc__rZ config_class*TF_OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZbase_model_prefixrrrrrZsra .. note:: TF 2.0 models accepts two formats as inputs: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional arguments. This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having all the tensors in the first argument of the model call function: :obj:`model(inputs)`. If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the first positional argument : - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})` Parameters: config (:class:`~transformers.OpenAIGPTConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.GPT2Tokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. training (:obj:`boolean`, `optional`, defaults to :obj:`False`): Whether to activate dropout modules (if set to :obj:`True`) during training or to de-activate them (if set to :obj:`False`) for evaluation. zcThe bare OpenAI GPT transformer model outputing raw hidden-states without any specific head on top.cs,eZdZfddZeeddZZS)TFOpenAIGPTModelcs&tj|f||t|dd|_dSNrrrr$r%ryrrr7rrr%szTFOpenAIGPTModel.__init__cKs|j|f|}|S)as Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.OpenAIGPTConfig`) and inputs: last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the last layer of the model. hidden_states (:obj:`tuple(tf.Tensor)` `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import OpenAIGPTTokenizer, TFOpenAIGPTModel tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt') model = TFOpenAIGPTModel.from_pretrained('openai-gpt') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple )r)r2rMr5rVrrrrbszTFOpenAIGPTModel.call)rcrdrer%rOPENAI_GPT_INPUTS_DOCSTRINGrbrgrrr7rrs rzOpenAI GPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). cs4eZdZfddZddZeeddZZS)TFOpenAIGPTLMHeadModelcs&tj|f||t|dd|_dSrrrr7rrr%szTFOpenAIGPTLMHeadModel.__init__cCs|jjSrrrzrrrrget_output_embeddingssz,TFOpenAIGPTLMHeadModel.get_output_embeddingscKs<|j|f|}|d}|jj|dd}|f|dd}|S)aa Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.OpenAIGPTConfig`) and inputs: prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: import tensorflow as tf from transformers import OpenAIGPTTokenizer, TFOpenAIGPTLMHeadModel tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt') model = TFOpenAIGPTLMHeadModel.from_pretrained('openai-gpt') input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :] # Batch size 1 outputs = model(input_ids) logits = outputs[0] rlinearrrNr)r2rMr5transformer_outputsr lm_logitsrVrrrrbs zTFOpenAIGPTLMHeadModel.call rcrdrer%rrrrbrgrrr7rrs ratOpenAI GPT Model transformer with a language modeling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks. The two heads are two linear layers. The language modeling head has its weights tied to the input embeddings, the classification head takes as input the input of a specified classification token index in the input sequence). cs6eZdZfddZddZeed ddZZS) TFOpenAIGPTDoubleHeadsModelcs>tj|f||d|_t|dd|_t||jdd|_dS)Nrrrrmultiple_choice_headr )r$r%Z num_labelsryrrr!rrr7rrr%sz$TFOpenAIGPTDoubleHeadsModel.__init__cCs|jjSrrrrrrrsz1TFOpenAIGPTDoubleHeadsModel.get_output_embeddingsNFc Csdt|ttfr|d} t|dkr*|dn|}t|dkrB|dn|}t|dkrZ|dn|}t|dkrr|dn|}t|dkr|dn|}t|dkr|dn|}t|dkstd nzt|tr2|d } |d |}|d |}|d |}|d|}|d|}|d|}t|dks6td n|} | dk rJt| } nt|dd} | d} | dk r|t | d| fnd} |dk rt |d| fnd} |dk rt |d| fnd}|dk rt |d| fnd}| | ||||g}|j ||d}|d}t || t|dd}|j j |dd}|j ||g|d}tj |dd}||f|dd}|S)a# mc_token_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, num_choices)`, `optional`, default to index of the last token of the input) Index of the classification token in each input sequence. Selected in the range ``[0, input_ids.size(-1) - 1[``. Return: :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.OpenAIGPTConfig`) and inputs: lm_prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, num_choices, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). mc_prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, num_choices)`): Prediction scores of the multiple choice classification head (scores for each choice before SoftMax). past (:obj:`List[tf.Tensor]` of length :obj:`config.n_layers` with each tensor of shape :obj:`(2, batch_size, num_heads, sequence_length, embed_size_per_head)`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `past` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: # For example purposes. Not runnable. import tensorflow as tf from transformers import OpenAIGPTTokenizer, TFOpenAIGPTDoubleHeadsModel tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt') model = TFOpenAIGPTDoubleHeadsModel.from_pretrained('openai-gpt') # Add a [CLS] to the vocabulary (we should train it also!) # This option is currently not implemented in TF 2.0 raise NotImplementedError tokenizer.add_special_tokens({'cls_token': '[CLS]'}) model.resize_token_embeddings(len(tokenizer)) # Update the model embeddings with the new vocabulary size print(tokenizer.cls_token_id, len(tokenizer)) # The newly token the last token of the vocabulary choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"] input_ids = tf.constant([tokenizer.encode(s) for s in choices])[None, :] # Batch size 1, 2 choices mc_token_ids = tf.constant([input_ids.size(-1), input_ids.size(-1)])[None, :] # Batch size 1 outputs = model(input_ids, mc_token_ids=mc_token_ids) lm_prediction_scores, mc_prediction_scores = outputs[:2] rrrrrurrrrrQrrrRr mc_token_idsNrCrFrrrE)rrrrr(rrr rrJrrzrZsqueeze)r2rMrQrrrRrrrGrZ input_shapesZ seq_lengthZflat_input_idsZflat_attention_maskZflat_token_type_idsZflat_position_idsZ flat_inputsrrrZ mc_logitsrVrrrrbsT@           z TFOpenAIGPTDoubleHeadsModel.call)NNNNNNFrrrr7rr s r)+rloggingZnumpyrZ tensorflowrZconfiguration_openairZ file_utilsrrZmodeling_tf_utilsrrrr r r Ztokenization_utilsr getLoggerrcloggerrrrr+r,Z ActivationZ activationsrZACT_FNSZLayerrrhrpryrZOPENAI_GPT_START_DOCSTRINGrrrrrrrrsL       ^ )(1