U &c=@sdZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZddlmZeeZiZdZd Zed eGd d d e ZGd dde ZedeGddde ZedeGddde ZdS)z TF 2.0 Flaubert model. N)FlaubertConfig)add_start_docstrings)TFXLMForSequenceClassificationTFXLMMainLayer TFXLMModelTFXLMWithLMHeadModel get_masks shape_list) BatchEncodingao This model is a `tf.keras.Model `__ sub-class. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.FlaubertConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ langs (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): A parallel sequence of tokens to be used to indicate the language of each token in the input. Indices are languages ids which can be obtained from the language names by using two conversion mappings provided in the configuration of the model (only provided for multilingual models). More precisely, the `language name -> language id` mapping is in `model.config.lang2id` (dict str -> int) and the `language id -> language name` mapping is `model.config.id2lang` (dict int -> str). See usage examples detailed in the `multilingual documentation `__. token_type_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ lengths (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Length of each sentence that can be used to avoid performing attention on padding token indices. You can also use `attention_mask` for the same result (see above), kept here for compatbility. Indices selected in ``[0, ..., input_ids.size(-1)]``: cache (:obj:`Dict[str, tf.Tensor]`, `optional`, defaults to :obj:`None`): dictionary with ``tf.Tensor`` that contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `cache` output below). Can be used to speed up sequential decoding. The dictionary object will be modified in-place during the forward pass to add newly computed hidden-states. head_mask (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`tf.Tensor` or :obj:`Numpy array` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. zbThe bare Flaubert Model transformer outputting raw hidden-states without any specific head on top.cs$eZdZeZeZfddZZS)TFFlaubertModelcs&tj|f||t|dd|_dSN transformer)namesuper__init__TFFlaubertMainLayerrselfconfiginputskwargs __class__E/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_tf_flaubert.pyrmszTFFlaubertModel.__init__ __name__ __module__ __qualname__rZ config_class(TF_FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapr __classcell__rrrrr esr c s&eZdZfddZdddZZS)rcs4tj|f||t|dd|_t|dd|_dS)N layerdropgpre_normF)rrgetattrr#r$rrrrrsszTFFlaubertMainLayer.__init__NFc Cst|ttfr|d} t|dkr*|dn|}t|dkrB|dn|}t|dkrZ|dn|}t|dkrr|dn|}t|dkr|dn|}t|dkr|dn|}t|dkr|dn|}t|d kr|d n| } t|d kstd nt|ttfr~|d } |d |}|d|}|d|}|d|}|d|}|d|}|d|}|d| } t|d kstd n|} | dk r| dk rtdn@| dk rt | \} } n(| dk rt | dd\} } ntd|dkr.| dk rt j t j t | |jt jddd}nt | g| t j}t jt |d| t| ||j|d\}}|dkrzt jt | dd}nt jt || | g|dk rt jt || | g|dk rtn dg|j}|dk rd| dk rd| |d}| dd| df} |dd| df}|dk r8|dd| df}|dd| df}|dd| df}| dkrx|| } | ||}|dk r|jr|||}|dk r|||}||}|j|| d}||dt jf}d}d}t|jD]d}t !dd}| r"||j"kr"q|j#r4||f}|j$s|j%|||d|||g| d}|d}|j&rz||df}|j|| d}||}|j'||}nb|j'||}|j%|||d|||g| d}|d}|j&r||df}|j|| d}||}|j$s,||j(||}|j)||}n |j)||}||j(||}||dt jf}q|j#rt||f}|dk r|d|*d7<|f}|j#r||f}|j&r||f}|S)Nrr zToo many inputs. input_idsattention_masklangstoken_type_ids position_idslengthscache head_mask inputs_embedszDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embeds)Zdtype)Zaxis)Z padding_maskslen)training.r)+ isinstancetuplelistlenAssertionErrordictr get ValueErrorr tfZ reduce_sumcast not_equalZ pad_indexZint32Zconvert_to_tensor debuggingZ assert_equalr ZcausalZ expand_dimsrangeNotImplementedErrorZn_layersZ embeddingsZposition_embeddingsZ use_lang_embZlang_embeddingsZlayer_norm_embZdropoutZnewaxisrandomuniformr#Zoutput_hidden_statesr$ attentionsZoutput_attentionsZ layer_norm1ZffnsZ layer_norm2size)rrr/r0r1r2r3r4r5r6r8r.bsr7maskZ attn_maskZ_slenZtensorZ hidden_statesrIiZdropout_probabilityZ attn_outputsZattnZtensor_normalizedoutputsrrrcallxs              &                   zTFFlaubertMainLayer.call) NNNNNNNNF)rrr rrOr"rrrrrrs rzThe Flaubert Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). cs$eZdZeZeZfddZZS)TFFlaubertWithLMHeadModelcs&tj|f||t|dd|_dSr rrrrrr:sz"TFFlaubertWithLMHeadModel.__init__rrrrrrP1srPzFlaubert Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs$eZdZeZeZfddZZS)#TFFlaubertForSequenceClassificationcs&tj|f||t|dd|_dSr rrrrrrHsz,TFFlaubertForSequenceClassification.__init__rrrrrrQ?srQ)__doc__loggingrGZ tensorflowrAZconfiguration_flaubertrZ file_utilsrZmodeling_tf_xlmrrrrr r Ztokenization_utilsr getLoggerrloggerr!ZFLAUBERT_START_DOCSTRINGZFLAUBERT_INPUTS_DOCSTRINGr rrPrQrrrrs8      1 @