U &cA@s dZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z mZmZmZmZmZeeZdd d d d Zd ZdZe deGdddeZe deGdddeZe deGdddeZe deGdddeZe deGddde ZdS)z' PyTorch Flaubert model, based on XLM. N) functional)FlaubertConfig)add_start_docstrings add_start_docstrings_to_callable)XLMForQuestionAnsweringXLMForQuestionAnsweringSimpleXLMForSequenceClassificationXLMModelXLMWithLMHeadModel get_maskszJhttps://cdn.huggingface.co/flaubert/flaubert_small_cased/pytorch_model.binzKhttps://cdn.huggingface.co/flaubert/flaubert_base_uncased/pytorch_model.binzIhttps://cdn.huggingface.co/flaubert/flaubert_base_cased/pytorch_model.binzJhttps://cdn.huggingface.co/flaubert/flaubert_large_cased/pytorch_model.bin)zflaubert-small-casedzflaubert-base-uncasedzflaubert-base-casedzflaubert-large-casedar This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.FlaubertConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.BertTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1`` corresponds to a `sentence B` token `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ lengths (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Length of each sentence that can be used to avoid performing attention on padding token indices. You can also use `attention_mask` for the same result (see above), kept here for compatbility. Indices selected in ``[0, ..., input_ids.size(-1)]``: cache (:obj:`Dict[str, torch.FloatTensor]`, `optional`, defaults to :obj:`None`): dictionary with ``torch.FloatTensor`` that contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `cache` output below). Can be used to speed up sequential decoding. The dictionary object will be modified in-place during the forward pass to add newly computed hidden-states. head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. zbThe bare Flaubert Model transformer outputting raw hidden-states without any specific head on top.c s6eZdZeZeZfddZee dddZ Z S) FlaubertModelcs,t|t|dd|_t|dd|_dS)N layerdropgpre_normF)super__init__getattrrrselfconfig __class__B/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_flaubert.pyrrs zFlaubertModel.__init__Nc Cs@|dk r|\} } n| dd\} } |dkrb|dk rR||jkjdd}nt| g| }|d| kstt|| kstt | ||j |d\} } |dk r|j n| j }|dkrtj | tj|d}| d| | f}n|| | fkst|dk r|| | fkst|||jj}|dk r|dk r| |d}|dd| df}|dd| df}|dk r|dd| df}| dd| df} | dd| df} | dkr||} | ||| }|dk r|jr|jjdkr|||}|dk r,|||}||}tj||j|jd }|| d|j9}d }d }t|jD]j}t dd}|jr||j!krqr|j"r||f}|j#s|j$||| |||d }|d}|j%r||df}tj||j|jd }||}|j&||}nb|j&||}|j$||| |||d }|d}|j%r`||df}tj||j|jd }||}|j#s||j'||}|j(||}n |j(||}||j'||}|| d|j9}qr|j"r||f}|dk r|d|d7<|f}|j"r*||f}|j%r<||f}|S) a Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import FlaubertTokenizer, FlaubertModel import torch tokenizer = FlaubertTokenizer.from_pretrained('flaubert-base-cased') model = FlaubertModel.from_pretrained('flaubert-base-cased') input_ids = torch.tensor(tokenizer.encode("Le chat mange une pomme.", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple Nr)Zdimr)Z padding_mask)dtypedeviceslen)ptrainingr)cache head_mask))sizeZ pad_indexsumlongtorchZ LongTensorAssertionErrormaxitemr ZcausalrZarangeZ unsqueezeexpandZ get_head_maskrZn_layersZ embeddingsZposition_embeddingsZ expand_asZ use_lang_embZn_langsZlang_embeddingsZlayer_norm_embFZdropoutrtorrangerandomuniformrZoutput_hidden_statesr attentionsZoutput_attentionsZ layer_norm1ZffnsZ layer_norm2)rZ input_idsZattention_maskZlangsZtoken_type_idsZ position_idslengthsr r!Z inputs_embedsbsrmaskZ attn_maskrZ_slenZtensorZ hidden_statesr/iZdropout_probabilityZ attn_outputsZattnZtensor_normalizedoutputsrrrforwardws+               zFlaubertModel.forward) NNNNNNNNN) __name__ __module__ __qualname__r config_class%FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAPpretrained_model_archive_maprrFLAUBERT_INPUTS_DOCSTRINGr5 __classcell__rrrrr is r zThe Flaubert Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). cs(eZdZdZeZeZfddZZ S)FlaubertWithLMHeadModelz This class overrides :class:`~transformers.XLMWithLMHeadModel`. Please check the superclass for the appropriate documentation alongside usage examples. cs"t|t||_|dSNrrr Z transformerZ init_weightsrrrrr5s  z FlaubertWithLMHeadModel.__init__ r6r7r8__doc__rr9r:r;rr=rrrrr>'sr>zFlaubert Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs(eZdZdZeZeZfddZZ S)!FlaubertForSequenceClassificationz This class overrides :class:`~transformers.XLMForSequenceClassification`. Please check the superclass for the appropriate documentation alongside usage examples. cs"t|t||_|dSr?r@rrrrrIs  z*FlaubertForSequenceClassification.__init__rArrrrrC;srCzFlaubert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs(eZdZdZeZeZfddZZ S)"FlaubertForQuestionAnsweringSimplez This class overrides :class:`~transformers.XLMForQuestionAnsweringSimple`. Please check the superclass for the appropriate documentation alongside usage examples. cs"t|t||_|dSr?r@rrrrr]s  z+FlaubertForQuestionAnsweringSimple.__init__rArrrrrDOsrDzFlaubert Model with a beam-search span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs(eZdZdZeZeZfddZZ S)FlaubertForQuestionAnsweringz This class overrides :class:`~transformers.XLMForQuestionAnswering`. Please check the superclass for the appropriate documentation alongside usage examples. cs"t|t||_|dSr?r@rrrrrqs  z%FlaubertForQuestionAnswering.__init__rArrrrrEcsrE)rBloggingr-r%Ztorch.nnrr*Zconfiguration_flaubertrZ file_utilsrrZ modeling_xlmrrr r r r getLoggerr6loggerr:ZFLAUBERT_START_DOCSTRINGr<r r>rCrDrErrrrsR    /;