U &cG@sdZddlZddlZddlmZddlmZmZddlmZddl m Z e e Z GdddejZd Zd Zed eeGd d d e ZedeeGdddejZdS)zPyTorch MMBT model. N)CrossEntropyLossMSELoss)add_start_docstrings)ModuleUtilsMixincs*eZdZdZfddZdddZZS)ModalEmbeddingszUGeneric Modal Embeddings which takes in an encoder, and a transformer embedding. cs\t||_||_t|j|j|_|j |_ |j |_ |j |_ |j |_ tj |jd|_dS)N)p)super__init__configencodernnLinearZmodal_hidden_size hidden_sizeproj_embeddingsposition_embeddingstoken_type_embeddingsword_embeddings LayerNormDropouthidden_dropout_probdropout)selfr r embeddings __class__>/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_mmbt.pyr $s zModalEmbeddings.__init__Nc Cs|||}|d}|dk rL||}|d7}tj|d|gdd}|dk r~||} |d7}tj|| dgdd}|dkrtj|tj|j d}|d |d|}|dkrtj |d|ftj|j d}| |} | |} || | } || } || } | S)NrZdimdtypedevicer)rr sizertorchcatZ unsqueezeZarangelongr!expandzerosrrrr) r input_modal start_token end_token position_idstoken_type_idsZtoken_embeddingsZ seq_lengthZstart_token_embedsZend_token_embedsrrrrrrforward/s2         zModalEmbeddings.forward)NNNN__name__ __module__ __qualname____doc__r r- __classcell__rrrrr s raV MMBT model was proposed in `Supervised Multimodal Bitransformers for Classifying Images and Text`_ by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine. It's a supervised multimodal bitransformer model that fuses information from text and other image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks. This model is a PyTorch `torch.nn.Module`_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. .. _`Supervised Multimodal Bitransformers for Classifying Images and Text`: https://github.com/facebookresearch/mmbt .. _`torch.nn.Module`: https://pytorch.org/docs/stable/nn.html#module Parameters: config (:class:`~transformers.MMBTConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. transformer (:class: `~nn.Module`): A text transformer that is used by MMBT. It should have embeddings, encoder, and pooler attributes. encoder (:class: `~nn.Module`): Encoder for the second modality. It should take in a batch of modal inputs and return k, n dimension embeddings. a Inputs: **input_modal**: ``torch.FloatTensor`` of shape ``(batch_size, ***)``: The other modality data. It will be the shape that the encoder for that type expects. e.g. With an Image Encoder, the shape would be (batch_size, channels, height, width) **input_ids**: ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``: Indices of input sequence tokens in the vocabulary. It does not expect [CLS] token to be added as it's appended to the end of other modality embeddings. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details. **modal_start_tokens**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size,)``: Optional start token to be added to Other Modality Embedding. [CLS] Most commonly used for Classification tasks. **modal_end_tokens**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size,)``: Optional end token to be added to Other Modality Embedding. [SEP] Most commonly used. **attention_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, sequence_length)``: Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. **token_type_ids**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``: Segment token indices to indicate different portions of the inputs. **modal_token_type_ids**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, modal_sequence_length)``: Segment token indices to indicate different portions of the non-text modality. The embeddings from these tokens will be summed with the respective token embeddings for the non-text modality. **position_ids**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``: Indices of positions of each input sequence tokens in the position embeddings. **modal_position_ids**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, modal_sequence_length)``: Indices of positions of each input sequence tokens in the position embeddings for the non-text modality. **head_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(num_heads,)`` or ``(num_layers, num_heads)``: Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**. **inputs_embeds**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, sequence_length, embedding_dim)``: Optionally, instead of passing ``input_ids`` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. **encoder_hidden_states**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, sequence_length, hidden_size)``: Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. **encoder_attention_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, sequence_length)``: Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. zRThe bare MMBT Model outputting raw hidden-states without any specific head on top.c s:eZdZdZfddZd ddZddZd d ZZS) MMBTModela Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs: **last_hidden_state**: ``torch.FloatTensor`` of shape ``(batch_size, sequence_length, hidden_size)`` Sequence of hidden-states at the output of the last layer of the model. **pooler_output**: ``torch.FloatTensor`` of shape ``(batch_size, hidden_size)`` Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during Bert pretraining. This output is usually *not* a good summary of the semantic content of the input, you're often better with averaging or pooling the sequence of hidden-states for the whole input sequence. **hidden_states**: (`optional`, returned when ``config.output_hidden_states=True``) list of ``torch.FloatTensor`` (one for the output of each layer + the output of the embeddings) of shape ``(batch_size, sequence_length, hidden_size)``: Hidden-states of the model at the output of each layer plus the initial embedding outputs. **attentions**: (`optional`, returned when ``config.output_attentions=True``) list of ``torch.FloatTensor`` (one for each layer) of shape ``(batch_size, num_heads, sequence_length, sequence_length)``: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: # For example purposes. Not runnable. transformer = BertModel.from_pretrained('bert-base-uncased') encoder = ImageEncoder(args) mmbt = MMBTModel(config, transformer, encoder) cs*t||_||_t|||j|_dSN)r r r transformerrr modal_encoderrr r6r rrrr s zMMBTModel.__init__NcCs|dk r| dk rtdn4|dk r,|}n"| dk rF| dd}ntd|dk r\|jn| j}|j|||| |d}|dd}|dkrtj|tj|d}|jj|||| d}t ||gd}|dd}|dkrtj||d}n tj tj||tjd |gdd }| dkr&tj||d} ntj tj||d| gdd } | |||j}| | }| | |j j} |jj||| | |d }|d }|j|}||f|dd}|S) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embeds)r)r*r+r,r) input_idsr+r, inputs_embedsr)r!)r!r r)attention_mask head_maskencoder_hidden_statesencoder_attention_maskr) ValueErrorr"r!r7r#Zonesr%r6rr$Zget_extended_attention_maskZinvert_attention_maskZ get_head_maskr Znum_hidden_layersr Zpooler)rr(r:modal_start_tokensmodal_end_tokensr<r,modal_token_type_idsr+modal_position_idsr=r;r>r?Zinput_txt_shaper!Zmodal_embeddingsZinput_modal_shapeZtxt_embeddingsZembedding_outputZ input_shapeZextended_attention_maskZencoder_extended_attention_maskZencoder_outputsZsequence_output pooled_outputoutputsrrrr-sl      zMMBTModel.forwardcCs|jjSr5rr)rrrrget_input_embeddings szMMBTModel.get_input_embeddingscCs ||j_dSr5rG)rvaluerrrset_input_embeddingsszMMBTModel.set_input_embeddings) NNNNNNNNNNNN) r/r0r1r2r r-rHrJr3rrrrr4s"  Rr4zMMBT Model with a sequence classification/regression head on top (a linear layer on top of the pooled output)c s*eZdZdZfddZdddZZS)MMBTForClassificationa **labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size,)``: Labels for computing the sequence classification/regression loss. Indices should be in ``[0, ..., config.num_labels - 1]``. If ``config.num_labels == 1`` a regression loss is computed (Mean-Square loss), If ``config.num_labels > 1`` a classification loss is computed (Cross-Entropy). Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs: **loss**: (`optional`, returned when ``labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``: Classification (or regression if config.num_labels==1) loss. **logits**: ``torch.FloatTensor`` of shape ``(batch_size, config.num_labels)`` Classification (or regression if config.num_labels==1) scores (before SoftMax). **hidden_states**: (`optional`, returned when ``config.output_hidden_states=True``) list of ``torch.FloatTensor`` (one for the output of each layer + the output of the embeddings) of shape ``(batch_size, sequence_length, hidden_size)``: Hidden-states of the model at the output of each layer plus the initial embedding outputs. **attentions**: (`optional`, returned when ``config.output_attentions=True``) list of ``torch.FloatTensor`` (one for each layer) of shape ``(batch_size, num_heads, sequence_length, sequence_length)``: Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: # For example purposes. Not runnable. transformer = BertModel.from_pretrained('bert-base-uncased') encoder = ImageEncoder(args) model = MMBTForClassification(config, transformer, encoder) outputs = model(input_modal, input_ids, labels=labels) loss, logits = outputs[:2] csDt|j|_t||||_t|j|_t |j |j|_ dSr5) r r num_labelsr4mmbtr rrrrr classifierr8rrrr 8s  zMMBTForClassification.__init__Nc  Cs|j||||||||| | | d } | d}||}||}|f| dd} | dk r|jdkr~t}||d| d}n t}||d|j| d}|f| } | S)N) r(r:rArBr<r,rCr+rDr=r;rr9)rMrrNrLrviewr)rr(r:rArBr<r,rCr+rDr=r;labelsrFrEZlogitsZloss_fctZlossrrrr-@s2    zMMBTForClassification.forward) NNNNNNNNNNNr.rrrrrKs rK)r2loggingr#Ztorch.nnr rrZ file_utilsrZmodeling_utilsr getLoggerr/loggerModulerZMMBT_START_DOCSTRINGZMMBT_INPUTS_DOCSTRINGr4rKrrrrs,    .-{