U &c@@sZdZddlZddlmZddlmZddlmZddlm Z e e Z Gdd d e Z dS) z2 Classes to support Encoder-Decoder architectures N)Optional)EncoderDecoderConfig)PretrainedConfig)PreTrainedModelc seZdZdZeZdeeeeeedfdd Z ddZ dd Z d d Z d d Z ddZedeeedddZdddZddZddZZS)EncoderDecoderModela :class:`~transformers.EncoderDecoder` is a generic model class that will be instantiated as a transformer architecture with one of the base model classes of the library as encoder and another one as decoder when created with the `AutoModel.from_pretrained(pretrained_model_name_or_path)` class method for the encoder and `AutoModelWithLMHead.from_pretrained(pretrained_model_name_or_path)` class method for the decoder. N)configencoderdecodercs|dk s |dk r|dk s td|dkr:t|j|j}nt||jsXtd||jt||dkrddl m }| |j }|dkrddl m }| |j}||_ ||_|j dkstddS)NzEEither a configuration or an Encoder and a decoder has to be providedzconfig: {} has to be of type {}r AutoModelAutoModelWithLMHeadzLThe encoder {} should not have a LM Head. Please use a model without LM Head)AssertionErrorrZfrom_encoder_decoder_configsr isinstance config_classformatsuper__init__Z transformersr Z from_configr rr get_output_embeddings)selfrr r r r __class__I/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_encoder_decoder.pyr's4      zEncoderDecoderModel.__init__cCsdSNrrrrr tie_weightsIszEncoderDecoderModel.tie_weightscCs|jSr)r rrrr get_encoderMszEncoderDecoderModel.get_encodercCs|jSr)r rrrr get_decoderPszEncoderDecoderModel.get_decodercCs |jSr)r get_input_embeddingsrrrrr Ssz(EncoderDecoderModel.get_input_embeddingscCs |jSr)r rrrrrrVsz)EncoderDecoderModel.get_output_embeddings)%encoder_pretrained_model_name_or_path%decoder_pretrained_model_name_or_pathreturnc Osdd|D}dd|D}|dd}|dkrf|dk sHtdddlm}|j|f||}d |j_|dd} | dkr|dk std dd lm} | j|f|} d | j_||| d } | S)a# Instantiates an encoder and a decoder from one or two base classes of the library from pre-trained model checkpoints. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, you need to first set it back in training mode with `model.train()`. Params: encoder_pretrained_model_name_or_path (:obj: `str`, `optional`, defaults to `None`): information necessary to initiate the encoder. Either: - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``. - a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``. - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/encoder``. - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. decoder_pretrained_model_name_or_path (:obj: `str`, `optional`, defaults to `None`): information necessary to initiate the decoder. Either: - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``. - a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``. - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/decoder``. - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. model_args: (`optional`) Sequence of positional arguments: All remaning positional arguments will be passed to the underlying model's ``__init__`` method kwargs: (`optional`) Remaining dictionary of keyword arguments. Can be used to update the configuration object (after it being loaded) and initiate the model. (e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or automatically loaded: Examples:: from tranformers import EncoderDecoder model = EncoderDecoder.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert cSs,i|]$\}}|dr|tdd|qS)Zencoder_N startswithlen.0argumentvaluerrr s zGEncoderDecoderModel.from_encoder_decoder_pretrained..cSs,i|]$\}}|dr|tdd|qSdecoder_Nr$r'rrrr+s modelNzeIf `model` is not defined as an argument, a `encoder_pretrained_model_name_or_path` has to be definedrr FzmIf `decoder_model` is not defined as an argument, a `decoder_pretrained_model_name_or_path` has to be definedr T)r r ) itemspoprZ modeling_autor Zfrom_pretrainedrZ is_decoderr) clsr!r"Z model_argskwargskwargs_encoderkwargs_decoderr r r rr.rrrfrom_encoder_decoder_pretrainedYs4,     z3EncoderDecoderModel.from_encoder_decoder_pretrainedc  Ksxdd| D} dd| D}|dkrF|jf||||d| }|d}|jf|| ||||| | d|}||S)a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary for the encoder. Indices can be obtained using :class:`transformers.PretrainedTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details. inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices for the encoder. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. head_mask: (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules for the encoder. Mask values selected in ``[0, 1]``: ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**. encoder_outputs (:obj:`tuple(tuple(torch.FloatTensor)`, `optional`, defaults to :obj:`None`): Tuple consists of (`last_hidden_state`, `optional`: `hidden_states`, `optional`: `attentions`) `last_hidden_state` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. decoder_input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`, defaults to :obj:`None`): Provide for sequence to sequence training to the decoder. Indices can be obtained using :class:`transformers.PretrainedTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details. decoder_attention_mask (:obj:`torch.BoolTensor` of shape :obj:`(batch_size, tgt_seq_len)`, `optional`, defaults to :obj:`None`): Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default. decoder_head_mask: (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules for the decoder. Mask values selected in ``[0, 1]``: ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**. decoder_inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, target_sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`decoder_input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. masked_lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the masked language modeling loss for the decoder. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Labels for computing the left-to-right language modeling loss (next word prediction) for the decoder. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` kwargs: (`optional`) Remaining dictionary of keyword arguments. Keyword arguments come in two flavors: - Without a prefix which will be input as `**encoder_kwargs` for the encoder forward function. - With a `decoder_` prefix which will be input as `**decoder_kwargs` for the decoder forward function. Examples:: from transformers import EncoderDecoderModel, BertTokenizer import torch tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert # forward input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids=input_ids, decoder_input_ids=input_ids) # training loss, outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, lm_labels=input_ids)[:2] # generation generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id) cSs i|]\}}|ds||qS)r-)r%r'rrrr+s z/EncoderDecoderModel.forward..cSs,i|]$\}}|dr|tdd|qSr,r$r'rrrr+s N) input_idsattention_mask inputs_embeds head_maskr)r6r8r7Zencoder_hidden_statesZencoder_attention_maskr9 lm_labelsmasked_lm_labels)r/r r )rr6r8r7r9encoder_outputsdecoder_input_idsdecoder_attention_maskZdecoder_head_maskZdecoder_inputs_embedsr;r:r2r3r4Z hidden_statesZdecoder_outputsrrrforwards6X  zEncoderDecoderModel.forwardcKsJ|dk stdt|tkr"|}n|f}|j|}||d|d|dS)Nz*past has to be defined for encoder_outputsr7r6)r7r>r=r<)rtypetupler prepare_inputs_for_generation)rr6pastr7r2r<Zdecoder_inputsrrrrB s  z1EncoderDecoderModel.prepare_inputs_for_generationcCs|Srr)rrCZbeam_idxrrr_reorder_cache2sz"EncoderDecoderModel._reorder_cache)NNN)NN) NNNNNNNNNNN)__name__ __module__ __qualname____doc__rrrrrrrrrr r classmethodstrr5r?rBrD __classcell__rrrrrsJ"P xr)rHloggingtypingrZconfiguration_encoder_decoderrZconfiguration_utilsrZmodeling_utilsr getLoggerrEloggerrrrrrs