U &c@sddlZddlZddlZddlmZmZmZmZddlZddl m Z ddl m Z m Z dddd d Zd Zd d eDZGddde ZeeeefdddZdS)N)DictListOptionalUnionS3_BUCKET_PREFIX) BatchEncodingPreTrainedTokenizerz source.spmz target.spmz vocab.jsonztokenizer_config.json) source_spm target_spmvocabZtokenizer_config_file)z opus-mt-en-decs$i|]\}|fddtDqS)cs"i|]}|td|dqS)z/Helsinki-NLP//r.0mfnameD/tmp/pip-unpacked-wheel-ymerj3tt/transformers/tokenization_marian.py sz.) MODEL_NAMESrkrrrrsrc seZdZdZeZeZddeDZdgZ e dZ d*fd d Z eed ddZddZedddZeeedddZeedddZeeedddZd+eeddd Zeedd!d"Zd,eeeeeeeeeed%d&d'Zeedd(d)ZZS)-MarianTokenizeraSentencepiece tokenizer for marian. Source and target languages have different SPM models. The logic is use the relevant source_spm or target_spm to encode txt as pieces, then look up each piece in a vocab dictionary. Examples:: from transformers import MarianTokenizer tok = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de') src_texts = [ "I am a small frog.", "Tom asked his teacher for advice."] tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional batch_enc: BatchEncoding = tok.prepare_translation_batch(src_texts, tgt_texts=tgt_texts) # keys [input_ids, attention_mask, decoder_input_ids, decoder_attention_mask]. # model(**batch) should work cCsi|] }|dqS)rrrrrr+szMarianTokenizer.Zattention_maskz>>.+<rc stj| |||dt||_|j|jkr2td|j|jksBtdd|jD|_ ||_ ||_ t |_|j|t |_|j|dd|jD|_zddlm} | ||_Wn(tk rtd d d |_YnXdS) N)max_len eos_token unk_token pad_tokenz token must be in vocabcSsi|]\}}||qSrr)rrvrrrrGsz,MarianTokenizer.__init__..cSs$g|]}|dr|dr|qS)z>>z<<) startswithendswithrrrr Ts z,MarianTokenizer.__init__..r)MosesPunctuationNormalizerz'Recommended: pip install mosestokenizercSs|SNr)xrrr\z*MarianTokenizer.__init__..)super__init__ load_jsonencoderr!KeyErrorr"AssertionErroritemsdecoder source_lang target_lang sentencepieceZSentencePieceProcessor spm_sourceLoad spm_targetZsupported_language_codesZmosestokenizerr'punc_normalizer ImportErrorwarningswarn) selfr r r r4r5r!r r"rr' __class__rrr-/s0         zMarianTokenizer.__init__)r)returncCs|r||SdS)zHCover moses empty string edge case. They return empty list for '' input!)r:)r>r)rrr normalize^szMarianTokenizer.normalizecCs|j||j|jSr()r/getr!)r>tokenrrr_convert_token_to_idbsz$MarianTokenizer._convert_token_to_id)textcCs2|j|}|r|dgng}||jd|fS)z6Remove language codes like <> before sentencepiecerrB)language_code_rematchgroupsub)r>rGrIcoderrrremove_language_codees z$MarianTokenizer.remove_language_code)rGrAcCs"||\}}|j|}||Sr()rM current_spmZEncodeAsPieces)r>rGrLpiecesrrr _tokenizeks zMarianTokenizer._tokenize)indexrAcCs|j||jS)z?Converts an index (integer) in a token (str) using the encoder.)r3rDr!)r>rQrrr_convert_id_to_tokenpsz$MarianTokenizer._convert_id_to_token)tokensrAcCs |j|S)z(Uses target language sentencepiece model)r9Z DecodePieces)r>rSrrrconvert_tokens_to_stringtsz(MarianTokenizer.convert_tokens_to_string)rAcCs$|dkr||jgS|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N)Z eos_token_id)r>Z token_ids_0Z token_ids_1rrr build_inputs_with_special_tokensxs z0MarianTokenizer.build_inputs_with_special_tokensc sfdd|DS)Ncsg|]}j|fqSr)decode)ridskwargsr>rrr&sz0MarianTokenizer.batch_decode..r)r>Z token_idsrYrrXr batch_decodeszMarianTokenizer.batch_decodeTpt) src_texts tgt_texts max_lengthpad_to_max_lengthreturn_tensorsrAc sd|krtd|j_fdd|D}j|d|||d}|dkrP|Sj_j|d|||d}|D]\}} | |d|<qtj_|S) aPrepare model inputs for translation. For best performance, translate one sentence at a time. Arguments: src_texts: list of src language texts tgt_texts: list of tgt language texts max_length: (None) defer to config (1024 for mbart-large-en-ro) pad_to_max_length: (bool) return_tensors: (str) default "pt" returns pytorch tensors, pass None to return lists. Returns: BatchEncoding: with keys [input_ids, attention_mask, decoder_input_ids, decoder_attention_mask] all shaped bs, seq_len. (BatchEncoding is a dict of string -> tensor or lists). If no tgt_text is specified, the only keys will be input_ids and attention_mask. rBz!found empty string in src_texts: csg|]}|qSr)rC)rtr>rrr&sz=MarianTokenizer.prepare_translation_batch..T)Zadd_special_tokensr`r^r_NZdecoder_) ValueErrorr7rNZbatch_encode_plusr9r2) r>r\r]r^r_r`Z model_inputsZdecoder_inputsrr#rrbrprepare_translation_batchs2z)MarianTokenizer.prepare_translation_batchcCs t|jSr()lenr/rbrrr vocab_sizeszMarianTokenizer.vocab_size) NNNNNrrrr)N)NNTr[) __name__ __module__ __qualname____doc__vocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_maprZmax_model_input_sizesZmodel_input_namesrecompilerHr-strrCrFrMrrPintrRrTrUrZrboolr rdpropertyrf __classcell__rrr?rrsL /  0r)pathrAc Cs,t|d}t|W5QRSQRXdS)Nr)openjsonload)rtfrrrr.s r.)rwrmr<typingrrrrr6Z file_utilsrZtokenization_utilsr r rkrr2rlrror.rrrrs$