U &c @sdZddlZddlZddlZddlmZddlmZee Z dZ ddiZ dd d d d d d iZ d d d d d d ZGd d d eZdS)z! Tokenization class for model T5.N)copyfile)PreTrainedTokenizeru▁ vocab_filez spiece.modelzChttps://s3.amazonaws.com/models.huggingface.co/bert/t5-spiece.model)zt5-smallzt5-basezt5-largezt5-3bzt5-11bics|eZdZdZeZeZeZ dfdd Z e d d Z d d Z d dZddZdddZddZddZddZddZZS) T5Tokenizera) Constructs an XLNet tokenizer. Based on `SentencePiece `__ . This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the methods. Users should refer to the superclass for more information regarding methods. Args: vocab_file (:obj:`string`): `SentencePiece `__ file (generally has a `.spm` extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (:obj:`string`, `optional`, defaults to ""): The end of sequence token. .. note:: When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the :obj:`sep_token`. unk_token (:obj:`string`, `optional`, defaults to ""): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (:obj:`string`, `optional`, defaults to ""): The token used for padding, for example when batching sequences of different lengths. extra_ids (:obj:`List[str]`, `optional`, defaults to :obj:`100`): Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginnning ("" is the last token in the vocabulary like in T5 preprocessing see: https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117) additional_special_tokens (:obj:`List[str]`, `optional`, defaults to :obj:`None`): Additional special tokens used by the tokenizer. dNc  s|dkr,|dkrg}|ddt|Dtjf||||d|z ddl}Wn tk rttdYnX||_||_ | |_ |j |dS)NrcSsg|]}d|qS) )format.0ir@/tmp/pip-unpacked-wheel-ymerj3tt/transformers/tokenization_t5.py psz(T5Tokenizer.__init__..) eos_token unk_token pad_tokenadditional_special_tokenszuYou need to install SentencePiece to use T5Tokenizer:https://github.com/google/sentencepiecepip install sentencepiece) extendrangesuper__init__ sentencepiece ImportErrorloggerwarningr _extra_idsSentencePieceProcessorsp_modelLoad) selfrrrrZ extra_idsrkwargsspm __class__rrrbs.   zT5Tokenizer.__init__cCs|j|jS)N)r!get_piece_sizerr#rrr vocab_sizeszT5Tokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr)Zconvert_ids_to_tokensr r)rr sz)T5Tokenizer.get_vocab..)rr*updateZadded_tokens_encoder)r#Zvocabrr)r get_vocabs zT5Tokenizer.get_vocabcCs|j}d|d<|S)Nr!)__dict__copy)r#staterrr __getstate__s zT5Tokenizer.__getstate__cCsP||_z ddl}Wn tk r2tdYnX||_|j|jdS)NrzvYou need to install SentencePiece to use T5Tokenizer: https://github.com/google/sentencepiecepip install sentencepiece) r.rrrrr r!r"r)r#dr%rrr __setstate__s  zT5Tokenizer.__setstate__FcCs&|s|j|}n|j|dd}|S)zZ Take as input a string and return a list of strings (tokens) for words/sub-words @g?)r!ZEncodeAsPiecesZSampleEncodeAsPieces)r#textsamplepiecesrrr _tokenizeszT5Tokenizer._tokenizecCs>|dr2td|}t|d}|j|dS|j|S)z2 Converts a token (str) in an id using the vocab. z r) startswithrematchintgroupr*r!Z piece_to_id)r#tokenr;numrrr_convert_token_to_ids   z T5Tokenizer._convert_token_to_idcCs4||jkr|j|}nd|jd|}|S)z=Converts an index (integer) in a token (str) using the vocab.r r)r!r(Z IdToPiecer r*)r#indexr>rrr_convert_id_to_tokensz T5Tokenizer._convert_id_to_tokencCs|j|}|S)z< Converts a sequence of tokens (string) in a single string. )r!Z decode_pieces)r#tokensZ out_stringrrrconvert_tokens_to_strings z$T5Tokenizer.convert_tokens_to_stringcCs^tj|s td|dStj|td}tj|j tj|krXt |j ||fS)zt Save the sentencepiece vocabulary (copy original file) and special tokens file to a directory. z*Vocabulary path ({}) should be a directoryNr) ospathisdirrerrorr joinVOCAB_FILES_NAMESabspathrr)r#Zsave_directoryZout_vocab_filerrrsave_vocabularys  zT5Tokenizer.save_vocabulary)rrr r N)F)__name__ __module__ __qualname____doc__rJZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesrpropertyr*r-r1r3r8r@rBrDrL __classcell__rrr&rr>s((  r)rPloggingrEr:shutilrZtokenization_utilsr getLoggerrMrZSPIECE_UNDERLINErJrQrRrrrrrs.