U &c^@sdZddlZddlZddlZddlmZeeZdddZ dd d d d d dddd dZ ddddd Z ddiddiddiddid Z ddZ GdddeZdS)z0Tokenization classes for Flaubert, based on XLM.N) XLMTokenizerz vocab.jsonz merges.txt)Z vocab_fileZ merges_filez\https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/vocab.jsonz]https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/vocab.jsonz[https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/vocab.jsonz\https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/vocab.json)zflaubert-small-casedzflaubert-base-uncasedzflaubert-base-casedzflaubert-large-casedz\https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/merges.txtz]https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/merges.txtz[https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/merges.txtz\https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/merges.txti do_lowercaseFTcCsddd}||dddS)zQ Converts `text` to Unicode (if it's not already), assuming UTF-8 input. utf-8strictcSs<t|tjr|||St|tjr(|Stdt|dS)Nznot expecting type '%s') isinstancesix binary_typedecode text_type TypeErrortype)sencodingerrorsrF/tmp/pip-unpacked-wheel-ymerj3tt/transformers/tokenization_flaubert.pysix_ensure_textDs    z+convert_to_unicode..six_ensure_textignore)rr)rrr)textrrrrconvert_to_unicode?s rcsDeZdZdZeZeZeZ e Z d fdd Z ddZ d ddZZS) FlaubertTokenizera BPE tokenizer for Flaubert - Moses preprocessing & tokenization - Normalize all inputs text - argument ``special_tokens`` and function ``set_special_tokens``, can be used to add additional symbols (ex: "__classify__") to a vocabulary - `do_lowercase` controle lower casing (automatically set for pretrained vocabularies) This tokenizer inherits from :class:`~transformers.XLMTokenizer`. Please check the superclass for usage examples and documentation regarding arguments. Fc stjf|||_d|_dS)NF)super__init__rZdo_lowercase_and_remove_accent)selfrkwargs __class__rrrbszFlaubertTokenizer.__init__cCs:|dddd}t|}td|}|jr6|}|S)Nz``"z''NFC)replacer unicodedata normalizerlower)rrrrrpreprocess_textgs  z!FlaubertTokenizer.preprocess_textcCsd}|r"|jr"||jkr"td|r0|}n&||}|j||d}|j||d}g}|D](}|r^|dd||dDq^|S)a Tokenize a string given language code using Moses. Details of tokenization: - [sacremoses](https://github.com/alvations/sacremoses): port of Moses - Install with `pip install sacremoses` Args: - bypass_tokenizer: Allow users to preprocess and tokenize the sentences externally (default = False) (bool). If True, we only apply BPE. Returns: List of tokens. frzSupplied language code not found in lang2id mapping. Please check that your language is supported by the loaded pretrained model.)langcSsg|]}|qSrr).0trrr sz/FlaubertTokenizer._tokenize.. ) Zlang2idloggererrorsplitr$Zmoses_pipelineZmoses_tokenizeextendZbpe)rrZbypass_tokenizerr&Z split_tokenstokenrrr _tokenizeqs  "zFlaubertTokenizer._tokenize)F)F)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_mapPRETRAINED_INIT_CONFIGURATIONZpretrained_init_configuration&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesrr$r0 __classcell__rrrrrOs  r)r4loggingr!rZtokenization_xlmr getLoggerr1r+r5r6r8r7rrrrrrs>