U &ºcáã@s¤dZddlZddlmZmZe e¡ZddiZdddd d d d d œiZ ddddddd œZ ddiddiddiddiddiddid œZ Gdd„deƒZ Gdd„deƒZ dS)z$Tokenization classes for DistilBERT.éNé)Ú BertTokenizerÚBertTokenizerFastZ vocab_filez vocab.txtzOhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txtzPhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txtzMhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txtzNhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-vocab.txtzZhttps://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-german-cased-vocab.txtzZhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt)zdistilbert-base-uncasedz'distilbert-base-uncased-distilled-squadzdistilbert-base-casedz%distilbert-base-cased-distilled-squadzdistilbert-base-german-casedz"distilbert-base-multilingual-casediZ do_lower_caseTFc@s&eZdZdZeZeZeZ e Z dgZ dS)ÚDistilBertTokenizeraW Constructs a DistilBertTokenizer. :class:`~transformers.DistilBertTokenizer is identical to :class:`~transformers.BertTokenizer` and runs end-to-end tokenization: punctuation splitting + wordpiece. Refer to superclass :class:`~transformers.BertTokenizer` for usage examples and documentation concerning parameters. Úattention_maskN© Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚVOCAB_FILES_NAMESZvocab_files_namesÚPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_mapÚ&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesÚPRETRAINED_INIT_CONFIGURATIONZpretrained_init_configurationZmodel_input_names©rrúH/tmp/pip-unpacked-wheel-ymerj3tt/transformers/tokenization_distilbert.pyr:s  rc@s&eZdZdZeZeZeZ e Z dgZ dS)ÚDistilBertTokenizerFastaš Constructs a "Fast" DistilBertTokenizer (backed by HuggingFace's `tokenizers` library). :class:`~transformers.DistilBertTokenizerFast` is identical to :class:`~transformers.BertTokenizerFast` and runs end-to-end tokenization: punctuation splitting + wordpiece. Refer to superclass :class:`~transformers.BertTokenizerFast` for usage examples and documentation concerning parameters. rNrrrrrrLs  r)r ÚloggingZtokenization_bertrrÚ getLoggerrÚloggerr r rrrrrrrrÚs: úÿ ú ú