U &c @sdZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl m Z ddlmZmZmZmZmZmZmZmZddlmZddlmZddlmZdd lm Z!d d l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(e&rddl)Z*e'rddl+Z+e,e-Z.d Z/d Z0dZ1e2dZ3e2dZ4e5Z6ee5Z7ee2Z8ee5e5fZ9eee5ee5fZ:eee2ee2fZ;GdddeZe e!e2e2e5e?e5e2e2e5d ddZ@Gddde ZAGdddZBGdd d eBZCGd!d"d"eCZDd%d#d$ZEdS)&zvTokenization classes for python and fast tokenizers. Fast tokenizers are provided by HuggingFace's tokenizers library.N)UserDict defaultdict)contextmanager)AnyDictList NamedTupleOptionalSequenceTupleUnion) AddedToken)Encoding)Decoder) BaseTokenizer) cached_path hf_bucket_url is_remote_urlis_tf_availableis_torch_availabletorch_requiredzspecial_tokens_map.jsonzadded_tokens.jsonztokenizer_config.jsonꌠ9Y>)Fg@xDc@s"eZdZUdZeed<eed<dS)CharSpanz Character span in the original string Args: start: index of the first character in the original string end: index of the character following the last character in the original string startendN__name__ __module__ __qualname____doc__int__annotations__r#r#C/tmp/pip-unpacked-wheel-ymerj3tt/transformers/tokenization_utils.pyr<s rc@s"eZdZUdZeed<eed<dS) TokenSpanz Token span in an encoded string (list of tokens) Args: start: index of the first token in the span end: index of the token following the last token in the span rrNrr#r#r#r$r%Hs r%xcCsttj|gS)z Flatten the provided (potentially nested) sequence Args: x (Sequence): Potentially nested sequence to flatten Returns: list: Flattened sequence ) functoolsreduceoperatoriconcatr&r#r#r$flattenTs r, tokenizer max_lengthstridestrategypad_to_max_length padding_side pad_token_idpad_token_type_id pad_tokenc cs|dk r|j|||d|r>|r>|dkr>|j|||||dn|rTtd||dV|dk rj||r|r|dkr|dS)a This contextmanager is in charge of defining the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. This contextmanager assumes the provider tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding/truncation when exiting the managed section. Args: tokenizer (BaseTokenizerFast): The tokenizer which will be used max_length (int): The maximum size of the sequence stride (int): The stride to use when handling overflow strategy (str): Overflowing logic to use pad_to_max_length (bool): Boolean indicating if the output needs to be padded up to max_length padding_side (str): "left" or "right" indicating the direction the output sequence will be padded pad_token_id (int): The integer representation of the padding token to use pad_token_type_id (int): The integer representation of the padding token type to use pad_token (str): The string representation of the padding token to use N)r0r1r)r/ directionZpad_idZ pad_type_idr6zDisabled padding because no padding token set (pad_token: {}, pad_token_id: {}). To remove this error, you can add a new pad token and then resize model embedding: tokenizer.pad_token = '' model.resize_token_embeddings(len(tokenizer)))Zenable_truncationZenable_paddingloggerwarningformatZ no_truncationZ no_paddingr-r#r#r$truncate_and_padbs,!r;cspeZdZdZd-eeeefeee e e fdfdd Z ee efe dddZ ed d d Zd d ZddZddZeeee dddZd.e ee dddZd/e eee dddZd0e ee e dddZd1e ee eddd Zd2e ee edd!d"Zd3e ee e d#d$d%Zd4e ee edd&d'Zd5e ee e d#d(d)Zeed*d+d,Z Z!S)6 BatchEncodinga BatchEncoding hold the output of the encode and batch_encode methods (tokens, attention_masks, etc). This class is derived from a python Dictionary and can be used as a dictionnary. In addition, this class expose utility methods to map from word/char space to token space. Args: data (:obj:`dict`): Dictionary of lists/arrays returned by the encode/batch_encode methods ('input_ids', 'attention_mask'...) encoding (:obj:`EncodingFast`, :obj:`list(EncodingFast)`, `optional`, defaults to :obj:`None`): If the tokenizer is a fast tokenizer which outputs additional informations like mapping from word/char space to token space the `EncodingFast` instance or list of instance (for batches) hold these informations. N)dataencodingcs&t|t|tr|g}||_dSN)super__init__ isinstance EncodingFast _encodings)selfr=r> __class__r#r$rAs  zBatchEncoding.__init__)itemreturncCs4t|tr|j|S|jdk r(|j|StddS)z If the key is a string, get the value of the dict associated to `key` ('input_ids', 'attention_mask'...) If the key is an integer, get the EncodingFast for batch item with index `key` NzIndexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers)rBstrr=rDKeyErrorrErHr#r#r$ __getitem__s    zBatchEncoding.__getitem__)rHcCs |j|Sr?)r=rLr#r#r$ __getattr__szBatchEncoding.__getattr__cCs |jSr?)r=keysrEr#r#r$rOszBatchEncoding.keyscCs |jSr?)r=valuesrPr#r#r$rQszBatchEncoding.valuescCs |jSr?r=itemsrPr#r#r$rSszBatchEncoding.itemsrIcCs|jS)z Return the list all encoding from the tokenization process Returns: List[EncodingFast] or None if input was tokenized through Python (i.e. not fast) tokenizer )rDrPr#r#r$ encodingsszBatchEncoding.encodingsr) batch_indexrIcCs|jstd|j|jS)Nzsz$BatchEncoding.to..rR)rErmr#rlr$rnszBatchEncoding.to)NN)r)r)N)N)N)N)N)N)"rrrr r rrJrr rCr rAr!rMrNrOrQrSpropertyrrUrXrZr_r%rcrrerhrjrkrrn __classcell__r#r#rFr$r<s0 '-%#%#r<c@seZdZdZdddddddd gZd d Zed d ZeddZeddZ eddZ eddZ eddZ eddZ eddZddZejdd ZejddZe jd dZ e jd!dZ e jd"dZ e jd#dZ e jd$dZ ejd%dZed&d'Zed(d)Zed*d+Zed,d-Zed.d/Zed0d1Zed2d3Zed4d5Zed6d7Zed8d9Zed:d;ZedS)?SpecialTokensMixinai SpecialTokensMixin is derived by ``PreTrainedTokenizer`` and ``PreTrainedTokenizerFast`` and handles specific behaviors related to special tokens. In particular, this class hold the attributes which can be used to directly access to these special tokens in a model-independant manner and allow to set and update the special tokens. bos_token eos_token unk_token sep_tokenr6 cls_token mask_tokenadditional_special_tokenscKsd|_d|_d|_d|_d|_d|_d|_d|_g|_| D]\}}||j kr>|dkr~t |t t frxtdd|Dstq>t |trt||t|q>t |trt|||q>td|t|q>dS)Nrr}css|]}t|tVqdSr?rBrJrptr#r#r$ sz.SpecialTokensMixin.__init__..zCspecial token {} has to be either str or AddedTokenFast but got: {}) _bos_token _eos_token _unk_token _sep_token _pad_token _cls_token _mask_token_pad_token_type_id_additional_special_tokensrSSPECIAL_TOKENS_ATTRIBUTESrBlisttupleallAssertionErrorAddedTokenFastsetattrrJ TypeErrorr:type)rEkwargskeyvaluer#r#r$rAs( &  zSpecialTokensMixin.__init__cCs|jdkrtd|jS)zW Beginning of sentence token (string). Log an error if used while not having been set. Nz'Using bos_token, but it is not set yet.)rr8errorrPr#r#r$rws  zSpecialTokensMixin.bos_tokencCs|jdkrtd|jS)zQ End of sentence token (string). Log an error if used while not having been set. Nz'Using eos_token, but it is not set yet.)rr8rrPr#r#r$rx s  zSpecialTokensMixin.eos_tokencCs|jdkrtd|jS)zI Unknown token (string). Log an error if used while not having been set. Nz'Using unk_token, but it is not set yet.)rr8rrPr#r#r$rys  zSpecialTokensMixin.unk_tokencCs|jdkrtd|jS)z Separation token (string). E.g. separate context and query in an input sequence. Log an error if used while not having been set. Nz'Using sep_token, but it is not set yet.)rr8rrPr#r#r$rzs  zSpecialTokensMixin.sep_tokencCs|jdkrtd|jS)zI Padding token (string). Log an error if used while not having been set. Nz'Using pad_token, but it is not set yet.)rr8rrPr#r#r$r6s  zSpecialTokensMixin.pad_tokencCs|jdkrtd|jS)z Classification token (string). E.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set. Nz'Using cls_token, but it is not set yet.)rr8rrPr#r#r$r{%s  zSpecialTokensMixin.cls_tokencCs|jdkrtd|jS)z Mask token (string). E.g. when training a model with masked-language modeling. Log an error if used while not having been set. Nz(Using mask_token, but it is not set yet.)rr8rrPr#r#r$r|,s  zSpecialTokensMixin.mask_tokencCs|jdkrtd|jS)zz All the additional special tokens you may want to use (list of strings). Log an error if used while not having been set. Nz7Using additional_special_tokens, but it is not set yet.)rr8rrPr#r#r$r}3s  z,SpecialTokensMixin.additional_special_tokenscCsdS)zL To be overriden by derived class if a backend tokenizer has to be updated. Nr#rErr#r#r$_maybe_update_backend:sz(SpecialTokensMixin._maybe_update_backendcCs||_||gdSr?)rrrr#r#r$rw>scCs||_||gdSr?)rrrr#r#r$rxCscCs||_||gdSr?)rrrr#r#r$ryHscCs||_||gdSr?)rrrr#r#r$rzMscCs||_||gdSr?)rrrr#r#r$r6RscCs||_||gdSr?)rrrr#r#r$r{WscCs||_||gdSr?)rrrr#r#r$r|\scCs||_||dSr?)rrrr#r#r$r}ascCs ||jS)zj Id of the beginning of sentence token in the vocabulary. Log an error if used while not having been set. )convert_tokens_to_idsrwrPr#r#r$ bos_token_idfszSpecialTokensMixin.bos_token_idcCs ||jS)zd Id of the end of sentence token in the vocabulary. Log an error if used while not having been set. )rrxrPr#r#r$ eos_token_idkszSpecialTokensMixin.eos_token_idcCs ||jS)z\ Id of the unknown token in the vocabulary. Log an error if used while not having been set. )rryrPr#r#r$ unk_token_idpszSpecialTokensMixin.unk_token_idcCs ||jS)z Id of the separation token in the vocabulary. E.g. separate context and query in an input sequence. Log an error if used while not having been set. )rrzrPr#r#r$ sep_token_iduszSpecialTokensMixin.sep_token_idcCs ||jS)z\ Id of the padding token in the vocabulary. Log an error if used while not having been set. )rr6rPr#r#r$r4zszSpecialTokensMixin.pad_token_idcCs|jS)z0 Id of the padding token type in the vocabulary.)rrPr#r#r$r5sz$SpecialTokensMixin.pad_token_type_idcCs ||jS)z Id of the classification token in the vocabulary. E.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set. )rr{rPr#r#r$ cls_token_idszSpecialTokensMixin.cls_token_idcCs ||jS)z Id of the mask token in the vocabulary. E.g. when training a model with masked-language modeling. Log an error if used while not having been set. )rr|rPr#r#r$ mask_token_idsz SpecialTokensMixin.mask_token_idcCs ||jS)z Ids of all the additional special tokens in the vocabulary (list of integers). Log an error if used while not having been set. )rr}rPr#r#r$additional_special_tokens_idssz0SpecialTokensMixin.additional_special_tokens_idscCs.i}|jD]}t|d|}|r |||<q |S)z A dictionary mapping special token class attribute (cls_token, unk_token...) to their values ('', ''...) _)rgetattr)rEset_attrattr attr_valuer#r#r$special_tokens_maps   z%SpecialTokensMixin.special_tokens_mapcCsHg}|j}|D]$}|t|ttfr.t|n|g}qtt|}|S)z} List all the special tokens ('', ''...) mapped to class attributes (cls_token, unk_token...). )rrQrBrrset)rEall_toksrrr#r#r$all_special_tokenss  " z%SpecialTokensMixin.all_special_tokenscCs|j}||}|S)z List the vocabulary indices of the special tokens ('', ''...) mapped to class attributes (cls_token, unk_token...). )rr)rErZall_idsr#r#r$all_special_idss z"SpecialTokensMixin.all_special_idsN)rrrr rrArtrwrxryrzr6r{r|r}rsetterrrrrr4r5rrrrrrr#r#r#r$rvs                            rvcseZdZUdZiZeeefed<iZeeeeeffed<iZ eeeee ffed<iZ eee fed<ddgZ eeed<d Zeed <d Zd Zee d ddZeddZeddZeddZeddZejddZejddZddZdjfdd Zdd Zed!d"Zed#d$Zd%d&Zd'd(Z d)d*Z!dkd,d-Z"d.d/Z#e$d0d1d2Z%d3d4Z&d5d6Z'd7d8Z(d9d:Z)dle*e$e+e,fe-e*e$e+e,fe.e-e e ee.e-ed>d?d@Z/dme*e$e+e,fe-e*e$e+e,fe.e-e e ee.e.e-ee-e.e-e.e.e.e.e0dAdBdCZ1dne*ee$ee2ee+ee3ee,ee4fe.e-e e ee.e.e-ee-e.e-e.e.e.e.e.e0dDdEdFZ5e6eddGdHdIZ7doee e-ee e-e e.e ee.e-ee-e.e-e.e.e.e.e0dJdKdLZ8eedMdNdOZ9dpee e-ee e ee e:ee ee ee fdPdQdRZ;dqee-eee dSdTdUZdte*e ee fe.e*e ee fd[d\d]Z?e ed^d_d`Z@eeedadbdcZAduee e.e.edddedfZBeCeedgdhdiZDZES)vPreTrainedTokenizeraX Base class for all tokenizers. Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary. This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Class attributes (overridden by derived classes): - ``vocab_files_names``: a python ``dict`` with, as keys, the ``__init__`` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - ``pretrained_vocab_files_map``: a python ``dict of dict`` the high-level keys being the ``__init__`` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` (string) of the pretrained models with, as associated values, the `url` (string) to the associated pretrained vocabulary file. - ``max_model_input_sizes``: a python ``dict`` with, as keys, the `short-cut-names` (string) of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or None if the model has no maximum input size. - ``pretrained_init_configuration``: a python ``dict`` with, as keys, the `short-cut-names` (string) of the pretrained models, and as associated values, a dictionnary of specific arguments to pass to the ``__init__``method of the tokenizer class for this pretrained model when loading the tokenizer with the ``from_pretrained()`` method. Args: - ``model_max_length``: (`Optional`) int: the maximum length in number of tokens for the inputs to the transformer model. When the tokenizer is loaded with `from_pretrained`, this will be set to the value stored for the associated model in ``max_model_input_sizes`` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`). no associated max_length can be found in ``max_model_input_sizes``. - ``padding_side``: (`Optional`) string: the side on which the model should have padding applied. Should be selected between ['right', 'left'] - ``model_input_names``: (`Optional`) List[string]: the list of the forward pass inputs accepted by the model ("token_type_ids", "attention_mask"...). - ``bos_token``: (`Optional`) string: a beginning of sentence token. Will be associated to ``self.bos_token`` and ``self.bos_token_id`` - ``eos_token``: (`Optional`) string: an end of sentence token. Will be associated to ``self.eos_token`` and ``self.eos_token_id`` - ``unk_token``: (`Optional`) string: an unknown token. Will be associated to ``self.unk_token`` and ``self.unk_token_id`` - ``sep_token``: (`Optional`) string: a separation token (e.g. to separate context and query in an input sequence). Will be associated to ``self.sep_token`` and ``self.sep_token_id`` - ``pad_token``: (`Optional`) string: a padding token. Will be associated to ``self.pad_token`` and ``self.pad_token_id`` - ``cls_token``: (`Optional`) string: a classification token (e.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model). Will be associated to ``self.cls_token`` and ``self.cls_token_id`` - ``mask_token``: (`Optional`) string: a masking token (e.g. when training a model with masked-language modeling). Will be associated to ``self.mask_token`` and ``self.mask_token_id`` - ``additional_special_tokens``: (`Optional`) list: a list of additional special tokens. Adding all special tokens here ensure they won't be split by the tokenization process. Will be associated to ``self.additional_special_tokens`` and ``self.additional_special_tokens_ids`` vocab_files_namespretrained_vocab_files_mappretrained_init_configurationmax_model_input_sizestoken_type_idsattention_maskmodel_input_namesrightr3zNo padding token is set for this model, therefore no batch can be made with uneven sequences. Set a padding token or adjust the lengths of the sequences building the batch so that every sequence is of the same length.zThe sequences building the batch are not of the same size, no tensor can be built. Set `pad_to_max_length=True` to pad the smaller sequencesup to the larger sequence's length.rTcCstdS)z8 Size of the base vocabulary (without the added tokens) NNotImplementedErrorrPr#r#r$ vocab_sizeszPreTrainedTokenizer.vocab_sizecCsdS)NFr#rPr#r#r$is_fastszPreTrainedTokenizer.is_fastcCs|jS)zq Kept here for backward compatibility. Now renamed to `model_max_length` to avoid ambiguity. )model_max_lengthrPr#r#r$max_len szPreTrainedTokenizer.max_lencCs|j|jddS)NFpairrnum_special_tokens_to_addrPr#r#r$max_len_single_sentencesz+PreTrainedTokenizer.max_len_single_sentencecCs|j|jddSNTrrrPr#r#r$max_len_sentences_pairsz*PreTrainedTokenizer.max_len_sentences_paircCs.||j|jddkr"tdntddS)zM For backward compatibility, allow to try to setup 'max_len_single_sentence' FrzXSetting 'max_len_single_sentence' is now deprecated. This value is automatically set up.Nrrr8r9rWrr#r#r$rscCs.||j|jddkr"tdntddS)zL For backward compatibility, allow to try to setup 'max_len_sentences_pair' TrzWSetting 'max_len_sentences_pair' is now deprecated. This value is automatically set up.Nrrr#r#r$r$scCs tdS)z Returns the vocabulary as a dict of {token: index} pairs. `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the vocab. NrrPr#r#r$ get_vocab0szPreTrainedTokenizer.get_vocabNc stjf||dk r|n |dd}|dk r2|nt|_|d|j|_|jdksbtd|j|d|j|_i|_t |_ i|_ d|_ i|_ dS)Nrr3)rleftzKPadding side should be selected between 'right' and 'left', current value: rr#)r@rApopVERY_LARGE_INTEGERrr3rradded_tokens_encoderrunique_added_tokens_encoderadded_tokens_decoder init_inputs init_kwargs)rErrrFr#r$rA4s  zPreTrainedTokenizer.__init__cCs|jt|jS)z3 Size of the full vocabulary with the added tokens )rlenrrPr#r#r$__len__MszPreTrainedTokenizer.__len__cOs |j||S)a Instantiate a :class:`~transformers.PreTrainedTokenizer` (or a derived class) from a predefined tokenizer. Args: pretrained_model_name_or_path: either: - a string with the `shortcut name` of a predefined tokenizer to load from cache or download, e.g.: ``bert-base-uncased``. - a string with the `identifier name` of a predefined tokenizer that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``. - a path to a `directory` containing vocabulary files required by the tokenizer, for instance saved using the :func:`~transformers.PreTrainedTokenizer.save_pretrained` method, e.g.: ``./my_model_directory/``. - (not applicable to all derived classes, deprecated) a path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (e.g. Bert, XLNet), e.g.: ``./my_model_directory/vocab.txt``. cache_dir: (`optional`) string: Path to a directory in which a downloaded predefined tokenizer vocabulary files should be cached if the standard cache should not be used. force_download: (`optional`) boolean, default False: Force to (re-)download the vocabulary files and override the cached versions if they exists. resume_download: (`optional`) boolean, default False: Do not delete incompletely recieved file. Attempt to resume the download if such a file exists. proxies: (`optional`) dict, default None: A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request. inputs: (`optional`) positional arguments: will be passed to the Tokenizer ``__init__`` method. kwargs: (`optional`) keyword arguments: will be passed to the Tokenizer ``__init__`` method. Can be used to set special tokens like ``bos_token``, ``eos_token``, ``unk_token``, ``sep_token``, ``pad_token``, ``cls_token``, ``mask_token``, ``additional_special_tokens``. See parameters in the doc string of :class:`~transformers.PreTrainedTokenizer` for details. Examples:: # We can't instantiate directly the base class `PreTrainedTokenizer` so let's show our examples on a derived class: BertTokenizer # Download vocabulary from S3 and cache. tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') # Download vocabulary from S3 (user-uploaded) and cache. tokenizer = BertTokenizer.from_pretrained('dbmdz/bert-base-german-cased') # If vocabulary files are in a directory (e.g. tokenizer was saved using `save_pretrained('./test/saved_model/')`) tokenizer = BertTokenizer.from_pretrained('./test/saved_model/') # If the tokenizer uses a single vocabulary file, you can point directly to this file tokenizer = BertTokenizer.from_pretrained('./test/saved_model/my_vocab.txt') # You can link tokens to special vocabulary when instantiating tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', unk_token='') # You should be sure '' is in the vocabulary when doing that. # Otherwise use tokenizer.add_special_tokens({'unk_token': ''}) instead) assert tokenizer.unk_token == '' )_from_pretrained)clsinputsrr#r#r$from_pretrainedQs5z#PreTrainedTokenizer.from_pretrainedc$ Os|dd}|dd}|dd}|dd}|dd}t|j} i} i} || kr|jD]\} } | || | <qd|jr||jkr|j|} nt d |d | |t j |st|rt|jd krtd |jtd |jt|jd } || | <n|tttd }|j|D]`\} }t j |rxt j ||}t j |st d |d}nt||dd}|| | <q0zHi}| D]6\} }|dkrd|| <nt||||||d|| <qWnNtk r(|| krd}n d |d | |t|j}t|YnXtdd|Drftd |d | |t|j| D]@\} }||| krt d |nt d ||| qn|dd}|dk rt|dd}t !|}W5QRX|dd}|s|}n| }|"|||jkrX|j|}|dk rXt#|t$t%frXt&|'dt$d||d<|dd}|d d}|D]\}}||krx|||<qx|dk rt|dd}t !|}W5QRX|D]\}}||kr|||<qz|||} Wnt(k rt(d!YnX|| _)|| _*| j+"t,| j-|dk rt|dd}!t !|!}"W5QRXd"d#|"D}#| j."|"| j/"|#| j+"t,| j.| S)$N cache_dirforce_downloadFresume_downloadproxieslocal_files_onlyzModel name '{}' not found in model shortcut name list ({}). Assuming '{}' is a path, a model identifier, or url to a directory containing tokenizer files.z, rzCalling {}.from_pretrained() with the path to a single file or url is not supported.Use a model identifier or the path to a directory instead.zPCalling {}.from_pretrained() with the path to a single file or url is deprecatedr)added_tokens_filespecial_tokens_map_filetokenizer_config_filez&Didn't find file {}. We won't load it.)filenameZuse_cdn)rrrrrz;Couldn't reach server at '{}' to download vocabulary files.zModel name '{}' was not found in tokenizers model name list ({}). We assumed '{}' was a path or url to a directory containing vocabulary files named {}, but couldn't find such vocabulary files at this path or url.css|]}|dkVqdSr?r#)rpfull_file_namer#r#r$rsz7PreTrainedTokenizer._from_pretrained..zModel name '{}' was not found in tokenizers model name list ({}). We assumed '{}' was a path, a model identifier, or url to a directory containing vocabulary files named {} but couldn't find such vocabulary files at this path or url.zloading file {}z loading file {} from cache at {}rutf-8r>rr#rrrrzoUnable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted.cSsi|]\}}||qSr#r#ror#r#r$rs1sz8PreTrainedTokenizer._from_pretrained..)0rrrrOrrSrcopyr8infor:joinospathisfilerrrrWrr9ADDED_TOKENS_FILESPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILEisdirexistsrrEnvironmentErrorrQropenjsonloadupdaterBr!floatmingetOSErrorrrrrrrr)$rZpretrained_model_name_or_pathrrrrrrrZ s3_models vocab_filesZinit_configurationfile_idZmap_listZadditional_files_names file_namerZresolved_vocab_files file_pathmsgrZtokenizer_config_handlerZsaved_init_inputsrrrZ args_nameZspecial_tokens_map_handlerrrr.Zadded_tokens_handleadded_tok_encoderadded_tok_decoderr#r#r$rs                               z$PreTrainedTokenizer._from_pretrainedc CsHtj|s td|dStj|t}tj|t}tj|t }t |j }t |jdkrtt |j|d<|jD]}||dq~t|ddd}|tj|dd W5QRXt|ddd}|tj|jdd W5QRXt |jdkr2t|ddd }tj|jdd }||W5QRX||} | ||fS) ac Save the tokenizer vocabulary files together with: - added tokens, - special-tokens-to-class-attributes-mapping, - tokenizer instantiation positional and keywords inputs (e.g. do_lower_case for Bert). Warning: This won't save modifications you may have applied to the tokenizer after the instantiation (e.g. modifying tokenizer.do_lower_case after creation). This method make sure the full tokenizer can then be re-loaded using the :func:`~transformers.PreTrainedTokenizer.from_pretrained` class method. z+Saving directory ({}) should be a directoryNrrwrrF) ensure_ascii)rrrr8rr:rrrrrdeepcopyrrrrrOrrwriterdumpsrrsave_vocabulary) rEsave_directoryrrrZtokenizer_configrfZout_strrr#r#r$save_pretrained8s*    z#PreTrainedTokenizer.save_pretrainedcCstdS)a Save the tokenizer vocabulary to a directory. This method does *NOT* save added tokens and special token mappings. Please use :func:`~transformers.PreTrainedTokenizer.save_pretrained` `()` to save the full Tokenizer state if you want to reload it using the :func:`~transformers.PreTrainedTokenizer.from_pretrained` class method. Nr)rErr#r#r$rasz#PreTrainedTokenizer.save_vocabularycs|sdSt|ts|g}g}|D]p}t|ts2tjddrR|jkrR|}|jkr | jkr ||kr | |t d|q t fddt|D}dd|D}j|tjtj_j|t|S) aK Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Args: new_tokens: string or list of string. Each string is a token to add. Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the ``unk_token`` to them). Returns: Number of tokens added to the vocabulary. Examples:: # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = BertModel.from_pretrained('bert-base-uncased') num_added_toks = tokenizer.add_tokens(['new_tok1', 'my_new-tok2']) print('We have added', num_added_toks, 'tokens') model.resize_token_embeddings(len(tokenizer)) # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer. r do_lower_caseFzAdding %s to the vocabularyc3s"|]\}}|t|fVqdSr?r)rpitokrPr#r$rsz1PreTrainedTokenizer.add_tokens..cSsi|]\}}||qSr#r#ror#r#r$rssz2PreTrainedTokenizer.add_tokens..)rBrrJrrrrlowerryrappendr8rdict enumeraterSrrrrOunionrrr)rE new_tokensZ to_add_tokenstokenrrr#rPr$ add_tokensks.    zPreTrainedTokenizer.add_tokensFcCs g}g}t|||r|ndS)a) Returns the number of added tokens when encoding a sequence with special tokens. Note: This encodes inputs and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair: Returns the number of added tokens in the case of a sequence pair if set to True, returns the number of added tokens in the case of a single sequence if set to False. Returns: Number of tokens added to sequences N)r build_inputs_with_special_tokens)rEr token_ids_0 token_ids_1r#r#r$rsz-PreTrainedTokenizer.num_special_tokens_to_addcCs|sdSd}|D]\}}||jks*t|dkrft|ttfrRtdd|DsVt|||7}nt|tstt|||g7}t d||t |||q|S)a Add a dictionary of special tokens (eos, pad, cls...) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary). Using `add_special_tokens` will ensure your special tokens can be used in several ways: - special tokens are carefully handled by the tokenizer (they are never split) - you can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This makes it easy to develop model-agnostic training and fine-tuning scripts. When possible, special tokens are already registered for provided pretrained models (ex: BertTokenizer cls_token is already registered to be '[CLS]' and XLM's one is also registered to be '') Args: special_tokens_dict: dict of string. Keys should be in the list of predefined special attributes: [``bos_token``, ``eos_token``, ``unk_token``, ``sep_token``, ``pad_token``, ``cls_token``, ``mask_token``, ``additional_special_tokens``]. Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the ``unk_token`` to them). Returns: Number of tokens added to the vocabulary. Examples:: # Let's see how to add a new classification token to GPT-2 tokenizer = GPT2Tokenizer.from_pretrained('gpt2') model = GPT2Model.from_pretrained('gpt2') special_tokens_dict = {'cls_token': ''} num_added_toks = tokenizer.add_special_tokens(special_tokens_dict) print('We have added', num_added_toks, 'tokens') model.resize_token_embeddings(len(tokenizer)) # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer. assert tokenizer.cls_token == '' rr}css|]}t|tVqdSr?r~rr#r#r$rsz9PreTrainedTokenizer.add_special_tokens..z+Assigning %s to the %s key of the tokenizer) rSrrrBrrrrrJr8rr)rEspecial_tokens_dict added_tokensrrr#r#r$add_special_tokenss%$z&PreTrainedTokenizer.add_special_tokens)textc s`jj|f|}fdd}jddr6||}ddfdd}j}|||}|S) a Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Take care of added tokens. Args: text (:obj:`string`): The sequence to be encoded. **kwargs (:obj: `dict`): Arguments passed to the model-specific `prepare_for_tokenization` preprocessing method. cs6ddD}dd|dd}t|dd|S) NcSsg|]}t|qSr#)reescape)rpZs_tokr#r#r$ szHPreTrainedTokenizer.tokenize..lowercase_text..(|z)|z(.+?)cSs|dp|dSNrr)groupsr)mr#r#r$zFPreTrainedTokenizer.tokenize..lowercase_text..)rrsub)rZescaped_special_tokspattern)rr#r$lowercase_textsz4PreTrainedTokenizer.tokenize..lowercase_textrFcSs~g}||}t|D]b\}}|}|dkr>|s>||g7}q|t|dkr`|rx||g7}qxq|rn||g7}||g7}q|Sr)splitr rstripr)rrresultZ split_textrsub_textr#r#r$split_on_tokens      z4PreTrainedTokenizer.tokenize..split_on_tokencs|s gS|s|Sg}|g}|D]:}g}|D](}|jkrR|||7}q4||g7}q4|}q(ttjfdd|DS)Nc3s(|] }|jkr|n|gVqdSr?)r _tokenize)rpr rPr#r$r"szHPreTrainedTokenizer.tokenize..split_on_tokens..)stripr(rr itertoolschain from_iterable)Ztok_listrtokenized_textZ text_listrr&)rEr'r#r$split_on_tokenss(    z5PreTrainedTokenizer.tokenize..split_on_tokens)rprepare_for_tokenizationrrr)rErrr"r.rr-r#)rrEr'r$tokenizes   zPreTrainedTokenizer.tokenizecKstdS)a Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Do NOT take care of added tokens. NrrErrr#r#r$r(-szPreTrainedTokenizer._tokenizecCsB|dkr dSt|tr ||Sg}|D]}|||q(|S)z Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. N)rBrJ#_convert_token_to_id_with_added_vocr)rErXidsr r#r#r$r6s  z)PreTrainedTokenizer.convert_tokens_to_idscCs*|dkr dS||jkr |j|S||Sr?)r_convert_token_to_idrEr r#r#r$r2Es   z7PreTrainedTokenizer._convert_token_to_id_with_added_voccCstdSr?rr5r#r#r$r4Msz(PreTrainedTokenizer._convert_token_to_idTr longest_first)r text_pairrr/r0truncation_strategyr2return_tensorsc Ks*|j|f|||||||d| } | dS)a@ Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``. Args: text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method) text_pair (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`, `optional`, defaults to :obj:`None`): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method) add_special_tokens (:obj:`bool`, `optional`, defaults to :obj:`True`): If set to ``True``, the sequences will be encoded with the special tokens relative to their model. max_length (:obj:`int`, `optional`, defaults to :obj:`None`): If set to a number, will limit the total sequence returned so that it has a maximum length. If there are overflowing tokens, those will be added to the returned dictionary. You can set it to the maximal input size of the model with `max_length = tokenizer.model_max_length`. stride (:obj:`int`, `optional`, defaults to ``0``): If set to a number along with max_length, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. truncation_strategy (:obj:`str`, `optional`, defaults to `longest_first`): String selected in the following options: - 'longest_first' (default) Iteratively reduce the inputs sequence until the input is under max_length starting from the longest one at each token (when there is a pair of input sequences) - 'only_first': Only truncate the first sequence - 'only_second': Only truncate the second sequence - 'do_not_truncate': Does not truncate (raise an error if the input sequence is longer than max_length) pad_to_max_length (:obj:`bool`, `optional`, defaults to :obj:`False`): If set to True, the returned sequences will be padded according to the model's padding side and padding index, up to their max length. If no max length is specified, the padding is done up to the model's max length. The tokenizer padding sides are handled by the class attribute `padding_side` which can be set to the following strings: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences Defaults to False: no padding. return_tensors (:obj:`str`, `optional`, defaults to :obj:`None`): Can be set to 'tf' or 'pt' to return respectively TensorFlow :obj:`tf.constant` or PyTorch :obj:`torch.Tensor` instead of a list of python integers. **kwargs: passed to the `self.tokenize()` method )r7r/rr0r8r2r9 input_ids) encode_plus) rErr7rr/r0r8r2r9rencoded_inputsr#r#r$encodePs:  zPreTrainedTokenizer.encode)rr7rr/r0r8r2is_pretokenizedr9return_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mappingrIc spfdd}|rtd|r2jdkr2td||}|dk rJ||nd}j||||||| | | | | d S)aI Returns a dictionary containing the encoded sequence or sequence pair and additional information: the mask for sequence classification and the overflowing elements if a ``max_length`` is specified. Args: text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]` (the later only for not-fast tokenizers)): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method) text_pair (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`, `optional`, defaults to :obj:`None`): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method) add_special_tokens (:obj:`bool`, `optional`, defaults to :obj:`True`): If set to ``True``, the sequences will be encoded with the special tokens relative to their model. max_length (:obj:`int`, `optional`, defaults to :obj:`None`): If set to a number, will limit the total sequence returned so that it has a maximum length. If there are overflowing tokens, those will be added to the returned dictionary You can set it to the maximal input size of the model with `max_length = tokenizer.model_max_length`. stride (:obj:`int`, `optional`, defaults to ``0``): If set to a number along with max_length, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. truncation_strategy (:obj:`str`, `optional`, defaults to `longest_first`): String selected in the following options: - 'longest_first' (default) Iteratively reduce the inputs sequence until the input is under max_length starting from the longest one at each token (when there is a pair of input sequences) - 'only_first': Only truncate the first sequence - 'only_second': Only truncate the second sequence - 'do_not_truncate': Does not truncate (raise an error if the input sequence is longer than max_length) pad_to_max_length (:obj:`bool`, `optional`, defaults to :obj:`False`): If set to True, the returned sequences will be padded according to the model's padding side and padding index, up to their max length. If no max length is specified, the padding is done up to the model's max length. The tokenizer padding sides are handled by the class attribute `padding_side` which can be set to the following strings: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences Defaults to False: no padding. is_pretokenized (:obj:`bool`, defaults to :obj:`False`): Set to True to indicate the input is already tokenized return_tensors (:obj:`str`, `optional`, defaults to :obj:`None`): Can be set to 'tf' or 'pt' to return respectively TensorFlow :obj:`tf.constant` or PyTorch :obj:`torch.Tensor` instead of a list of python integers. return_token_type_ids (:obj:`bool`, `optional`, defaults to :obj:`None`): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the :obj:`return_outputs` attribute. `What are token type IDs? <../glossary.html#token-type-ids>`_ return_attention_mask (:obj:`bool`, `optional`, defaults to :obj:`none`): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the :obj:`return_outputs` attribute. `What are attention masks? <../glossary.html#attention-mask>`__ return_overflowing_tokens (:obj:`bool`, `optional`, defaults to :obj:`False`): Set to True to return overflowing token information (default False). return_special_tokens_mask (:obj:`bool`, `optional`, defaults to :obj:`False`): Set to True to return special tokens mask information (default False). return_offsets_mapping (:obj:`bool`, `optional`, defaults to :obj:`False`): Set to True to return (char_start, char_end) for each token (default False). If using Python's tokenizer, this method will raise NotImplementedError. This one is only available on fast tokenizers inheriting from PreTrainedTokenizerFast. **kwargs: passed to the `self.tokenize()` method Return: A Dictionary of shape:: { input_ids: list[int], token_type_ids: list[int] if return_token_type_ids is True (default) attention_mask: list[int] if return_attention_mask is True (default) overflowing_tokens: list[int] if a ``max_length`` is specified and return_overflowing_tokens is True num_truncated_tokens: int if a ``max_length`` is specified and return_overflowing_tokens is True special_tokens_mask: list[int] if ``add_special_tokens`` if set to ``True`` and return_special_tokens_mask is True } With the fields: - ``input_ids``: list of token ids to be fed to a model - ``token_type_ids``: list of token type ids to be fed to a model - ``attention_mask``: list of indices specifying which tokens should be attended to by the model - ``overflowing_tokens``: list of overflowing tokens if a max length is specified. - ``num_truncated_tokens``: number of overflowing tokens a ``max_length`` is specified - ``special_tokens_mask``: if adding special tokens, this is a list of [0, 1], with 0 specifying special added tokens and 1 specifying sequence tokens. cst|tr*j|fdi}|St|ttfr\t|dkr\t|dtr\|St|ttfrt|dkrt|dtr|StddSNrrz\Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers. rBrJr0rrrrr!rWrrXrrrEr#r$ get_input_idss  ( (z6PreTrainedTokenizer.encode_plus..get_input_ids return_offset_mapping is not available when using Python tokenizers.To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast.More information on available tokenizers at https://github.com/huggingface/transformers/pull/2674NUnable to set proper padding strategy as the tokenizer does not have a padding token. In this case please set the `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via the function add_special_tokens if you want to use a padding strategy) pair_idsr/r2rr0r8r9r@r?rArB)rr4rWprepare_for_model)rErr7rr/r0r8r2r>r9r?r@rArBrCrrH first_ids second_idsr#rGr$r;s2k  zPreTrainedTokenizer.encode_plus)batch_text_or_text_pairsrr/r0r8r2r>r9r?return_attention_masksrAreturn_special_tokens_masksrCreturn_lengthsrIc sRfdd}|r&jdkr&td| r2tdg}|D]`}t|ttfrft|dkrf|sf|\}}n |d}}||}|dk r||nd}|||fq:|dkr|rʇfddtfd d |D}i}|D]`\}}j ||||||| | | | |dd }| D]*\}}||kr g||<|||qq|dk rJ ||t |S) a& Returns a dictionary containing the encoded sequence or sequence pair and additional information: the mask for sequence classification and the overflowing elements if a ``max_length`` is specified. Args: batch_text_or_text_pairs (:obj:`List[str]`, :obj:`List[Tuple[str, str]]`, :obj:`List[List[str]]`, :obj:`List[Tuple[List[str], List[str]]]`, and for not-fast tokenizers, also: :obj:`List[List[int]]`, :obj:`List[Tuple[List[int], List[int]]]`): Batch of sequences or pair of sequences to be encoded. This can be a list of string/string-sequences/int-sequences or a list of pair of string/string-sequences/int-sequence (see details in encode_plus) add_special_tokens (:obj:`bool`, `optional`, defaults to :obj:`True`): If set to ``True``, the sequences will be encoded with the special tokens relative to their model. max_length (:obj:`int`, `optional`, defaults to :obj:`None`): If set to a number, will limit the total sequence returned so that it has a maximum length. If there are overflowing tokens, those will be added to the returned dictionary stride (:obj:`int`, `optional`, defaults to ``0``): If set to a number along with max_length, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. truncation_strategy (:obj:`str`, `optional`, defaults to `longest_first`): String selected in the following options: - 'longest_first' (default) Iteratively reduce the inputs sequence until the input is under max_length starting from the longest one at each token (when there is a pair of input sequences) - 'only_first': Only truncate the first sequence - 'only_second': Only truncate the second sequence - 'do_not_truncate': Does not truncate (raise an error if the input sequence is longer than max_length) pad_to_max_length (:obj:`bool`, `optional`, defaults to :obj:`False`): If set to True, the returned sequences will be padded according to the model's padding side and padding index, up to their max length. If no max length is specified, the padding is done up to the model's max length. The tokenizer padding sides are handled by the class attribute `padding_side` which can be set to the following strings: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences Defaults to False: no padding. is_pretokenized (:obj:`bool`, defaults to :obj:`False`): Set to True to indicate the input is already tokenized return_tensors (:obj:`str`, `optional`, defaults to :obj:`None`): Can be set to 'tf' or 'pt' to return respectively TensorFlow :obj:`tf.constant` or PyTorch :obj:`torch.Tensor` instead of a list of python integers. return_token_type_ids (:obj:`bool`, `optional`, defaults to :obj:`None`): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the :obj:`return_outputs` attribute. `What are token type IDs? <../glossary.html#token-type-ids>`_ return_attention_masks (:obj:`bool`, `optional`, defaults to :obj:`none`): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the :obj:`return_outputs` attribute. `What are attention masks? <../glossary.html#attention-mask>`__ return_overflowing_tokens (:obj:`bool`, `optional`, defaults to :obj:`False`): Set to True to return overflowing token information (default False). return_special_tokens_masks (:obj:`bool`, `optional`, defaults to :obj:`False`): Set to True to return special tokens mask information (default False). return_offsets_mapping (:obj:`bool`, `optional`, defaults to :obj:`False`): Set to True to return (char_start, char_end) for each token (default False). If using Python's tokenizer, this method will raise NotImplementedError. This one is only available on Rust-based tokenizers inheriting from PreTrainedTokenizerFast. return_lengths (:obj:`bool`, `optional`, defaults to :obj:`False`): If set the resulting dictionary will include the length of each encoded inputs **kwargs: passed to the `self.tokenize()` method Return: A Dictionary of shape:: { input_ids: list[List[int]], token_type_ids: list[List[int]] if return_token_type_ids is True (default) attention_mask: list[List[int]] if return_attention_mask is True (default) overflowing_tokens: list[List[int]] if a ``max_length`` is specified and return_overflowing_tokens is True num_truncated_tokens: List[int] if a ``max_length`` is specified and return_overflowing_tokens is True special_tokens_mask: list[List[int]] if ``add_special_tokens`` if set to ``True`` and return_special_tokens_mask is True } With the fields: - ``input_ids``: list of token ids to be fed to a model - ``token_type_ids``: list of token type ids to be fed to a model - ``attention_mask``: list of indices specifying which tokens should be attended to by the model - ``overflowing_tokens``: list of overflowing tokens if a max length is specified. - ``num_truncated_tokens``: number of overflowing tokens a ``max_length`` is specified - ``special_tokens_mask``: if adding special tokens, this is a list of [0, 1], with 0 specifying special added tokens and 1 specifying sequence tokens. cst|tr*j|fdi}|St|ttfr\t|dkr\t|dtr\|St|ttfrt|dkrt|dtr|StddSrDrErFrGr#r$rHs  ( (z.get_input_idsNrJrIcs4|\}}t||dkrnt|jddSr)rr)Z input_pairsrMrNrPr#r$total_sequence_lengths  zDPreTrainedTokenizer.batch_encode_plus..total_sequence_lengthcsg|] }|qSr#r#)rpr3)rTr#r$rsz9PreTrainedTokenizer.batch_encode_plus..) rKr/r2rr0r8r@r?rArBrRr9) r4rWrrBrrrrmaxrLrSconvert_to_tensors_r<)rErOrr/r0r8r2r>r9r?rPrArQrCrRrrHr:Zids_or_pair_idsr3rKrMrN batch_outputsoutputsrrr#)rrrErTr$batch_encode_plus3sXq        z%PreTrainedTokenizer.batch_encode_plus)rWr9rIc Cs|D]\}}|dkrptrpzt|||<Wqtk rlddd|Dkr^t|jn t|jYqXq|dkrtrzt |||<Wqtk rt|jYqt k rddd|Dkrt|jnYqXq|dk rt d |qdS)NtfcSsg|]}|D]}|q qSr#r#rpsequencerHr#r#r$rsz;PreTrainedTokenizer.convert_to_tensors_..ptcSsg|]}|D]}|q qSr#r#r[r#r#r$rsVUnable to convert output to tensors format {}, PyTorch or TensorFlow is not available.)rSrrZconstantrWNO_PAD_TOKEN_FOR_BATCH_MSGUNEVEN_SEQUENCES_FOR_BATCH_MSGrtorchtensor RuntimeErrorr8r9r:)rErWr9rrr#r#r$rVs.   z'PreTrainedTokenizer.convert_to_tensors_)r3rKr/rr0r8r2r9r?r@rArBrRrIcCsft|dk }t|}|r t|nd}| dkr6d|jk} | dkrHd|jk} i}|||rb|j|dnd}|r||kr|j||||||d\}}}| r||d<|||d<|r|||}|||}n4|r||n|}dgt||rd gt|ng}||d <| r||d<| rB|r0||||d <ndgt||d <|dksbt|d |ksbt|dkrt|d |j krt d t||j |o|rt|d |kp|dkot|d |j ko|j t k}|r|dkr|j t krt d |rJ|dk r|n|j t|d }|jdkr| r\d gt|d dg||d<| rz|d|jg||d<| r|d d g||d <|d |jg||d <n|jdkr6| rdg|d gt|d |d<| r|jg||d|d<| rd g||d |d <|jg||d |d <ntdt|jn| rfd gt|d |d<| r|t|d |d<|dkrtrt|d g|d <d|krt|dg|d<d|kr^t|dg|d<n~|dkrDtrDt|d g|d <d|kr$t|dg|d<d|kr^t|dg|d<n|dk r^t d |t|S)a  Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens Args: ids: list of tokenized input ids. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids: Optional second list of input ids. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. max_length: maximum length of the returned list. Will truncate by taking into account the special tokens. add_special_tokens: if set to ``True``, the sequences will be encoded with the special tokens relative to their model. stride: window stride for overflowing tokens. Can be useful to remove edge effect when using sequential list of inputs. The overflowing token will contains a part of the previous window of tokens. truncation_strategy: string selected in the following options: - 'longest_first' (default) Iteratively reduce the inputs sequence until the input is under max_length starting from the longest one at each token (when there is a pair of input sequences) - 'only_first': Only truncate the first sequence - 'only_second': Only truncate the second sequence - 'do_not_truncate': Does not truncate (raise an error if the input sequence is longer than max_length) pad_to_max_length: if set to True, the returned sequences will be padded according to the model's padding side and padding index, up to their max length. If no max length is specified, the padding is done up to the model's max length. The tokenizer padding sides are handled by the following strings: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences Defaults to False: no padding. return_tensors: (optional) can be set to 'tf' or 'pt' to return respectively TensorFlow tf.constant or PyTorch torch.Tensor instead of a list of python integers. return_token_type_ids: (optional) Set to False to avoid returning token_type_ids (default: set to model specifics). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) return_overflowing_tokens: (optional) Set to True to return overflowing token information (default False). return_special_tokens_mask: (optional) Set to True to return special tokens mask information (default False). return_lengths (:obj:`bool`, `optional`, defaults to :obj:`False`): If set the resulting dictionary will include the length of each encoded inputs Return: A Dictionary of shape:: { input_ids: list[int], token_type_ids: list[int] if return_token_type_ids is True (default) overflowing_tokens: list[int] if a ``max_length`` is specified and return_overflowing_tokens is True num_truncated_tokens: int if a ``max_length`` is specified and return_overflowing_tokens is True special_tokens_mask: list[int] if ``add_special_tokens`` if set to ``True`` and return_special_tokens_mask is True length: int if return_lengths is True } With the fields: - ``input_ids``: list of token ids to be fed to a model - ``token_type_ids``: list of token type ids to be fed to a model - ``overflowing_tokens``: list of overflowing tokens if a max length is specified. - ``num_truncated_tokens``: number of overflowing tokens a ``max_length`` is specified - ``special_tokens_mask``: if adding special tokens, this is a list of [0, 1], with 0 specifying special added tokens and 1 specifying sequence tokens. - ``length``: this is the length of ``input_ids`` Nrrrr)rKnum_tokens_to_remover8r0overflowing_tokensZnum_truncated_tokensrr:special_tokens_maskzToken indices sequence length is longer than the specified maximum sequence length for this model ({} > {}). Running this sequence through the model will result in indexing errorszdSequence can't be padded as no maximum length is specified and the model maximum length is too high.rrzInvalid padding strategy:lengthrZr]r^)boolrrrtruncate_sequencesr$create_token_type_ids_from_sequencesget_special_tokens_maskrrr8r9r: LARGE_INTEGERr3r5r4rWrJrrZr_rrbrcr<)rEr3rKr/rr0r8r2r9r?r@rArBrRrZlen_idsZ len_pair_idsr<Z total_lenrfr\rZneeds_to_be_padded differencer#r#r$rLsI       $             z%PreTrainedTokenizer.prepare_for_model)rrIcKs|S)z< Performs any necessary transformations before tokenization r#r1r#r#r$r/sz,PreTrainedTokenizer.prepare_for_tokenization)r3rKrer8r0rIc CsX|dkr||gfS|dkrg}t|D]D}|dksBt|t|kr^|dg|}|dd}q&|dd}q&tt||}|dkr|| d|}n|dkrt||ksttt|||}|| d}|d| }np|dkr2|dk rt||ksttt|||}|| d}|d| }n|dkrFtdntd |||fS) a, Truncates a sequence pair in place to the maximum length. Args: ids: list of tokenized input ids. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids: Optional second list of input ids. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. num_tokens_to_remove (:obj:`int`, `optional`, defaults to ``0``): number of tokens to remove using the truncation strategy truncation_strategy: string selected in the following options: - 'longest_first' (default) Iteratively reduce the inputs sequence until the input is under max_length starting from the longest one at each token (when there is a pair of input sequences). Overflowing tokens only contains overflow from the first sequence. - 'only_first': Only truncate the first sequence. raise an error if the first sequence is shorter or equal to than num_tokens_to_remove. - 'only_second': Only truncate the second sequence - 'do_not_truncate': Does not truncate (raise an error if the input sequence is longer than max_length) stride (:obj:`int`, `optional`, defaults to ``0``): If set to a number along with max_length, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. rr6NZ only_firstZ only_secondZdo_not_truncatezPInput sequence are too long for max_length. Please select a truncation strategy.zkTruncation_strategy should be selected in ['longest_first', 'only_first', 'only_second', 'do_not_truncate'])rangerrrrW) rEr3rKrer8r0rfrZ window_lenr#r#r$rjs8     z&PreTrainedTokenizer.truncate_sequences)rrrIcCs2|dkrt|dgSdgt|dgt|SrrrErrr#r#r$rksz8PreTrainedTokenizer.create_token_type_ids_from_sequencescCs|dkr |S||S)a2 Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A RoBERTa sequence has the following format: single sequence: X pair of sequences: A B Nr#rqr#r#r$r sz4PreTrainedTokenizer.build_inputs_with_special_tokens)rralready_has_special_tokensrIcCsdg|rt|ndt|S)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer ``prepare_for_model`` or ``encode_plus`` methods. Args: token_ids_0: list of ids (must not contain special tokens) token_ids_1: Optional list of ids (must not contain special tokens), necessary when fetching sequence ids for sequence pairs already_has_special_tokens: (default False) Set to True if the token list is already formated with special tokens for the model Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. rr)rErrrrr#r#r$rl,sz+PreTrainedTokenizer.get_special_tokens_mask)r3skip_special_tokensrIcCs~t|tr(||jkr|j|S||Sg}|D]H}t|}|rL||jkrLq0||jkrh||j|q0|||q0|S)a' Converts a single index or a sequence of indices (integers) in a token " (resp.) a sequence of tokens (str), using the vocabulary and added tokens. Args: skip_special_tokens: Don't decode special tokens (self.all_special_tokens). Default: False )rBr!r_convert_id_to_tokenrr)rEr3rsrXindexr#r#r$convert_ids_to_tokens?s     z)PreTrainedTokenizer.convert_ids_to_tokensrurIcCstdSr?rrErur#r#r$rtXsz(PreTrainedTokenizer._convert_id_to_token)rXrIcCsd||S)z Converts a sequence of tokens (string) in a single string. The most simple way to do it is ' '.join(self.convert_ids_to_tokens(token_ids)) but we often want to remove sub-word tokenization artifacts at the same time.  )rrv)rErXr#r#r$convert_tokens_to_string[sz,PreTrainedTokenizer.convert_tokens_to_string token_idsrsclean_up_tokenization_spacesrIc Cs|j||d}g}g}|D]L}|r.||jkr.q||jkr\|rP|||g}||q||q|r||||d|}|r||} | S|SdS)aH Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing ``self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))``. Args: token_ids: list of tokenized input ids. Can be obtained using the `encode` or `encode_plus` methods. skip_special_tokens: if set to True, will replace special tokens. clean_up_tokenization_spaces: if set to True, will clean up the tokenization spaces. )rsryN)rvrrrrzrclean_up_tokenization) rEr|rsr}Zfiltered_tokensZ sub_textsZcurrent_sub_textr r clean_textr#r#r$decodebs&      zPreTrainedTokenizer.decode) out_stringrIcCsX|ddddddddd d d d d ddddddd}|S)zx Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms. z ..z ??z !!z ,,z ' 'z n'tzn'tz 'mz'mz 'sz'sz 'vez'vez 'rez're)replace)rr#r#r$r~s<   z)PreTrainedTokenizer.clean_up_tokenization)N)F)NTNrr6FN) NTNrr6FFNNNFFF) TNrr6FFNNNFFFF) NNTrr6FNNNFFF)Nrr6r)N)N)NF)F)FT)Frrrr rrrJr"rrrrr!rrr3r`rartrrrrrrrrAr classmethodrrrrrrr TextInputr0r(rr2r4r PreTokenizedInput EncodedInputr rir=r<r; TextInputPairPreTokenizedInputPairEncodedInputPairrYr rVrLr/r rjrkrrlrvrtrzr staticmethodr~rur#r#rFr$rs 6        6 0) 1 6H   K  & D   I  <    (rcs(eZdZdZedfdd ZeedddZeeddd Z ee dd d Z ee dd d Z e dddZddZd;eee ee ee e e e eeefdddZe edddZe eedddZde ee e eed)d*d+Z!d?eee ee"ee#ee$fe ee e ee e eeee ee e e e e e%d/d0d1Z&d@ee e#feee e#fe ee e e ee ee ee ee e e e e%d2d3d4Z'dAee e e ed5d6d7Z(ee)ed8d9d:Z*Z+S)BPreTrainedTokenizerFasta2 Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherit from PreTrainedTokenizer. Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary. This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Class attributes (overridden by derived classes): - ``vocab_files_names``: a python ``dict`` with, as keys, the ``__init__`` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - ``pretrained_vocab_files_map``: a python ``dict of dict`` the high-level keys being the ``__init__`` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` (string) of the pretrained models with, as associated values, the `url` (string) to the associated pretrained vocabulary file. - ``max_model_input_sizes``: a python ``dict`` with, as keys, the `short-cut-names` (string) of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or None if the model has no maximum input size. - ``pretrained_init_configuration``: a python ``dict`` with, as keys, the `short-cut-names` (string) of the pretrained models, and as associated values, a dictionnary of specific arguments to pass to the ``__init__``method of the tokenizer class for this pretrained model when loading the tokenizer with the ``from_pretrained()`` method. Args: - ``tokenizer`` (`BaseTokenizerFast`): A Fast tokenizer from the HuggingFace tokenizer library (in low level Rust language) - ``model_max_length``: (`Optional`) int: the maximum length in number of tokens for the inputs to the transformer model. When the tokenizer is loaded with `from_pretrained`, this will be set to the value stored for the associated model in ``max_model_input_sizes`` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`). no associated max_length can be found in ``max_model_input_sizes``. - ``padding_side``: (`Optional`) string: the side on which the model should have padding applied. Should be selected between ['right', 'left'] - ``model_input_names``: (`Optional`) List[string]: the list of the forward pass inputs accepted by the model ("token_type_ids", "attention_mask"...). - ``bos_token``: (`Optional`) string: a beginning of sentence token. Will be associated to ``self.bos_token`` and ``self.bos_token_id`` - ``eos_token``: (`Optional`) string: an end of sentence token. Will be associated to ``self.eos_token`` and ``self.eos_token_id`` - ``unk_token``: (`Optional`) string: an unknown token. Will be associated to ``self.unk_token`` and ``self.unk_token_id`` - ``sep_token``: (`Optional`) string: a separation token (e.g. to separate context and query in an input sequence). Will be associated to ``self.sep_token`` and ``self.sep_token_id`` - ``pad_token``: (`Optional`) string: a padding token. Will be associated to ``self.pad_token`` and ``self.pad_token_id`` - ``cls_token``: (`Optional`) string: a classification token (e.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model). Will be associated to ``self.cls_token`` and ``self.cls_token_id`` - ``mask_token``: (`Optional`) string: a masking token (e.g. when training a model with masked-language modeling). Will be associated to ``self.mask_token`` and ``self.mask_token_id`` - ``additional_special_tokens``: (`Optional`) list: a list of additional special tokens. Adding all special tokens here ensure they won't be split by the tokenization process. Will be associated to ``self.additional_special_tokens`` and ``self.additional_special_tokens_ids`` )r.c s*t|tstd||_tjf|dS)NzZTokenizer should be an instance of a Tokenizer provided by HuggingFace tokenizers library.)rBBaseTokenizerFastrW _tokenizerr@rA)rEr.rrFr#r$rAs  z PreTrainedTokenizerFast.__init__rTcCs|jSr?)rrPr#r#r$backend_tokenizersz)PreTrainedTokenizerFast.backend_tokenizercCs |jjjSr?)rdecoderrPr#r#r$rszPreTrainedTokenizerFast.decodercCsdS)NTr#rPr#r#r$rszPreTrainedTokenizerFast.is_fastcCs|jjddS)NFZwith_added_tokensrZget_vocab_sizerPr#r#r$rsz"PreTrainedTokenizerFast.vocab_sizecCs|jjddS)NTrrrPr#r#r$rszPreTrainedTokenizerFast.__len__cCs|j|dS)zc Update the backend fast tokenizer. Override method from base class SpecialTokensMixin N)rrrr#r#r$rsz-PreTrainedTokenizerFast._maybe_update_backendNF)r>r9r?r@rArBrCrIc Cs0|dkrd|jk}|dkr$d|jk}|r@|jdk r@|g|j}n|g}tt} |D]d} | d| j|rz| d| j|r| d| j|r| d| j|rR| d| j qR|dk r,| D]`\} } |dkrt rt | | | <q|dkrtrt| | | <q|dk rtd |q| S) a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). If return_tensors is not None, these lists of lists are converted to 2-D tensors for input_ids, token_type_ids and attention_mask. Output shape: (overflows, sequence length) Nrrr:rgZoffset_mappingrZr]r^)rZ overflowingrrrr3Ztype_idsrrgoffsetsrSrrZr_rrbrcr8r9r:) rEr>r9r?r@rArBrCrUZ encoding_dicterrr#r#r$_convert_encodings>   z)PreTrainedTokenizerFast._convert_encoding)r rIcCs|j|}|dkr|jS|Sr?)rZ token_to_idr)rEr rur#r#r$r21 s z;PreTrainedTokenizerFast._convert_token_to_id_with_added_vocrwcCs|jt|Sr?)rZ id_to_tokenr!rxr#r#r$rt7 sz,PreTrainedTokenizerFast._convert_id_to_token)rXrsrIcCs|j||Sr?)rr)rErXrsr#r#r$rz: sz0PreTrainedTokenizerFast.convert_tokens_to_string)r rIcCst|tr|g}|j|S)ax Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Args: new_tokens: string or list of string or AddedTokenFast. Each string is a token to add. Tokens are only added if they are not already in the vocabulary. AddedTokenFast wrap a string token to let you personnalize it's behavior (Whether this token should only match against single word, whether this token should strip all potential whitespaces on the left side, Whether this token should strip all potential whitespaces on the right side...). See details for AddedToken in HuggingFace tokenizers library. Returns: Number of tokens added to the vocabulary. Examples:: # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased') model = BertModel.from_pretrained('bert-base-uncased') num_added_toks = tokenizer.add_tokens(['new_tok1', 'my_new-tok2']) print('We have added', num_added_toks, 'tokens') model.resize_token_embeddings(len(tokenizer)) # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer. )rBrJrr)rEr r#r#r$r= s z"PreTrainedTokenizerFast.add_tokens)rrIcs(t|}t|}|j||Sr?)r@rr,rQr)rErZnum_added_tokensrXrFr#r$rX s   z*PreTrainedTokenizerFast.add_special_tokens)rrIcCs |j|Sr?)rr)rErr#r#r$rf sz1PreTrainedTokenizerFast.num_special_tokens_to_add)rrrrIcCs|j|||jSr?)rr=rX)rErrrr#r#r$r0i sz PreTrainedTokenizerFast.tokenizeTrr6)rOrr/r0r8r2r>r9r?r@rArBrCrRrIc sht|tstdt||p2dk o2t|dk}|rJjdkrJtdtj||||j jj j d B|rHg}t |D]\}}t|tt fstd|t|tt|dkot|dtt f}tjjj|r|dn|ddd d }|r&tjjjd d |dDddd d }nd}j|||}||qnft|dkrt|dt tfrjj|dd |i}njj|d|d}|g}njj||d}W5QRXfdd |D}i}|dD]Rfdd |D}dkrtj|dd}ndkr.tj|dd}||<qr^tdd t |D}||d<t||S)Nz2batch_text_or_text_pairs has to be a list (got {})rzTUnable to set proper padding strategy as the tokenizer does not have a padding tokenr-zbatch_encode_plus(..., is_pretokenized=True) requires batch_text_or_text_pairs to be either List[List[str]] or List[Tuple[List[str], List[str]]] but sample at index {} is of type {}rFrTZgrowing_offsetscSsg|] }d|fqSr#)rpsr#r#r$r sz=PreTrainedTokenizerFast.batch_encode_plus..rc s$g|]}j|dqS))r>r9r?r@rArBrC)r)rpr>)r@rCrArBr9r?rEr#r$r s csg|]}|D]}|qqSr#r#)rprHr)rr#r$r s rZ)Zaxisr]ZdimcSs"g|]\}}|gt|dqS)r:r)rprencr#r#r$r soverflow_to_sample_mapping)rBrrWr:rrr4r;rr3r5rr rrrirCmerge encode_batch post_processrr=rOrZstackrbr,r<)rErOrr/r0r8r2r>r9r?r@rArBrCrRrrUrsampleZis_pairZencodings_textZencodings_pairr>rXZ sanitizedrrr#)rr@rCrArBr9r?rEr$rYn s   "    z)PreTrainedTokenizerFast.batch_encode_plus)rr7rr/r2r0r8r>r9r?r@rArBrCrIcKs>|rt|trt|dkr|jj|dd}tj|dd}t|trl|jjdd|Ddd}tj|dd}n&|dkrzd}ntdt |t ||j |||}t |j || | | | | |d |}ntd t |t |n>|r||fgn|g}|j |f||||| | | | | ||d |}| s:t d d |D|j}|S)NrFrTrcSsg|] }d|fqSrr#)rppr#r#r$r sz7PreTrainedTokenizerFast.encode_plus..zrencode_plus(..., is_pretokenized=True) requires text and text_pair to be List[str] but got (text={}, text_pair={}))r9r?r@rArBrCzdencode_plus(..., is_pretokenized=True) requires text to be List[str] but got (text={}, text_pair={})) rr/r0r8r9r?r@rArBrCr2cSs8i|]0\}}|t|dkr0t|dtr0|dn|qS)r)rrBr)rprrr#r#r$rsK sz7PreTrainedTokenizerFast.encode_plus..)rBrrrrrCrrr:rrr<rrYrSrU)rErr7rr/r2r0r8r>r9r?r@rArBrCrr>Z encoding_pairZbatched_outputZ batched_inputr#r#r$r; s~     z#PreTrainedTokenizerFast.encode_plusr{cCs(|j||}|r ||}|S|SdSr?)rrr~)rEr|rsr}rrr#r#r$rT s  zPreTrainedTokenizerFast.decode)rrIcCsJtj|r|j|}n(tjtj|\}}|jj||d}t|S)N)name)rrrrsaver#abspathr)rErfilesfolderfiler#r#r$r_ s  z'PreTrainedTokenizerFast.save_vocabulary)NNNFFF)F)F)NF) TNrr6FFNNNFFFF) NTNFrr6FNNNFFF)FT),rrrr rrArtr DecoderFastrrirr!rrrrCr rrJrrr2rtrrzr rrr rrrr0rrrr<rYr;rr rrur#r#rFr$rs9  8    ] rcCsN||jdd}|dkr*|dd|fS|dd|f|dd|ffSdS)z=Remove columns that are populated exclusively by pad_token_idrrN)neany)r:r4rZkeep_column_maskr#r#r$ trim_batchi sr)N)Fr rr(r*rloggingr*rr collectionsrr contextlibrtypingrrrrr r r r Z tokenizersr rrrCZtokenizers.decodersrrZtokenizers.implementationsrrZ file_utilsrrrrrrZ tensorflowrZrb getLoggerrr8rrrr!rrmrJrrrrrrrr%r,rir;r<rvrrrr#r#r#r$s (         @7^sP