U #c/@s^ddlmZmZmZddlmZmZddlmZm Z m Z m Z m Z e e e fZGdddZdS)) TokenizerEncoding AddedToken)TokenizedSequenceTokenizedSequenceWithOffsets)ListUnionTupleOptionalDictc@seZdZdEedddZddZeeddd ZdFee e efd d d Z dGeed ddZ dHe e e ee ee e e edddZddZdIee ee e dddZddZeee efeddd Zeee efed!d"d#Ze e d$d%d&ZdJeeefeed'd(d)ZdKeeeeefeeed*d+d,ZdLe e e eed-d.d/ZdMeee ee e ffeeed0d1d2ZdNeee ee d3d4d5Z dOeeee ee d6d7d8Z!e e ed9d:d;Z"ee e d<d=d>Z#dPe e e d?d@dAZ$dQee eeedBdCdDZ%dS)R BaseTokenizerN) tokenizercCs||_|dk r|ni|_dS)N) _tokenizer _parameters)selfr parametersrM/tmp/pip-unpacked-wheel-zq7pv0lh/tokenizers/implementations/base_tokenizer.py__init__ szBaseTokenizer.__init__cCs(d|jddd|jDS)Nz!Tokenizer(vocabulary_size={}, {})z, css"|]\}}|dt|VqdS)=N)str).0kvrrr sz)BaseTokenizer.__repr__..)formatrget_vocab_sizejoinritemsrrrr__repr__szBaseTokenizer.__repr__)is_pairreturncCs |j|S)z Return the number of special tokens that would be added for single/pair sentences. :param is_pair: Boolean indicating if the input would be a single sentence or a pair :return: )rnum_special_tokens_to_add)rr!rrrr#sz'BaseTokenizer.num_special_tokens_to_addT)with_added_tokensr"cCs|jj|dS)z Returns the vocabulary Args: with_added_tokens: boolean: Whether to include the added tokens in the vocabulary Returns: The vocabulary r$)r get_vocabrr$rrrr&s zBaseTokenizer.get_vocabcCs|jj|dS)z Return the size of vocabulary, with or without added tokens. Args: with_added_tokens: (`optional`) bool: Whether to count in added special tokens or not Returns: Size of vocabulary r%)rrr'rrrr(s zBaseTokenizer.get_vocab_sizerightr[PAD] directionpad_id pad_type_id pad_token max_lengthcCs|jj|||||dS)a Change the padding strategy Args: direction: (`optional`) str: Can be one of: `right` or `left` pad_id: (`optional`) unsigned int: The indice to be used when padding pad_type_id: (`optional`) unsigned int: The type indice to be used when padding pad_token: (`optional`) str: The pad token to be used when padding max_length: (`optional`) unsigned int: If specified, the length at which to pad. If not specified we pad using the size of the longest sequence in a batch r*)renable_padding)rr+r,r-r.r/rrrr04szBaseTokenizer.enable_paddingcCs |jS)z Disable padding )r no_paddingrrrrr1WszBaseTokenizer.no_padding longest_first)r/stridestrategycCs|jj|||dS)a Change the truncation options Args: max_length: unsigned int: The maximum length at which to truncate stride: (`optional`) unsigned int: The length of the previous first sequence to be included in the overflowing sequence strategy: (`optional) str: Can be one of `longest_first`, `only_first` or `only_second` )r3r4)renable_truncation)rr/r3r4rrrr5[szBaseTokenizer.enable_truncationcCs |jS)z Disable truncation )r no_truncationrrrrr6mszBaseTokenizer.no_truncation)tokensr"cCs |j|S)aQ Add the given tokens to the vocabulary Args: tokens: List[Union[str, AddedToken]]: A list of tokens to add to the vocabulary. Each token can either be a string, or an instance of AddedToken Returns: The number of tokens that were added to the vocabulary )r add_tokens)rr7rrrr8qs zBaseTokenizer.add_tokens)special_tokensr"cCs |j|S)a Add the given special tokens to the vocabulary, and treat them as special tokens. The special tokens will never be processed by the model, and will be removed while decoding. Args: tokens: List[Union[str, AddedToken]]: A list of special tokens to add to the vocabulary. Each token can either be a string, or an instance of AddedToken Returns: The number of tokens that were added to the vocabulary )radd_special_tokens)rr9rrrr:~sz BaseTokenizer.add_special_tokens)sequencer"cCs |j|S)z Normalize the given sequence Args: sequence: str: The sequence to normalize Returns: The normalized string )r normalize)rr;rrrr<s zBaseTokenizer.normalize)r;type_idr"cCs|jj||S)a6 Encode the given sequence. Let us skip the Normalizer and PreTokenizer by providing already tokenized substrings. A sequence can either be: - `TokenizedSequence`: (`List[str]`) - `TokenizedSequenceWithOffsets: (`List[Tuple[str, Offsets]]`) where Offsets is a Tuple[int, int]. If the Offsets are not provided, they will be automatically generated, making the hypothesis that all the tokens in the `TokenizedSequence` are contiguous in the original string. Args: sequence: Union[TokenizedSequence, TokenizedSequenceWithOffsets] Either a TokenizedSequence or a TokenizedSequenceWithOffsets type_id: int: The type id of the given sequence Returns: An Encoding )rmodelencode)rr;r=rrrencode_tokenizedszBaseTokenizer.encode_tokenized) sequencesr=r"cCs|jj||S)a Encode the given batch of sequence. Let us skip the Normalizer and PreTokenizer by providing already tokenized substrings. A sequence can either be: - `TokenizedSequence`: (`List[str]`) - `TokenizedSequenceWithOffsets: (`List[Tuple[str, Offsets]]`) where Offsets is a Tuple[int, int]. If the Offsets are not provided, they will be automatically generated, making the hypothesis that all the tokens in the `TokenizedSequence` are contiguous in the original string. Args: sequences: Union[List[TokenizedSequence], List[TokenizedSequenceWithOffsets]] A list of sequence. Each sequence is either a TokenizedSequence or a TokenizedSequenceWithOffsets type_id: int: The type if of the given sequence Returns: A list of Encoding )rr> encode_batch)rrAr=rrrencode_tokenized_batchsz$BaseTokenizer.encode_tokenized_batch)r;pairr:r"cCs |dkrtd|j|||S)aa Encode the given sequence Args: sequence: str: The sequence to encode pair: (`optional`) Optional[str]: The optional pair sequence add_special_tokens: bool: Whether to add the special tokens while encoding. Returns: An Encoding Nz,None input is not valid. Should be a string.) ValueErrorrr?)rr;rDr:rrrr?szBaseTokenizer.encode)rAr:r"cCs|dkrtd|j||S)a Encode the given sequences or pair of sequences Args: sequences: List[Union[str, Tuple[str, str]]]: A list of sequences or pair of sequences. The list can contain both at the same time. add_special_tokens: bool: Whether to add the special tokens while encoding. Returns: A list of Encoding NzSNone input is not valid. Should be a list of strings or a list of tuple of strings.)rErrB)rrAr:rrrrBs zBaseTokenizer.encode_batch)idsskip_special_tokensr"cCs |dkrtd|jj||dS)aT Decode the given list of ids to a string sequence Args: ids: List[unsigned int]: A list of ids to be decoded skip_special_tokens: (`optional`) boolean: Whether to remove all the special tokens from the output string Returns: The decoded string Nz6None input is not valid. Should be a list of integers.rG)rErdecode)rrFrGrrrrIs zBaseTokenizer.decode)rArGr"cCs |dkrtd|jj||dS)a} Decode the list of sequences to a list of string sequences Args: sequences: List[List[unsigned int]]: A list of sequence of ids to be decoded skip_special_tokens: (`optional`) boolean: Whether to remove all the special tokens from the output strings Returns: A list of decoded strings Nzsave)rrOrPrrrrQ>s zBaseTokenizer.save)encodingrDr:r"cCs|j|||S)ak Apply all the post-processing steps to the given encodings. The various steps are: 1. Truncate according to global params (provided to `enable_truncation`) 2. Apply the PostProcessor 3. Pad according to global params. (provided to `enable_padding`) Args: encoding: Encoding: The main Encoding to post process pair: Optional[Encoding]: An optional pair Encoding add_special_tokens: bool: Whether to add special tokens Returns: The resulting Encoding )r post_process)rrRrDr:rrrrSJszBaseTokenizer.post_process)N)T)T)r(rrr)N)rr2)r)r)NT)T)T)T)N)NT)&__name__ __module__ __qualname__rrr boolintr#r rr&rr r0r1r5r6rrrr8r:r<rrrr@rCr?r rBrIrJrLrNrQrSrrrrr s  #             r N)Z tokenizersrrrZtokenizers.modelsrrtypingrrr r r rXZOffsetsr rrrrs