U &cU&@sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z e e Zddd Zd d id d id Zd d iZddZddZGddde ZGddde ZdS)z$Tokenization classes for OpenAI GPT.N)CharBPETokenizer)BasicTokenizer)PreTrainedTokenizerPreTrainedTokenizerFastz vocab.jsonz merges.txt) vocab_file merges_filez openai-gptzIhttps://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-vocab.jsonzIhttps://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-merges.txticCs6t}|d}|ddD]}|||f|}q|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rrN)setadd)wordpairsZ prev_charcharrD/tmp/pip-unpacked-wheel-ymerj3tt/transformers/tokenization_openai.py get_pairs.s rcCsn|dd}|dd}|dd}|dd}|dd}td d |}td d |}td d|}|S)zq fixes some issues the spacy tokenizer had on books corpus also does some whitespace standardization u—-u–u―u…z...´'zD(-+|~+|!+|"+|;+|\?+|\++|,+|\)+|\(+|\\+|\/+|\*+|\[+|\]+|}+|{+|\|+|_+)z \1 z\s*\n\s*z z[^\S\n]+ )replaceresubstrip)textrrrtext_standardize;s     rcsreZdZdZeZeZeZ dfdd Z e ddZ ddZ d d Zd d Zd dZddZddZddZZS)OpenAIGPTTokenizera BPE tokenizer. Peculiarities: - lower case all inputs - uses SpaCy tokenizer and ftfy for pre-BPE tokenization if they are installed, fallback to BERT's BasicTokenizer if not. This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the methods. Users should refer to the superclass for more information regarding methods. Args: vocab_file (:obj:`str`): Path to the vocabulary file. merges_file (:obj:`str`): Path to the merges file. unk_token (:obj:`string`, `optional`, defaults to ""): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. c stjfd|i|z4ddl}ddlm}|}|j||_|j|_Wn0t k rzt dt dd|_d|_YnXt |dd}t||_W5QRXd d |jD|_t |dd} | d d d } W5QRXdd| D} tt| tt| |_i|_dS)N unk_tokenr)EnglishzQftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.T)Z do_lower_caseutf-8encodingcSsi|]\}}||qSrr).0kvrrr tsz/OpenAIGPTTokenizer.__init__.. rcSsg|]}t|qSr)tuplesplit)r"mergerrr wsz/OpenAIGPTTokenizer.__init__..)super__init__ftfyZ spacy.lang.enrZDefaultsZcreate_tokenizernlpfix_text ImportErrorloggerwarningropenjsonloadencoderitemsdecoderreadr)dictziprangelen bpe_rankscache) selfrrrkwargsr.rZ_nlpZ vocab_handleZ merges_handleZmerges __class__rrr-cs&      zOpenAIGPTTokenizer.__init__cCs t|jSN)r>r7rArrr vocab_size{szOpenAIGPTTokenizer.vocab_sizecCst|jf|jSrE)r;r7Zadded_tokens_encoderrFrrr get_vocabszOpenAIGPTTokenizer.get_vocabc st|dd|ddf}|jkr2j|St|}|sF|dSt|fddd}|jkrhqf|\}}g}d}|t|krcsj|tdS)Ninf)r?getfloat)pairrFrrz(OpenAIGPTTokenizer.bpe..keyrrrz z ) r(r@rminr?r>index ValueErrorextendappendjoin) rAtokenr r ZbigramfirstsecondZnew_wordijrrFrbpesF    2      zOpenAIGPTTokenizer.bpecCsg}|jdkrF|j|}|D]$}|dd||dDqnD|t||}|D]*}|dd||jdDq^|S)z Tokenize a string. NcSsg|]}|qSrrr"trrrr+sz0OpenAIGPTTokenizer._tokenize..rcSsg|]}|qSrrr_rrrr+s) r0r/tokenizerVr^r)rrlower)rArZ split_tokensrYrrr _tokenizes  $(zOpenAIGPTTokenizer._tokenizecCs|j||j|jS)z2 Converts a token (str) in an id using the vocab. )r7rKr)rArYrrr_convert_token_to_idsz'OpenAIGPTTokenizer._convert_token_to_idcCs|j||jS)z0Converts an id in a token (BPE) using the vocab.)r9rKr)rArTrrr_convert_id_to_tokensz'OpenAIGPTTokenizer._convert_id_to_tokencCsd|dd}|S)z< Converts a sequence of tokens (string) in a single string. rIr)rXrr)rAtokensZ out_stringrrrconvert_tokens_to_stringsz+OpenAIGPTTokenizer.convert_tokens_to_stringc Cstj|s td|dStj|td}tj|td}t|ddd}| t j |j dd W5QRXd }t|dddh}| d t |jd d dD]@\}}||krtd||}| d|d|d7}qW5QRX||fS)a Save the vocabulary and special tokens file to a directory. Args: save_directory (:obj:`str`): The directory in which to save the vocabulary. Returns: :obj:`Tuple(str)`: Paths to the files saved. z*Vocabulary path ({}) should be a directoryNrrwrr F) ensure_asciirz#version: 0.2 cSs|dS)Nrr)kvrrrrNrOz4OpenAIGPTTokenizer.save_vocabulary..rPzqSaving vocabulary to {}: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rr&r)ospathisdirr2errorformatrXVOCAB_FILES_NAMESr4writer5dumpsr7sortedr?r8r3) rAZsave_directoryrZ merge_filefrTwriterZ bpe_tokensZ token_indexrrrsave_vocabularys*   z"OpenAIGPTTokenizer.save_vocabulary)r)__name__ __module__ __qualname____doc__rqvocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesr-propertyrGrHr^rcrdrerhrw __classcell__rrrCrrKs ,rcs.eZdZdZeZeZeZ dfdd Z Z S)OpenAIGPTTokenizerFastaM Construct a "Fast" BPE tokenizer for OpenAI GPT (backed by HuggingFace's `tokenizers` library). Peculiarities: - lower case all inputs - uses SpaCy tokenizer and ftfy for pre-BPE tokenization if they are installed, fallback to BERT's BasicTokenizer if not. This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the methods. Users should refer to the superclass for more information regarding methods. Args: vocab_file (:obj:`str`): Path to the vocabulary file. merges_file (:obj:`str`): Path to the merges file. unk_token (:obj:`string`, `optional`, defaults to ""): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. rc s,|d|tjt|||ddf|dS)NrT)rrrZ lowercase) setdefaultr,r-r)rArrrrBrCrrr-s  zOpenAIGPTTokenizerFast.__init__)r) rxryrzr{rqr|r}r~rrr-rrrrCrrs r)r{r5loggingrlrZ tokenizersrZtokenization_bertrZtokenization_utilsrr getLoggerrxr2rqr}rrrrrrrrrs,    $