U &c&@sdZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z e e ZddiZddd d d d iZd d d d d Zddddddddddddddddd ZGddde ZGdddZGdddeZdS)zTokenization classes.N)Optional)BasicTokenizer BertTokenizerWordpieceTokenizer load_vocab vocab_filez vocab.txtzZhttps://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese/vocab.txtzmhttps://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-whole-word-masking/vocab.txtz_https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char/vocab.txtzrhttps://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking/vocab.txt)zbert-base-japanesez%bert-base-japanese-whole-word-maskingzbert-base-japanese-charz*bert-base-japanese-char-whole-word-maskingiFmecab wordpiece) do_lower_caseword_tokenizer_typesubword_tokenizer_type characterc s:eZdZdZeZeZeZ e Z dfd d Z ddZ ZS)BertJapaneseTokenizerz BERT tokenizer for Japanese textFTbasicr N[UNK][SEP][PAD][CLS][MASK]c stt|jf|| | | | d|tj|ssz2BertJapaneseTokenizer.__init__..rF)r never_splitZtokenize_chinese_charsr )r r!z.Invalid word_tokenizer_type '{}' is specified.r )vocabrrz1Invalid subword_tokenizer_type '{}' is specified.N)superr__init__ospathisfile ValueErrorformatrr" collections OrderedDictitemsZ ids_to_tokensdo_word_tokenizerword_tokenizerMecabTokenizerdo_subword_tokenizerrsubword_tokenizerCharacterTokenizer)selfrr r-r0r r r!rrrrrZ mecab_kwargskwargs __class__rrr$OsP"     zBertJapaneseTokenizer.__init__csBjrjj|jd}n|g}jr:fdd|D}n|}|S)N)r!cs"g|]}j|D]}|qqSr)r1tokenize)rtokenZ sub_tokenr3rrr sz3BertJapaneseTokenizer._tokenize..)r-r.r7Zall_special_tokensr0)r3texttokensZ split_tokensrr9r _tokenizeszBertJapaneseTokenizer._tokenize) FTTrr NrrrrrN)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_mapPRETRAINED_INIT_CONFIGURATIONZpretrained_init_configuration&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesr$r< __classcell__rrr5rrGs&Jrc@s.eZdZdZd eedddZd dd ZdS) r/z8Runs basic tokenization with MeCab morphological parser.FNT) mecab_optioncCsF||_|dk r|ng|_||_ddl}|dk r8||n||_dS)aConstructs a MecabTokenizer. Args: **do_lower_case**: (`optional`) boolean (default True) Whether to lower case the input. **never_split**: (`optional`) list of str Kept for backward compatibility purposes. Now implemented directly at the base class level (see :func:`PreTrainedTokenizer.tokenize`) List of token not to split. **normalize_text**: (`optional`) boolean (default True) Whether to apply unicode normalization to text before tokenization. **mecab_option**: (`optional`) string passed to `MeCab.Tagger` constructor (default "") Nr)r r!normalize_textMeCabZTaggerr )r3r r!rGrFrHrrrr$s zMecabTokenizer.__init__c Ks|jrtd|}|j|dk r"|ng}g}|j|}d}|dD]Z}|dkrVq|d\}} |||} | t|} |j r||kr| }| || }qF|S)zTokenizes a piece of text.NFKCNr ZEOS ) rG unicodedata normalizer!r parsesplitindexlenr lowerappend) r3r:r!r4r;Z mecab_outputcursorliner8_Z token_startZ token_endrrrr7s"     zMecabTokenizer.tokenize)FNTN)N)r=r>r?r@rstrr$r7rrrrr/sr/c@s"eZdZdZdddZddZdS) r2zRuns Character tokenziation.TcCs||_||_||_dS)agConstructs a CharacterTokenizer. Args: **vocab**: Vocabulary object. **unk_token**: str A special symbol for out-of-vocabulary token. **normalize_text**: (`optional`) boolean (default True) Whether to apply unicode normalization to text before tokenization. N)r"rrG)r3r"rrGrrrr$s zCharacterTokenizer.__init__cCsN|jrtd|}g}t|D]*\}}||jkr>||jq||q|S)aeTokenizes a piece of text into characters. For example: input = "apple" output = ["a", "p", "p", "l", "e"] Args: text: A single token or whitespace separated tokens. This should have already been passed through `BasicTokenizer`. Returns: A list of characters. rI)rGrLrM enumerater"rSr)r3r:Z output_tokensicharrrrr7s     zCharacterTokenizer.tokenizeN)T)r=r>r?r@r$r7rrrrr2s r2)r@r*loggingr%rLtypingrZtokenization_bertrrrr getLoggerr=loggerrArBrDrCrr/objectr2rrrrsP    `4