U #c @shddlmZmZmZmZmZddlmZddlm Z ddl m Z ddl m Z mZmZGddde Zd S) ) Tokenizer AddedTokenpre_tokenizersdecoderstrainers)BPE)NFKC) BaseTokenizer)OptionalListUnionc seZdZdZdeeeeeeefeeee dfdd Z d d dgd gdfeee efe e e eeefe e eed d dZ ZS)SentencePieceBPETokenizerzs SentencePiece BPE Tokenizer Represents the BPE algorithm, with the pretokenization used by SentencePiece N▁T) vocab_file merges_file unk_token replacementadd_prefix_spacedropoutc s|dk r&|dk r&tt||||d}n tt}|t|dk rR|t|gt|_tj||d|_ t j||d|_ d||||d}t ||dS)N)rr)rrZSentencePieceBPE)modelrrrr)rrZ token_to_idstrZadd_special_tokensrZ normalizerrZ MetaspaceZ pre_tokenizerrdecodersuper__init__) selfrrrrrr tokenizer parameters __class__P/tmp/pip-unpacked-wheel-zq7pv0lh/tokenizers/implementations/sentencepiece_bpe.pyrs.  z"SentencePieceBPETokenizer.__init__i0ui)files vocab_size min_frequencyspecial_tokenslimit_alphabetinitial_alphabet show_progressc Cs8tj||||||d}t|tr&|g}|j||dS)z' Train the model using the given files )r%r&r'r(r)r*N)rZ BpeTrainer isinstancer _tokenizertrain) rr$r%r&r'r(r)r*Ztrainerr!r!r"r-4s  zSentencePieceBPETokenizer.train)NNrrTN)__name__ __module__ __qualname____doc__r rr rboolfloatrr intr- __classcell__r!r!rr"r s< (rN)Z tokenizersrrrrrZtokenizers.modelsrZtokenizers.normalizersrZbase_tokenizerr typingr r r rr!r!r!r"s