U "ºcùHã@sÐddlmZddlZddlmZmZddlmZddlm Z ddl m Z ddl m Z ddlmZmZddlZejdd kržddlZddlZejZe ed ƒ¡e ƒZGd d „d eƒZGd d„deƒZd dgZdS)é)Úprint_functionN)Ú defaultdictÚCounter)Úpartial)Úchain)Ú text_type)Ú Perluniprops)Úparallelize_preprocessÚgrouperézTYou should really be using Python3!!! Tick tock, tick tock, https://pythonclock.org/csÐeZdZdZed e d¡¡ƒZed e d¡¡ƒZ ed e d¡¡ƒZ d$‡fdd„ Z d%d d „Z d&d d„Z d'dd„Zd(dd„Zd)dd„Zd*dd„Zd+dd„Zedd„ƒZdd„Zdd„Zd d!„Zd"d#„Z‡ZS),ÚMosesTruecaserzò This is a Python port of the Moses Truecaser from https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/train-truecaser.perl https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/truecase.perl ÚÚLowercase_LetterÚUppercase_LetterNÚutf8csztt|ƒ ¡t d |j|j|j¡¡|_ t d¡|_ ddddh|_ ddd d d d d dh|_ ||_ ||_|rv| |¡|_dS)a. :param load_from: :type load_from: :param is_asr: A flag to indicate that model is for ASR. ASR input has no case, make sure it is lowercase, and make sure known are cased eg. 'i' to be uppercased even if i is known. :type is_asr: bool z[{}{}{}]z(<.*(?<=>))(.*)((?=]*>)Ú.ú:ú?ú!ú(ú[ú"ú'ú'ú"ú[ú]N)Úsuperr Ú__init__ÚreÚcompileÚformatrrÚTitlecase_LetterÚSKIP_LETTERS_REGEXZXML_SPLIT_REGXÚSENT_ENDÚDELAYED_SENT_STARTÚencodingÚis_asrÚ _load_modelÚmodel)ÚselfZ load_fromr'r&©Ú __class__©ú7/tmp/pip-unpacked-wheel-43qgbez0/sacremoses/truecase.pyr-s. ÿÿ ø zMosesTruecaser.__init__FcCs°d}g}t|ƒD]š\}}t d|¡r(qn ||jkr4q|sH||jkrHd}q|j |¡sZd}qd}|shd}n"|rŠ|d ¡r~d}n |dkrŠd}d}|dkr| | ¡||f¡q|S)z This function checks through each tokens in a sentence and returns the appropriate weight of each surface token form. Tú (<\S[^>]*>)Frégš™™™™™¹?) Ú enumeraterÚsearchr%r$r#ÚislowerÚappendÚlower)r*ÚtokensÚpossibly_use_first_tokenÚ is_first_wordZtruecase_weightsÚiÚtokenZcurrent_word_weightr-r-r.Úlearn_truecase_weightsTs2    z%MosesTruecaser.learn_truecase_weightsr0c Cshttƒ}t|j|d}tt||||dŽ}|D]\} } } || | | 7<q.|r^| ||¡| |¡S)aN :param document_iterator: The input document, each outer list is a sentence, the inner list is the list of tokens for each sentence. :type document_iterator: iter(list(str)) :param possibly_use_first_token: When True, on the basis that the first word of a sentence is always capitalized; if this option is provided then: a) if a sentence-initial token is *not* capitalized, then it is counted, and b) if a capitalized sentence-initial token is the only token of the segment, then it is counted, but with only 10% of the weight of a normal token. :type possibly_use_first_token: bool :returns: A dictionary of the best, known objects as values from `_casing_to_model()` :rtype: {'best': dict, 'known': Counter} )r7©Ú progress_bar)rrrr;rr Ú_save_model_from_casingÚ_casing_to_model) r*Údocument_iteratorÚsave_tor7Ú processesr=ÚcasingZtrain_truecaserZ token_weightsZlowercase_tokenZ surface_tokenZweightr-r-r.Ú_train‚s$þÿÿ zMosesTruecaser._traincCs"d|_|j|||||d|_|jS)z\ Default duck-type of _train(), accepts list(list(str)) as input documents. Nr<)r)rD)r*Z documentsrAr7rBr=r-r-r.Útrain¬s ûzMosesTruecaser.trainc CsNt||jd}dd„| ¡Dƒ}W5QRXd|_|j|||||d|_|jS)zj Duck-type of _train(), accepts a filename to read as a `iter(list(str))` object. ©r&css|]}| ¡VqdS©N©Úsplit©Ú.0Úliner-r-r.Ú Ïsz1MosesTruecaser.train_from_file..Nr<)Úopenr&Ú readlinesr)rD)r*ÚfilenamerAr7rBr=Úfinr@r-r-r.Útrain_from_fileÁs ÿûzMosesTruecaser.train_from_filecCs4dd„| ¡Dƒ}d|_|j|||||d|_|jS)zm Duck-type of _train(), accepts a file object to read as a `iter(list(str))` object. css|]}| ¡VqdSrGrHrJr-r-r.rMész8MosesTruecaser.train_from_file_object..Nr<)rOr)rD)r*Z file_objectrAr7rBr=r@r-r-r.Útrain_from_file_objectÜs ÿûz%MosesTruecaser.train_from_file_objectc Cs$tdƒ}t|dƒst|ƒ‚d}g}| |¡}t|ƒD]Ú\}} t d| ¡rT| | ¡q4| dksf|  d¡rr| | ¡q4t d| ¡  ¡\} } |j r”|   ¡} |j d  |   ¡d¡} |j d   | d¡} |rÊ| rÊ| } n| rÜ|rÖ| n| } n| rä| } | | } | | ¡| |jk}| |jkr4d }q4|r d  |¡S|S) a  Truecase a single sentence / line of text. :param text: A single string, i.e. sentence text. :type text: str :param use_known: Use the known case if a word is a known word but not the first word. :type use_known: bool zV Use Truecaser.train() to train a model. Or use Truecaser('modefile') to load a model.r)Tr/ú|z ^([^\|]+)(.*)ÚbestNÚknownFú )ÚstrÚhasattrÚAssertionErrorÚ split_xmlr1rr2r4Ú startswithÚgroupsr'r5r)Úgetr$r%Újoin) r*ÚtextÚ return_strZ use_knownZcheck_model_messager8Útruecased_tokensr6r9r:Z other_factorsZ best_caseZ known_caser-r-r.Útruecaseös@ ÿ    ÿ   zMosesTruecaser.truecaseTc csJt||jd2}|D]&}| | ¡¡}|r4d |¡n|VqW5QRXdS)NrFrW)rNr&rcÚstripr_)r*rPrarQrLrbr-r-r.Ú truecase_file>szMosesTruecaser.truecase_filecCs*| ¡}g}|r&t d|¡}t d|¡}t d|¡}|rÄ| ¡\}}t d|¡r°t|ƒdkr°t d|d¡r°|d|7<t d|¡}|r¾|d| d ¡7<| d ¡}n| |d ¡|}nP|rä| | d ¡¡| d ¡}n0|r| | d ¡¡| d ¡}ntd  |¡ƒ‚|d ¡|d<q |S) a  Python port of split_xml function in Moses' truecaser: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/truecaser.perl :param line: Input string, should be tokenized, separated by space. :type line: str z^\s*(<\S[^>]*>)(.*)$z^\s*([^\s<>]+)(.*)$z^\s*(\S+)(.*)$z^\Srz\|$éÿÿÿÿz ^(\|+)(.*)$r0érWzERROR: huh? {}) rdrr2r]ÚlenÚgroupr4Ú Exceptionr!)rLr6Zhas_xmlZ is_non_xmlZ xml_cognatesZ potential_xmlZ line_nextZ is_factorr-r-r.r[EsD      ÿ þý  ÿ ÿ zMosesTruecaser.split_xmlc Csli}tƒ}|D]L}|| ¡}|dd||<|js|dd…D]\}}||d7<q@q|||dœ}|S)zg :returns: A tuple of the (best, known) objects. :rtype: tuple(dict, Counter) rr0N)rUrVrC)rÚ most_commonr') r*rCrUrVZ token_lowerr6r:Úcountr)r-r-r.r?xs  zMosesTruecaser._casing_to_modelcCs| |jd|¡dS)NrC)r>r))r*rPr-r-r.Ú save_modelŽszMosesTruecaser.save_modelc Cs t|d|jd†}|D]z}t|| ¡ƒ}g}t|| ¡ƒD]<\}\}}|dkrbd |||¡} nd |||¡} | | ¡q>td  |¡d|dqW5QRXd S) a" Outputs the truecaser model file in the same output format as https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/train-truecaser.perl :param casing: The dictionary of tokens counter from `train()`. :type casing: default(Counter) ÚwrFrz {} ({}/{})z{} ({})rWÚ )ÚendÚfileN) rNr&ÚsumÚvaluesr1rkr!r4Úprintr_) r*rCrPZfoutr:Ztotal_token_countZ tokens_countsr9rlZ out_tokenr-r-r.r>‘s z&MosesTruecaser._save_model_from_casingc Cs~ttƒ}t||jdX}|D]L}| ¡ ¡}t|dƒD]0\}}| d¡d d¡}t|ƒ|| ¡|<q6qW5QRX|  |¡S)zÄ Loads pre-trained truecasing file. :returns: A dictionary of the best, known objects as values from `_casing_to_model()` :rtype: {'best': dict, 'known': Counter} rFrgú/rz()) rrrNr&rdrIr Úintr5r?)r*rPrCrQrLr:rlr-r-r.r(¥s "zMosesTruecaser._load_model)NNr)F)NFr0F)NFr0F)NFr0F)NFr0F)FF)T)Ú__name__Ú __module__Ú __qualname__Ú__doc__rr_Ú perlunipropsÚcharsrrr"rr;rDrErRrSrcreÚ staticmethodr[r?rmr>r(Ú __classcell__r-r-r+r.r !sD' 1ú -ú ú ú  H  2r cs&eZdZ‡fdd„Zddd„Z‡ZS)ÚMosesDetruecaserc#s‚tt|ƒ ¡ddddh|_ddddd d d d h|_d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/h#|_dS)0NrrrrrrrrrrrrÚaÚafterZagainstzal-.+ÚandÚanyÚasÚatÚbeZbecauseZbetweenZbyZduringzel-.+ÚforÚfromZhisÚinÚisZitsÚlastÚnotZofÚoffÚonZthanZtheZtheirÚthisÚtoÚwasÚwereÚwhichZwillÚwith)rrrr$r%Ú ALWAYS_LOWER©r*r+r-r.r¸s^ø ÝzMosesDetruecaser.__init__FcsŠg}d}| ¡D]P}|r4|dd… ¡|dd…n|}| |¡|ˆjkrRd}q|ˆjkrd}q|rx‡fdd„|Dƒ}|r†d |¡S|S)z¼ Detruecase the translated files from a model that learnt from truecased tokens. :param text: A single string, i.e. sentence text. :type text: str TNr0Fcs6g|].}|ˆjkr|n|dd… ¡|dd…‘qS)Nr0)r•Úupper)rKr:r–r-r.Ú sÿz/MosesDetruecaser.detruecase..rW)rIr—r4r$r%r_)r*r`Z is_headlineraZ cased_tokensZsentence_startr:r-r–r.Ú detruecaseîs  $    þzMosesDetruecaser.detruecase)FF)rwrxryrr™r~r-r-r+r.r·s 6r)Ú __future__rrÚ collectionsrrÚ functoolsrÚ itertoolsrÚsixrZsacremoses.corpusrZsacremoses.utilr r ÚsysÚ version_infoÚioÚwarningsrNÚwarnrXr{Úobjectr rÚ__all__r-r-r-r.Ús0     ÿÿT