U "c!@szddlmZWn ek r0ddlmZYnXddlmZddlmZmZddlm Z m Z ddl m Z Gddde Z d d Zd d Zd dZddZdddZdddZdS)) zip_longest) izip_longest)tee)escapeunescape)Paralleldelayed)tqdmc @sZeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd Zeeeeee e e e eeg ZdS)CJKCharsz An object that enumerates the code points of the CJK characters as listed on http://en.wikipedia.org/wiki/Basic_Multilingual_Plane#Basic_Multilingual_Plane iii.iϤi@iiiiii0iOiei)ioioipiii/ipiiiN)__name__ __module__ __qualname____doc__Z Hangul_JamoZ CJK_RadicalsZPhags_PaZHangul_SyllablesZCJK_Compatibility_IdeographsZCJK_Compatibility_FormsZKatakana_Hangul_HalfwidthZ#Ideographic_Symbols_And_PunctuationZTangutZKana_SupplementZNushuZSupplementary_Ideographic_Planerangesrr3/tmp/pip-unpacked-wheel-43qgbez0/sacremoses/util.pyr s2r cstfdddDS)u This checks for CJK character. >>> CJKChars().ranges [(4352, 4607), (11904, 42191), (43072, 43135), (44032, 55215), (63744, 64255), (65072, 65103), (65381, 65500), (94208, 101119), (110592, 110895), (110960, 111359), (131072, 196607)] >>> is_cjk(u'㏾') True >>> is_cjk(u'﹟') False :param character: The character that needs to be checked. :type character: char :return: bool cs,g|]$\}}|tko"|knqSr)ord).0startend characterrr sszis_cjk..) r r r rrrrrrrr)anyr!rr!ris_cjkcs  r%cCst|dddddddS)a This function transforms the input text into an "escaped" version suitable for well-formed XML formatting. Note that the default xml.sax.saxutils.escape() function don't escape some characters that Moses does so we have to manually add them to the entities dictionary. >>> input_str = ''')| & < > ' " ] [''' >>> expected_output = ''')| & < > ' " ] [''' >>> escape(input_str) == expected_output True >>> xml_escape(input_str) ')| & < > ' " ] [' :param text: The text that needs to be escaped. :type text: str :rtype: str '"|[])'"|[]entities)rtextrrr xml_escapesr4cCst|dddddddS)ai This function transforms the "escaped" version suitable for well-formed XML formatting into humanly-readable string. Note that the default xml.sax.saxutils.unescape() function don't unescape some characters that Moses does so we have to manually add them to the entities dictionary. >>> from xml.sax.saxutils import unescape >>> s = ')| & < > ' " ] [' >>> expected = ''')| & < > ' " ] [''' >>> xml_unescape(s) == expected True :param text: The text that needs to be unescaped. :type text: str :rtype: str r+r,r-r.r/)r&r'r(r)r*r0)rr2rrr xml_unescapesr5cCs t|\}}t|dt||S)zp From https://docs.python.org/3/library/itertools.html#recipes s -> (s0,s1), (s1,s2), (s2, s3), ... N)rnextzip)iterableabrrrpairwises  r;NcCst|g|}t|d|iS)ziCollect data into fixed-length chunks or blocks from https://stackoverflow.com/a/16789869/610569 fillvalue)iterr)r8nr<argsrrrgroupersr@Fcs>|r t|n|}|dkr"t|St|dfdd|DS)N)Zn_jobsc3s|]}t|VqdS)N)r)rlinefuncrr sz)parallelize_preprocess..)r mapr)rDiteratorZ processes progress_barrrCrparallelize_preprocesss rI)N)F) itertoolsr ImportErrorrrxml.sax.saxutilsrrZjoblibrrr objectr r%r4r5r;r@rIrrrrs  S#