U <c' @s,UddlZddlZddlmZmZmZmZmZmZee e e dddZ e e e eee ee fdddZ ee ee ee d d d Z e e d d dZe e d ddZe eee e fee e fee fdddZee eee e fee e fee fdddZd6e e ee dddZe e d ddZd7ee ee fee ee e ee e fe ee ddd Zd!Ze ed"<d#d$hZee ed%<eZee ed&<d'd'd(Zee e fed)<d*Ze ed+<d#d$hZee ed,<eZee ed-<d.d/iZee e fed0<dd1d2d3Ze d4kr(e!d5edS)8N)AnyDictListSetTupleUnion)lines indentationreturncCsFd}dd|}t|D](\}}|dkr0||7}||d|7}q|S)N  r) enumeratereplace)rr outputZnew_line_with_indentilinerF/tmp/pip-unpacked-wheel-gikjz4vx/torch/utils/data/datapipes/gen_pyi.pymaterialize_liness rdir template_name output_name replacementsc Cstj||}tj||}t|d}|}W5QRX|D]<\}} } t|d"}||t| | }||W5QRXq>dS)Nrw)ospathjoinopenreadrrwrite) rrrrZ template_pathZ output_pathfcontent placeholderrr rrrgen_from_templates  r&) dir_pathsfiles_to_excluder csVt}|D]Ft}dd|D}fdd|D}|fdd|Dq |S)z When given a path to a directory, returns the paths to the relevant files within it. This function does NOT recursive traverse to subdirectories. cSs h|]}d|ddkr|qS)z.pyNr.0fnamerrr %sz"find_file_paths..csh|]}|kr|qSrrr*r(rrr-&scsh|]}tj|qSrrrrr*)dir_pathrrr-'s)setrlistdirupdate)r'r(paths all_filesZ python_filesZ filter_filesr)r0r(rfind_file_pathss r6)rr cCs\d|krd\}}n d|kr$d\}}ntd|||t|||}}|||S)zb Extracts method name from decorator in the form of "@functional_datapipe({method_name})" (")r7z")(')r8z')z4Unable to find appropriate method name within line: ) RuntimeErrorfindlenrZ start_tokenZ end_tokenstartendrrrextract_method_name+s  r?cCs2d}d}||t|||}}|||S)z` Extracts class name from class definition in the form of "class {CLASS_NAME}({Type}):" class ()r:r;r<rrrextract_class_name9srB) file_pathr c Cs~iit}}}t|R}d}d\}}}d} |D].} | dddkrV| } | s8d| krdq8d| krvt| }q8|rd| krt| }q8|rd | ksd | krd | kr|||d7}| d td } | | d } |dkr8|| d 7}|| d 8}|dkr@| d } || d | 7}t |||<|||<d\}}}q8|dkrTt dq8|| d d7}q8W5QRX|||fS)zp Given a path to file, parses the file and returns a dictionary of method names to function signatures. r)r r r Fz"""z@functional_datapiper@z def __init__(z def __new__(rAN)z7open parenthesis count < 0. This shouldn't be possible.r r ) r1r readlinescountr?rBaddr:r;rfindprocess_signaturer9strip) rCmethod_to_signaturemethod_to_class_nameZspecial_output_typer#Zopen_paren_count method_name class_name signatureskiprr=r>rrrparse_datapipe_fileCsF             rS) file_pathsr cCsRiit}}}|D]0}t|\}}}||||||q|||fS)N)r1rSr3)rTmethods_and_signaturesmethods_and_class_namesZ!methods_with_special_output_typesrrMrNZ$methods_needing_special_output_typesrrrparse_datapipe_filesms   rW,)r delimiterr cCspd}d}g}|D]P}|dkr&|d7}n2|dkr8|d8}n ||krX|dkrX||d}q||7}q|||S)z\ Given a line of text, split it on comma unless the comma is within a bracket '[]'. rr [rE])append)rrYZ bracket_countZ curr_tokenrescharrrrsplit_outside_bracketws     r_cCst|}t|D]\}}|d||<|dkr8d||<q|dkrjd||dkrj||ddkrjd||<qd|kr|d d \}}|dd ||<qd d |D}d|}|S)z Given a raw function signature, clean it up by removing the self-referential datapipe argument, default arguments of input functions, newlines, and spaces. r clsselfrrE*r z Callable ==rDz= ...cSsg|]}|dkr|qS)r r)r+trrr sz%process_signature..z, )r_rrLrsplitr)rtokensrtokenheadZ default_argrrrrKs (  rKr )rCr(deprecated_filesdefault_output_typemethod_to_special_output_typerootr c sdkrtttjt|tr,|gn|}fdd|D}t|||d}t |\}}} |D]} | | krf| | qfg} | D]J\} } || }| | kr|| }n|}| d|d| d| d|d q| j d d d | S) a  .pyi generation for functional DataPipes Process # 1. Find files that we want to process (exclude the ones who don't) # 2. Parse method name and signature # 3. Remove first argument after self (unless it is "*datapipes"), default args, and spaces r csg|]}tj|qSrr/)r+rrmrrresz*get_method_definitions..r.z# Functional form of 'z' def rAz) -> z: ...cSs|ddS)Nr rE)split)srrrz(get_method_definitions..)key)strpathlibPath__file__parentresolve isinstancer6unionrWrIitemsr\sort)rCr(rjrkrlrmrTrUrVZmethods_w_special_output_typesfn_nameZmethod_definitionsrO argumentsrP output_typerrnrget_method_definitionss*   &riteriterDP_file_pathz __init__.pyzutils.pyiterDP_files_to_excludeiterDP_deprecated_fileszList[IterDataPipe])Zdemuxfork$iterDP_method_to_special_output_typemapmapDP_file_pathmapDP_files_to_excludemapDP_deprecated_filesshuffle IterDataPipe#mapDP_method_to_special_output_type)r cCs\ttttdt}ttttdt}t t j }d|dfd|dfg}tt|dd|dd S) a # Inject file into template datapipe.pyi.in TODO: The current implementation of this script only generates interfaces for built-in methods. To generate interface for user-defined DataPipes, consider changing `IterDataPipe.register_datapipe_as_function`. rZ MapDataPipez${IterDataPipeMethods}z${MapDataPipeMethods}zdatapipe.pyi.inz datapipe.pyirN)rrrrrrrrrrurvrwrxryr&rt)Ziter_method_definitionsZmap_method_definitionsrrrrrmains"r__main__z2Generating Python interface file 'datapipe.pyi'...)rX)r )"rrutypingrrrrrrrtintrr&r6r?rBrSrWr_rKrr__annotations__rr1rrrrrrr__name__printrrrrsB  "  .*2   (