U &c# @s6ddlZddlmZmZddlmZmZmZddlm Z ddl m Z ddl m Z mZzLddlmZddlmZmZmZdd lmZdd lmZdd lmZd ZWn(eefk reZd dZdZYnXedZ edddZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%Gddde Z&dS)N)ArgumentParser Namespace)AnyListOptional)Pipeline)BaseTransformersCLICommand)SUPPORTED_TASKSpipeline)run)FastAPI HTTPExceptionBody)APIRoute) BaseModel) JSONResponseTcOsdS)N)xyrrA/tmp/pip-unpacked-wheel-ymerj3tt/transformers/commands/serving.pyrsrFztransformers-cli/serving)argscCs:t|j|jr|jnd|j|j|jd}t||j|j|j S)z} Factory function used to instantiate serving server from provided command line arguments. :return: ServeCommand N)taskmodelconfig tokenizerdevice) r rrrrr ServeCommandhostportworkers)rZnlprrrserve_command_factorysr c@seZdZUdZeed<dS)ServeModelInfoResultz" Expose model information infosN)__name__ __module__ __qualname____doc__dict__annotations__rrrrr!-s r!c@s.eZdZUdZeeed<eeeed<dS)ServeTokenizeResultz Tokenize result model tokens tokens_idsN) r#r$r%r&rstrr(rintrrrrr)5s  r)c@seZdZUdZeed<dS)ServeDeTokenizeResultz! DeTokenize result model textN)r#r$r%r&r,r(rrrrr.>s r.c@seZdZUdZeed<dS)ServeForwardResultz Forward result model outputN)r#r$r%r&rr(rrrrr0Fs r0c@seZdZeedddZeeeedddZ ddZ d d Z e d d d e dd d fee dddZe d d d e dd d e d d d feee e dddZe d d d fddZd S)r)parsercCs|jddd}|jdttdd|jdtdd d |jd td d d |jdtddd |jdtdd|jdtdd|jdtdd|jdtddd |jtddS)z Register this command to argparse so it's available for the transformer-cli :param parser: Root parser to register command-specific arguments :return: ZservezFCLI tool to run inference requests through REST and GraphQL endpoints.)helpz--taskzThe task to run the pipeline on)typechoicesr3z--host localhostz$Interface the server will listen on.)r4defaultr3z--porti"z Port the serving will listen to.z --workerszNumber of http workersz--modelz%Model's name or path to stored model.)r4r3z--configz,Model's config name or path to stored model.z --tokenizerzTokenizer name to use.z--devicezSIndicate the device to run onto, -1 indicates CPU, >= 0 indicates GPU (default: -1))funcN) add_parser add_argumentr,r keysr- set_defaultsr )r2Z serve_parserrrrregister_subcommandOs.z ServeCommand.register_subcommand)r rrrc Cs||_||_||_||_ts&tdnptd||t t d|j t t dgdt d|jtt dgdt d|jtt dgdt d |jtt dgdgd d |_dS) NzUsing serve command requires FastAPI and unicorn. Please install transformers with [serving]: pip install "transformers[serving]".Or install FastAPI and unicorn separately.zServing model over {}:{}/GET)Zresponse_modelresponse_classmethodsz /tokenizePOSTz /detokenizez/forwardiX)Zroutestimeout) _pipelinerrr_serve_dependencies_installed RuntimeErrorloggerinfoformatr r model_infor!rtokenizer) detokenizer.forwardr0_app)selfr rrrrrr__init__jsRzServeCommand.__init__cCst|j|j|j|jddS)N)rrr)r rPrrrrQrrrr szServeCommand.runcCstt|jjjdS)N)r")r!varsrFrrrSrrrrLszServeCommand.model_infoNT)embedF) text_input return_idsc Cs|z>|jj|}|r0|jj|}t||dWSt|dWSWn8tk rv}ztddt|ddW5d}~XYnXdS)z Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to tokenize - **return_ids**: Boolean flags indicating if the tokens have to be converted to their integer mapping. )r*r+)r*rerror status_codeZdetailN)rFrrMZconvert_tokens_to_idsr) Exceptionr r,)rQrVrWZ tokens_txtr+errrrMszServeCommand.tokenize)r+skip_special_tokenscleanup_tokenization_spacesc Cs^z |jj|||}td|dWStk rX}ztddt|ddW5d}~XYnXdS)aB Detokenize the provided tokens ids to readable text: - **tokens_ids**: List of tokens ids - **skip_special_tokens**: Flag indicating to not try to decode special tokens - **cleanup_tokenization_spaces**: Flag indicating to remove all leading/trailing spaces and intermediate ones. rY)rr/rXrZr\N)rFrdecoder.r^r r,)rQr+r`raZ decoded_strr_rrrrNs zServeCommand.detokenizec sht|dkrtggdSz||}t|dWStk rb}ztddt|iW5d}~XYnXdS)zV **inputs**: **attention_mask**: **tokens_type_ids**: r)r1Z attention)r1rXr[N)lenr0rFr^r r,)rQinputsr1r_rrrrOs    zServeCommand.forward)r#r$r% staticmethodrr?rr,r-rRr rLrboolrMrrNrOrrrrrNs2&    r)'loggingargparserrtypingrrrZ transformersrZtransformers.commandsrZtransformers.pipelinesr r Zuvicornr Zfastapir r rZfastapi.routingrZpydanticrZstarlette.responsesrrG ImportErrorAttributeErrorobject getLoggerrIr r!r)r.r0rrrrrs.