U &c[-@sddlZddlZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl mZmZddlmZmZmZmZmZmZmZmZddlZddlmZmZdd lmZdd l m!Z!m"Z"dd l#m$Z$m%Z%dd l&m'Z'dd l(m)Z)ddl*m+Z+ddl,m-Z-e$r0ddl.Z/ddl0m1Z1m2Z2m3Z3m4Z4m5Z5e%r\ddl6Z6ddl7m8Z8m9Z9m:Z:m;Z;mZ?dIddZ@GdddeZAGdddeAZBGdddZCGdddeCZDGdddeCZEGdddeCZFGd d!d!eZGGd"d#d#eGZHGd$d%d%eHZIGd&d'd'eHZJGd(d)d)eHZKGd*d+d+eHZLGd,d-d-eHZMeMZNGd.d/d/eAZOGd0d1d1eHZPGd2d3d3eHZQGd4d5d5eHZReIe$re1nde%re8ndd6d6d7dd6d8d9eKe$re2nde%re9ndd:d:d7d:d;d8d9eMe$re4nde%re;nddd>d7dd6d?d@ifd8d9eLe$r`e5nde%rne{sz6DefaultArgumentHandler.handle_args..zBInvalid input type {}. Pipeline supports Union[str, Iterable[str]]) rAr(r)rBallr r from_iterable ValueErrorformattype)r9r0r0r1rDms      z"DefaultArgumentHandler.handle_argscOsDt|dkr t|dkr tdt|dkr6t|St|SdS)Nrz,Pipeline cannot handle mixed args and kwargs)rArLr?rFrDr7r0r0r1r;s   zDefaultArgumentHandler.__call__N) r+r<r=r> staticmethodr r rFr rr)rDr;r0r0r0r1r?_s r?c@seZdZdZdddgZdeeeeeedddZed d Z ee d d d Z e e e e fedddZedeeeeeeedddZdS)PipelineDataFormata Base class for all the pipeline supported data format both for reading and writing. Supported data formats currently includes: - JSON - CSV - stdin/stdout (pipe) PipelineDataFormat also includes some utilities to work with multi-columns like mapping from datasets columns to pipelines keyword arguments through the `dataset_kwarg_1=dataset_column_1` format. jsoncsvpipeF output_path input_pathcolumncCs||_||_|dk r|dndg|_t|jdk|_|jrNdd|jD|_|dk rx|sxtt|jrxtd |j|dk rtt|jstd |jdS)N,rcSs*g|]"}d|krt|dn||fqS)=)tuplesplit)rGcr0r0r1rIsz/PipelineDataFormat.__init__..z{} already exists on diskz{} doesnt exist on disk) rUrVr\rWrAis_multi_columnsrrOSErrorrMr8rUrVrW overwriter0r0r1__init__s zPipelineDataFormat.__init__cCs tdSr4r5r8r0r0r1__iter__szPipelineDataFormat.__iter__datacCs tdS)z Save the provided data object with the representation for the current `DataFormat`. :param data: data to store :return: Nr5r8rfr0r0r1saveszPipelineDataFormat.saverfr@c CsJtj|j\}}tjj|df}t|d}t||W5QRX|S)z Save the provided data object as a pickle-formatted binary data on the disk. :param data: data to store :return: (str) Path where the data has been saved picklezwb+) ospathsplitextrUextsepjoinopenrjdump)r8rfrl_Z binary_pathZf_outputr0r0r1 save_binarys  zPipelineDataFormat.save_binary)rMrUrVrWcCsZ|dkrt||||dS|dkr0t||||dS|dkrHt||||dStd|dS)NrQrarRrSz6Unknown reader {} (Available reader are json/csv/pipe))JsonPipelineDataFormatCsvPipelineDataFormatPipedPipelineDataFormatKeyErrorrM)rMrUrVrWrar0r0r1from_strszPipelineDataFormat.from_strN)F)F)r+r<r=r>ZSUPPORTED_FORMATSr r)rbrrddictrhrr rsrOryr0r0r0r1rPs*   rPcsNeZdZd eeeeeedfdd ZddZeeddd Z Z S) rvFrTcstj||||ddS)Nrtsuperrbr`r*r0r1rbszCsvPipelineDataFormat.__init__c #s^t|jdH}t|}|D]2|jr>fdd|jDVq|jdVqW5QRXdS)Nrcsi|]\}}||qSr0r0rGkr]rowr0r1 sz2CsvPipelineDataFormat.__iter__..r)rprVrR DictReaderr^rW)r8freaderr0rr1rds  zCsvPipelineDataFormat.__iter__rec CsRt|jd<}t|dkrDt|t|d}|||W5QRXdS)Nwr) rprUrArR DictWriterrBkeys writeheader writerows)r8rfrwriterr0r0r1rhs  zCsvPipelineDataFormat.save)F) r+r<r=r r)rbrdr rzrh __classcell__r0r0r}r1rvs rvcsJeZdZd eeeeeedfdd ZddZeddd ZZ S) ruFrTc s:tj||||dt|d}t||_W5QRXdS)Nrtr~)r|rbrprQload_entries)r8rUrVrWrarr}r0r1rbs zJsonPipelineDataFormat.__init__c#s>|jD]2|jr(fdd|jDVq|jdVqdS)Ncsi|]\}}||qSr0r0rentryr0r1rsz3JsonPipelineDataFormat.__iter__..r)rr^rWrcr0rr1rds zJsonPipelineDataFormat.__iter__rec Cs(t|jd}t||W5QRXdS)Nr)rprUrQrq)r8rfrr0r0r1rhszJsonPipelineDataFormat.save)F) r+r<r=r r)rbrdrzrhrr0r0r}r1rusrucsJeZdZdZddZedddZeeeefe dfdd Z Z S) rwz Read data from piped input to the python process. For multi columns data, columns should separated by If columns are provided, then the output will be a dictionary with {column_x: value_x} ccsTtjD]H}d|krH|d}|jr.)sysstdinr\rWzipr[)r8liner0r0r1rds   z PipedPipelineDataFormat.__iter__recCs t|dSr4)printrgr0r0r1rh szPipedPipelineDataFormat.saverics|jdkrtdt|S)NzWhen using piped input on pipeline outputting large object requires an output file path. Please provide such output path through --output argument.)rUrxr|rsrgr}r0r1rs#s  z#PipedPipelineDataFormat.save_binary) r+r<r=r>rdrzrhrr r)rsrr0r0r}r1rwsrwc@s(eZdZdZeddZeddZdS) _ScikitCompatzA Interface layer for the Scikit and Keras compatibility. cCs tdSr4r5r8Xr0r0r1 transform2sz_ScikitCompat.transformcCs tdSr4r5rr0r0r1predict6sz_ScikitCompat.predictN)r+r<r=r>rrrr0r0r0r1r-s  rc @seZdZdZdZdedeeeee e e e e ddd Z d d Zd d ZddZeddZddZddddZddZdddZdS)PipelineaI The Pipeline class is the class from which all pipelines inherit. Refer to this class for methods shared across different pipelines. Base class implementing pipelined operations. Pipeline workflow is defined as a sequence of the following operations: Input -> Tokenization -> Model Inference -> Post-Processing (Task dependent) -> Output Pipeline supports running on CPU or GPU through the device argument. Users can specify device argument as an integer, -1 meaning "CPU", >= 0 referring the CUDA device ordinal. Some pipeline, like for instance FeatureExtractionPipeline ('feature-extraction') outputs large tensor object as nested-lists. In order to avoid dumping such large structure as textual data we provide the binary_output constructor argument. If set to True, the output will be stored in the pickle format. Arguments: model (:obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. tokenizer (:obj:`~transformers.PreTrainedTokenizer`): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from :class:`~transformers.PreTrainedTokenizer`. modelcard (:obj:`str` or :class:`~transformers.ModelCard`, `optional`, defaults to :obj:`None`): Model card attributed to the model for this pipeline. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. args_parser (:class:`~transformers.pipelines.ArgumentHandler`, `optional`, defaults to :obj:`None`): Reference to the object in charge of parsing supplied pipeline parameters. device (:obj:`int`, `optional`, defaults to :obj:`-1`): Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, >=0 will run the model on the associated CUDA device id. binary_output (:obj:`bool`, `optional`, defaults to :obj:`False`): Flag indicating if the output the pipeline should happen in a binary format (i.e. pickle) or as raw text. Return: :obj:`List` or :obj:`Dict`: Pipeline returns list or dictionary depending on: - Whether the user supplied multiple samples - Whether the pipeline exposes multiple fields in the output object NrYFZPreTrainedModelZTFPreTrainedModel)r. tokenizer modelcardr/task args_parserdevice binary_outputc Cs|dkrt}||_||_||_||_|dkr2|nt|dkrBdnd||_||_|p^t |_ |jdkr|jj dkr|j |j|_|jj j} | dk r|| kr|jj | |dS)Nr&rcpuzcuda:{}r'cuda)r2r.rrr/torchrrMrr? _args_parserrNtoconfigtask_specific_paramsupdateget) r8r.rrr/rrrrrr0r0r1rbns *  zPipeline.__init__cCsRtj|s td|dS|j||j||j dk rN|j |dS)zY Save the pipeline's model and tokenizer to the specified save_directory z(Provided path ({}) should be a directoryN) rkrlisdirloggererrorrMr.save_pretrainedrr)r8Zsave_directoryr0r0r1rs    zPipeline.save_pretrainedcCs ||dSzn Scikit / Keras interface to transformers' pipelines. This method will forward to __call__(). )rr0rr0r0r1rszPipeline.transformcCs ||dSrr0rr0r0r1rszPipeline.predictc csb|jdkr>t|jdkrdn d|j dVW5QRXn |jjdkrXtj|jdVdS)a Context Manager allowing tensor allocation on the user-specified device in framework agnostic way. example: # Explicitly ask for tensor allocation on CUDA device :0 nlp = pipeline(..., device=0) with nlp.device_placement(): # Every framework specific tensor allocation will be done on the request device output = nlp(...) Returns: Context manager r&rz/CPU:0z/device:GPU:{}Nr)r/r&rrMrNrrZ set_devicercr0r0r1device_placements " zPipeline.device_placementc sfdd|DS)zm Ensure PyTorch tensors are on the specified device. :param inputs: :return: csi|]\}}||jqSr0)rr)rGnametensorrcr0r1rsz4Pipeline.ensure_tensor_on_device..)items)r8inputsr0rcr1ensure_tensor_on_devicesz Pipeline.ensure_tensor_on_deviceTpad_to_max_lengthcOs&|j||}|jj|d|j|d}|S)z. Parse arguments and tokenize T)Zadd_special_tokensreturn_tensorsr)rrZbatch_encode_plusr/)r8rr9r:rr0r0r1_parse_and_tokenizes zPipeline._parse_and_tokenizecOs|j||}||Sr4)r_forward)r8r9r:rr0r0r1r;s zPipeline.__call__c Cs||Z|jdkr*|j|jddd}n4t&|jf|}|jf|d}W5QRXW5QRX|rp|S|SdS)aH Internal framework specific forward dispatching. Args: inputs: dict holding all the keyworded arguments for required by the model forward method. return_tensors: Whether to return native framework (pt/tf) tensors rather than numpy array. Returns: Numpy array r&F)ZtrainingrN) rr/r.rfrno_gradrrnumpy)r8rr predictionsr0r0r1rs    (zPipeline._forward)NNrYNrF)F)r+r<r=r>default_input_namesrrr rr)r3intboolrbrrrrrrrr;rr0r0r0r1r;s60    rc sLeZdZdZd edeeeeee e edfdd Z fd d Z Z S) FeatureExtractionPipelineao Feature extraction pipeline using Model head. This pipeline extracts the hidden states from the base transformer, which can be used as features in downstream tasks. This feature extraction pipeline can currently be loaded from the :func:`~transformers.pipeline` method using the following task identifier(s): - "feature-extraction", for extracting features of a sequence. All models may be used for this pipeline. See a list of all models, including community-contributed models on `huggingface.co/models `__. Arguments: model (:obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. tokenizer (:obj:`~transformers.PreTrainedTokenizer`): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from :class:`~transformers.PreTrainedTokenizer`. modelcard (:obj:`str` or :class:`~transformers.ModelCard`, `optional`, defaults to :obj:`None`): Model card attributed to the model for this pipeline. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. args_parser (:class:`~transformers.pipelines.ArgumentHandler`, `optional`, defaults to :obj:`None`): Reference to the object in charge of parsing supplied pipeline parameters. device (:obj:`int`, `optional`, defaults to :obj:`-1`): Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, >=0 will run the model on the associated CUDA device id. NrrYrr.rrr/rrrc s tj||||||d|ddSNTr.rrr/rrrrr{)r8r.rrr/rrrr}r0r1rbs z"FeatureExtractionPipeline.__init__cstj||Sr4)r|r;tolistr7r}r0r1r;'sz"FeatureExtractionPipeline.__call__)NNNrrYr+r<r=r>rrr rr)r3rrbr;rr0r0r}r1rs 'rc @s@eZdZdZdZddddddd d d d d g ZddddddZdS)TextGenerationPipelinea Language generation pipeline using any ModelWithLMHead head. This pipeline predicts the words that will follow a specified text prompt. This language generation pipeline can currently be loaded from the :func:`~transformers.pipeline` method using the following task identifier(s): - "text-generation", for generating text from a specified prompt. The models that this pipeline can use are models that have been trained with an autoregressive language modeling objective, which includes the uni-directional models in the library (e.g. gpt2). See the list of available community models on `huggingface.co/models `__. aIn 1991, the remains of Russian Tsar Nicholas II and his family (except for Alexei and Maria) are discovered. The voice of Nicholas's young son, Tsarevich Alexei Nikolaevich, narrates the remainder of the story. 1883 Western Siberia, a young Grigori Rasputin is asked by his father and a group of men to perform magic. Rasputin has a vision and denounces one of the men as a horse thief. Although his father initially slaps him for making such an accusation, Rasputin watches as the man is chased outside and beaten. Twenty years later, Rasputin sees a vision of the Virgin Mary, prompting him to become a priest. Rasputin quickly becomes famous, with people, even a bishop, begging for his blessing. XLNetLMHeadModelTransfoXLLMHeadModelZReformerModelWithLMHeadZGPT2LMHeadModelZOpenAIGPTLMHeadModelZCTRLLMHeadModelZTFXLNetLMHeadModelZTFTransfoXLLMHeadModelZTFGPT2LMHeadModelZTFOpenAIGPTLMHeadModelZTFCTRLLMHeadModelFTr return_textclean_up_tokenization_spacesc Os|jjj|jkr(td|jjj|j|j|}g}|D]b}||jjjdkrn|j|j |dd} n|j|dd} | dj ddkrd| d<d| d<|j d kr| ddk r|j f| } | d} | dks| j dd kst d |jjfd| i|} W5QRXg} | D]} | } i}|r0| |d <|r|jj| d |d}| dkrXd}nt|jj| dd |d}|||d|d<| |q|| g7}q:t|d kr|dS|S)Nz[Generation is currently not supported for {}. Please select a model from {} for generation.)rrFr input_idsrrattention_maskr'rz~Batch generation is currently not supported. See https://github.com/huggingface/transformers/issues/3021 for more information.Zgenerated_token_idsTZskip_special_tokensrZgenerated_text)r.r*r+ALLOWED_MODELSr6rMrrr PADDING_TEXTshaper/rAssertionErrorgeneraterrrdecoderAappend)r8rrrr9generate_kwargsZ text_inputsresultsZ prompt_textrrZoutput_sequencesresultZgenerated_sequencerecordtextZ prompt_lengthr0r0r1r;Wsl         zTextGenerationPipeline.__call__N)r+r<r=r>rrr;r0r0r0r1r+s$ rcs eZdZdZfddZZS)TextClassificationPipelinea Text classification pipeline using ModelForSequenceClassification head. See the `sequence classification usage <../usage.html#sequence-classification>`__ examples for more information. This text classification pipeline can currently be loaded from the :func:`~transformers.pipeline` method using the following task identifier(s): - "sentiment-analysis", for classifying sequences according to positive or negative sentiments. The models that this pipeline can use are models that have been fine-tuned on a sequence classification task. See the up-to-date list of available models on `huggingface.co/models `__. Arguments: model (:obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. tokenizer (:obj:`~transformers.PreTrainedTokenizer`): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from :class:`~transformers.PreTrainedTokenizer`. modelcard (:obj:`str` or :class:`~transformers.ModelCard`, `optional`, defaults to :obj:`None`): Model card attributed to the model for this pipeline. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. args_parser (:class:`~transformers.pipelines.ArgumentHandler`, `optional`, defaults to :obj:`None`): Reference to the object in charge of parsing supplied pipeline parameters. device (:obj:`int`, `optional`, defaults to :obj:`-1`): Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, >=0 will run the model on the associated CUDA device id. cs>tj||}t|t|jddd}fdd|DS)NrTZkeepdimscs,g|]$}jjj||dqS))labelscore)r.rid2labelargmaxmaxitem)rGrrcr0r1rIsz7TextClassificationPipeline.__call__..)r|r;npexpsum)r8r9r:outputsscoresr}rcr1r;sz#TextClassificationPipeline.__call__)r+r<r=r>r;rr0r0r}r1rs$rc sHeZdZdZd edeeeeee e edfdd Z d d Z Z S) FillMaskPipelinea Masked language modeling prediction pipeline using ModelWithLMHead head. See the `masked language modeling usage <../usage.html#masked-language-modeling>`__ examples for more information. This mask filling pipeline can currently be loaded from the :func:`~transformers.pipeline` method using the following task identifier(s): - "fill-mask", for predicting masked tokens in a sequence. The models that this pipeline can use are models that have been trained with a masked language modeling objective, which includes the bi-directional models in the library. See the up-to-date list of available models on `huggingface.co/models `__. Arguments: model (:obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. tokenizer (:obj:`~transformers.PreTrainedTokenizer`): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from :class:`~transformers.PreTrainedTokenizer`. modelcard (:obj:`str` or :class:`~transformers.ModelCard`, `optional`, defaults to :obj:`None`): Model card attributed to the model for this pipeline. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. args_parser (:class:`~transformers.pipelines.ArgumentHandler`, `optional`, defaults to :obj:`None`): Reference to the object in charge of parsing supplied pipeline parameters. device (:obj:`int`, `optional`, defaults to :obj:`-1`): Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, >=0 will run the model on the associated CUDA device id. NrrYrrc s&tj||||||d|d||_dSr)r|rbtopk) r8r.rrr/rrrrr}r0r1rbs  zFillMaskPipeline.__init__cOs|j||}|j|dd}g}|jdkr2|jdn|d}t|D]*}|d|}g} |jdkrt||jj k } ||| ddf} tj | } tjj| |jd} | j | j }}nB||jj k } ||| ddf} | j dd} | |j\}}t||D]J\}}| }||| <|t||jjk}| |j|||dq|| g7}qDt|d kr|dS|S) NT)rr&rrr)Zdim)sequencertokenr)rrr/rsizeranger&whererZ mask_token_idrrnnZsoftmaxmathZtop_krrCindicesZnonzerorrr pad_token_idrrrA)r8r9r:rrrZ batch_sizeirrZ masked_indexZlogitsZprobsrrCrvptokensr0r0r1r; s4      zFillMaskPipeline.__call__)NNNrrrYrr0r0r}r1rs")rc s^eZdZdZdZddddddgdfedeeeee e e e e d fd d Z d d ZZS) NerPipelinea Named Entity Recognition pipeline using ModelForTokenClassification head. See the `named entity recognition usage <../usage.html#named-entity-recognition>`__ examples for more information. This token recognition pipeline can currently be loaded from the :func:`~transformers.pipeline` method using the following task identifier(s): - "ner", for predicting the classes of tokens in a sequence: person, organisation, location or miscellaneous. The models that this pipeline can use are models that have been fine-tuned on a token classification task. See the up-to-date list of available models on `huggingface.co/models `__. Arguments: model (:obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. tokenizer (:obj:`~transformers.PreTrainedTokenizer`): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from :class:`~transformers.PreTrainedTokenizer`. modelcard (:obj:`str` or :class:`~transformers.ModelCard`, `optional`, defaults to :obj:`None`): Model card attributed to the model for this pipeline. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. args_parser (:class:`~transformers.pipelines.ArgumentHandler`, `optional`, defaults to :obj:`None`): Reference to the object in charge of parsing supplied pipeline parameters. device (:obj:`int`, `optional`, defaults to :obj:`-1`): Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, >=0 will run the model on the associated CUDA device id. sequencesNrFOrYrrc s2tj|||||||| dtdd|_||_dS)NrF)Z do_lower_case)r|rbrZ_basic_tokenizer ignore_labels) r8r.rrr/rrrrrr}r0r1rbXs  zNerPipeline.__init__c Os|j||}g}|D]P}||jj|d|j|jjd}|jdkrr||jdd}|dd}nPt B|j f|}|jf|dd }|d d}W5QRXW5QRXt |t |jddd} | jdd } g} t| D]X\} } |jjj| |jkr| |jt|| | | | |jjj| d g7} q|| g7}qt|d kr~|dS|S) NF)Zreturn_attention_maskr max_lengthr&rrrTr)Zaxis)wordrentityr)rrrZ encode_plusr/max_lenr.rfrrrrrrrrr enumeraterrrZconvert_ids_to_tokensrrrA)r8r9r:ranswersZsentencerentitiesrrZ labels_idxansweridxZ label_idxr0r0r1r;rs@      (   zNerPipeline.__call__)r+r<r=r>rrrr rr)r3rrrbr;rr0r0r}r1r1s($rc@seZdZdZddZdS) QuestionAnsweringArgumentHandlera6 QuestionAnsweringPipeline requires the user to provide multiple arguments (i.e. question & context) to be mapped to internal SquadExample / SquadFeature structures. QuestionAnsweringArgumentHandler manages all the possible to create SquadExample from the command-line supplied arguments. cs|dk r:t|dkr:t|dkr.|d|d<n t||d<d|ksJd|krd|krZ|dn|d}t|trt|g}ndd|D}t|D]f\}ttrtfddd Drtd tjf||<qtt st d d|krdndqn|d |krbd |krbt|d t r&|d g|d <t|d t rD|d g|d <ddt |d |d D}nt d |t|ts|g}|S)NrrrrfcSsg|]}|qSr0r0)rGrr0r0r1rIsz=QuestionAnsweringArgumentHandler.__call__..c3s|]}|kVqdSr4r0rGrrr0r1 sz.questioncontextzFYou need to provide a dictionary with keys {question:..., context:...}zO{} argument needs to be of type (list[SquadExample | dict], SquadExample, dict)rrcSsg|]\}}t||qSr0)QuestionAnsweringPipeline create_samplerGqr]r0r0r1rIszUnknown arguments {})rArBr(rzranyrxrrrrLrMr)r)r8r9r:rrr0rr1r;s@      z)QuestionAnsweringArgumentHandler.__call__Nr+r<r=r>r;r0r0r0r1r sr cseZdZdZdZdedeeeee e e dfdd Z e ee e e fee e e feee efd d d Zd dZejeje e edddZe e e dddZZS)ra Question Answering pipeline using ModelForQuestionAnswering head. See the `question answering usage <../usage.html#question-answering>`__ examples for more information. This question answering can currently be loaded from the :func:`~transformers.pipeline` method using the following task identifier(s): - "question-answering", for answering questions given a context. The models that this pipeline can use are models that have been fine-tuned on a question answering task. See the up-to-date list of available models on `huggingface.co/models `__. Arguments: model (:obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. tokenizer (:obj:`~transformers.PreTrainedTokenizer`): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from :class:`~transformers.PreTrainedTokenizer`. modelcard (:obj:`str` or :class:`~transformers.ModelCard`, `optional`, defaults to :obj:`None`): Model card attributed to the model for this pipeline. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. args_parser (:class:`~transformers.pipelines.ArgumentHandler`, `optional`, defaults to :obj:`None`): Reference to the object in charge of parsing supplied pipeline parameters. device (:obj:`int`, `optional`, defaults to :obj:`-1`): Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, >=0 will run the model on the associated CUDA device id. zquestion,contextNrrYr)r.rrr/rrc s(tjf||||t||d|dS)Nr)r|rbr )r8r.rrr/rrr:r}r0r1rbs z"QuestionAnsweringPipeline.__init__)rrr@cCs4t|trddt||DStd||dddSdS)a9 QuestionAnsweringPipeline leverages the SquadExample/SquadFeatures internally. This helper method encapsulate all the logic for converting question(s) and context(s) to SquadExample(s). We currently support extractive question answering. Arguments: question: (str, List[str]) The question to be ask for the associated context context: (str, List[str]) The context in which we will look for the answer. Returns: SquadExample initialized with the corresponding question and context. c Ss"g|]\}}td||dddqSr4)rrr0r0r1rI-sz;QuestionAnsweringPipeline.create_sample..N)r(rBrrrr0r0r1rs z'QuestionAnsweringPipeline.create_samplec sddddddddd d d d ddkrftd dddkrtddj|}fdd|D}g}t||D] \jjdg}fdd|D}jdkr&dd| D} |\}} | | }} nTt Ffdd| D}j f|\}} | |  }} W5QRXW5QRXd} g} t|| D]\} } t| tt| } t| tt| } | ttjd| ttjd} } d r0t| | d| d} d| d<| d<| | dd\}}}tj| fddt|||D7} qd r| | ddddt| ddddd d} || 7}qt|dkr|dS|S)!a7 Args: We support multiple use-cases, the following are exclusive: X: sequence of SquadExample data: sequence of SquadExample question: (str, List[str]), batch of question(s) to map along with context context: (str, List[str]), batch of context(s) associated with the provided question keyword argument Returns: dict: {'answer': str, 'score": float, 'start": int, "end": int} answer: the textual answer in the intial context score: the score the current answer scored for the model start: the character index in the original string corresponding to the beginning of the answer' span end: the character index in the original string corresponding to the ending of the answer' span rr doc_stridemax_answer_len max_seq_lenimax_question_len@Zhandle_impossible_answerFz&topk parameter should be >= 1 (got {})z0max_answer_len parameter should be >= 1 (got {})c s2g|]*}t|gjddddddqS)rrrF)Z tqdm_enabled)rr)rGexample)r:r8r0r1rIPs z6QuestionAnsweringPipeline.__call__..rcs i|]fddDqS)csg|]}|jqSr0)__dict__)rGfeaturerr0r1rI_szAQuestionAnsweringPipeline.__call__...r0)rG)featuresrr1r_sz6QuestionAnsweringPipeline.__call__..r&cSsi|]\}}|t|qSr0)r&ZconstantrGrrr0r0r1rdscs"i|]\}}|tj|jdqS))r)rrrr$rcr0r1rjsi@Brc s|g|]t\}}}|tj|kddtj|kdddjj|j|ddqS)rr rrstartendr )rrrZtoken_to_orig_maproZ doc_tokens)rGser) char_to_wordr r"r0r1rIs rYr&cSs|dS)Nrr0)xr0r0r1z4QuestionAnsweringPipeline.__call__..T)keyreverseN) setdefaultrLrMrrrmodel_input_namesrr/rr.rrrrrrrabsarrayZp_maskminrrZchar_to_word_offsetrsortedrA)r8r9r:ZexamplesZ features_listZ all_answersr2Zfw_argsr'r(Zmin_null_scorerZstart_end_ZstartsZendsrr0)r+r r"r#r:r8r1r;1sd              .    z"QuestionAnsweringPipeline.__call__)r'r(rrr@c Cs|jdkr|d}|jdkr$|d}tt|dt|d}tt||d}|}|dkrtt|g}nDt||krt | }n*t | |d|} | t ||  }t ||j dd\}}|||d||ffS)a Take the output of any QuestionAnswering head and will generate probalities for each span to be the actual answer. In addition, it filters out some unwanted/impossible cases like answer len being greater than max_answer_len or answer end position being before the starting position. The method supports output the k-best answer through the topk argument. Args: start: numpy array, holding individual start probabilities for each token end: numpy array, holding individual end probabilities for each token topk: int, indicates how many possible answer span(s) to extract from the model's output max_answer_len: int, maximum size of the answer to extract from the model's output rNrr) ndimrmatmulZ expand_dimsZtrilZtriuflattenrrAZargsortZ argpartitionZ unravel_indexr) r8r'r(rrouter candidatesZ scores_flatZidx_sortr r0r0r1rs   z QuestionAnsweringPipeline.decode)rr'r(c Csg}d}}}}t|dD]~\} } |j| } ||krJ|krxnn*||krZ|}||krn|t| }|| g7}||krq|t| 7}|t| d7}q"d|td|tt||dS)a When decoding from token probalities, this method maps token indexes to actual word in the initial context. Args: text: str, the actual context to extract the answer from start: int, starting answer token index end: int, ending answer token index Returns: dict: {'answer': str, 'start': int, 'end': int} rr%r)r r'r()rr\rtokenizerArorr5) r8rr'r(wordsZ token_idxZchar_start_idxZ char_end_idxZ chars_idxrrrr0r0r1span_to_answers$      z(QuestionAnsweringPipeline.span_to_answer)NNrrY)r+r<r=r>rrrr rr)rrbrOr rrr;rZndarrayrrr?rr0r0r}r1rs,$i(rc@s"eZdZdZddddddZdS)SummarizationPipelinea$ Summarize news articles and other documents Usage:: # use bart in pytorch summarizer = pipeline("summarization") summarizer("Sam Shleifer writes the best docstring examples in the whole world.", min_length=5, max_length=20) # use t5 in tf summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf") summarizer("Sam Shleifer writes the best docstring examples in the whole world.", min_length=5, max_length=20) The models that this pipeline can use are models that have been fine-tuned on a summarization task, which is currently, '`bart-large-cnn`', '`t5-small`', '`t5-base`', '`t5-large`', '`t5-3b`', '`t5-11b`'. See the up-to-date list of available models on `huggingface.co/models `__. Arguments: model (:obj:`str` or :obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`, `optional`, defaults to :obj:`None`): The model that will be used by the pipeline to make predictions. This can be :obj:`None`, a string checkpoint identifier or an actual pre-trained model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. If :obj:`None`, the default of the pipeline will be loaded. tokenizer (:obj:`str` or :obj:`~transformers.PreTrainedTokenizer`, `optional`, defaults to :obj:`None`): The tokenizer that will be used by the pipeline to encode data for the model. This can be :obj:`None`, a string checkpoint identifier or an actual pre-trained tokenizer inheriting from :class:`~transformers.PreTrainedTokenizer`. If :obj:`None`, the default of the pipeline will be loaded. modelcard (:obj:`str` or :class:`~transformers.ModelCard`, `optional`, defaults to :obj:`None`): Model card attributed to the model for this pipeline. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. args_parser (:class:`~transformers.pipelines.ArgumentHandler`, `optional`, defaults to :obj:`None`): Reference to the object in charge of parsing supplied pipeline parameters. device (:obj:`int`, `optional`, defaults to :obj:`-1`): Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, >=0 will run the model on the associated CUDA device id. FTrc s|s|stdt|dks$td|jdkrDd|jjjkrDtd|jjjdk r\|jjjndt |dt r|j j dk std fd d |dDf}d }n4t |dt rˆ|df}d }ntd|d|4|j|d|i}|jdkr|jf|}|djd}n"|jdkr:t|dd}|d|jjj} || dkrltd| ||d|jjj} || krtd| ||jj|dfd|di|} g} | D]>} i}|r| |d<|r|j j| d |d|d<| |q| W5QRSQRXdS)a Args: *documents: (list of strings) articles to be summarized return_text: (bool, default=True) whether to add a decoded "summary_text" to each result return_tensors: (bool, default=False) whether to return the raw "summary_token_ids" to each result clean_up_tokenization_spaces: (`optional`) bool whether to include extra spaces in the output **generate_kwargs: extra kwargs passed to `self.model.generate`_ Returns: list of dicts with 'summary_text' and/or 'summary_token_ids' for each document_to_summarize .. _`self.model.generate`: https://huggingface.co/transformers/model_doc/bart.html#transformers.BartForConditionalGeneration.generate 8You must specify return_tensors=True or return_text=Truerz&Please provide a document to summarizer&ZBartForConditionalGenerationzRTensorflow is not yet supported for Bart. Please consider using T5, e.g. `t5-base`NrYOPlease make sure that the tokenizer has a pad_token_id when using a batch inputcsg|] }|qSr0r0)rGdocumentprefixr0r1rIAsz2SummarizationPipeline.__call__..TF\ `documents[0]`: {} have the wrong format. The should be either of type `str` or type `list`rr'rr min_lengthzYour min_length is set to {}, but you input_length is only {}. You might consider decreasing min_length manually, e.g. summarizer('...', min_length=10)rzYour max_length is set to {}, but you input_length is only {}. You might consider decreasing max_length manually, e.g. summarizer('...', max_length=50)rZsummary_token_idsrZ summary_text)rrAr/r.r*r+r6rrEr(rBrrr)rLrMrrrrr&rrrGrwarningrrrr)r8rrrZ documentsrrr input_lengthrGrZ summariesrsummaryrr0rDr1r;s       zSummarizationPipeline.__call__Nrr0r0r0r1r@s 0r@c@s"eZdZdZddddddZdS)TranslationPipelinea Translates from one language to another. Usage:: en_fr_translator = pipeline("translation_en_to_fr") en_fr_translator("How old are you?") The models that this pipeline can use are models that have been fine-tuned on a translation task, currently: "t5-small", "t5-base", "t5-large", "t5-3b", "t5-11b" See the up-to-date list of available models on `huggingface.co/models `__. Arguments: model (:obj:`str` or :obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`, `optional`, defaults to :obj:`None`): The model that will be used by the pipeline to make predictions. This can be :obj:`None`, a string checkpoint identifier or an actual pre-trained model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. If :obj:`None`, the default of the pipeline will be loaded. tokenizer (:obj:`str` or :obj:`~transformers.PreTrainedTokenizer`, `optional`, defaults to :obj:`None`): The tokenizer that will be used by the pipeline to encode data for the model. This can be :obj:`None`, a string checkpoint identifier or an actual pre-trained tokenizer inheriting from :class:`~transformers.PreTrainedTokenizer`. If :obj:`None`, the default of the pipeline will be loaded. modelcard (:obj:`str` or :class:`~transformers.ModelCard`, `optional`, defaults to :obj:`None`): Model card attributed to the model for this pipeline. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. args_parser (:class:`~transformers.pipelines.ArgumentHandler`, `optional`, defaults to :obj:`None`): Reference to the object in charge of parsing supplied pipeline parameters. device (:obj:`int`, `optional`, defaults to :obj:`-1`): Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, >=0 will run the model on the associated CUDA device id. FTrc s|s|std|jjjdk r(|jjjndt|dtrl|jjdk sNtdfdd|dDf}d}n4t|dtr|df}d }nt d |d| |j |d |i}|j d kr|jf|}|d jd}n"|j dkrt|d d}|d|jjj} |d| kr6td || |jj|d fd|di|} g} | D]>} i} |rt| | d<|r|jj| d|d| d<| | q^| W5QRSQRXdS)a Args: *args: (list of strings) texts to be translated return_text: (bool, default=True) whether to add a decoded "translation_text" to each result return_tensors: (bool, default=False) whether to return the raw "translation_token_ids" to each result **generate_kwargs: extra kwargs passed to `self.model.generate`_ Returns: list of dicts with 'translation_text' and/or 'translation_token_ids' for each text_to_translate .. _`self.model.generate`: https://huggingface.co/transformers/model_doc/bart.html#transformers.BartForConditionalGeneration.generate rANrYrrBcsg|] }|qSr0r0)rGrrDr0r1rIsz0TranslationPipeline.__call__..TFrFrr'rrr&rg?zYour input_length: {} is bigger than 0.9 * max_length: {}. You might consider increasing your max_length manually, e.g. translator('...', max_length=400)rZtranslation_token_idsrZtranslation_text)rr.rrEr(rBrrr)rLrMrrr/rrr&rrrrrIrrr)r8rrrr9rrrrJrZ translationsr translationrr0rDr1r;sh      zTranslationPipeline.__call__Nrr0r0r0r1rLxs 'rLzdistilbert-base-cased)r'r&r.rr)implr&r'defaultz/distilbert-base-uncased-finetuned-sst-2-englishzdistilbert-base-uncasedz0dbmdz/bert-large-cased-finetuned-conll03-englishzbert-large-casedz%distilbert-base-cased-distilled-squadZuse_fastFzdistilroberta-basezbart-large-cnnzt5-baseZgpt2) zfeature-extractionzsentiment-analysisZnerzquestion-answeringz fill-maskZ summarizationZtranslation_en_to_frZtranslation_en_to_deZtranslation_en_to_roztext-generation)rr.rrr/r@c  s|tkr td|tt|p*t|}t|d|}}|dkrnfdddD\}}}||}|dkrt|tr|tkr|}n t|tr|tkr|}nt dd} t|tr|} nt|tr|} t|tt frt|t rt j |df|d }n t |}t|tr$t |}t| tr:t | } t|tri} |d krt|d rtd | d <tdn(|dkr|drd | d<td|j |fd|i| }|f||| ||d|S)aj Utility factory method to build a pipeline. Pipeline are made of: - A Tokenizer instance in charge of mapping raw textual input to token - A Model instance - Some (optional) post processing for enhancing model's output Args: task (:obj:`str`): The task defining which pipeline will be returned. Currently accepted tasks are: - "feature-extraction": will return a :class:`~transformers.FeatureExtractionPipeline` - "sentiment-analysis": will return a :class:`~transformers.TextClassificationPipeline` - "ner": will return a :class:`~transformers.NerPipeline` - "question-answering": will return a :class:`~transformers.QuestionAnsweringPipeline` - "fill-mask": will return a :class:`~transformers.FillMaskPipeline` - "summarization": will return a :class:`~transformers.SummarizationPipeline` - "translation_xx_to_yy": will return a :class:`~transformers.TranslationPipeline` model (:obj:`str` or :obj:`~transformers.PreTrainedModel` or :obj:`~transformers.TFPreTrainedModel`, `optional`, defaults to :obj:`None`): The model that will be used by the pipeline to make predictions. This can be :obj:`None`, a model identifier or an actual pre-trained model inheriting from :class:`~transformers.PreTrainedModel` for PyTorch and :class:`~transformers.TFPreTrainedModel` for TensorFlow. If :obj:`None`, the default for this pipeline will be loaded. config (:obj:`str` or :obj:`~transformers.PretrainedConfig`, `optional`, defaults to :obj:`None`): The configuration that will be used by the pipeline to instantiate the model. This can be :obj:`None`, a model identifier or an actual pre-trained model configuration inheriting from :class:`~transformers.PretrainedConfig`. If :obj:`None`, the default for this pipeline will be loaded. tokenizer (:obj:`str` or :obj:`~transformers.PreTrainedTokenizer`, `optional`, defaults to :obj:`None`): The tokenizer that will be used by the pipeline to encode data for the model. This can be :obj:`None`, a model identifier or an actual pre-trained tokenizer inheriting from :class:`~transformers.PreTrainedTokenizer`. If :obj:`None`, the default for this pipeline will be loaded. framework (:obj:`str`, `optional`, defaults to :obj:`None`): The framework to use, either "pt" for PyTorch or "tf" for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to PyTorch. Returns: :class:`~transformers.Pipeline`: Class inheriting from :class:`~transformers.Pipeline`, according to the task. Examples:: from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer # Sentiment analysis pipeline pipeline('sentiment-analysis') # Question answering pipeline, specifying the checkpoint identifier pipeline('question-answering', model='distilbert-base-cased-distilled-squad', tokenizer='bert-base-cased') # Named entity recognition pipeline, passing in a specific model and tokenizer model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english") tokenizer = AutoTokenizer.from_pretrained("bert-base-cased") pipeline('ner', model=model, tokenizer=tokenizer) z'Unknown task {}, available tasks are {}rONcsg|]}d|qS)rPr0r Z targeted_taskr0r1rIszpipeline..rNzImpossible to guess which tokenizer to use. Please provided a PretrainedTokenizer class or a path/identifier to a pretrained tokenizer.rrr'z.h5TZfrom_tfz}Model might be a TensorFlow model (ending with `.h5`) but TensorFlow is not available. Trying to load the model with PyTorch.r&z.binZfrom_ptz{Model might be a PyTorch model (ending with `.bin`) but PyTorch is not available. Trying to load the model with Tensorflow.r)r.rrr/r)SUPPORTED_TASKSrxrMrBrr2r(r)r Exceptionr[rZfrom_pretrainedrrendswithrrI) rr.rrr/r:Z task_classZ model_classmodelsrZ model_kwargsr0rQr1pipelineSsVK          rV)N)NNNN)VrRrQloggingrkrjrabcrr contextlibr itertoolsros.pathrrtypingrr r r r r rrrrZconfiguration_autorrZconfiguration_utilsrrfrrZ file_utilsrrrrZtokenization_autorZtokenization_bertrZtokenization_utilsrZ tensorflowr&Zmodeling_tf_autorrrrr rZ modeling_autor!r"r#r$r% getLoggerr+rr2r3r?rPrvrurwrrrrrrrZTokenClassificationPipeliner rr@rLrRr)rVr0r0r0r1s  (        3J%4=w+do> s             l