U &c5@sddlZddlZddlZddlZddlmZddlmZmZmZddl m Z m Z e e ZeGdddZedd Gd d d ZGd d d ZGdddeZdS)N) dataclass)ListOptionalUnion)is_tf_availableis_torch_availablec@sJeZdZUdZeed<eed<dZeeed<dZeeed<ddZ dS) InputExamplea5 A single training/test example for simple sequence classification. Args: guid: Unique id for the example. text_a: string. The untokenized text of the first sequence. For single sequence tasks, only this sequence must be specified. text_b: (Optional) string. The untokenized text of the second sequence. Only must be specified for sequence pair tasks. label: (Optional) string. The label of the example. This should be specified for train and dev examples, but not for test examples. guidtext_aNtext_blabelcCstjt|dddS)*Serializes this instance to a JSON string.)indent jsondumps dataclassesZasdictselfrF/tmp/pip-unpacked-wheel-ymerj3tt/transformers/data/processors/utils.pyto_json_string2szInputExample.to_json_string) __name__ __module__ __qualname____doc__str__annotations__r rr rrrrrr s  r T)frozenc@sfeZdZUdZeeed<dZeeeed<dZ eeeed<dZ ee ee fed<ddZ dS) InputFeaturesa A single set of features of data. Property names are the same names as the corresponding inputs to a model. Args: input_ids: Indices of input sequence tokens in the vocabulary. attention_mask: Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: Usually ``1`` for tokens that are NOT MASKED, ``0`` for MASKED (padded) tokens. token_type_ids: (Optional) Segment token indices to indicate first and second portions of the inputs. Only some models use them. label: (Optional) Label corresponding to the input. Int for classification problems, float for regression problems. input_idsNattention_masktoken_type_idsr cCstt|dS)rrrrrrrrMszInputFeatures.to_json_string)rrrrrintr r$rr%r rfloatrrrrrr"7s  r"c@sFeZdZdZddZddZddZdd Zd d Ze dd dZ d S) DataProcessorzEBase class for data converters for sequence classification data sets.cCs tdS)zGets an example from a dict with tensorflow tensors Args: tensor_dict: Keys and values should match the corresponding Glue tensorflow_dataset examples. NNotImplementedError)rZ tensor_dictrrrget_example_from_tensor_dictUsz*DataProcessor.get_example_from_tensor_dictcCs tdS)z7Gets a collection of `InputExample`s for the train set.Nr)rdata_dirrrrget_train_examples]sz DataProcessor.get_train_examplescCs tdS)z5Gets a collection of `InputExample`s for the dev set.Nr)r,rrrget_dev_examplesaszDataProcessor.get_dev_examplescCs tdS)z*Gets the list of labels for this data set.Nr)rrrr get_labelseszDataProcessor.get_labelscCs(t|dkr$|t|j|_|S)zSome tensorflow_datasets datasets are not formatted the same way the GLUE datasets are. This method converts examples to the correct format.)lenr0r&r )rexamplerrrtfds_mapiszDataProcessor.tfds_mapNc Cs:t|ddd"}ttj|d|dW5QRSQRXdS)z!Reads a tab separated value file.rz utf-8-sig)encoding ) delimiter quotecharN)openlistcsvreader)clsZ input_filer9frrr _read_tsvpszDataProcessor._read_tsv)N) rrrrr+r.r/r0r4 classmethodr@rrrrr(Rsr(c@sdeZdZdZdddZddZd d ZedddZedddZ dddZ dddZ dddZ dS)%SingleSentenceClassificationProcessorzA Generic processor for a single sentence classification data set.NclassificationFcCs4|dkr gn||_|dkrgn||_||_||_dSN)labelsexamplesmodeverbose)rrErFrGrHrrr__init__zsz.SingleSentenceClassificationProcessor.__init__cCs t|jSrD)r2rFrrrr__len__sz-SingleSentenceClassificationProcessor.__len__cCs(t|trt|j|j|dS|j|S)N)rErF) isinstanceslicerBrErF)ridxrrr __getitem__s z1SingleSentenceClassificationProcessor.__getitem__rr1c Ks(|f|}|j||||||ddd|S)NT) split_name column_label column_text column_idskip_first_rowoverwrite_labelsoverwrite_examples)add_examples_from_csv) r> file_namerPrQrRrSrTkwargs processorrrrcreate_from_csvs  z5SingleSentenceClassificationProcessor.create_from_csvcKs|f|}|j||d|S)N)rE) add_examples)r>texts_or_text_and_labelsrErYrZrrrcreate_from_exampless z:SingleSentenceClassificationProcessor.create_from_examplesc Cs||} |r| dd} g} g} g} t| D]^\} }| ||| |||dk rj| ||q.|rzd|| fnd| }| |q.|j| | | ||dS)Nr1z%s-%sz%s)rUrV)r@ enumerateappendr\)rrXrPrQrRrSrTrUrVlinesZtextsrEidsiliner rrrrWs(   z;SingleSentenceClassificationProcessor.add_examples_from_csvc Cs|dkst|t|kst|dks8t|t|ks8t|dkrNdgt|}|dkrddgt|}g}t}t|||D]N\}} } t|ttfr| dkr|\} } n|} || |t | | d| dqz|r||_ n |j ||rt||_ ntt|j  ||_ |j S)N)r r r r )r2AssertionErrorsetziprKtupler;addr`r rFextendrEunion) rr]rErbrUrVrFZ added_labelsZtext_or_text_and_labelr r textrrrr\s*    z2SingleSentenceClassificationProcessor.add_examplesTcs|dkr|j}ddt|jD}g}t|jD]D\} } | ddkrPtd| |j| jdt||jd} | | q0t d d |D} gtt ||jD]\} \} } | ddkrtd | t |jf|rd ndgt | } | t | }|r|g|| } |rdnd g|| } n&| |g|} | |r8dnd g|} t | | ksft d t | | t | | kst d t | | |jdkr|| j}n"|jdkrt| j}n t|j| dkr>|jr>tdtd| jtdddd| Dtdddd| Dtd| j|f t| | |dq|dkrbS|dkrts|tdddl}fdd}|jj||j|jd|jf|dg|dgd|gf}|S|d krt std!ddl!}dd"l"m#}|j$d#dD|j%d$}|j$d%dD|j%d$}|jdkr`|j$d&dD|j%d$}n&|jdkr|j$d'dD|jd$}||||}|Std(dS))a Convert examples in a list of ``InputFeatures`` Args: tokenizer: Instance of a tokenizer that will tokenize the examples max_length: Maximum example length task: GLUE task label_list: List of labels. Can be obtained from the processor using the ``processor.get_labels()`` method output_mode: String indicating the output mode. Either ``regression`` or ``classification`` pad_on_left: If set to ``True``, the examples will be padded on the left rather than on the right (default) pad_token: Padding token mask_padding_with_zero: If set to ``True``, the attention mask will be filled by ``1`` for actual values and by ``0`` for padded values. If set to ``False``, inverts it (``1`` for padded values, ``0`` for actual values) Returns: If the ``examples`` input is a ``tf.data.Dataset``, will return a ``tf.data.Dataset`` containing the task-specific features. If the input is a list of ``InputExamples``, will return a list of task-specific ``InputFeatures`` which can be fed to the model. NcSsi|]\}}||qSrr).0rcr rrr szFSingleSentenceClassificationProcessor.get_features..i'rzTokenizing example %dT)Zadd_special_tokens max_lengthcss|]}t|VqdSrD)r2)rmr#rrr szESingleSentenceClassificationProcessor.get_features..zWriting example %d/%dr1z Error with input length {} vs {}rCZ regressionz*** Example ***zguid: %sz input_ids: %s cSsg|] }t|qSrrrmxrrr /szFSingleSentenceClassificationProcessor.get_features..zattention_mask: %scSsg|] }t|qSrrsrtrrrrv0szlabel: %s (id = %d)r#r$r tfz?return_tensors set to 'tf' but TensorFlow 2.0 can't be importedc3s$D]}|j|jd|jfVqdS)Nr#r$rw)exfeaturesrrgen<sz?SingleSentenceClassificationProcessor.get_features..genryptz8return_tensors set to 'pt' but PyTorch can't be imported) TensorDatasetcSsg|] }|jqSr)r#rmr?rrrrvLs)ZdtypecSsg|] }|jqSr)r$rrrrrvMscSsg|] }|jqSrr rrrrrvOscSsg|] }|jqSrrrrrrrvQsz,return_tensors should be one of 'tf' or 'pt')&max_lenr_rErFloggerinfoencoder minr`maxrgr2reformatrGr r' ValueErrorrHr joinr"r RuntimeErrorZ tensorflowdataZDatasetZfrom_generatorZint32Zint64Z TensorShapertorchZtorch.utils.datarZtensorlong)r tokenizerroZ pad_on_leftZ pad_tokenZmask_padding_with_zeroZreturn_tensorsZ label_mapZ all_input_idsZex_indexr3r#Z batch_lengthr$Zpadding_lengthr rxr}ZdatasetrrZall_attention_maskZ all_labelsrr{r get_featuress               "     z2SingleSentenceClassificationProcessor.get_features)NNrCF)rOrr1NF)N)rOrr1NFFF)NNFF)NFrTN) rrrrrIrJrNrAr[r^rWr\rrrrrrBws@     $rB)r<rrloggingrtypingrrrZ file_utilsrr getLoggerrrr r"r(rBrrrrs  %