U &c@sddlmZmZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z ddlmZGdd d eZe d eZeGd d d eZeGd ddeZdS))ABCabstractmethod) dataclass)AnyDictListNewTypeTupleN) pad_sequence)PreTrainedTokenizerc@s,eZdZdZeeeejfdddZ dS) DataCollatorz A `DataCollator` is responsible for batching and pre-processing samples of data as requested by the training loop. )returncCsdS)z Take a list of samples from a Dataset and collate them into a batch. Returns: A dictionary of tensors NselfrrC/tmp/pip-unpacked-wheel-ymerj3tt/transformers/data/data_collator.py collate_batchszDataCollator.collate_batchN) __name__ __module__ __qualname____doc__rrstrtorchTensorrrrrrr sr InputDataClassc@s.eZdZdZeeeeej fdddZ dS)DefaultDataCollatora Very simple data collator that: - simply collates batches of dict-like objects - Performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object - does not do any additional preprocessing i.e., Property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. )featuresrcs*|d}t|drj|jdk rjt|jtkrFtjdd|Dtjd}ntjdd|Dtjd}d|i}njt|dr|jdk rt|jdtkrtjd d|Dtjd}ntjd d|Dtjd}d|i}ni}t | D]D\}d kr|dk rt |t stjfd d|Dtjd|<q|S) NrlabelcSsg|] }|jqSrr.0frrr 9sz5DefaultDataCollator.collate_batch..ZdtypecSsg|] }|jqSrrr rrrr#;slabels label_idscSsg|] }|jqSrr&r rrrr#?scSsg|] }|jqSrr'r rrrr#As)rr&csg|]}t|qSr)getattrr krrr#Js) hasattrrtypeintrtensorlongfloatr&varsitems isinstancer)rrfirstr%batchvrr)rr-s   $z!DefaultDataCollator.collate_batchN) rrrrrrrrrrrrrrrrs rc@seZdZUdZeed<dZeed<dZe ed<e e j e ee j fddd Ze e j e j dd d Ze j ee j e j fd d dZdS)DataCollatorForLanguageModelingz Data collator used for language modeling. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for masked language modeling tokenizerTmlmg333333?mlm_probability)examplesrcCs6||}|jr(||\}}||dS||dSdS)N) input_idsZmasked_lm_labels)r<r%)_tensorize_batchr9 mask_tokens)rr;r5inputsr%rrrrZs   z-DataCollatorForLanguageModeling.collate_batchcsn|ddtfdd|D}|r6tj|ddS|jjdkrXtd|jjjdt |d|jj dSdS) Nrc3s|]}|dkVqdS)rN)size)r!xZlength_of_firstrr dszCDataCollatorForLanguageModeling._tensorize_batch..)ZdimzCYou are attempting to pad samples but the tokenizer you are using (z) does not have one.T)Z batch_firstZ padding_value) r@allrstackr8 _pad_token ValueError __class__rr pad_token_id)rr;Zare_tensors_same_lengthrrBrr=bs z0DataCollatorForLanguageModeling._tensorize_batch)r?rc sjjdkrtd|}t|jj}fdd|D}|j tj |tj dddjj dk r| jj}|j |ddt| }d||<tt|jd  |@}jjj||<tt|jd  |@|@}tjtj|jtjd} | |||<||fS) zw Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. NzThis tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.csg|]}jj|ddqS)T)Zalready_has_special_tokens)r8Zget_special_tokens_mask)r!valrrrr#|sz?DataCollatorForLanguageModeling.mask_tokens..r$g)valueig?g?)r8Z mask_tokenrGclonerfullshaper:tolistZ masked_fill_r.boolrFeqrIZ bernoulliZconvert_tokens_to_idsrandintlenr/) rr?r%Zprobability_matrixZspecial_tokens_maskZ padding_maskZmasked_indicesZindices_replacedZindices_randomZ random_wordsrrrr>os*    " z+DataCollatorForLanguageModeling.mask_tokensN)rrrrr __annotations__r9rPr:r0rrrrrrr=r r>rrrrr7Ns     r7)abcrrZ dataclassesrtypingrrrrr rZtorch.nn.utils.rnnr Ztokenization_utilsr r rrr7rrrrs    .