U &c@sddlZddlZddlZddlmZmZddlmZmZmZm Z ddl m Z m Z m Z e rbddlZzddlmmZdZWnek rdZYnXe dd ZeeZeGd d d ZdS) N) dataclassfield)AnyDictOptionalTuple)cached_propertyis_torch_availabletorch_requiredTFcCstS)N)_has_tpur r >/tmp/pip-unpacked-wheel-ymerj3tt/transformers/training_args.pyis_tpu_availablesrc@sBeZdZUdZeddidZeed<edddidZe ed <eddd idZ e ed <eddd idZ e ed <edddidZ e ed<edddidZ e ed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<ed dd!idZeed"<ed#dd$idZeed%<ed&dd'idZeed(<ed)dd*idZeed+<ed,dd-idZeed.<ed/dd0idZeeed1<eddd2idZe ed3<ed4dd5idZeed6<ed4dd7idZeed8<ed/dd9idZeeed:<eddd;idZ e ed<<ed=dd>idZ!eed?<eddd@idZ"e edA<edBddCidZ#eedD<ed)ddEidZ$eedF<ed/ddGidZ%eeedH<edddIidZ&e edJ<e'edKdLdMZ(e'edKdNdOZ)e*e+e,dPefdKdQdRZ-e'e+dPdKdSdTZ.e'e+dUdVZ/dWdXZ0e1ee2fdKdYdZZ3d/S)[TrainingArgumentsa TrainingArguments is the subset of the arguments we use in our example scripts **which relate to the training loop itself**. Using `HfArgumentParser` we can turn this class into argparse arguments to be able to specify them on the command line. helpzQThe output directory where the model predictions and checkpoints will be written.)metadata output_dirFz{Overwrite the content of the output directory.Use this to continue training if output_dir points to a checkpoint directory.)defaultroverwrite_output_dirzWhether to run training.do_trainz#Whether to run eval on the dev set.do_evalz+Whether to run predictions on the test set. do_predictz4Run evaluation during training at each logging step.evaluate_during_trainingz$Batch size per GPU/CPU for training.per_gpu_train_batch_sizez&Batch size per GPU/CPU for evaluation.per_gpu_eval_batch_sizerzONumber of updates steps to accumulate before performing a backward/update pass.gradient_accumulation_stepsg-C6 ?z#The initial learning rate for Adam. learning_rategzWeight decay if we apply some. weight_decayg:0yE>zEpsilon for Adam optimizer. adam_epsilong?zMax gradient norm. max_grad_normg@z+Total number of training epochs to perform.num_train_epochszQIf > 0: set total number of training steps to perform. Override num_train_epochs. max_stepsrz Linear warmup over warmup_steps. warmup_stepsNzTensorboard log dir. logging_dirz"Log and eval the first global_steplogging_first_stepizLog every X updates steps. logging_stepsz&Save checkpoint every X updates steps. save_stepszwLimit the total amount of checkpoints.Deletes the older checkpoints in the output_dir. Default is unlimited checkpointssave_total_limitz)Do not use CUDA even when it is availableno_cuda*zrandom seed for initializationseedzOWhether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bitfp16ZO1zFor fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3'].See details at https://nvidia.github.io/apex/amp.htmlfp16_opt_levelz$For distributed training: local_rank local_rankzBTPU: Number of TPU cores (automatically passed by launcher script) tpu_num_coresz#TPU: Whether to print debug metricstpu_metrics_debug)returncCs|jtd|jSNr)rmaxn_gpuselfr r rtrain_batch_sizetsz"TrainingArguments.train_batch_sizecCs|jtd|jSr4)rr5r6r7r r reval_batch_sizexsz!TrainingArguments.eval_batch_sizez torch.devicecCstd|jr td}d}nbtr4t}d}nN|jdkrbttj rPdnd}tj }n tj j ddtd|j}d}||fS) NzPyTorch: setting up devicescpurr#cudaZnccl)backendr)loggerinfor+torchdevicerxmZ xla_devicer0r<Z is_availableZ device_countZ distributedZinit_process_group)r8rAr6r r r_setup_devices|s    z TrainingArguments._setup_devicescCs |jdS)NrrCr7r r rrAszTrainingArguments.devicecCs |jdSr4rDr7r r rr6szTrainingArguments.n_gpucCstjt|ddS)z< Serializes this instance to a JSON string. )indent)jsondumps dataclassesasdictr7r r rto_json_stringsz TrainingArguments.to_json_stringcs>t|}ttttgtr(tj fdd| DS)uM Sanitized serialization to use with TensorBoard’s hparams cs*i|]"\}}|t|kr|nt|qSr )typestr).0kvZ valid_typesr r sz7TrainingArguments.to_sanitized_dict..) rIrJboolintfloatrMr appendr@ZTensoritems)r8dr rQrto_sanitized_dicts    z#TrainingArguments.to_sanitized_dict)4__name__ __module__ __qualname____doc__rrrM__annotations__rrSrrrrrrTrrrrUrr r!r"r$r%r&rr'r(r)r*r+r-r.r/r0r1r2propertyr9r:r r rrCrAr6rKrrrYr r r rrs      r)rIrGloggingrrtypingrrrrZ file_utilsr r r r@Ztorch_xla.core.xla_modelcoreZ xla_modelrBr ImportErrorr getLoggerrZr>rr r r rs"