U &c@sdZddlZddlZddlZddlmZddlmZee Z dddZ ddd Z dd d Z dd dZdddZGdddeZdS)z$PyTorch optimization for BERT model.N) OptimizerLambdaLRcCst|dd|dS)z6 Create a schedule with a constant learning rate. cSsdS)N)_rr=/tmp/pip-unpacked-wheel-ymerj3tt/transformers/optimization.pyz'get_constant_schedule.. last_epochr) optimizerr rrr get_constant_schedulesrcsfdd}t|||dS)z Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate increases linearly between 0 and 1. cs"|krt|ttdSdS)N?floatmax current_stepnum_warmup_stepsrr lr_lambda'sz4get_constant_schedule_with_warmup..lr_lambdar r)rrr rrrr !get_constant_schedule_with_warmup"s rcsfdd}t|||S)zz Create a schedule with a learning rate that decreases linearly after linearly increasing during a warmup period. csB|krt|ttdStdt|ttdS)Nrrrnum_training_stepsrrr r4s z2get_linear_schedule_with_warmup..lr_lambdar)rrrr rrrr get_linear_schedule_with_warmup/sr?csfdd}t|||S)z Create a schedule with a learning rate that decreases following the values of the cosine function between 0 and `pi * cycles` after a warmup period during which it increases linearly between 0 and 1. c sf|krt|ttdSt|ttd}tdddttjtd|S)Nrrrrg@rrmathcospirprogress num_cyclesrrrr rDsz2get_cosine_schedule_with_warmup..lr_lambdarrrrr&r rrr%r get_cosine_schedule_with_warmup>sr(rcsfdd}t|||S)z Create a schedule with a learning rate that decreases following the values of the cosine function with several hard restarts, after a warmup period during which it increases linearly between 0 and 1. c sr|krt|ttdSt|ttd}|dkrHdStdddttjt|dS)Nrrrrrr#r%rr rUs zEget_cosine_with_hard_restarts_schedule_with_warmup..lr_lambdarr'rr%r 2get_cosine_with_hard_restarts_schedule_with_warmupMsr)cs,eZdZdZd fdd Zd d d ZZS)AdamWa Implements Adam algorithm with weight decay fix. Parameters: lr (float): learning rate. Default 1e-3. betas (tuple of 2 floats): Adams beta parameters (b1, b2). Default: (0.9, 0.999) eps (float): Adams epsilon. Default: 1e-6 weight_decay (float): Weight decay. Default: 0.0 correct_bias (bool): can be set to False to avoid correcting bias in Adam (e.g. like in Bert TF repository). Default True. MbP?g?g+?ư>rTcs|dkrtd|d|dkr.dksBntd|dd|dkrZdksnntd|dd|kstd|t|||||d}t||dS) Nrz,Invalid learning rate: {} - should be >= 0.0rrz4Invalid beta parameter: {} - should be in [0.0, 1.0[rz,Invalid epsilon value: {} - should be >= 0.0)lrbetaseps weight_decay correct_bias) ValueErrorformatdictsuper__init__)selfparamsr.r/r0r1r2defaults __class__rr r7kszAdamW.__init__NcCsd}|dk r|}|jD]b}|dD]R}|jdkr8q&|jj}|jrNtd|j|}t|dkrd|d<t|j|d<t|j|d<|d|d}}|d\} } |dd 7<| |  d | || |  d | ||| |d } |d } |d r>d | |d} d | |d}| t || } |j| || |ddkr&|j |d  |d|jq&q|S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr9zJAdam does not support sparse gradients, please consider SparseAdam insteadrstepexp_avg exp_avg_sqr/rrr0r.r2r1r)Z param_groupsgraddataZ is_sparse RuntimeErrorstatelentorchZ zeros_likeZmul_Zadd_Zaddcmul_sqrtr Zaddcdiv_)r8closureZlossgrouppr@rCr>r?Zbeta1Zbeta2ZdenomZ step_sizeZbias_correction1Zbias_correction2rrr r=ws<       "z AdamW.step)r+r,r-rT)N)__name__ __module__ __qualname____doc__r7r= __classcell__rrr;r r*`s  r*)r)r)r)rr)rr)rMloggingr rEZ torch.optimrZtorch.optim.lr_schedulerr getLoggerrJloggerrrrr(r)r*rrrr s