U &ºcœã@s2dZddlZddlZddlmZddlmmZddlm Z ddl m Z m Z ddl mZddlmZe e¡Zdd iZd d „Zd d „ZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGdd„deƒZdZdZe deƒGdd„deƒƒZ e deƒGd d!„d!eƒƒZ!dS)"z PyTorch Transformer XL model. Adapted from https://github.com/kimiyoung/transformer-xl. In particular https://github.com/kimiyoung/transformer-xl/blob/master/pytorch/mem_transformer.py éNé)ÚTransfoXLConfig)Úadd_start_docstringsÚ add_start_docstrings_to_callable)ÚProjectedAdaptiveLogSoftmax)ÚPreTrainedModelztransfo-xl-wt103z=https://cdn.huggingface.co/transfo-xl-wt103-pytorch_model.binc Cs,i}t|dƒr²| |jj|jjdœ¡tt|jj|jj|j ƒƒD]j\}\}}}d|}|j rr| |d|j i¡n"t ‚| |d|j |d|j i¡|s@| |d|i¡q@|j}tt|jj|jjƒƒD]0\}\}}d|}| |d|j |d|i¡qÈt|jƒD]Æ\}} d |}| |d | jjj |d | jjj |d | jjj |d | jjj |d| jjj |d| jjj |d| jjj |d| jjdj |d| jjdj |d| jjdj |d| jjdj i ¡q|jrg} g} |jD]"} |  | jj¡|  | jj¡qân|jg} |jg} | | | dœ¡|S)zš A map of modules from TF to PyTorch. This time I use a map to keep the PyTorch model as identical to the original PyTorch model as possible. Ú transformer)z/transformer/adaptive_softmax/cutoff_0/cluster_Wz/transformer/adaptive_softmax/cutoff_0/cluster_bz'transformer/adaptive_softmax/cutoff_%d/ÚbZ lookup_tableÚprojz%transformer/adaptive_embed/cutoff_%d/Zproj_Wztransformer/layer_%d/zrel_attn/LayerNorm/gammazrel_attn/LayerNorm/betazrel_attn/o/kernelzrel_attn/qkv/kernelzrel_attn/r/kernelzff/LayerNorm/gammazff/LayerNorm/betazff/layer_1/kernelrzff/layer_1/biaszff/layer_2/kernelézff/layer_2/bias)ztransformer/r_r_biasztransformer/r_w_bias)ÚhasattrÚupdateÚcritÚcluster_weightÚ cluster_biasÚ enumerateÚzipÚ out_layersÚ out_projsÚ tie_projsÚ tie_weightÚbiasÚNotImplementedErrorÚweightrÚword_embÚ emb_layersÚ emb_projsÚlayersÚdec_attnÚ layer_normÚo_netÚqkv_netÚr_netÚpos_ffÚCoreNetÚuntie_rÚappendÚr_r_biasÚr_w_bias) ÚmodelÚconfigÚ tf_to_pt_mapÚiZout_lZproj_lÚtie_projZ layer_strZembed_lr Zr_r_listZr_w_list©r.úD/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_transfo_xl.pyÚbuild_tf_to_pytorch_map)st þÿÿ"    õÿ r0c CsLzddl}ddl}Wn tk r4t d¡‚YnXt||ƒ}|j |¡}i}|D]0\}} t d  || ¡¡|j  ||¡} | ||<qT|  ¡D]œ\}} ||ks¤t ‚||} d|ks¼d|krÆ|  | ¡} d|ksØd|krt| ƒd krt| ƒ| jdksüt ‚t| ƒD]ˆ\} } | | d f}z| j|jks,t ‚Wn<t k rj}z|j| j|jf7_‚W5d}~XYnXt d   || ¡¡t |¡| _qnpz| j| jks¤t ‚Wn<t k râ}z|j| j| jf7_‚W5d}~XYnXt d   |¡¡t | ¡| _| |d¡| |d d¡| |dd¡qŽt d  d | ¡¡¡¡|S)z, Load tf checkpoints in a pytorch model rNzšLoading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z"Loading TF weight {} with shape {}Úkernelr r'r(r.z)Initialize PyTorch weight {} for layer {}zInitialize PyTorch weight {}z/Adamz/Adam_1z'Weights not copied to PyTorch model: {}z, )ZnumpyZ tensorflowÚ ImportErrorÚloggerÚerrorr0ZtrainZlist_variablesÚinfoÚformatZ load_variableÚitemsÚAssertionErrorÚ transposeÚlenÚshaperÚargsÚtorchZ from_numpyÚdataÚpopÚjoinÚkeys)r)r*Ztf_pathÚnpÚtfr+Z init_varsZ tf_weightsÚnamer;ÚarrayÚpointerr,Zp_iZarr_iÚer.r.r/Úload_tf_weights_in_transfo_xllsV ÿ          rHcs&eZdZ‡fdd„Zddd„Z‡ZS)ÚPositionalEmbeddingcs:tƒ ¡||_ddt d|d¡|}| d|¡dS)Nri'çg@Úinv_freq)ÚsuperÚ__init__Údembr=ÚarangeZregister_buffer)ÚselfrNrK©Ú __class__r.r/rM§s zPositionalEmbedding.__init__NcCsjt ||j¡}tj| ¡| ¡gdd}|dk rP|dd…ddd…f d|d¡S|dd…ddd…fSdS)Néÿÿÿÿ©Údim)r=ZgerrKÚcatÚsinÚcosÚexpand)rPÚpos_seqÚbszZ sinusoid_inpÚpos_embr.r.r/Úforward¯s  zPositionalEmbedding.forward)N©Ú__name__Ú __module__Ú __qualname__rMr]Ú __classcell__r.r.rQr/rI¦s rIcs&eZdZd‡fdd„ Zdd„Z‡ZS)ÚPositionwiseFFFçñh㈵øä>c sntƒ ¡||_||_||_t t ||¡tjddt  |¡t ||¡t  |¡¡|_ tj ||d|_ ||_ dS)NT)Zinplace©Zeps)rLrMÚd_modelÚd_innerÚdropoutÚnnZ SequentialÚLinearZReLUÚDropoutr$Ú LayerNormrÚ pre_lnorm)rPrfrgrhrmÚlayer_norm_epsilonrQr.r/rMºs    ûzPositionwiseFF.__init__cCs<|jr | | |¡¡}||}n| |¡}| ||¡}|S©N)rmr$r)rPÚinpÚcore_outÚoutputr.r.r/r]Ís   zPositionwiseFF.forward)Frdr^r.r.rQr/rc¹srcc s0eZdZd ‡fdd„ Zdd„Zd d d „Z‡ZS) Ú RelPartialLearnableMultiHeadAttnrNFrdcstƒ ¡| |_||_||_||_||_tj|d||dd|_ t  |¡|_ t  |¡|_ tj|||dd|_ tj|| d|_d|d|_| |_| dks¤| dkrÖt t |j|j¡¡|_t t |j|j¡¡|_n | |_| |_tj|j|j|jdd|_dS)Nr F)rrerçà?)rLrMÚoutput_attentionsÚn_headrfÚd_headrhrirjr!rkÚdropÚdropattr rlrÚscalermÚ Parameterr=Ú FloatTensorr'r(r")rPrvrfrwrhryÚtgt_lenÚext_lenÚmem_lenrmr'r(rurnrQr.r/rMßs&   z)RelPartialLearnableMultiHeadAttn.__init__cCsŒ| d¡df| ¡dd…}tj||j|jd}tj||gdd}| d¡d| d¡f| ¡dd…}|j|Ž}|dd… |¡}|S)Nrré©ÚdeviceÚdtyperT)Úsizer=Úzerosr‚rƒrVÚviewZview_as)rPÚxZzero_pad_shapeZzero_padZx_paddedZx_padded_shaper.r.r/Ú _rel_shift s( z+RelPartialLearnableMultiHeadAttn._rel_shiftcCsF| d¡| d¡| d¡}}}|dk rŒt ||gd¡} |jrR| | | ¡¡} n | | ¡} | |¡} tj| ddd\} } }| | d…} nB|jr¤| | |¡¡} n | |¡} | |¡} tj| ddd\} } }|  d¡}|  |||j |j ¡} |  |||j |j ¡} | |||j |j ¡}|  ||j |j ¡} | |j }t  d|| f¡}| |j }t  d|| f¡}| |¡}||}| |j¡|dk r˜t |¡ ¡r˜|dk}| ¡dkrt| ¡ƒjtjkrê| ¡ |ddd…dd…dfd ¡ |¡}n*| ¡ |ddd…dd…dfd ¡ |¡}n‚| ¡dkr˜t| ¡ƒjtjkrj| ¡ |dd…dd…dd…dfd ¡ |¡}n.| ¡ |dd…dd…dd…dfd ¡ |¡}tj|dd}| |¡}|dk rÂ||}t  d ||f¡}| ¡ | d¡| d¡|j |j ¡}| |¡}|  |¡}|jr ||g}n| ||¡g}|j!rB| "|¡|S) Nrrr rSrTzibnd,jbnd->ijbnzibnd,jnd->ijbnr€iÿÿgꌠ9Y>)Æzijbn,jbnd->ibnd)#r„r=rVrmr!rr"Úchunkr†rvrwr(Zeinsumr'rˆÚmul_rzÚsumÚitemrUÚnextÚ parametersrƒZfloat16ÚfloatZ masked_fillZtype_asÚFZsoftmaxryÚ contiguousr rxrur&)rPÚwÚrÚ attn_maskÚmemsÚ head_maskÚqlenZrlenr[rVZw_headsZr_head_kZw_head_qZw_head_kZw_head_vÚklenZ rw_head_qZACZ rr_head_qZBDZ attn_scoreZ attn_probZattn_vecZattn_outÚoutputsr.r.r/r]sf"         (ÿ,0.  &    z(RelPartialLearnableMultiHeadAttn.forward) rNNNFNNFrd)NNN)r_r`rarMrˆr]rbr.r.rQr/rsÞsò- rscs(eZdZd‡fdd„ Zddd„Z‡ZS) ÚRelPartialLearnableDecoderLayerrdc sDtƒ ¡t||||fd|i|—Ž|_t|||| d¡|d|_dS)Nrnrm)rmrn)rLrMrsrrcÚgetr#)rPrvrfrwrgrhrnÚkwargsrQr.r/rMrs$ ÿÿÿÿz(RelPartialLearnableDecoderLayer.__init__Nc Cs8|j|||||d}| |d¡}|g|dd…}|S)N)r”r•r–rr)rr#) rPZdec_inpr“Ú dec_attn_maskr•r–Z attn_outputsZ ff_outputr™r.r.r/r]|sz'RelPartialLearnableDecoderLayer.forward)rd)NNNr^r.r.rQr/ršqs ršcs&eZdZd‡fdd„ Zdd„Z‡ZS)ÚAdaptiveEmbeddingrFc stƒ ¡||_||_||g|_||_||_|d|_dg|j|_t   ¡|_ t   ¡|_ |dkr¢|j  t j|||dkd¡||kr |j  t  t ||¡¡¡nltt|jƒƒD]\}|j||j|d}} |||} |j  t  | || ¡¡|j  t  t || ¡¡¡q°dS)Nrtrr)Úsparse)rLrMÚn_tokenÚd_embedÚcutoffsÚdiv_valÚd_projÚ emb_scaleÚ cutoff_endsriÚ ModuleListrZ ParameterListrr&Ú Embeddingr{r=r|Úranger:) rPr r¡r¤r¢r£Úsample_softmaxr,Úl_idxÚr_idxZd_emb_irQr.r/rM‡s&      zAdaptiveEmbedding.__init__cCs.|jdkr8|jd|ƒ}|j|jkr6t ||jd¡}næt| ¡ƒ}|  d¡}t j |  d¡|jg|j |jd}tt|jƒƒD]†}|j||j|d}}||k||k@} |  ¡ ¡} |  ¡dkrÄq|| d| ¡|} |j|| ƒ} t | |j|¡} | d| | ¡q||  ¡|jf} |  | ¡}| |j¡|S)NrrrS©rƒr‚)r£rr¤r¡rZlinearrrrŽr†r=r…r„rƒr‚r©r:r¢r¦ZnonzeroZsqueezeZnumelZ index_selectZ index_copy_rŠr¥)rPrpÚembedÚparamZinp_flatZemb_flatr,r«r¬Zmask_iZ indices_iZinp_iZemb_iZ embed_shaper.r.r/r]¢s*         zAdaptiveEmbedding.forward)rFr^r.r.rQr/rž†sržc@s8eZdZdZeZeZeZ dZ dd„Z dd„Z dd„Z d S) ÚTransfoXLPreTrainedModelz† An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rcCsL|jjdkr(tj ||jj |jj¡n |jjdkrHtj |d|jj¡dS)NÚuniformÚnormalrJ)r*ÚinitriZuniform_Z init_rangeÚnormal_Úinit_std)rPrr.r.r/Ú _init_weightÌs  z%TransfoXLPreTrainedModel._init_weightcCstj |d¡dS)NrJ)rir³Z constant_)rPrr.r.r/Ú _init_biasÒsz#TransfoXLPreTrainedModel._init_biascCs<|jj}| d¡dkrZt|dƒr6|jdk r6| |j¡t|dƒrV|jdk rV| |j¡nÞ| d¡dkr²t|dƒr®tt |j ƒƒD],}|j |dk r€t j   |j |d|jj¡q€n†| d ¡dkrÚt|dƒrÖ| |j¡n^| d ¡dkr€t|d ƒr|jdk r| |j¡t|d ƒr2|jdk r2| |j¡t|d ƒr8tt |jƒƒD]0}|j|dk rLt j   |j|d|jj¡qLn¸| d¡dkrØt|dƒr²t j   |jd|jj¡t|dƒr8|jdk r8| |j¡n`t|dƒrð| |j¡t|dƒr| |j¡t|dƒr | |j¡t|dƒr8| |j¡dS)z! Initialize the weights. rjrSrNrržrrJr¨rrrrrlgð?Úr_embr(r'Úr_bias)rRr_Úfindr rr¶rr·r©r:rrir³r´r*Z proj_init_stdrrrrµr¸r(r'r¹)rPÚmÚ classnamer,r.r.r/Ú _init_weightsÕsH                z&TransfoXLPreTrainedModel._init_weightsN)r_r`raÚ__doc__rZ config_classÚ'TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_maprHZload_tf_weightsZbase_model_prefixr¶r·r½r.r.r.r/r°Âsr°as This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.TransfoXLConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`transformers.TransfoXLTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `mems` output below). Can be used to speed up sequential decoding. The token ids which have their mems given to this model should not be passed as input ids as they have already been computed. head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. z^The bare Bert Model transformer outputting raw hidden-states without any specific head on top.csfeZdZ‡fdd„Zdd„Zdd„Zdd„Zd d „Zd d „Zd d„Z dd„Z e e ƒddd„ƒZ ‡ZS)ÚTransfoXLModelcsªtƒ |¡|j|_|j|_|j|_|j|_|j|_|j|_|j |_ t |j|j|j|j |j d|_ t |j¡|_|j|_|j|_|j|_|j|_|j|j|j|_|j|_|jsât t |j|j ¡¡|_t t |j|j ¡¡|_t ¡|_|jdkrlt |jƒD]f}|j !t"|j|j|j |j#|j|j|j|j|j$|j%|jrBdn|j|jrRdn|j|j|j&d¡qnt'‚|j(|_(|j)|_)|jdkršt*|jƒ|_+nt'‚| ,¡dS)N©r£r) r}r~rryrmr(r'rurn)-rLrMruÚoutput_hidden_statesÚ vocab_sizer r¡rfrvrwržr¢r£rrirkrhrxÚn_layerr}rr~Zmax_klenÚ attn_typer%r{r=r|r(r'r§rr©r&ršrgryrmrnrÚ same_lengthÚ clamp_lenrIr\Ú init_weights)rPr*r,rQr.r/rM)sj ÿ  òÿ  zTransfoXLModel.__init__cCs|jSro©r©rPr.r.r/Úget_input_embeddingsjsz#TransfoXLModel.get_input_embeddingscCs ||_dSrorÉ)rPZnew_embeddingsr.r.r/Úset_input_embeddingsmsz#TransfoXLModel.set_input_embeddingscCs d|_dS)NrS)rªrÊr.r.r/Úbackward_compatiblepsz"TransfoXLModel.backward_compatiblecCs||_||_||_dSro)r}rr~©rPr}r~rr.r.r/Ú reset_lengthsszTransfoXLModel.reset_lengthcCst d¡dS)Nz8Head pruning is not implemented for Transformer-XL model)r3r5)rPZheadsr.r.r/Ú _prune_headsxs zTransfoXLModel._prune_headscCs^|jdkrVg}t| ¡ƒ}t|jƒD],}tj|j||jj|j |j d}|  |¡q$|SdSdS)Nrr­) rrrŽr©rÄr=r…r*rfrƒr‚r&)rPr[r•r¯r,Úemptyr.r.r/Ú init_mems|s   zTransfoXLModel.init_memsc Csª|dkr dSt|ƒt|ƒks$tdƒ‚t ¡tg}|td|d|jƒ}td||jƒ}tt|ƒƒD]4}tj||||gdd} |  | ||…  ¡¡qfW5QRX|S)Nzlen(hids) != len(mems)rrT) r:r8r=Zno_gradÚmaxr~rr©rVr&Údetach) rPÚhidsr•Úmlenr—Únew_memsZend_idxZbeg_idxr,rVr.r.r/Ú _update_memsˆs "zTransfoXLModel._update_memsNcCsV|dk r|dk rtdƒ‚n^|dk r@| dd¡ ¡}| ¡\}}n8|dk rp| dd¡ ¡}|jd|jd}}ntdƒ‚|dkrŠ| |¡}|dk r | ¡dkrÒ| d¡ d¡ d¡ d¡}| |j dddd¡}n"| ¡dkrô| d¡ d¡ d¡}|j t |  ¡ƒj d}n dg|j }|dk r(|}n | |¡}|dk rJ|d d¡nd}||} |jrÆ|j|| ftjd} | |j} | dkr|| } n|} t | d|¡t | | ¡dd…dd…df} n4tj|j|| ftjdd|d dd…dd…df} g}g}|jdkrÎtj| ddd |j|j d }|jdkrD|j|jd | |¡}| |¡}| |¡}t|jƒD]^\}}| |¡|dkrŒdn||}|||| |||d }|d}|j rl| |d¡qlnt!‚| |¡}| "||||¡}| dd¡ ¡|g}|j#r.| |¡t$dd„|Dƒƒ}| |¡|j rRt$dd„|Dƒƒ}| |¡|S)aë Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.TransfoXLConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the last layer of the model. mems (:obj:`List[torch.FloatTensor]` of length :obj:`config.n_layers`): Contains pre-computed hidden-states (key and values in the attention blocks). Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as input ids as they have already been computed. hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import TransfoXLTokenizer, TransfoXLModel import torch tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103') model = TransfoXLModel.from_pretrained('transfo-xl-wt103') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states, mems = outputs[:2] NzDYou cannot specify both input_ids and inputs_embeds at the same timerrú5You have to specify either input_ids or inputs_embedsrSr€)rƒ)Zdiagonalgð¿r)rÓ)rr•r–css|]}| dd¡ ¡VqdS)rrN)r9r‘©Ú.0Útr.r.r/Ú sz)TransfoXLModel.forward..css"|]}| dddd¡ ¡VqdS)r€r rrN)Zpermuter‘rÚr.r.r/rÝs)%Ú ValueErrorr9r‘r„r;rÒrUZ unsqueezerYrÄÚtorrŽrƒrrÆZnew_onesr=Zuint8rZtriuZtrilrÅrOr‚rÇZclamp_r\rxrrr&rurrØrÂÚlist)rPÚ input_idsr•r–Ú inputs_embedsr—r[rrÖr˜Úall_onesZmask_lenZmask_shift_lenrrÕZ attentionsrZr\rqr,ZlayerZmems_iZ layer_outputsr×r™r.r.r/r] sŒ%      ÿ      2,ÿ      ÿ    zTransfoXLModel.forward)NNNN)r_r`rarMrËrÌrÍrÏrÐrÒrØrÚTRANSFO_XL_INPUTS_DOCSTRINGr]rbr.r.rQr/rÀ$s A rÀz‡The Transformer-XL Model with a language modeling head on top (adaptive softmax with weights tied to the adaptive input embeddings)csVeZdZ‡fdd„Zdd„Zdd„Zdd„Zeeƒdd d „ƒZ d d „Z dd„Z ‡Z S)ÚTransfoXLLMHeadModelcsZtƒ |¡t|ƒ|_|j|_|jdks0tdƒ‚t|j|j|j |j |j d|_ |  ¡dS)Nrz†Sampling from the softmax is not implemented yet. Please look at issue: #3310: https://github.com/huggingface/transformers/issues/3310rÁ)rLrMrÀrrªr8rrÃr¡rfr¢r£rrÈ)rPr*rQr.r/rM(s  ÿþÿzTransfoXLLMHeadModel.__init__cCs|jjrsD     ÿC:%<= þ{ý