U &c#@sdZddlZddlZddlmZddlmZddlmZddl Z ddl Z ddl m Z ddl mZddlmZd d lmZmZmZmZd d lmZd d lmZmZmZmZd d lmZmZe e!Z"dddZ#ddZ$ee j j%j&eeee$dZ'eddddgZ(edddgZ)eddddgZ*edddddgZ+edddddgZ,eddd d!gZ-d"d#Z.Gd$d%d%e j/Z0Gd&d'd'e j/Z1Gd(d)d)e j/Z2Gd*d+d+Z3Gd,d-d-e j/e3Z4Gd.d/d/eZ5Gd0d1d1e j/e3Z6Gd2d3d3e j/Z7Gd4d5d5e j/Z8Gd6d7d7e j/Z9Gd8d9d9e j/Z:Gd:d;d;e j/Z;Gdd?d?eZ=Gd@dAdAe j/Z>GdBdCdCe j/Z?GdDdEdEeZ@dFZAdGZBedHeAGdIdJdJe@ZCedKeAGdLdMdMe@ZDdS)NzPyTorch REFORMER model. N) namedtuple)reduce)mul)nn)Function)CrossEntropyLoss)gelu gelu_fastgelu_newswish)ReformerConfig) DUMMY_INPUTS DUMMY_MASKadd_start_docstrings add_start_docstrings_to_callable)PreTrainedModelapply_chunking_to_forwardzQhttps://cdn.huggingface.co/google/reformer-crime-and-punishment/pytorch_model.binzChttps://cdn.huggingface.co/google/reformer-enwik8/pytorch_model.bin)z$google/reformer-crime-and-punishmentzgoogle/reformer-enwik8cCs|ttj|SN)torchtanhr functionalZsoftplus)xrB/tmp/pip-unpacked-wheel-ymerj3tt/transformers/modeling_reformer.pymish,sr)r relur r r rLSHSelfAttentionOutput hidden_statesattention_probsbucketsLocalSelfAttentionOutputAttentionOutputReformerOutput attn_outputReformerBackwardOutputgrad_attn_outputgrad_hidden_statesReformerEncoderOutputall_hidden_statesall_attentionscCs|j}t|}t|dkr,|ddkr,|jSt|dkrJ|ddkrJ|jSt|dkrv|tddgkrvt|j|jStd|jdS)NrrlshlocalzOnly attn layer types 'lsh' and 'local' exist, but `config.attn_layers`: {}. Select attn layer types from ['lsh', 'local'] only.) attn_layerssetlenlsh_attn_chunk_lengthlocal_attn_chunk_lengthnpZlcmNotImplementedErrorformat)configZ attn_typesZattn_types_setrrr _get_least_common_mult_chunk_lenEsr7cs(eZdZdZfddZddZZS)AxialPositionEmbeddingszlConstructs axial position embeddings. Useful for very long input sequences to save memory and time. c st|j|_|j|_|j|_t||_t |_ t |j|j ksZt d|j|j t|jD]R\}}dgt|j}|j|||<t||f}|j ttj|tjdqddS)NzOMake sure that config.axial_pos_embds factors: {} sum to config.hidden_size: {}r)dtype)super__init__axial_pos_shapeZaxial_pos_embds_dimhidden_dropout_probdropoutr7least_common_mult_chunk_lengthrZ ParameterListweightssum hidden_sizeAssertionErrorr5 enumerater0tupleappend ParameterronesZfloat32)selfr6ZaxisZaxial_pos_embd_dimZax_shape __class__rrr;[s$   z AxialPositionEmbeddings.__init__c s<|jd|jdfddjD}jdkrttjksdtdjjttjjdkrt j |dd}| d d}t j j|jjd }| d d}t |df}nt j fd d|Ddd}nZttjkstd jjt j |dd}|d|jddddf}|S) Nrrcs,g|]$}|fj|jddqS)N)expandr<shape.0weight) batch_sizerIrr ysz3AxialPositionEmbeddings.forward..TzIf training, make sure that config.axial_pos_shape factors: {} multiply to sequence length. Got prod({}) != sequence_length: {}. You might want to consider padding your sequence length to {} or changing config.axial_pos_shape.rLdimr-ptrainingcsg|]}t|dfqSrL)rreshaperO)rRsequence_lengthrrrSszMake sure that config.axial_pos_shape factors: {} multiply at least to max(sequence_length, least_common_mult_chunk_length): max({}, {}))rNr@rXrrr<rCr5r>rcat transposerrZ dropout2drZr?view)rI position_idsZbroadcasted_weightsr@Ztransposed_weightsZdropped_transposed_weightsZdropped_weightsZposition_encodingsr)rRrIr[rforwardtsT        zAxialPositionEmbeddings.forward__name__ __module__ __qualname____doc__r;r` __classcell__rrrJrr8Vs r8cs(eZdZdZfddZddZZS)PositionEmbeddingsz^Constructs conventional position embeddings of shape `[max_pos_embeddings, hidden_size]`. cs(t|j|_t|j|j|_dSr) r:r;r=r>r Embeddingmax_position_embeddingsrB embeddingrIr6rJrrr;s zPositionEmbeddings.__init__cCs$||}tjj||j|jd}|SNrV)rjrrr>rX)rIr_position_embeddingsrrrr`s zPositionEmbeddings.forwardrarrrJrrgs rgcs*eZdZdZfddZdddZZS)ReformerEmbeddingszLConstruct the embeddings from word, position and token_type embeddings. csHt|j|_|j|_t|j|j|_ |j r:t |nt ||_ dSr)r:r;rir=r>rrh vocab_sizerBword_embeddingsZaxial_pos_embdsr8rgrmrkrJrrr;s  zReformerEmbeddings.__init__Nc Cs|dk r|}|j}n|dd}|j}|d}|dkr`tj|tj|d}|d|}|dkrr||}|jd|j kst d |jd|j t j j||j|jd}||}||}|S)NrLrr9devicerzRSequence Length: {} has to be larger equal than config.max_position_embeddings: {}rV)sizerrrarangelong unsqueezerMrprNrirCr5rrr>rXrm) rI input_idsr_ inputs_embeds input_shaperrZ seq_length embeddingsrmrrrr`s,  zReformerEmbeddings.forward)NNNrarrrJrrns rnc@s2eZdZdZddZddZddZd d d ZdS) EfficientAttentionMixinzL A few utilities for nn.Modules in Reformer, to be used as a mixin. c Cs|dkr|dkr|Sg}t| |dD]`}|dkr@||q(|tj|dddd|ddf|ddddd|dfgddq(tj|ddS)a Used to implement attention between consecutive chunks. Args: vectors: array of shape [batch_size, num_attention_heads, n_chunks, chunk_len, ...] num_chunks_before: chunks before current chunk to include in attention num_chunks_after: chunks after current chunk to include in attention Returns: tensor of shape [num_chunks, N * chunk_length, ...], where N = (1 + num_chunks_before + num_chunks_after). rrN.r-rT)rangerFrr\)rIvectorsnum_chunks_beforenum_chunks_afterZslicesirrr_look_adjacents  Jz&EfficientAttentionMixin._look_adjacentcCs.|dd||f}|j|}|ddS)zS splits hidden_size dim into attn_head_size and num_attn_heads NrLr-r)rsr^r])rIrnum_attn_headsattn_head_sizeZ new_x_shaperrr_split_hidden_size_dims z.EfficientAttentionMixin._split_hidden_size_dimcCs.|dddd}t||dd||fS)zW merges attn_head_size dim and num_attn_heads dim into hidden_size rr-rr|rL)ZpermuterrZrs)rIrrrrrr_merge_hidden_size_dimssz/EfficientAttentionMixin._merge_hidden_size_dimsNcCsh|jd}||||f}t|jdkr6t|||fSt|jdkrPt||Stdt|jdS)zg splits sequence length dim of vectors into `dim_factor_1` and `dim_factor_2` dims rr|z5Input vector rank should be one of [3, 4], but is: {}N)rNr0rrZ ValueErrorr5)rIr~Z dim_factor_1Z dim_factor_2rrrRZsplit_dim_shaperrr_split_seq_length_dim_to s   z0EfficientAttentionMixin._split_seq_length_dim_to)N)rbrcrdrerrrrrrrrr{s r{csheZdZfddZdddZddZd d Zd d Zd dZddZ ddZ dddZ ddZ Z S)LSHSelfAttentioncst|j|_|j|_|j|_|j|_|j|_ |j |_ |j |_ |j |_ |j |_|j|_|j|_|j|j|_|j|_tj|j|jdd|_tj|j|jdd|_|dtd|dtd|dtd|d td dS) NFbiasself_mask_value_float16g@self_mask_value_float32gjmask_value_float16mask_value_float32e)r:r;r1 chunk_length num_hashes num_bucketsZlsh_num_chunks_beforerZlsh_num_chunks_afterr hash_seed is_decoderriZ lsh_attention_probs_dropout_probr>num_attention_headsattention_head_size all_head_sizerBrLinear query_keyvalueregister_bufferrtensorrkrJrrr;s( zLSHSelfAttention.__init__NFcKs|jd}|jd} |dk r |n|j}||} ||} ~|| |j|j} || |j|j} | jd|jkstd| jd|j| jd|jkstd| jd|j|j dkr| ||dkr| | |}t |jd||ks td|jd||| |||\} } | |} || | |} || | |} || d|j|j|j} || d|j|j|j} |jdkr|jdkr|jdkstd|| }|j| || | ||d\}}}~ ~~ t||| | |j\}}|dkrJ|||||j|j}|||||j|jd}t|tj|d d d }tj||d d }~~|j| |j||jfksltd |||j|j}|dkrd}t|||dS)NrrrL5last dim of query_key_vectors is {} but should be {}.z1last dim of value_vectors is {} but should be {}.z+last dim of buckets is {}, but should be {}tIf `config.chunk_length` is `None`, make sure `config.num_chunks_after` and `config.num_chunks_before` are set to 0.) query_vectors key_vectors value_vectorssorted_bucket_idxattention_mask head_maskr-TrUkeepdimrTzuout_vectors have be of shape `[batch_size, config.num_attention_heads, sequence_length, config.attention_head_size]`.Frrrr )rNrrrrrrrCr5r_set_num_buckets _hash_vectorsint1_get_sorted_bucket_idx_and_undo_sorted_bucket_idx_gather_by_expansionrrrr_len_and_dim_norm_attend ReverseSortapplyrvrexp logsumexprArr)rIrrrrdo_output_attentionsr kwargsr[rRZquery_key_vectorsrrundo_sorted_bucket_idxr out_vectorslogitsrZ probs_vectorsrrrr`8s             zLSHSelfAttention.forwardcCs|jd}t|jtrB|jddks4td|j|j}|j}n>d\}}|jD].}|ddksntd|||}||}qP|}|jdk rt |j|j |jd||df}tj ||j |j d}td||} t|jtst|jd krtj| | gdd } tj| dd } nd \} } } |jD]v}| d | | |df} | |d} tj| | gdd } | dkr~tj| dd } n| | tj| dd } | |} q(tj||j d }||d}|||j f|jdd}| |jddd}|S)Nrr-zEThere should be an even number of bucktes, but `self.num_bucktes`: {})rrz:The number of buckets should be even, but `num_bucket`: {}rLrrr9zbmtd,mdhr->bmhtrrrT)Nrr.rr)rrrLrr|Z start_dimZend_dim)rN isinstancerrrCr5detachrr manual_seedrZrandnrrr9Zeinsumr0r\Zargmaxrtr^rMflatten)rIr~rrRZ rotation_sizerZ bucket_factorZrotations_shapeZrandom_rotationsZrotated_vectorsr Zcur_sumZ cur_productZrotated_vectors_factoroffsetsZoffset_bucketsrrrrsL             zLSHSelfAttention._hash_vectorsc Cst|jd}tj|||jdddd}|||j|jd}||||}|}tj |dd}tj|jd|jdddd|j}|j | } | d||W5QRX|| fS)NrrrrLrT) rno_gradrNrtrrr^rMrrZargsortnewrsZscatter_) rIr[r rrRZ orig_indicesZscaled_bucketsrindicesrrrrrs$  zBLSHSelfAttention._get_sorted_bucket_idx_and_undo_sorted_bucket_idxcCs`d||j}tt|j|jd|j}|d|krF|||dg}td|||_dS)Nr-g?rzBconfig.num_buckets is not set. Setting config.num_buckets to {}...)rmaxrriloggerwarningr5r)rIr[rZnum_buckets_limitrrrrs  z!LSHSelfAttention._set_num_bucketscCsh|||j|j}|||j|j}t||dd}~~||d|j|j}|||j|j} |j tj kr|j } |j } n |j} |j} ||| |} | dk rt| || }~ t|d| d|j} t| || }~ tj|ddd}t||}~tjj||j|jd}|dk r.||}t||}~|jdddd}|jddd}|||fS) NrLrTrrVr-r|r)rrrrmatmulr]rrrr9float16rhalfrrr_compute_attn_maskwherenervtorrrrrrr>rXrZsqueeze)rIrrrrrrquery_key_dotsZquery_bucket_idxZkey_value_bucket_idxZself_mask_value mask_valuemaskZ self_maskrrrrrrr"sH      zLSHSelfAttention._attendcCsd}|jr*t|d|d|j}|dk r|tjddddddf}||jddd}t |d|}t |d|}|d|d}~~~|dk r||}n|}|S)NrLrrY) rrgervrrruint8rMrNgather)rI query_indices key_indicesrrZ key_attn_maskZquery_attn_mask attn_maskrrrrqs   z#LSHSelfAttention._compute_attn_maskcCs.||}|ttj|j|j|jd}|S)zF length and attention head size dim normalization r) _len_normrrsqrtrrrrr9)rIr~rrrrs  z"LSHSelfAttention._len_and_dim_normư>cCs*tj|dddd}|t||}|S)z* length normalization r-rLT)r)rmeanr)rIrepsilonZvarianceZnorm_xrrrrszLSHSelfAttention._len_normcCs6|dddd|j}|dd|d}t|d|S)zO expand dims of idxs and vectors for all hashes and gather rLrr-)rvrMrrepeatrr)rIr~ZidxsrZ expanded_idxsrrrrsz%LSHSelfAttention._gather_by_expansion)NNNFN)r)rbrcrdr;r`rrrrrrrrrfrrrJrrs  ~A O rc@s(eZdZdZeddZeddZdS)ra After chunked attention is applied which sorted clusters, original ordering has to be restored. Since customized backward function is used for Reformer, the gradients of the output vectors have to be explicitely sorted here. c CsVt@||_||_|d|j}t|d|}t|d|}W5QRX||fS)NrLr-)rrrrrvrMrNr)ctxrrrrrZexpanded_undo_sort_indicesrrrr`s zReverseSort.forwardcCs|j}|j}|j}|j}||dd|df}||dd|df|dd}t||jdd|df}|d|j}t|d|}t|d|}t||}t||}||dddfS)Nr-rLr|) rrrNr^rrZrvrMr)rZgrad_out_vectorsZ grad_logitsrrZgrad_logits_shapeZgrad_out_vectors_shapeZexpanded_sort_indicesrrrbackwards  zReverseSort.backwardNrbrcrdre staticmethodr`rrrrrrs  rcs.eZdZfddZd ddZddZZS) LocalSelfAttentioncst|j|_|j|_|j|_|j|_|j |_ |j |_ |j |_ |j|j |_ |j |_ tj|j |j dd|_tj|j |j dd|_tj|j |j dd|_|j|_|dtd|dtddS)NFrrrrr)r:r;rr2rZlocal_num_chunks_beforerZlocal_num_chunks_afterrr pad_token_idrrrBrrquerykeyrZ"local_attention_probs_dropout_probr>rrrrkrJrrr;s  zLocalSelfAttention.__init__NFcKs|jd}|jd}||}||} ||} |||j|j}|| |j|j} || |j|j} |jd|jkstd|jd|j| jd|jkstd| jd|j| jd|jkstd| jd|j|j dkr |j dkr|j dks td| t t j|j| j| jd} ||d|j |j|j}|| d|j |j|j} || d|j |j|j} t j||jd||jd} || d|j |j} || d|j |j} || |j |j } || |j |j } || |j |j } t || dd}~~ || | ||j}|dk rP|jt jkr<|j}n|j}t |||}~t j|dd d }t ||}~tj j!||j!|j"d }|dk r||}t || }~ |j#d d d}|j||j||jfkst|$||j|j}|dkrd}t%||dS)NrrrLrrrrrTrrVr-r|rFr)rr)&rNrrrrrrrCr5rrrrsqrtrrrr9rrtrrrr]rrrrrrrrrrr>rXrrr!)rIrrrrrr[rRrrrrrrrrrrrrrrrr`s           zLocalSelfAttention.forwardcCsd}|dk rN|tjdddddf}||d|jd}|||j|j}|jdkrxt | d| d|j }|dk r| d| d |}|dk r||}n|}|S)NrLrTr) rrrrrrrrrrrvrrrM)rIrrrZquery_key_dots_shaperZattention_mask_keyrrrrrns   z%LocalSelfAttention._compute_attn_mask)NNF)rbrcrdr;r`rrfrrrJrrs  srcs$eZdZfddZddZZS)ReformerSelfOutputcs6t|j|j}|j|_tj||jdd|_ dS)NFr) r:r;rrr=r>rrrBdense)rIr6rrJrrr;s  zReformerSelfOutput.__init__cCs$||}tjj||j|jd}|Srlrrrr>rXrIrrrrr`s zReformerSelfOutput.forwardrbrcrdr;r`rfrrrJrrs rcs(eZdZdfdd Zd ddZZS) ReformerAttentionrcst||_|j|_tj|j|jd|_t t |jdkrX|jddkrXt ||_ nt t |jdkr|jddkrt ||_ n`t t |jdkrt |jt ddgkr|j|jdkrt ||_ qt ||_ ntd|jt||_dS)NZepsrrr+r,r-zOnly attn layer types 'lsh' and 'local' exist, but got `config.attn_layers`: {}. Select attn layer types from ['lsh', 'local'] only.)r:r;layer_idr.r LayerNormrBlayer_norm_eps layer_normr0r/rself_attentionrr4r5routputrIr6rrJrrr;s$     (  zReformerAttention.__init__NFc CsR||}|j||||||d}||j}t|dr>|j}nd}t||j|dS)N)rrrrrr r r)rrrrhasattrr r"r) rIrrrrrr Zself_attention_outputsattention_outputrrrr`s$   zReformerAttention.forward)r)NNNFNrrrrJrrsrcs$eZdZfddZddZZS)ReformerFeedForwardDensecsJt|j|_t|jtr,t|j|_n|j|_t |j |j |_ dSr)r:r;r=r>rZ hidden_actstrACT2FNact_fnrrrBfeed_forward_sizerrkrJrrr;s   z!ReformerFeedForwardDense.__init__cCs.||}tjj||j|jd}||}|Srl)rrrr>rXrrrrrr`s  z ReformerFeedForwardDense.forwardrrrrJrrs rcs$eZdZfddZddZZS)ReformerFeedForwardOutputcs(t|j|_t|j|j|_dSr) r:r;r=r>rrrrBrrkrJrrr;s z"ReformerFeedForwardOutput.__init__cCs$||}tjj||j|jd}|Srlrrrrrr`s z!ReformerFeedForwardOutput.forwardrrrrJrrs rcs,eZdZfddZddZddZZS)ChunkReformerFeedForwardcsDt|j|_d|_tj|j|jd|_t ||_ t ||_ dS)Nrr) r:r;chunk_size_feed_forward seq_len_dimrrrBrrrrrrrkrJrrr;s   z!ChunkReformerFeedForward.__init__cCst|j|j|j|Sr)rrr forward_chunk)rIrrrrr`s z ChunkReformerFeedForward.forwardcCs||}||}||Sr)rrrrrrrrs  z&ChunkReformerFeedForward.forward_chunkrbrcrdr;r`rrfrrrJrrs rcsBeZdZdfdd ZddZddZdd d Zdd d ZZS) ReformerLayerrcs0tt|||_d|_d|_t||_dSr)r:r;r attentionattention_seedfeed_forward_seedr feed_forwardrrJrrr;s   zReformerLayer.__init__cCsdt|jjdkr@tj}tjj||_ tj |j n t tt j |_ t |j dS)a This function sets a new seed for the attention layer to make dropout deterministic for both forward calls: 1 normal forward call and 1 forward call in backward to recalculate activations. cudaN)next parametersrrtyperr current_devicedefault_generatorsseedrrrsysmaxsizerIZ device_idxrrr_init_attention_seeds  z"ReformerLayer._init_attention_seedcCsdt|jjdkr@tj}tjj||_ tj |j n t tt j |_ t |j dS)a This function sets a new seed for the feed forward layer to make dropout deterministic for both forward calls: 1 normal forward call and 1 forward call in backward to recalculate activations. r N)r r rrr rr r rrrrrrrrrrr_init_feed_forward_seed$s  z%ReformerLayer._init_feed_forward_seedNFc CsjtH||j|||||d}|j}||}~||||}W5QRXt|||j|j dS)N)rrrrr)r$rrr ) rrrrrrrr#rr ) rIprev_attn_outputrrrrrZ attn_outputsr$rrrr`8s( zReformerLayer.forwardc Cst0d|_t|j||}|j|ddW5QRXt ||}~||j}d|_W5QRXt:d|_t|j |j ||||dj } | j|ddW5QRXt*|| } ~ ~||j}d|_| }W5QRXt | |||dS)NT)Z retain_graph)rrrr r$rr&r')rZ enable_gradZ requires_gradrrrrrZgradrrrrr%) rInext_attn_outputrr&r'rrr Zres_hidden_statesrr$rrr backward_passds@         zReformerLayer.backward_pass)r)NNNF)NNN) rbrcrdr;rrr`rrfrrrJrrs  2rc@s(eZdZdZeddZeddZdS)_ReversibleFunctionak To prevent PyTorch from performing the usual backpropagation, a customized backward function is implemented here. This way it is made sure that no memory expensive activations are saved during the forward pass. This function is heavily inspired by https://github.com/lucidrains/reformer-pytorch/blob/master/reformer_pytorch/reversible.py c  Csd} tj|ddd\}} t||D]V\} } |dkr<||| | ||| || d}|j} |j}| |jf} | r"||jq"|dkr|||| | ||_ | |_ ||_ ||_ tj| |gddS)Nrr-rLrTT)rrrrrr)rchunkziprFr$rr rZsave_for_backwardrlayers all_bucketsrrr\)rrrrrrr)r*do_output_hidden_statesrrr$layerZlayer_head_maskZ layer_outputsrrrr`s4    z_ReversibleFunction.forwardc Cstj|ddd\}}|j\}}t||||d}~~~~|j}|j}|j}|j} t|dddD]L\} } |d} |dd}| j |j |j |j |j |t|| d| | d}q`|dkstdtj|j |j gdd}|ddddddddf S) Nr-rLrTrr)rrr&r'rrr rz.buckets have to be empty after backpropagation)rrZ saved_tensorsr%rrrrrDrr$rr&r'r0rCr\) rr'r&r$rrrrrridxrr rrrrs8   z_ReversibleFunction.backwardNrrrrrrs  2rcs&eZdZfddZdddZZS)ReformerEncodercsNtj|_tfddtjD|_tj dj j d|_ dS)Ncsg|]}t|qSr)r)rPrr6rrrSsz,ReformerEncoder.__init__..r-r) r:r;r=r>rZ ModuleListr}num_hidden_layersrrrBrrrkrJr"rr; s  zReformerEncoder.__init__NFc Csdg}g}tj||gdd}t||j||||||| }||}tjj||j|j d}t |||dS)NrLrTrV)rr)r*) rr\rrrrrrr>rXr() rIrrrrrrr)r*rrrr`s*  zReformerEncoder.forward)NNNFFrrrrJrr! s r!cs,eZdZfddZddZddZZS)ReformerOnlyLMHeadcsTtd|_|j|_tjd|j|jdd|_t t |j|_ |j |j_ dS)Nrr-Fr) r:r;rchunk_size_lm_headrrrBrodecoderrGrzerosrrkrJrrr;=s  zReformerOnlyLMHead.__init__cCst|j|j|j|Sr)rr%rrrrrrr`IszReformerOnlyLMHead.forwardcCs||}|Sr)r&rrrrrLs z ReformerOnlyLMHead.forward_chunkrrrrJrr$<s r$c@s0eZdZdZeZeZdZe ddZ ddZ dS)ReformerPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. reformercCs"tt}tt}||d}|S)N)rwr)rrrr)rIrwZ input_mask dummy_inputsrrrr*Zs   z$ReformerPreTrainedModel.dummy_inputscCst|tr.|jD]}tjjj||jjdqnnt|tj rR|j j jd|jj dnJt|tj rv|j j jd|jj dn&t|tjr|jj |j j dt|tj r|jdk r|jj dS)z Initialize the weights )stdg)rr+g?N)rr8r@rrinitZnormal_r6Zaxial_norm_stdrhrQdataZinitializer_rangerrrZzero_Zfill_)rImodulerQrrr _init_weightsds      z%ReformerPreTrainedModel._init_weightsN) rbrcrdrer Z config_class%REFORMER_PRETRAINED_MODEL_ARCHIVE_MAPZpretrained_model_archive_mapZbase_model_prefixpropertyr*r/rrrrr(Qs r(uJ Reformer was proposed in `Reformer: The Efficient Transformer`_ by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya. .. _`Reformer: The Efficient Transformer`: https://arxiv.org/abs/2001.04451 This model is a PyTorch `torch.nn.Module `_ sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.ReformerConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. During training the input_ids sequence_length has to be a multiple of the relevant model's chunk lengths (lsh's, local's or both). During evaluation, the indices are automatically padded to be a multiple of the chunk length. Indices can be obtained using :class:`transformers.ReformerTokenizer`. See :func:`transformers.PreTrainedTokenizer.encode` and :func:`transformers.PreTrainedTokenizer.encode_plus` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens. `What are attention masks? <../glossary.html#attention-mask>`__ position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**. inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. num_hashes (:obj:`int`, `optional`, defaults to :obj:`None`): `num_hashes` is the number of hashing rounds that should be performed during bucketing. Setting `num_hashes` overwrites the default `num_hashes` defined in `config.num_hashes`. For more information, see `num_hashes` in :class:`transformers.ReformerConfig`. zaThe bare Reformer Model transformer outputting raw hidden-stateswithout any specific head on top.c sPeZdZfddZddZddZddZeedd d Z dd dZ Z S) ReformerModelcsFt|||_|jjdks&tdt||_t||_| dS)NrzS`config.attn_layers` is empty. Select at least one attn layer form ['lsh', 'local']) r:r;r6r#rCrnrzr!encoder init_weightsrkrJrrr;s    zReformerModel.__init__cCs|jjSrrzrprIrrrget_input_embeddingssz"ReformerModel.get_input_embeddingscCs ||j_dSrr5)rIrrrrset_input_embeddingssz"ReformerModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr3rrZ prune_heads)rIZheads_to_prunerZheadsrrr _prune_headsszReformerModel._prune_headsNFc  Cs|jj}|jj}|dk r*|dk r*tdn@|dk rB|} |j} n(|dk rb|dd} |j} ntdt| dkstd| |j ||jj dd}| d} t |j} | d| d k} | r| | d| }|j dkrtd | d| | d||j ||||| || | d \}}}}} |j|||d }|j||||||d }|j}| rb|ddd| f}|f}|dkr~||jf}|dkr||jf}|S)a Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. all_hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. all_attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``do_output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import ReformerModel, ReformerTokenizer import torch tokenizer = ReformerTokenizer.from_pretrained('google/reformer-crime-and-punishment') model = ReformerModel.from_pretrained('google/reformer-crime-and-punishment') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids) last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple NzDYou cannot specify both input_ids and inputs_embeds at the same timerLz5You have to specify either input_ids or inputs_embedsr-zO`input_ids` have be of shape `[batch_size, sequence_length]`, but got shape: {}T)Zis_attention_chunkedrzIf training, sequence Length {} has to be a multiple of least common multiple chunk_length {}. Please consider padding the input to a length of {}.)rxrr_rypadding_lengthpadded_seq_lengthrr)rwr_rx)rrrrrr)r6Zoutput_attentionsZoutput_hidden_statesrrsrrr0rCr5Z get_head_maskr#r7rX_pad_to_mult_of_chunk_lengthrzr3rr)r*)rIrwrr_rrxrrrryrrZorig_sequence_lengthr?Zmust_pad_to_match_chunk_lengthr;Zembedding_outputZencoder_outputssequence_outputoutputsrrrr`sr,          zReformerModel.forwardc CsRtd|d|d||tj|d|f|jj|tjd} |dk rptj|tj |d|||j dgdd}n6tjtj ||tj dtj |d|f|tj dgdd}|dk rtj|| gdd}| }|dk rtj|d|tj|d} |d|d|} tj|| gdd}|dk rD|| |} tj|| gdd}| }|||||fS)Nz^Input ids are automatically padded from {} to {} to be a multiple of `config.chunk_length`: {}rLrrrTrqr)rinfor5rfullr6rrur\r'r9rHrrsrtrvrMrz) rIrwrxrr_ryr;r<rrZpadded_input_idsZpadded_position_idsZpadded_inputs_embedsrrrr=FsL       z*ReformerModel._pad_to_mult_of_chunk_length)NNNNNNFF)NNNNNNN) rbrcrdr;r7r8r:rREFORMER_INPUTS_DOCSTRINGr`r=rfrrrJrr2s,  xr2z7Reformer Model with a `language modeling` head on top. c sFeZdZfddZddZddZeed d d Zd d Z Z S)ReformerModelWithLMHeadcs,t|t||_t||_|dSr)r:r;r2r)r$lm_headr4rkrJrrr;s   z ReformerModelWithLMHead.__init__cCs|jjSr)rDr&r6rrrget_output_embeddingssz-ReformerModelWithLMHead.get_output_embeddingscCsdSrrr6rrr tie_weightssz#ReformerModelWithLMHead.tie_weightsNFc  Cs|j|||||||| d} | d} || } | f| dd} |dk r| dddddf}|dddf}t}||d|jj|d}|f| } | S)a5 labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[-100, 0, ..., config.vocab_size - 1]`. All labels set to ``-100`` are ignored (masked), the loss is only computed for labels in ``[0, ..., config.vocab_size]`` Return: :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs: loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided): Classification loss (cross entropy). prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). all_hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. all_attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``do_output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Examples:: from transformers import ReformerModelWithLMHead, ReformerTokenizer import torch tokenizer = ReformerTokenizer.from_pretrained('google/reformer-crime-and-punishment') model = ReformerModelWithLMHead.from_pretrained('google/reformer-crime-and-punishment') input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) # Batch size 1 outputs = model(input_ids, labels=input_ids) loss, prediction_scores = outputs[:2] )r_rrrxrrrrrN.rL)r)rD contiguousrr^r6ro)rIrwr_rrrxrlabelsrrZreformer_outputsr>rr?Z shift_logitsZ shift_labelsZloss_fctZlossrrrr`s(4   zReformerModelWithLMHead.forwardcKs d|i}d|kr|d|d<|S)Nrwrr)rIrwZpastrZ inputs_dictrrrprepare_inputs_for_generations z5ReformerModelWithLMHead.prepare_inputs_for_generation) NNNNNNNFF) rbrcrdr;rErFrrBr`rIrfrrrJrrCs  LrC)Ereloggingr collectionsr functoolsroperatorrZnumpyr3rrZtorch.autograd.functionrZtorch.nnrZ activationsr r r r Zconfiguration_reformerr Z file_utilsrrrrZmodeling_utilsrr getLoggerrbrr0rrrrrr!r"r#r%r(r7Moduler8rgrnr{rrrrrrrrrrr!r$r(ZREFORMER_START_DOCSTRINGrBr2rCrrrrs          O-:9'; k0&(L