U &ºc"ã@s>dZddlZddlmZe e¡ZddiZGdd„deƒZdS) z Transformer XL configuration éNé)ÚPretrainedConfigztransfo-xl-wt103zPhttps://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-config.jsoncs´eZdZdZeZdZddddgdddd d d d d dddddddddddddddddddf‡fdd„ Zedd„ƒZ edd„ƒZ e j d d„ƒZ ed!d"„ƒZ ed#d$„ƒZ ed%d&„ƒZ‡ZS)'ÚTransfoXLConfiga™ This is the configuration class to store the configuration of an :class:`~transformers.TransfoXLModel`. It is used to instantiate a Transformer XL model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the `Transformer XL `__ architecture. Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information. Args: vocab_size (:obj:`int`, optional, defaults to 267735): Vocabulary size of the Transformer XL model. Defines the different tokens that can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.TransfoXLModel`. cutoffs (:obj:`List[int]`, optional, defaults to :obj:`[20000, 40000, 200000]`): Cutoffs for the adaptive softmax d_model (:obj:`int`, optional, defaults to 1024): Dimensionality of the model's hidden states. d_embed (:obj:`int`, optional, defaults to 1024): Dimensionality of the embeddings n_head (:obj:`int`, optional, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. d_head (:obj:`int`, optional, defaults to 64): Dimensionality of the model's heads. d_inner (:obj:`int`, optional, defaults to 4096): Inner dimension in FF div_val (:obj:`int`, optional, defaults to 4): Divident value for adapative input and softmax pre_lnorm (:obj:`boolean`, optional, defaults to :obj:`False`): Apply LayerNorm to the input instead of the output n_layer (:obj:`int`, optional, defaults to 18): Number of hidden layers in the Transformer encoder. tgt_len (:obj:`int`, optional, defaults to 128): Number of tokens to predict ext_len (:obj:`int`, optional, defaults to 0): Length of the extended context mem_len (:obj:`int`, optional, defaults to 1600): Length of the retained previous heads clamp_len (:obj:`int`, optional, defaults to 1000): use the same pos embeddings after clamp_len same_length (:obj:`boolean`, optional, defaults to :obj:`True`): Use the same attn length for all tokens proj_share_all_but_first (:obj:`boolean`, optional, defaults to :obj:`True`): True to share all but first projs, False not to share. attn_type (:obj:`int`, optional, defaults to 0): Attention type. 0 for Transformer-XL, 1 for Shaw et al, 2 for Vaswani et al, 3 for Al Rfou et al. sample_softmax (:obj:`int`, optional, defaults to -1): number of samples in sampled softmax adaptive (:obj:`boolean`, optional, defaults to :obj:`True`): use adaptive softmax tie_weight (:obj:`boolean`, optional, defaults to :obj:`True`): tie the word embedding and softmax weights dropout (:obj:`float`, optional, defaults to 0.1): The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler. dropatt (:obj:`float`, optional, defaults to 0): The dropout ratio for the attention probabilities. untie_r (:obj:`boolean`, optional, defaults to :obj:`True`): Untie relative position biases init (:obj:`string`, optional, defaults to `normal`): Parameter initializer to use init_range (:obj:`float`, optional, defaults to 0.01): Parameters initialized by U(-init_range, init_range). proj_init_std (:obj:`float`, optional, defaults to 0.01): Parameters initialized by N(0, init_std) init_std (:obj:`float`, optional, defaults to 0.02): Parameters initialized by N(0, init_std) layer_norm_epsilon (:obj:`float`, optional, defaults to 1e-5): The epsilon to use in the layer normalization layers Example:: from transformers import TransfoXLConfig, TransfoXLModel # Initializing a Transformer XL configuration configuration = TransfoXLConfig() # Initializing a model from the configuration model = TransfoXLModel(configuration) # Accessing the model configuration configuration = model.config Attributes: pretrained_config_archive_map (Dict[str, str]): A dictionary containing all the available pre-trained checkpoints. z transfo-xli×i Ni@œi@ iéé@iéFéé€ri@ièTéÿÿÿÿgš™™™™™¹?gÚnormalg{®Gáz„?g{®Gáz”?gñh㈵øä>c sþtƒjfd|i|—Ž||_g|_|j |¡||_|rRdgdgt|jƒ|_ndgdgt|jƒ|_||_||_ ||_ ||_ ||_ | |_ | |_||_| |_| |_| |_||_||_||_||_||_||_||_||_||_||_||_||_||_dS)NÚ eos_token_idFT) ÚsuperÚ__init__Ú vocab_sizeÚcutoffsÚextendÚ tie_weightÚlenZ tie_projsÚd_modelÚd_embedÚd_headÚd_innerÚdiv_valÚ pre_lnormÚn_layerÚn_headÚtgt_lenÚext_lenÚmem_lenÚ same_lengthÚ attn_typeÚ clamp_lenÚsample_softmaxÚadaptiveÚdropoutÚdropattÚuntie_rÚinitÚ init_rangeÚ proj_init_stdÚinit_stdÚlayer_norm_epsilon)Úselfrrrrrrrrrrrrrr!rZproj_share_all_but_firstr r"r#rr$r%r&r'r(r)r*r+r Úkwargs©Ú __class__©úI/tmp/pip-unpacked-wheel-ymerj3tt/transformers/configuration_transfo_xl.pyrzs@! zTransfoXLConfig.__init__cCs|j|j|jS©N)rrr©r,r0r0r1Úmax_position_embeddings¾sz'TransfoXLConfig.max_position_embeddingscCs|jSr2©rr3r0r0r1Ún_tokenÂszTransfoXLConfig.n_tokencCs ||_dSr2r5)r,Úvaluer0r0r1r6ÆscCs|jSr2)rr3r0r0r1Ú hidden_sizeÊszTransfoXLConfig.hidden_sizecCs|jSr2)rr3r0r0r1Únum_attention_headsÎsz#TransfoXLConfig.num_attention_headscCs|jSr2)rr3r0r0r1Únum_hidden_layersÒsz!TransfoXLConfig.num_hidden_layers)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú(TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAPZpretrained_config_archive_mapZ model_typerÚpropertyr4r6Úsetterr8r9r:Ú __classcell__r0r0r.r1rsZWâD     r) r>ÚloggingZconfiguration_utilsrÚ getLoggerr;Úloggerr?rr0r0r0r1Ús  ÿ