U Uc@sdZddlmZddlmZddlZddlZddlZddlZddlm Z ddl Z ddl Z ddlmZddlmZmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZGddde Z!Gddde Z"Gddde Z#Gddde Z$ddZ%ddZ&ddZ'ddZ(d d!Z)d"d#Z*d$d%Z+d&d'Z,d(d)Z-d*d+Z.d,d-Z/d.d/Z0d0d1Z1d2d3Z2d4d5Z3d6d7Z4d8d9Z5d:d;Z6dz%Shared utils among inference plugins.)division)print_functionN)logging) json_format) binary_type string_types integer_types) iteritems)zip) signature) common_utils)platform_utils)classification_pb2) inference_pb2)regression_pb2c@seZdZdZddZdS) VizParamsaLight-weight class for holding UI state. Attributes: x_min: The minimum value to use to generate mutants for the feature (as specified the user on the UI). x_max: The maximum value to use to generate mutants for the feature (as specified the user on the UI). examples: A list of examples to scan in order to generate statistics for mutants. num_mutants: Int number of mutants to generate per chart. feature_index_pattern: String that specifies a restricted set of indices of the feature to generate mutants for (useful for features that is a long repeated field. See `convert_pattern_to_indices` for more details. c Cs|dd}dd}dd}|||_|||_||_|||_g|_|rxz|||_Wn"tk rv} zW5d} ~ XYnXdS)zDInits VizParams may raise InvalidUserInputError for bad user inputs.c Ss*z t|WSttfk r$YdSXdSN)float ValueError TypeErrorxrQ/tmp/pip-unpacked-wheel-15h0ro02/tensorboard_plugin_wit/_utils/inference_utils.pyto_float_or_none<s z,VizParams.__init__..to_float_or_nonec Ss@z t|WSttfk r:}zt|W5d}~XYnXdSr)intrrr InvalidUserInputError)rerrrto_intBs z"VizParams.__init__..to_intcSstdd|dD}g}|D]N}d|krXdd|ddD\}}|t||dq|t|qt|S)acConverts a printer-page-style pattern and returns a list of indices. Args: pattern: A printer-page-style pattern with only numeric characters, commas, dashes, and optionally spaces. For example, a pattern of '0,2,4-6' would yield [0, 2, 4, 5, 6]. Returns: A list of indices represented by the pattern. cSsg|] }|qSr)strip).0tokenrrr TszJVizParams.__init__..convert_pattern_to_indices..,-cSsg|]}t|qSr)rr)r rrrrr"Xs)splitextendrangeappendrrsorted)patternpiecesindicesZpiecelowerupperrrrconvert_pattern_to_indicesHs z6VizParams.__init__..convert_pattern_to_indicesN)x_minx_maxexamples num_mutantsfeature_indicesr) selfr1r2r3r4Zfeature_index_patternrrr0rrrr__init__8s    zVizParams.__init__N__name__ __module__ __qualname____doc__r7rrrrr(src@seZdZdZddZdS)OriginalFeatureListaLight-weight class for holding the original values in the example. Should not be created by hand, but rather generated via `parse_original_feature_from_example`. Just used to hold inferred info about the example. Attributes: feature_name: String name of the feature. original_value: The value of the feature in the original example. feature_type: One of ['int64_list', 'float_list']. Raises: ValueError: If OriginalFeatureList fails init validation. cCs4||_dd|D|_||_tdd|D|_dS)zInits OriginalFeatureList.cSsg|] }t|qSr)ensure_not_binaryr valuerrrr"sz0OriginalFeatureList.__init__..css|] }dVqdS)r%Nr)r _rrr sz/OriginalFeatureList.__init__..N) feature_nameoriginal_value feature_typesumlength)r6rCrDrErrrr7~s zOriginalFeatureList.__init__Nr8rrrrr=nsr=c@seZdZdZddZdS)MutantFeatureValueaLight-weight class for holding mutated values in the example. Should not be created by hand but rather generated via `make_mutant_features`. Used to represent a "mutant example": an example that is mostly identical to the user-provided original example, but has one feature that is different. Attributes: original_feature: An `OriginalFeatureList` object representing the feature to create mutants for. index: The index of the feature to create mutants for. The feature can be a repeated field, and we want to plot mutations of its various indices. mutant_value: The proposed mutant value for the given index. Raises: ValueError: If MutantFeatureValue fails init validation. cCsht|tstdt|||_|dk rFt|tsFtdt|||_t|tr^| n||_ dS)zInits MutantFeatureValue.zMoriginal_feature should be `OriginalFeatureList`, but had unexpected type: {}Nz8index should be None or int, but had unexpected type: {}) isinstancer=rformattypeoriginal_featurerindexrencode mutant_value)r6rLrMrOrrrr7s$  zMutantFeatureValue.__init__Nr8rrrrrHsrHc@seZdZdZdddZdS) ServingBundlea<Light-weight class for holding info to make the inference request. Attributes: inference_address: An address (such as "hostname:port") to send inference requests to. model_name: The Servo model name. model_type: One of ['classification', 'regression']. model_version: The version number of the model as a string. If set to an empty string, the latest model will be used. signature: The signature of the model to infer. If set to an empty string, the default signuature will be used. use_predict: If true then use the servo Predict API as opposed to Classification or Regression. predict_input_tensor: The name of the input tensor to parse when using the Predict API. predict_output_tensor: The name of the output tensor to parse when using the Predict API. estimator: An estimator to use instead of calling an external model. feature_spec: A feature spec for use with the estimator. custom_predict_fn: A custom prediction function. Raises: ValueError: If ServingBundle fails init validation. Nc Cst|tstdt||dddd|_t|tsNtdt|||_|dkrjtd|||_|r|t |nd|_ |r|nd|_ ||_ ||_ ||_| |_| |_| |_dS) zInits ServingBundle.z&Invalid inference_address has type: {}zhttp://zhttps://zInvalid model_name has type: {})classification regressionzInvalid model_type: {}N)rIrrrJrKreplaceinference_address model_name model_typer model_versionr use_predictpredict_input_tensorpredict_output_tensor estimator feature_speccustom_predict_fn) r6rUrVrWrXr rYrZr[r\r]r^rrrr7s0   zServingBundle.__init__)NNNr8rrrrrPs rPcCs6zt|tr|n|WStk r0|YSXdS)z#Return non-binary version of value.N)rIrdecodeUnicodeDecodeError)r@rrrr>sr>cCsNt||}|dkr"td||d}|dkrBtd|t||jS)zCGet the value of a feature from Example regardless of feature type.Nz#Feature {} is not on example proto.kindz1Feature {} on example proto has no declared type.)get_example_featuresrrJ WhichOneofgetattrr@)examplerCfeaturerErrrproto_value_for_features  rgcCs,t||}|d}t||}t|||S)zReturns an `OriginalFeatureList` for the specified feature_name. Args: example: An example. feature_name: A string feature name. Returns: A filled in `OriginalFeatureList` object representing the feature. ra)rbrcrgr=)rerCrfrErDrrr#parse_original_feature_from_examples   rhcCsBt}t|tjr$|j|jnt|tj r>|j |j|S)zReturns packaged inference results from the provided proto. Args: inference_result_proto: The classification or regression response proto. Returns: An InferenceResult proto with the result from the response. ) rZInferenceResultrIrClassificationResponseZclassification_resultZCopyFromresultrRegressionResponseZregression_result)inference_result_protoZinference_protorrrwrap_inference_resultss  rmcs$dt|tfddDS)zReturns a list of feature names for float and int64 type features. Args: example: An example. Returns: A list of strings of the names of numeric features. ) float_list int64_listcs"g|]}|dkr|qS)rarcr rCfeaturesZnumeric_featuresrrr"-sz-get_numeric_feature_names..rbr*rerrrrget_numeric_feature_names"s rvcst|tfddDS)zReturns a list of feature names for byte type features. Args: example: An example. Returns: A list of categorical feature names (e.g. ['education', 'marital_status'] ) cs"g|]}|ddkr|qS)ra bytes_listrprqrsrrr"=sz1get_categorical_feature_names..rtrurrxrget_categorical_feature_names3s  rycCsNtt}|D],}t|D]}t||}|||jqqddt|DS)zReturns numerical features and their observed ranges. Args: examples: Examples to read to get ranges. Returns: A dict mapping feature_name -> {'observedMin': 'observedMax': } dicts, with a key for each numerical feature. cSs$i|]\}}|t|t|dqS))Z observedMinZ observedMax)minmax)r rCfeature_valuesrrr Ss z:get_numeric_features_to_observed_range..) collections defaultdictlistrvrhr'rDr )r3observed_featuresrerCrLrrr&get_numeric_features_to_observed_rangeCs  rc Cstt}|D],}t|D]}t||}|||jqqi}tt|D]2\}}ddt | |D}|rLd|i||<qL|S)aReturns categorical features and a sampling of their most-common values. The results of this slow function are used by the visualization repeatedly, so the results are cached. Args: examples: Examples to read to get feature samples. top_k: Max number of samples to return per feature. Returns: A dict of feature_name -> {'samples': ['Married-civ-spouse', 'Never-married', 'Divorced']}. There is one key for each categorical feature. Currently, the inner dict just has one key, but this structure leaves room for further expansion, and mirrors the structure used by `get_numeric_features_to_observed_range`. cSsg|]\}}|dkr|qS)r%r)r wordcountrrrr"ysz8get_categorical_features_to_sampling..samples) r~rrryrhr'rDr*r Counter most_common) r3Ztop_krrerCrLrjr|rrrr$get_categorical_features_to_sampling\s$   rc s|j}|j}|j}|j}jdkr@fddt|||DSjdkrtt|t||t }t t |}fdd|DSjdkrt ||}|j d}fdd|DStd jd S) zEReturn a list of `MutantFeatureValue`s that are variants of original.rncsg|]}t|qSrrHr?index_to_mutaterLrrr"sz(make_mutant_features..rocsg|]}t|qSrrr?rrrr"srwrcsg|]}td|qSrrr?)rLrrr"sz(Malformed original feature had type of: N)r1r2r3r4rEnpZlinspacerZastypetolistr*setrrCr) rLr viz_paramsr.r/r3r4 mutant_valuesZfeature_to_samplesrrrmake_mutant_featuress<        rc Cst|||}g}|D]}|D]}t|}|jj} zNt|| } |dkrN|j} nt| } |j| |<| dd=| | | |Wqt t fk r| |YqXqq||fS)aReturn a list of `MutantFeatureValue`s and a list of mutant Examples. Args: example_protos: The examples to mutate. original_feature: A `OriginalFeatureList` that encapsulates the feature to mutate. index_to_mutate: The index of the int64_list or float_list to mutate. viz_params: A `VizParams` object that contains the UI state of the request. Returns: A list of `MutantFeatureValue`s and a list of mutant examples. N) rcopydeepcopyrLrCrgrOrr'r)r IndexError) example_protosrLrrmutant_featuresmutant_examplesZ example_protomutant_featureZcopied_examplerCZ feature_listZ new_valuesrrrmake_mutant_tupless(     rc sfddztd|Wn0tk rT}zdgdWYSd}~XYnXjpdtj}jdkrtdnd}z|fd d |DdWStk r}zt|W5d}~XYnXdS) a Returns JSON formatted for rendering all charts for a feature. Args: example_proto: The example protos to mutate. feature_name: The string feature name to mutate. serving_bundles: One `ServingBundle` object per model, that contains the information to make the serving request. viz_params: A `VizParams` object that contains the UI state of the request. Raises: InvalidUserInputError if `viz_params.feature_index_pattern` requests out of range indices for `feature_name` within `example_proto`. Returns: A JSON-able dict for rendering a single mutant chart. parsed in `tf-inference-dashboard.html`. { 'chartType': 'numeric', # oneof('numeric', 'categorical') 'data': [A list of data] # parseable by vz-line-chart or vz-bar-chart } csDt|\}}g}D]$}t||\}}|t|||q|Sr)r run_inferencer)$make_json_formatted_for_single_chart)rrrchartsserving_bundlerlrA)rrLserving_bundlesrrrchart_for_indexs$z2mutant_charts_for_feature..chart_for_indexrZ categorical) chartTypedataNrwnumericcsg|] }|qSrr)r r)rrrr"sz-mutant_charts_for_feature..) rhrr5r(rGrErr r)rrCrrrZindices_to_mutateZ chart_typer)rrrLrrrmutant_charts_for_features0  rc s2dd}t|tjrPi}t|jjD]\}}||t|}t|jD]\}} | jdkrdt || _t|jdkr~| jdkr~qH| j} |r| d|7} | |kri|| <t |j } | || krg|| | <|| |  | j qHq&tt} t|D]Z\} } t| D]0\}}| |  ||t|tt|iq| | jfddd q| St|tjr&i}t|jjD]F\}}||t|}t |j } | |krg|| <||  |jqnd } |d kr| d|7} g}t|D],\}}| ||t|tt|iq|jfd dd | |iStd dS)aReturns JSON formatted for a single mutant chart. Args: mutant_features: An iterable of `MutantFeatureValue`s representing the X-axis. inference_result_proto: A ClassificationResponse or RegressionResponse returned by Servo, representing the Y-axis. It contains one 'classification' or 'regression' for every Example that was sent for inference. The length of that field should be the same length of mutant_features. index_to_mutate: The index of the feature being mutated for this chart. Returns: A JSON-able dict for rendering a single mutant chart, parseable by `vz-line-chart` or `vz-bar-chart`. stepscalarrQ0z (index %d)cs|SrrpZx_labelrrQz6make_json_formatted_for_single_chart..)keyr@rcs|Srrrrrrrkrz/Only classification and regression implemented.N)rIrri enumeraterjZclassificationslenclasseslabelstrr>rOr)Zscorer~rrr rFrsortrrkZ regressionsr@NotImplementedError)rrlrZy_labelseriesidxrRrZ class_indexZclassification_classrZ mutant_valZ return_seriesrr@Zy_listZpointsrSZlist_of_pointsrrrrs                rcCst|tjjr|jjS|jjS)z.T)rreverse)rrvaluesr(rabsrr*)rZ chart_dataZsorted_features_listrfrrZ max_measureZ is_numericmodelsZchartrZmeasureiZmin_yZmax_yvalrrrsort_eligible_featuress<    (   rc Cst|rpz8tjj|d}dd|DW5QRWSQRXWn2tjjk rn}ztd|W5d}~XYnXgS)z>Returns a list of label strings loaded from the provided path.rcSsg|]}|dqS) )rstrip)r linerrrr"sz#get_label_vocab..zerror reading vocab file: %sN)rioZgfileZGFileerrorsZ NotFoundErrorrerror)Z vocab_pathferrrrrget_label_vocabs&rc sdd}d}d}tjj|tjjdtjddi}tj||}tddddgtj }t d }||ft t |||}fd d } tj fd d | |||g| | tdgd} || d} | W5QRSQRXdS)zReturns an encoded sprite image for use in Facets Dive. Args: examples: A list of serialized example protos to get images for. Returns: An encoded PNG. cSst|d}ttt|}|d}|d}||}||}d}t|||g} t |D]\\} } | |} tt | |} | |}||}| |}||}| | ||||ddf<qftj tj | tjdS)z8Generates a sprite atlas image from a set of thumbnails.rr%N)Zdtype)rshapeevalrmathceilsqrtrzerosrfloorimageZ encode_pngcastZuint8)Z thumbnailsthumbnail_dimsZnum_thumbnailsZimages_per_rowZ thumb_heightZ thumb_widthZ master_heightZ master_widthZ num_channelsZmasterrrZleft_idxZtop_idxZ left_startZleft_endZ top_startZtop_endrrrgenerate_image_from_thubnailss"z:create_sprite_image..generate_image_from_thubnailsz image/encoded rrQ) default_valuer%rcsh||}tjj|dd}tj|}t|dtt|dfddfdd|d|fS)Nr)ZchannelsrcsSrrr)expanded_imagerrrrz8create_sprite_image..loop_body..cstgdS)Nr)rconcatrrimagesrrrrr%)rrZ decode_jpegresizeZ expand_dimsZcondequal)rencoded_imagesrZ encoded_imagerZ resized_image)rrr loop_body s   z&create_sprite_image..loop_bodycs t|Sr)rZless)rrr) num_examplesrrrrz%create_sprite_image..N)Zshape_invariantsr)rcompatZv1SessionrZFixedLenFeaturestring parse_examplerZfloat32ZconstantrZ while_loopZ get_shapeZ TensorShaper) r3rZimage_feature_nameZsprite_thumbnail_dim_pxZkeys_to_featuresparsedrrrrZloop_outZspriter)rrrcreate_sprite_images6     rc sDdjrjrjfdd}jr4jnd}g}|D]d}|dkrt|}t|dkrn|d}njdkr~d}nd }||krt d || ||q@t |dfSj r0tj }|j}t|dkr }t|d kr }d} t|tr|d }|} n|}t || fStdfSdS) aRun inference on examples given model information Args: examples: A list of examples that matches the model spec. serving_bundle: A `ServingBundle` object that contains the information to make the inference request. Returns: A tuple with the first entry being the ClassificationResponse or RegressionResponse proto and the second entry being a dictionary of extra data for each example, such as attributions, or None if no data exists. @cs*tjjtjddDjS)NcSsg|] }|qSr)ZSerializeToString)r exrrrr"5sz3run_inference....)rrZDatasetZfrom_tensor_slicesrrr]batchrZ batch_sizer3rrrr4s zrun_inference..Nr%rrRZ probabilitiesZ predictionsz."%s" not found in model predictions dictionaryr)r\r]ZpredictrYr[rkeysrrWKeyErrorr)r Zconvert_prediction_valuesr^r parametersrIdictpopr Z call_servo) r3rpredsZ key_to_userpredZ returned_keyssigparamsrrrrr#sT               r)8r< __future__rrr~rrrZabslrZnumpyrZ tensorflowrZgoogle.protobufrsixrrrr Z six.movesr inspectr Ztensorboard_plugin_wit._utilsr r Z6tensorboard_plugin_wit._vendor.tensorflow_serving.apisrrrobjectrr=rHrPr>rgrhrmrvryrrrrrrrbrrrrrrrrrrsT            F$;  '%+=b 5 F