Diagnosing Quantization Accuracy Loss with Neural Insights
Easily Identify the Operators Causing Accuracy Loss
Agata Radys, Zehao Huang, Suyue Chen, and Bartosz Myrcha, Intel Corporation
Accuracy loss is a common problem when quantizing deep learning models. There is a tradeoff between reducing the computational and storage requirements of a model while maintaining accuracy. In this article we will quantize the TensorFlow inception_v3 model with Intel Neural Compressor, then go through a step-by-step example to diagnose accuracy loss using Neural Insights, a component of Intel Neural Compressor.
More information about these tools can be found in a previous blog:
Preparation
First, install Intel Neural Compressor and Neural Insights:
# Install Neural Compressor
pip install neural-compressor
# Install Neural Insights
pip install neural-insightsThen, install the libraries needed by the TensorFlow inception_v3 model:
git clone https://github.com/intel/neural-compressor.git
cd examples/tensorflow/image_recognition/tensorflow_models/inception_v3/quantization/ptq
pip install -r requirements.txtDownload the pretrained model in PB format:
wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/inceptionv3_fp32_pretrained_model.pbDownload the dataset from ImageNet and convert it to the TFRecord format:
cd examples/tensorflow/image_recognition/tensorflow_models/
bash prepare_dataset.sh --output_dir=./inception_v3/quantization/ptq/data --raw_dir=/PATH/TO/img_raw/val/ --subset=validation
bash prepare_dataset.sh --output_dir=./inception_v3/quantization/ptq/data --raw_dir=/PATH/TO/img_raw/train/ --subset=train Quantization
Before applying quantization, we need to modify some code to enable Neural Insights. Set the argument “diagnosis” to be True in “PostTrainingQuantConfig” so that Neural Insights will dump weights and activations of quantizable operators (ops) in this model. Delete the “op_name_dict” argument because this will be the answer of our investigation. The configuration should look like this:
# set 'diagnosis' to True
config = PostTrainingQuantConfig(approach="static", quant_format="QOperator", quant_level=1, diagnosis=True) Execute Neural Insights from the command-line: neural_insights. You will get a webpage where you can find the visible graph of weights and activations:
Neural Insights Server started.
Open address […]Quantize the model with the following command:
bash run_tuning.sh - input_model=/PATH/TO/inceptionv3_fp32_pretrained_model.pb \
- output_model=./nc_inception_v3.pb - dataset_location=/path/to/ImageNet/Quantization Results
The default strategy for quantization is to convert all ops to int8. In this case, the accuracy drops significantly, from 74.62% to 21.03%:
Along with the quantization results, we can also get a summary of the activations. The table shows all ops sorted by MSE (Mean Square Error) with their activation minimum and maximum value:
Usually, MSE is a leading indicator of activations that cause accuracy losses, but this is not always the case. Another thing worth checking is the min-max data range. A wide data range usually leads to higher accuracy loss. It is also useful to find outliers and try to fall back these ops and test for quantization accuracy. We can also check the weights histogram from channel level because we may find some different layouts between channels. In this case, try per-channel (for weights only) quantization, so each channel could have its own zero point and scale factor.
In the second table, there is information about ops weights. They are also sorted by MSE with minimum, maximum, mean, and standard deviation values included:
In the next steps, we can verify if it is true that ops with the highest MSE cause the highest accuracy loss.
Accuracy Loss Diagnosis
As an experiment, we can disable the quantization of some ops with highest MSE in both tables (accuracy and weights) and rerun the quantization. If we disable the top-5 MSE ops, the result is unsatisfactory as we still have excessive accuracy drop (from 74.62% to 21.03%):
Therefore, we are going to refer to the distribution of weights and activations of this model shown in histograms in Neural Insights. Weights of the ops are usually distributed in one spike, as in the following charts:
These charts show the distribution of weight values per channel for each op. The horizontal axis stands for values of weights and the vertical axis represents the density or frequency of a weight value. During quantization, distributions like these mean that this small range can be mapped to a wide range of integers, providing more precision. Going through the weights histograms of all ops in this model, we find an op that is an outlier:
In this op, the minimum and maximum values of weights (data range) are high because there are many outliers, so the range cannot be clipped. The values near the zero point, which are the majority, will be mapped to a small range in int8. This leads to a huge accuracy loss. Additionally, because the min-max values vary across channels, the accuracy will decrease if we do not use channel-wise quantization.
Therefore, we can disable quantization for the op that we found in the previous step:
op_name_dict = {'v0/cg/conv0/conv2d/Conv2D': {'activation': {'dtype': ['fp32']}}}
conf = PostTrainingQuantConfig(calibration_sampling_size=[50, 100], op_name_dict=op_name_dict) Now, when we run quantization again, the results now look much better. The baseline accuracy and accuracy after quantization have the same value 74.62%. In other words, there is no accuracy loss anymore:
This is why the published version of this example already sets this op to stay as FP32. As you might imagine, figuring out the one op to disable quantization for can take a long time without having the insights provided by this tool.
Summary and Future Work
Our experiment shows that some common diagnostic rules are not always true (e.g., higher MSE means larger accuracy loss). Sometimes we must look for outliers. Neural Insights can help diagnose accuracy loss.
Neural Insights is currently available for TensorFlow and ONNX. We plan to support more frameworks in an upcoming release. Check out our GitHub repository for more information. If you have suggestions or feedback, feel free to create a pull request, submit issues, or contact us by email(neural.insights@intel.com).
