| Title |
Enhancing Low-bit Quantization of Vision Models via Rotation Transformation |
| Authors |
박재우(Jaewoo Park) ; 용윤정(Yunjeong Yong) ; 이지혜(Jihae Lee) |
| DOI |
https://doi.org/10.5573/ieie.2026.63.3.112 |
| Keywords |
Quantization; Object detection; Rotation transformation; Low-bit quantization |
| Abstract |
Recent vision models often suffer substantial performance degradation under low-bit quantization due to activation outliers. To mitigate this issue, we apply an orthogonal-matrix-based rotation technique, previously validated in LLMs, to RF-DETR, RT-DETR, and YOLOv12. Experimental results show that the rotation reduces layer-wise kurtosis, effectively alleviating the impact of long-tailed distributions and extreme outliers, and significantly decreasing detection accuracy drops in low-bit settings. In particular, RF-DETR recovers performance close to the baseline under W4A8, while RT-DETR achieves additional improvements over QRT-DETR, reducing sensitivity in low-bit regimes. YOLOv12 also demonstrates consistent gains in low-bit configurations, confirming that the proposed method helps mitigate quantization errors. Moreover, inference latency remains comparable after applying rotation (41.31 ms) relative to the baseline (43.75 ms), suggesting that the proposed technique does not meaningfully increase runtime overhead. |