| Title |
Effect of KV-cache quantization on long-context recall of Korean LLMs: a matched Korean-English comparison |
| Authors |
송종휘(JongHwi Song) ; 서병석(ByungSuk Seo) |
| DOI |
https://doi.org/10.5573/ieie.2026.63.10.135 |
| Keywords |
KV-cache quantization; Long-context recall; Korean LLM; Token inflation; On-premises inference |
| Abstract |
The key-value (KV) cache dominates memory during long-context LLM inference, and KV-cache quantization is a standard remedy shipped in mainstream inference frameworks. However, its accuracy has been evaluated almost exclusively in English, leaving its impact on Korean?a non-Latin-script language affected by token inflation?largely unknown. We design a matched Korean-English needle-in-a-haystack protocol and measure recall at 4k?32k tokens under BF16, 4-bit, 3-bit and 2-bit KV caches for three models whose Korean-to-English token ratios differ (EXAONE-3.5-7.8B, Llama-3.1-8B, Qwen3-8B). With 1,728 measurements per precision, we find that (1) 4-bit KV caches are lossless for both languages up to 32k in all three 8B-class models, but not in Qwen3 models of 4B or smaller; (2) below 4 bits, Korean recall is significantly lower than English in two of the three models (4?5 points at 3 bits, 12?15 points at 2 bits), while Llama-3.1 shows a smaller, non-significant gap of 4 points at 2 bits; and (3) the penalty persists at equal token length even in EXAONE, whose Korean text uses fewer tokens than English, showing that it is intrinsic to the language rather than a consequence of haystack token inflation. The mechanism differs across models: Qwen3 fails on the generation side with garbled Korean output, whereas EXAONE stays fluent but loses retrieval precision on digit strings. Placing the needle inside the full-precision residual window does not restore recall, indicating that the loss stems from attention interference by the quantized context rather than from forgetting the needle. |