| Title |
Autoregressive Inference Bottleneck and Memory Bandwidth-based Performance Determinant Analysis of Small Language Models on Edge AI Platforms |
| DOI |
https://doi.org/10.5573/ieie.2026.63.7.66 |
| Keywords |
Small language model; Autoregressive inference; Edge AI; Memory bandwidth; Roofline model |
| Abstract |
As Small Language Models are increasingly deployed on edge devices, identifying the performance bottlenecks of autoregressive inference on heterogeneous accelerators remains an open problem. Prior studies relied on static Roofline analysis on cloud GPUs or single-type edge devices and did not compare performance-determining factors across accelerator types. This study measures autoregressive inference performance of Qwen2-0.5B and TinyLlama-1.1B on five edge platforms comprising three GPUs (Jetson AGX Orin 275 TOPS, Orin NX 100 TOPS, Orin Nano Super 67 TOPS) and two CPUs (RK3588, Raspberry Pi 5), using the Roofline Model as a diagnostic tool. Results show that decode-phase token generation speed strongly correlates with effective memory bandwidth, which explains 93.0% of decode performance variance, whereas nominal compute capacity (TOPS) accounts for only 45.2%. On GPU platforms, decode throughput increased by up to 55.5% with longer output sequences, whereas CPU platforms showed a 6.1% decrease, indicating divergent scaling behaviors. The near-identical decode performance of Orin NX and Orin Nano Super despite a 1.49x TOPS difference confirms that TOPS is not a reliable predictor of autoregressive inference speed. We propose an Autoregressive Readiness Taxonomy that classifies accelerator support levels from AR-0 (native GPU acceleration) to AR-X (structurally unsupported). Q4_K_M quantization incurred only 1.79% perplexity degradation, confirming its practical viability for edge deployment. |