| Title |
Performance-improving Dimensionality Reduction with Tensor Decomposition and Integrated Positional Encoding |
| Authors |
이희열(Hee-Yeol Lee) ; 이승호(Seung-Ho Lee) |
| DOI |
https://doi.org/10.5573/ieie.2026.63.4.84 |
| Keywords |
Positional encoding; Tensor decomposition; Dimensionality reduction; Transformer; Embedding representation |
| Abstract |
This study proposes a tensor decomposition based dimensionality reduction method with integrated positional encoding to improve performance degradation caused by dimensionality reduction and improve performance in Transformer language models. Conventional low-dimensional projection approaches can degrade performance by weakening positional information during compression, while simply adding positional encodings may fail to interact effectively with compressed representations and destabilize training depending on dataset characteristics and hyperparameter settings. To address these issues, the proposed method projects high-dimensional token embeddings into a low-dimensional representation using Tucker decomposition and explicitly compensates positional information within the reduced representation by adaptively calibrating positional compensation strength through a combination of normalized sinusoidal positional features and a context-dependent gating mechanism. This design enables effective dimensionality reduction while robustly preserving sequential information and stabilizing training behavior under dimensionality reduction.. Language modeling experiments on the WikiText and IMDb datasets show that the proposed Tucker decomposition with context-adaptive positional compensation achieves lower cross-entropy loss and perplexity than the Transformer baseline. In particular, more pronounced improvements are observed on the IMDb dataset, which contains longer contexts, indicating the effectiveness of the proposed approach in preserving long-range dependencies. These results demonstrate that the proposed method enables stable performance improvement beyond simple dimensionality reduction and can serve as a performance-oriented dimensionality reduction module for various natural language processing tasks. |