1. Introduction

Customer service is one of the most difficult tasks in product marketing because it is not easy to satisfy customers while limiting costs ^[1]. Cheong et al. ^[2] showed that 53% of customers were not satisfied with customer support center service. The results also revealed that customers wanted their problems solved quickly by customer service representatives.

As one way to address this problem, a number of companies have constructed live chat systems that connect customers to representatives in real time through the Internet ^[3]. In addition, since costs increase with an increased number of representatives, attempts to replace people with chatbots have also been initiated in order to keep costs down. It is questionable, however, whether chatbots are capable of natural conversation and of understanding exactly what the customer needs ^[4,5].

Another way—text classification through deep learning—can be used to preliminarily classify customer questions before passing them to human representatives. However, the accuracy of such classification systems is not good enough to help representatives, and it is not easy to enhance such classification systems to obtain the necessary training data ^[6-10]. Attempts have been made to transform those classification problems into similarity evaluation problems based on recently proposed natural language processing (NLP) models, such as BERT ^[11], BiMPM ^[12], and Open AI GPT ^[13]. Nonetheless, although the NLP models were pre-trained to include extensive domain information, they are not efficient enough to be used for customer service, and a lot of additional data are required ^[14-18].

Consequently, in this paper, a novel representative assistance system is proposed to overcome the difficulties with the previous approaches and to improve customer service efficiency. The proposed system includes two main functions: FAQ recommendation and automatic data acquisition. For FAQ recommendation, the system calculates a similarity measure between an input question and every question in a well-defined customer service FAQ list. Then, it recommends the top $\textit{k}$ FAQs to the representative.

In fact, consumers frequently ask questions that have already been answered in the FAQ list, or that are similar. Thus, the recommended FAQs can help the representative to answer more quickly and accurately by transforming a subjective problem into an objective problem. Following this system, the representative chooses one of the recommendations, and the choice is automatically saved as new data. Consequently, the system is updated with newly collected data from the specific service domain, and the accuracy of the system is improved incrementally.

This paper is organized as follows. Section 2 explains the proposed system. In Section 3, the experimental results are evaluated. Finally, Section 4 presents the conclusions.

2. The Proposed System

2.1 Building a Baseline NLP Model

The first main function of the proposed system is to recommend the $\textit{k}$ FAQs that are more similar to a customer’s query than others from the list. Fig. 1 shows the overall flow of the proposed system. At first, it is necessary to train a baseline NLP model to recommend the most similar $\textit{k}$ FAQs. Here, for the baseline NLP model, a Quora Question Pairs (QQP) dataset ^[19] was used. The dataset contains roughly 400,000 sentence pairs with corresponding labels. The original dataset is structured as shown in Fig. 2(a) and has been modified for simplicity as shown in Fig. 2(b)

Fig. 1. The overall flow of the proposed system.

Fig. 2. An example of the data format.

2.2 Operating the System with the NLP Model

When a customer asks a question, the NLP model measures the similarity between the customer’s question and every FAQ in a well-defined FAQ list. Then, the system shows the representative the closest $\textit{k}$ FAQs. After that, the representative chooses one from among them that is similar to the customer’s question. New data are constructed from these choices and are stored in a training dataset. Fig. 3 shows an example of the data construction process with k=3.

Fig. 3. An example of the data-construction process.

3. Performance Evaluation

The environmental settings for the experiments are as follows. As the initial FAQ list for customer service, we chose 40 FAQs from the Facebook website. Customer questions were collected from the Facebook user community. We then divided the collected questions into two sets. One set was used for training the model, and the other for testing the performance of the system in each version. The training dataset was built through a role-playing simulation by five participants randomly recruited from among a population of graduate students who did not know the authors personally. The participants used the proposed system as if they were representatives, choosing responses from the recommended $\textit{k}$ FAQs when $\textit{k}$=5 for each query. The test dataset was built based on directly matching participants. Note that the training and test datasets included questions related to FAQs 1-20 and FAQs 1-40, respectively, which means the system did not learn information from FAQs 21-40.

In the experiments, BiMPM, OpenAI GPT, and BERT were employed as the NLP models, and the results were compared. Each model was pre-trained with the QQP dataset and used as the baseline model. The service’s operation and fine-tuning scenario was set considering the real-world customer service process illustrated in Fig. 4. The scenario consisted of four steps in operating/fine-tuning the pairs and one step in testing them, with 5,000 data entries gathered for each operation and the number of FAQs in the FAQ list increased at the beginning of Step 3. In Step 5 (the testing step), since versions 1 and 2 were trained with data from FAQs 1-10, they were tested with the test dataset that included FAQs 1-10 and then retested with the dataset that had FAQs 21-40. Similarly, versions 3 and 4 were tested with the test dataset including FAQs 1-20 and then retested with the dataset using FAQs 21-40.

Fig. 4. The test scenario of the experiments.

Fig. 5 shows the test accuracies for each model and version. Top $\textit{k}$ accuracy (the y-axes) indicates the probability that the best answer exists among the top $\textit{k}$ recommendations in the system. For every NLP model, the accuracy from the proposed system increased after each step in the scenario. Table 1 shows the test accuracy in detail. The most important thing is that for the BERT and OpenAI GPT models already pre-trained with relatively heavy data in their initial states, the test accuracies increased even with the test dataset excluding experienced information. Moreover, BiMPM showed significant accuracy improvement with the test dataset including the experienced information, and this is an advantage because additional data for the changed FAQ list can be readily and automatically accumulated during services with the proposed system, as shown in the test scenario. OpenAI GPT showed the best performance, along with the proposed system, under the test configuration.

Fig. 5. The resulting top $\textit{k}$ accuracies for each model.

4. Conclusion

In this paper, we proposed a novel system to assist customer service representatives in answering customer questions. Since the proposed system automatically accumulates new data during service calls with a representative, it can avoid the data-shortage problem common in various service fields. In addition, as the experimental results show, the more data gathered, the greater the accuracy becomes. This means the accuracy of the proposed system improves from the automatically accumulated data as time goes by. Above all, the proposed system transforms subjective problems into objective ones so that representatives can save time in answering, and so customers are more satisfied. Furthermore, this system can be applied to languages other than English.