2.1 Least-square-regression

Least square regression (LSR) is vital for prediction applications, including microarray missing value estimation, disease spread estimation, and weather condition estimation. In ^[6,7] proposed a novel algorithm called least square multi-splitting (LSMS) to solve large-scale regression problems in parallel. The authors designed a technique of partition for regression design based on cluster analysis. The LSMS approach can compete as a common function algorithm with the LSR. Similarly, the result shows that LSMS is a comprehensive and well performer technique for the partitioning of large-scale regression problems. For comparison, at every truncation, it is estimated that the partitions must utilize the cluster analysis based on three links: single, complete, and average. This work is the optimal choice for multicategory classification, and it is based on the newly proposed technique of group-wise retargeted least squares regression (GReLSR). This work is an extension of prior retargeted least square regression proposed by XuYao Lingfeng et al. The authors proposed a new reformulation of ReLSR for utilizing a groupwise regularization to restrict the translation values of ReLSR and produce the novel GReLSR technique. The performance of GReLSR was compared with the seven prior multicategory classification techniques. The result showed that the recently proposed GReLSR technique could be a state-of-the-art method and a better performer, unlike prior findings ^[8]. Drug availability has recently been an important part of human life. This study has been proposed to forecast drug availability for the next month through the LSR algorithm. For the sake of forecasting stock for medicine requirements, the authors collected data from January to November 2017 from Puskemas Community health center, East Kalimantan, Indonesia, to predict the future requirements of drug quantity. Nevertheless, the shortage of this work is that the authors did not compare LSR with other prediction algorithms to show the accuracy of their results ^[9]. Zhao Shuping et al. ^[10] proposed a novel discriminant and sparsity-based LSR. Compared to prior LSR techniques, the authors demonstrated the relationship among the training samples with L1 regularization to combinedly learn the discriminative projection matrix and the orthogonal relaxing term. Compared with other studies in the image classification area, a comprehensive experiment was done on the Yale B, LFW, and 15-Scene databases to demonstrate outperforming results. The work, together with eight prior algorithms, was compared. The results showed the efficiency of the proposed approach for classification. The COVID-19 outbreak is a global concern. O. Roseline et al. reported the predictive modelling of COVID-19 based on a linear regression model. The authors of this study have analyzed the impact of the traveling history and contacts regarding COVID-19 confirmed cases in Nigeria. The ordinary LSR was used as a tool to fit the data along with diagnostic checks datasets which were derived from the Nigeria Centre for Disease Control (NCDC) website starting from April 05 to April 13, 2020. In the end, they compared both results. The comparative results show that traveling history and contacts increase the likelihood of being infected with COVID-19 by 85% to 88%, respectively ^[11]. Generally, COVID-19 is increasing rapidly in countries where the population is dense. India is also one such country in the world. Work was done based on multiple linear regression to predict COVID-19 upcoming active cases in the last two weeks of August 2020 in Odisha and India. The authors suggested that the containment facilities in the India and Odisha should be reinforced by responsible authorities to keep people healthy and decrease the number of patients ^[12]. Qin Lei et al. proposed another prediction about COVID-19 cases of 2019. The data regarding patients’ categories like dry cough, fever, chest distress, coronavirus, and pneumonia was derived from the social media search index from 31 December 2019 until 17 February 2020. The authors proposed a method for COVID-19 prediction based on five regularized regression methods, including subset selection, forward selection, lasso regression, ridge regression, and elastic net to prevent overfitting while prediction. The comparative result regarding patients showed that by January 22 2020 the number of Coronavirus, Pneumonia and new suspected cases would increase sprightly and decrease slightly ^[13,14]. Table 1 lists the different regression analysis techniques.

2.2 Loss Function

A loss is the difference between the original outcome and the predicted outcome for a given instance. It is also called the error or residual. When the loss of a set of instances occurs, it is called the cost of the model. The sum of squared error (SSE) or cross-entropy (CE) is typically used as a cost function. The parameter values for different purposes in ^[15], such as sequential sampling, classification, and optimal control, were estimated using the cost function. Sequential sampling analysis is one of the important computation techniques in various fields, including economics, engineering, medical science, and statistics. K Jampachaisri et al. proposed an Empirical Bayes (EB) prediction method ^[15] for parameter estimation in a sequential sampling plan (SSP) based on squared error loss (SEL) and precautionary loss function (PL). The proposed EB approach was compared with the prior single-sampling statistical approach. The result shows that the EB in SSP via computation of SEL and PL affords the highest probability of acceptance and the smallest average sample number. The advantage of the proposed method is that the mean square error is always convex when it uses its parameter. Nevertheless, the authors did not compare the result of their proposed method with more than one sequential sampling technique.

DL algorithms will be an essential tool for everyday life. In particular, developing new techniques for training complex datasets quickly and accurately is on-demand. In ^[16] proposed a novel activation function called Reward cum Penalty loss function based Extreme Learning Machine (RPELM) to prevent those data points which do not fall on the targeted location and draft reward method for the data points which fall on the desired location. The authors compared the RP-ELM result with prior activation functions to demonstrate precise classification outputs. The result shows that the proposed RP-ELM activation function performs better than hinge and quadratic losses regarding fast and correct classification. The main objective of any activation function is to apply linear data to nonlinear data. S Ma et al. proposed a nonlinear optimization method called variable-step beetle antennae search algorithm (GBAS) to update the performance of the Huber loss function ^[17]. Currently, DL algorithms are de facto important tools in many areas ^[18-20]. Proposed novel ideas to contrivance multiple loss functions based on kernel density estimation (KDE) in ^[21] to predict the probability density function (PDF) regarding spoofing detection. The authors used a recent automatic speaker verification (ASV) spoof 2019 dataset to execute the actual scenario. The experimental results show that the proposed KDE-based loss functions are superior to the conventional loss functions, which were exploited for estimating anti-spoofing detection until now. The superiority of this finding suggests a new idea of KDE based on several loss functions in the field of DL. The weakness of the work was that the authors did not compare their proposed KDE-based loss functions with conventional loss functions to be clear to the reader. Traffic classification is also a long-term challenge in the research community. In ^[22] proposed a new loss function method called UniLoss. The authors of the work tried to classify the problem of an imbalanced dataset of VoIP traffic while occurring in a minority category with poor misclassification performance and high accuracy. In the experimental part of this finding, the newly proposed UniLoss function was run in four different types of deep neural networks (DNN), including convolutional neural network (CNN), recurrent neural network (RNN), ResNet, and FusionNet. Similarly, two conventional loss functions were run in the above-mentioned DNN algorithms for comparison. The research outcome showed that the proposed UniLoss function has a higher F1 score of single categories for better performance than the other two conventional loss functions. The supremacy of this finding is outperforming the novel UniLoss approach compared to other conventional loss functions. Unlike the drawbacks of the finding, authors did not experiment with other types of data for classification but only VoIP data. Despite the remarkable development in the field of biomedical technology DL algorithms is also becoming an important tool of prognosis in biomedical science. H Seo et al. proposed a generalized loss function (GLF) technique with functional parameters in ^[23] for optimal decision making in small target segmentation. The proposed method displayed more precise discovery and segmentation of lung and liver cancer tumors. An outcome of the work showed that the proposed GLF performs an accurate diagnosis compared to prior techniques in terms of detection and segmentation of lung and liver and tumors. Similarly, several studies have been done in the past using different types of loss functions in ^[24] and ^[25]. Table 2 lists different papers which were used the loss function for different applications.

2.3 Gradient Descent

Gradient descent (GD) is an optimization algorithm for finding the absolute minimum to minimize the error of the function while training fixed datasets. It has diverse applications, including image classification, entity clustering, weather condition, and disease spread prediction. D Zou et al. proposed a technique ^[26] for binary classification while training a deep, fully connected neural network in combination with Rectified Linear Unit (ReLU) activation function and cross-entropy loss function using GD. The result showed that with proper random weight initialization, GD could find absolute minima while training loss for an over-parametrized deep ReLU activation function under supposition on the training fixed amount of data. They compared the training performance of their findings with two prior proposed techniques. Their technique could discover global minima faster than the findings, which are exploited in the experimental part of the paper. An advantage of the finding is that the authors could prove their idea regarding Gaussian random initialization pursued through GD, which produces a sequence of iterations that remind inside a small disorder region placed at the initial weights. The disadvantage of the proposed method is that the authors did not compare their proposed method performance with more than two previous techniques. J Flynn et al. proposed a novel method for deducing multiplane image (MPI) scene representation in combination with Learned Gradient Descent (LGD) ^[27]. They compared the MPI results with a well-known method called Soft3D and some multiple methods in deep learning. The results showed that the proposed MPI scene representation, combined with the learned GD, showed better results than other traditional techniques, particularly in solving complicated, nonlinear inverse problems. The dominance of the proposed MPI technique was that authors could implement their novel idea based on LGD. The weak point of this work was the complication of implementing RAM requirements and the speed of LGD while training, which takes multiple days by utilizing more than one GPU. J Lee et al. proposed additional learning dynamics of GD in the theoretical approach for parameter space of deep nonlinear network and classification ^[28]. They compared their proposed technique with traditional SGD and CEL for classification purposes by exploiting MNIST and CIFAR datasets. The result showed that the proposed technique is more sufficient than the SGD and CEL techniques in training a wide DNN while gaining minimum error. The superiority of the work is that theoretical results are solely precise in the finding is authors did not experiment for comparison of their proposed classification technique with more than two classification datasets to prove the robustness of their technique. One continuous progress in the DL research community is reproducing tremendous works about updating convex, non-convex and concave functions. Two-time scale gradient descent ascent (GDA) for solving nonconvex-concave minimax problems has been proposed in ^[29]. The main aim of this work is to solve the nonconvex-concave minimax problem and compare the performance of GDA with two prior techniques Wasserstein robustness model (WRM) and Gradient Descent max (GDmA). The results showed that the proposed GDA outperforms both prior techniques, which were compared during the experiment by exploiting three types of classification datasets. An advantage of the finding is that the authors could prove the solution for the nonconvex-concave minimax problem while training combined classification datasets. On the other hand, the authors did not prove why they chose only three categories of classification datasets for a result comparison and showed the efficiency of GDA while there are many classification datasets. M M Amiri et al. in ^[30] proposed a novel analog distributed stochastic gradient descent (A-DSGD) to reduce noise from the channel bandwidth with a combination of parameter servers (PS). In addition, one of the prior techniques, called digital distributed stochastic gradient descent (DDSGD) was exploited for a performance comparison with A-DSGD. The results showed that A-DSGD has faster performance than D-DSGD because of its available channel bandwidth. In the last decade, many studies have been conducted using different GD types for different applications, such as classification ^[31], sparse linear problems ^[32], new task learning ^[33], data overfitting prevention ^[34], and inverse filtering ^[35].

2.4 Backpropagation

A combined Empirical Mode Decomposition-Variational Mode Decomposition-Genetic Algorithm-Backpropagation (EMD-VMD-GA-BP) model was introduced in ^[36] for carbon price prediction. The proposed model used several techniques for accurate carbon price estimation in the Hubei market, particularly the backpropagation (BP) neural network model combined with a genetic algorithm (GA) for accurate prediction purposes. The proposed model outperformed other prediction models. The advantage of this work is that the proposed model can meaningfully reduce the exertion of carbon price time series forecasting. In ^[37] Multi-input Multi-Output (MIMO) technique model for Wireless Sensor Network (WSN) was proposed, which addresses the Cluster Head (CH) recognition challenge for the MIMO sensor network. The proposed technique is based on a neural network BP algorithm. The main aim of the proposed model was to address the location identification problem of CH for the MIMO sensor network, which is utilizable in Intelligent Transportation System (ITS). The model minimizes the total estimation error compared to other proposed techniques. ML is used in several areas, such as agriculture. L Wang et al. proposed maize growth monitoring on the North China plain utilizing a hybrid genetic algorithm-based BP neural network ^[38]. An experimental part of the study was conducted through the remotely sensed leaf area index (LAI) and vegetation temperature condition index (VTCI). The data derived from the Global LAnd Surface Satellite (GLASS) and Moderate-resolution Imaging Spectroradiometer (MODIS) data were chosen as the key indicators of maize growth and a hybrid GA based on the BP neural network (BPNN).

The (GA-BPNN) model was designed to provide enough information on maize growth at the core growing stage. The proposed BPNN-based GA-BPNN performed satisfactorily compared to other techniques. A new method of ML algorithms combination was proposed called Genetic Algorithm-Back propagation called (GA-BP) ^[39,40]. The proposal aimed to predict clothing pressure. The proposed GA-BP algorithm does not require complex modeling compared to prior girdle pressure predicting models, such as General Regression Neural Network (GRNN) and grey BP. W Yang et al. proposed an analysis method for skeletal anthropology fields based on an improved BP algorithm. The main aim of the study was to determine if the selected skull was male or female from the compound of skeleton ^[40]. The proposed improved BP algorithm showed better classification accuracy with a 97.232 % training stage and 0.01 mean square error. The results for comparing the performance were trained with other prior techniques, including the Cranical Sagittal chord and Apical sagittal chord. Consequently, improved BP showed outstanding performance compared to two prior techniques.

3.1 Least Square Regression

The least square regression model aimed to determine the relationship between the dependent and independent variables. The large amount of missed data was assessed using the least square model. All data points of the dependent variable depend upon the alteration of the independent variable ^[41,42]. Fig. 1 shows how linear regression has fit the datapoints (X-input, Y-output), given in Table 4, using line function f(x) = mx + b.

Fig. 1. Fitting line on the data points.

The line equation on the data points was determined by estimating (approximating) the value for m and b. The initial step of the LSR was calculating the slope of the function, computing the y-intercept, and finally substituting the input values in the model explored. Before calculating the slope of the function, it is essential to know the mean of the input and output values, as shown in Table 5. As the first step of the goal is to determine b (slope) of the function, each observation from X and Y axes must be subtracted from the mean of each column and demonstrated in a separate column. Table 5 lists this process in detail. With the values calculated in Table 5, the slope (m) ^[42] can be found using the following formula

The next step after finding the slope is finding the y-intercept (b). Y-intercept is a point at which the trend line crosses on the y-axis and then moves upward,

The values of m and b were calculated manually. This can be computed using Excel and plot the line of best fit (trend line), as shown in Fig. 2.

Fig. 2. Line of best fit.

3.2 Loss Function

The main goal of the loss function is to minimize the data loss occurrence while training complex datasets ^[2,43]. Table 6 provides some examples of working hour training datasets for predicting the gain of a worker. An example is given that when a worker works approximately one hour, his salary will be 13 dollars. On the other hand, what happens when more than one hour is worked?

What happens when more than 10 hours are worked? What about if there is very larger amount of work data for prediction? For this type of training, multiple ML algorithms can be used to deal with existing challenges while researching. It is important to follow some simple equations or ML hypotheses. Table 7 provides an example of a prediction of a worker’s salary. As the working time is X integer and the salary of a worker is the Y variable, then the ML hypothesis must be followed for a simple prediction as below

where $\hat{Y}$ is an undefined prediction variable. The variable X is the input data that should be multiplied with variable W to update the whole input data for predicting an accurate output ^[44,45]. The next stage of computing loss is determined by exploiting the obtained loss formula as follows:

Tables 7-9 list the prediction of a worker's salary for various W values.

Tables 10(a) and (b) list the mean square error or total loss of each training. The loss decreases through every iteration. This example did not show the condition in which the loss equals 0. On the other hand, only an example of loss is shown, decreasing the total loss in every training. In addition, if L = 0, more training is needed to achieve the actual goal. Fig. 3 shows different $\hat{Y}$ on the data points during the training process. The graph shows the loss minimization using simple mathematical computations, including graphs and tables. The first random value W$^{1}$, which is always equal to W$^{1}$=1, should be pointed out before every phase. For example, if there are 1000 or 10000 random guesses, the computer can automatically provide a variable value and compute other random guesses of W$^{n}$ for updating the output ^[6]. Therefore, variable W has an important role in the neural networking field. There are three random guesses of W, each with a simple value of 1, 2, and 3, for easy understanding of the loss function ^[45]. Table 11 shows three values of the X and Y axes in each column. Similarly, three more columns for describing the output of the predictions are needed to minimize the loss while training. A straightforward mathematical computation was performed at the bottom using the same values in the prediction part. The hypothesis $\hat{y}=x\star w$ was exploited to update each training data of input x ^[8]. $\mathrm{Cost}=\sum (\hat{y}-y)^{2}$ was used for minimizing the loss function. For example,

As shown in the above examples, the loss in the first iteration was 140 but 35 in the second iteration.

Eventually, the loss approaches 0. Fig. 4 shows the loss for different possible lines. The red line indicates the gap between the true and error lines. The particular point where L = 0 in the graph denotes the local minima of a function ^[45].

Fig. 3. Several random guesses of w${\cap}$y to approach the true line.

Fig. 4. Loss for each line.

3.3 Gradient Descent and Backpropagation

Gradient descent is an iterative optimization algorithm for finding the minimum of the cost function ^[46] as shown in Fig. 5. This section reviews the concept of gradient descent with simple expressive calculations working on the cost function, the sum of squared error (SSE). The equation of SSE is given below

Table 12 explains the cost calculation on the given data.

For the first training example, the prediction variables $\hat{Y}$ are needed to calculate the loss function ^[46].

Fig. 5. Gradient Descent.

Fig. 6. Backpropagation.

The gradient of the cost function was calculated with respect to all the parameters (weights and biases) to find the global loss minimum ^[47]. The equation is given below.

where ${\omega}$ is the derivative for loss with respect to the weight parameters. ${\alpha}$ is the learning rate that controls the speed of convergence. The weight values were updated by subtracting the derivative of the loss function ^[48]. If there are three weight values in the network, their values are three weight values in the network, as shown in Fig. 6, their values are calculated as below,

We used 10000 images for test and 50000 images for training along through 10 epochs. In the execution, we regulate learning rate to 0.01. Total iteration is 1000 with approximately 95% of accuracy as shown in Fig. 7.

Fig. 7. Backpropagation and loss minimization.