2.1 Vocal Information Features

(1) Mel-frequency cepstral coefficient (MFCC)

The MFCC is a feature commonly used in speech recognition ^[9], which is similar to the perception of the human ear. Hence, it can play a larger role in vocal information processing ^[10]. When extracting the MFCC, it is first necessary to perform a pre-emphasis on the original audio signal:

where $n$ is the time point, and $\alpha $ is the pre-emphasis factor.

The Hamming window is then used:

Window function length $L=N+1$.

The preprocessed sequence $x\left(n\right)$ is then transformed to the frequency domain by a fast Fourier transform (FFT):

where $N$ is the frame length and $\mathrm{x}\left(\mathrm{k}\right)$ is the complex sequence of $\mathrm{N}$ points. The logarithmic energy calculated computed using the Mel filter:

where $E\left(k\right)$, $H_{m}\left(K\right)$, and $k$ are the spectral line energy, frequency domain response of the filter, and the k$^{\mathrm{th}}$ spectral line. The discrete cosine transform (DCT) is carried out on $S\left(m\right)$ to obtain the MFCC feature:

where $\mathrm{M}$ is for the number of filters, and $m$ refers to the $m$$^{\mathrm{th}}$ Mel filter.

(2) Colorimetric feature

The colorimetric feature is a feature based on the equal temperament ^[11]. In vocal information, the pitch is used to describe how high or low the pitch of a voice is, and an equal temperament can be used to calculate the pitch of a note, so chromatic features can be a good way of describing the pitch variations in a melody. The extracted chromatic feature is a 12-dimensional vector. The $j$$^{\mathrm{th}}$ audio frame in the sequence is assumed to be $v_{j}$. The $l$$^{\mathrm{th}}$ colorimetric feature is written as $c_{l}^{j}$, $l\in \left(0,1,\cdots ,11\right)$. First, the center frequency $f_{l}$ of each half-order tone class is calculated as follows:

where $f_{rep}$ refers to the fundamental frequency of the signal. The energy $chroma_{l}$ of the half-order tone class is then calculated:

where $A_{i}$ refers to the amplitude at the i$^{\mathrm{th}}$ amplitude peak point, and $w\left(l,f_{i}\right)$ means the weight of the signal with a frequency of $f_{i}$ in the half-order tone class. The $\mathrm{c}h\text{roma}_{\mathrm{l}}$ is then normalized to obtain the final colorimetric feature:

2.2 Conditional Random Field (CRF)

The CRF has a wide range of applications in sequence annotation ^[12]. Therefore, it can also be used in pitch sequence annotation. Suppose the input observation sequence is $X=\left(X_{1},X_{2},\cdots ,X_{n}\right)$, and the output labeled sequence is $Y=\left(Y_{1},Y_{2},\cdots ,Y_{n}\right)$, the conditional probability distribution based on CRF can be written as follows:

where $A_{{y_{i}},{y_{i+1}}}$ is the transfer characterization matrix, denoting the transfer score between labels. $B_{i,{y_{i}}}$ is the emission probability matrix, denoting the score of the i$^{\mathrm{th}}$ position being label $\mathrm{y}_{\mathrm{i}}$. The final result is

2.3 Main Melody Extraction Algorithm Design

A main melody extraction algorithm based on CRF was designed to process vocal information using MFCC and colorimetric features as inputs. The CRF was used to obtain the best melodic line. A convolutional neural network (CNN) was added before the CRF to realize the extraction of deeper melodic features and improve the extraction of the main melodic line. The CNN adopts the structure of a six-layer convolution and adds cavity convolution in the first two layers ^[13]. The first layer uses 32 cavity convolution blocks (5${\times}$5) with a dilation width of (1,10), while the second layer uses 64 cavity convolution blocks (3${\times}$5) with a dilation width of (12,1). The integration and compression of the features are achieved through a fully connected layer after the CNN layer, which serves as the input to the CRF layer.

In the main melody extraction, the CRF is used to calculate the pitch score at time $t$ and then output the melodic sequence. In this task,

where $A$ is the transfer feature matrix, and $A_{{y_{t-1}},{y_{t}}}$ represents the probability of pitch $y_{t-1}$ at time $t-1$ transferring to pitch $y_{t}$ at time $t$. $B$ represents the emission probability matrix, and $B_{t,{y_{t}}}$ is the probability when the pitch at time $t$ is $y_{t}$.

The CRF result was decoded using the Viterbi algorithm ^[14]. The maximum value of all possible values of pitch $f$ at time $t$ is denoted as $\omega _{t}\left(f\right)$, $f=1,2,\cdots ,K$. The pitch value at time $t$ when $\omega _{t}\left(f\right)$ reached the maximum value is denoted as $\delta _{t+1}\left(f\right)$. The Viterbi decoding process is expressed below.

The relevant parameters are initialized:

where $\mathrm{A}_{{\mathrm{y}_{0}},\mathrm{f}}$ is the probability of pitch transferring to $\mathrm{f}$, and $\mathrm{B}_{\mathrm{f},\mathrm{t}}$ is the probability of pitch being $\mathrm{f}$ at time $\mathrm{t}$.

After recursion using the above equation, the following equations can be obtained:

The optimal solution is obtained after backtracking:

The main melodic pitch sequence output by the CNN-CRF algorithm is $y'=\left(y'_{1},y'_{2},\cdots ,y'_{T}\right)$.

3.1 Experimental Setup

The experiment was performed using MATLAB 2019 and Python 3.6. The algorithm was implemented based on the TensorFlow framework. For the CNN-CRF model, the Adam optimization algorithm was used for optimization training. The learning rate was set to 0.001. The batch size was 20. The maximum iteration count was 100. Overfitting was mitigated by adding a dropout layer with a ratio of 0.5 after output, and the activation function utilized was the rectified linear unit (ReLU). In the experiment, the large MIR1K dataset ^[15] was used as the training and validation set to help the model learn more data patterns and features. The independent datasets ADC2004 and MIREX05, which were not encountered by the model during training and validation, were used as the test sets to evaluate the model performance objectively. Table 1 lists the three datasets.

[1] MIR1K includes 110 songs performed by 19 amateur singers, with each clip lasting 4-13 s.

[2] ADC2004 includes 20 pop music clips, including jazz and R&B, each lasting approximately 20 s.

[3] MIREX05 includes 13 clips of pop and pure music, each lasting 24-39 s.

3.2 Evaluation Indicators

The algorithms were evaluated based on the methodology described elsewhere ^[16], as shown in Table 2.

Table 3 lists the explanations of the parameters in Table 2.

According to Table 3, the following are defined.

TP: there is a melody, and it is detected correctly;

TN: no melody and correctly detected;

FP: no melody and wrong detection;

FN: no melody and correctly detected;

DV: detected melody;

DU: undetected melody;

GV: there is a melody in Ground Truth;

GU: no melody in Ground Truth;

TO: total.

In RPA and RCA, $c$ refers to the pitch, and $ch$ refers to the tone level.

3.3 Analysis of Results

Under the same conditions, the effects of the features selected in this paper on the main melody extraction results were analyzed, as shown in Fig. 1.

The main theme extraction algorithm achieved an OA and VFA of 85.74% and 7.43%, respectively, when using only the MFCC as the input feature for the CNN-CRF algorithm (Fig. 1). The OA and VFA were 83.62% and 7.26%, respectively, when exclusively using the chroma as the input feature. From this perspective, when using the chroma, the OA of the algorithm decreased, and the VFA increased. On the other hand, the RCA, RPA, and VR improved when using the chroma. This is because the chroma is a pitch-related feature. When the MFCC and chroma were used simultaneously as inputs for the CNN-CRF algorithm, the algorithm achieved an OA of 86.72%, showing 0.98% improvement compared to using the MFCC alone and 3.1% improvement compared to using the chroma alone. Moreover, the VFA of the algorithm was 6.84%, showing a 0.59% reduction compared to using the MFCC alone and a 0.42% reduction compared to using chroma alone. The RCA of the algorithm remained high at 85.76%, but it was slightly lower than using chroma alone. Nevertheless, both RPA and VR showed improvement.

Fig. 2 presents the results of the MIREX05 dataset.

When only MFCC was used, the algorithm achieved an OA and VFA of 83.56% and 8.14%, respectively. On the other hand, when only chroma was used, the OA was slightly lower at 82.34%, but the VFA was higher at 9.33%. Hence, the MFCC was more accurate in extracting the main melody in the MIREX05 dataset. This difference in performance may be attributed to the MIREX05 dataset containing both pop and pure music. MFCC was more sensitive to vocals, making it better at distinguishing pitch in vocal information.

Similar to the ADC2004 dataset, chroma performed better regarding RCA with a value of 84.06%. When both MFCC and chroma were used, the CNN-CRF algorithm achieved an OA and VFA of 85.21% and 11.16%, respectively, on the MIREX05 dataset. In addition, it demonstrated impressive performance on RCA, RPA, and VR with values of 83.91%, 82.56%, and 86.33%, respectively. Overall, including MFCC and chroma as input features improved the algorithm OA. Although the VFA showed a slight increase, the enhanced performance on RCA, RPA, and VR underscored the valuable contribution of vocal information features to the main melody extraction in this study.

The effectiveness of the CNN-CRF algorithm was evaluated by a comparison with other algorithms as follows:

(1) SegNet ^[17]: a model based on the encoder-decoder structure that locates melodic frequencies through pooling indices;

(2) frequency-temporal attention network (FTANet) ^[18]: a neural network based on a time-frequency attention structure that analyzes the time mode using the time attention module and selects the same band using the frequency attention module.

Table 4 lists the results of the two datasets.

The bold data in Table 4 are the optimal values. According to Table 4, on the ADC2004 dataset, the SegNet algorithm exhibited the highest VR at 88.24%. On the other hand, its VFA and OA were 19.93% and 82.72%, respectively. The FTANet algorithm showed a significantly lower VFA, only 10.53%, and achieved an OA of 84.99%. In contrast, the CNN-CRF algorithm achieved the lowest VFA, 6.84%, which was 13.09% lower than the SegNet algorithm and 3.69% lower than the FTANet algorithm. Its OA was the highest at 86.72%, showing 4% improvement over the SegNet algorithm and 1.73% improvement over the FTANet algorithm.

On the MIREX05 dataset, the CNN-CRF algorithm achieved the best values across all indicators. It achieved a VFA of 11.16%, which was 8.81% lower than the SegNet algorithm and 9.45% lower than the FTANet algorithm. Furthermore, it attained an OA of 85.21%, 8.92% higher than the SegNet algorithm and 7.01% higher than the FTANet algorithm. These results demonstrate the advantage of the CNN-CRF algorithm in extracting main melodies.

In summary, the CNN-CRF algorithm uses the MFCC and chroma as features to capture the human voice in vocal information effectively. Compared to the ADC2004 dataset, the MIREX05 dataset features a greater abundance of popular music, making the improvements in the results more pronounced. The addition of the chroma significantly improved the RCA of the algorithm, resulting in higher OA and lower VFA.