3.3.1 Merge Mode Inter Prediction

Regular merge mode: A merge candidate list consists of newly introduced history-based motion vector prediction (HMVP) and pairwise average MVP (PAMVP) following spatial merge candidates and temporal merge candidates similar to those in H.265/HEVC. HMVP candidates are the MVs of previously coded CUs mostly from non-adjacent CUs, that are stored using a five-entry table and updated using a first-in-first-out (FIFO) rule. The PAMVP is generated by averaging the MVs of the first and second candidates in the merge candidate list for each reference picture list, even if they have different reference pictures, and assigning the reference picture index of the first candidate to that of the PAMVP. If there is only one MV available, it is used directly. If no MV is available for one of the reference picture lists, the MV for the list is considered invalid.

Merge with motion vector difference (MMVD): In MMVD, one of the first two existing candidates in the merge candidate list is selected as the base motion. The base motion is refined with an MVD obtained from a predefined direction and a predefined distance. A predefined direction indicating the direction of MVD can be either horizontal or vertical direction, resulting in four directions. A direction index is signaled to indicate the selected MVD direction, as shown in Table 4. A predefined distance indicates a distance to be refined from the base motion. There are two predefined distance tables, as shown in Table 5. Based on one of the tables selected at the picture level and a signaled distance index for MVD, the distance from the base motion can be derived for the refined motion. MMVD improves the accuracy of MV from the general merge mode, even though it cannot provide MVs as accurate as those in AMVP mode.

Combined inter and intra prediction (CIIP): Similar to the bi-prediction, i.e., a linear superposition of two motion compensated predictions that can further reduce the energy of the prediction error than uni-prediction, a uni-directional inter prediction can be superposed with intra prediction. In CIIP mode, the predicted signal, $P_{CIIP}$, is generated by a weighted sum of inter-predicted signals using merge mode, $P_{inter}$, and intra-predicted signals using the planar mode, $P_{intra}$, as described in Eq. (3.3.1). The weights, a weight for inter prediction, $W_{inter}$, and weight for intra prediction, $W_{intra}$, are derived based on whether the above and left neighboring CUs are coded using intra or inter prediction mode. When both the above and left neighboring CUs are intra-coded or inter-coded ($W_{inter}$, $_{W_{intra}}$) is set to (1, 3) or (3, 1), respectively. Furthermore, it is set to (2, 2) if only one of both neighboring CUs is intra-coded.

Geometric partitioning mode (GPM): H.266/VVC does not employ geometric block partitioning that could increase the partitioning precision for moving objects boundaries because of implementation complexity. Instead, GPM mode has been adopted to provide a similar effect to geometric block partitioning. In GPM mode, a CU is partitioned into two parts using a straight line. The straight line is parametrized by an angle and an offset so that there are 64 partitions for a CU with size $w\times h=2^{m}\times 2^{n}$ with $m,n\in $\{3, {\ldots}, 6\} excluding 8${\times}$64 and 64${\times}$8. In each partition, its own MV is derived from performing a block-based motion compensation. The final prediction for the CU is generated by performing a blending process with adaptive weights based on the position of each sample relative to the geometry partitioning boundary, as shown in Fig. 8. A, MV for each partition is uni-directional, so the complexity of motion compensation for the CU is the same as that of bi-directional prediction. The MV for each partition is derived from the regular merge list, as shown in Fig. 9, with the signaled GPM merge index.

Subblock-based merge mode: In subblock-based merge mode, a CU with a height and width larger than or equal to eight luma samples is divided into 8${\times}$8 subblocks, and a different MV is derived for each subblock. A subblock-based merge mode consists of SbTMVP similar to TMVP in H.265/HEVC and motion vector prediction based on an affine motion model. SbTMVP is only applied to the subblock-based merge mode, and an affine motion model-based prediction is applied to the merge and AMVP modes. Section 3.3.3 describes the affine motion model-based prediction. Fig. 10 presents the derivation process of SbTMVP consisting of two steps: 1) derivation of a displacement vector (DV), and 2) derivation of motion information for each subblock based on the motion information derived by the DV. If the MV of the left-bottom neighboring block refers to the collocated picture, that MV is used as the DV. Otherwise, zero MV is used as the DV. The DV is applied to the central position of the current CU to locate the corresponding sample position in the collocated picture. the SbTMVP is considered available if the block containing the corresponding sample position in the collocated picture is inter-coded. If available, the motion information of the corresponding subblocks in the collocated picture is found by applying the DV to each subblock in the current CU. Finally, the motion vector of each subblock in the current CU is derived from the motion information of the corresponding subblock, similar to the TMVP process in H.265/HEVC, where temporal motion scaling is applied to align the reference pictures of the temporal motion vectors to those of the current CU ^[31]. The derived motion vectors for subblocks in the current CU become the SbTMVP.

Fig. 8. Derivation of predicted signal using GPM.

Fig. 9. GPM merge list construction.

Fig. 10. Derivation of SbTMVP[22].

3.3.2 AMVP Mode Inter Prediction

In the AMVP mode, the components of an MV are coded differentially using an MVP and MV difference (MVD). In H.266/VVC, the AMVP mode is extended using improved predictors, providing a more flexible MVD signaling to improve the tradeoff between the motion accuracy and overhead motion bits. These enhancements on MV coding and motion compensation, including the revised AMVP candidate list construction, AMVR, BCW, and SMVD, are described in the following.

General AMVP mode: The MV prediction algorithm of H.266/VVC is based on the AMVP of H.265/HEVC. The AMVP introduced in H.265/HEVC explicitly signals one of the two potential MVP candidates derived from five spatially neighboring and two temporally co-located MVs. In this way, the motion information, including reference picture indices, MVP indices, and MVDs for each reference picture list 0 and list 1, are singled in the AMVP mode. As a new feature, H.266/VVC adds an MV prediction of HMVP in the merge mode and AMVP candidate list. The HMVP allows the reuse of MVs of previously coded non-adjacent CUs. In addition, the AMVP candidate list construction process is revised in terms of complexity.

Adaptive motion vector resolution (AMVR): H.266/VVC increases the MV precision to 1/16 luma sample, while the HEVC uses only a quarter-luma-sample precision. On top of the higher precision MV representations, a CU-level AMVR method is applied to customize the balance between quality and the MV bit cost overhead. For a CU with translational motion in AMVP mode, MVDs can be coded in units of a quarter, half, integer, or four luma samples. For the affine AMVP mode, MVDs can be switched among the quarter, integer, or 1/16 luma samples. An alternative six-tap smoothing interpolation filter (IF) is used instead of the eight-tap IF from HEVC when a half-luma-sample MV accuracy is used in AMVP mode.

The MVP is rounded to the indicated precision before being added together with the MVD to ensure the reconstructed MV uses the same precision as the MVD. The CU-level MV resolution indication is conditionally signaled if the current CU has at least one non-zero MVD component. Quarter-luma-sample MVD resolution is inferred if all MVD components (i.e., horizontal and vertical MVDs for reference lists 0 and 1, respectively) are zero.

Bi-prediction with CU-level weights (BCW): BCW provides the weighted averaging of the two prediction signals for bi-prediction at the CU-level, in addition to the traditional weighted prediction (WP) for which the weights are specified at the slice level for each reference picture. In H.266/VVC, the legacy explicit-weighted prediction scheme is kept and extended with CU-level syntax control for the weighted averaging. Five weights are predefined, $w\in ${-2, 3, 4, 5, 10}, and an index (denoted as $wIdx$) is signaled at the CU level to specify the selected weight $w$ of the prediction block from list 1. All five weights are used when all the reference pictures are temporally preceding the current picture in display order. Otherwise, only the weights $w\in ${3, 4, 5} are used.

Each luma/chroma prediction sample of BCW is calculated as follows:

where $P_{bi-pred}$ is the final prediction and $P_{L0}$ and $P_{L1}$ are prediction samples pointed to by the MVs from reference picture lists 0 and 1, respectively.

BCW is only applied to CUs with a CU size larger than or equal to 256 luma samples. To avoid interactions between WP and BCW, if CU uses WP, then the $wIdx$ is not signaled, and $w$ is inferred to be 4 (i.e., equal weight is applied). For regular merge mode or affine merge mode, the $wIdx$ is inferred from neighboring blocks based on the merge candidate index. CIIP and BCW cannot be jointly applied for a CU. When a CU is coded with CIIP mode, the $wIdx$ is set to 2, e.g., equal weight. The DMVR and BDOF are both turned off when the weight is non-equal.

Symmetric motion vector difference (SMVD): When the motion of the current block is on a constant trajectory over a past and future reference picture in display order, the corresponding MVs and reference picture indices tend to be symmetrical. SMVD exploits this assumption of linear motion to save bits for MVDs and reference picture index signaling in the true bi-direction mode that uses past and future reference pictures, as shown in Fig. 11. When SMVD is applied for a CU, only the MVP indices of lists 0 and 1 and the MVD for list 0 are signaled. Other motion information is derived at the decoder side without signaling. That is, first, the MVD for list 1 $\left(mvdx_{L1},mvdy_{L1}\right)$ is set to the reverse of the list 0 MVD $(-mvdx_{L0},-mvdy_{L0})$, as shown below:

Second, the lists 0 and 1 reference picture indices are implicitly derived at the slice level. That is, each reference picture is the nearest picture among all pictures in its list, and they have opposite directions to each other.

Fig. 11. Illustration for SMVD mode.

3.3.3 Affine Motion Compensation

Because H.265/HEVC only considered the translational motion model for motion compensation prediction, it is inefficient when there is motion, such as zoom-in/-out, rotation, or perspective motions. H.266/VVC has adopted the 4-/6-parameter affine motion model for motion compensation prediction to improve the coding efficiency, especially for those cases.

Affine motion models based on control point MV (CPMV), 4-parameter affine motion model using two CPMVs, and 6-parameter affine motion model using three CPMVs, are described in Fig. 12. In the case of the 4-parameter affine motion model using 2 CPMVs which are $v_{0}$ and $v_{1}$, an MV ($mv_{x}$, $mv_{y}$) at sample location (x, y) in a CU whose size of $W\times H$ is derived as follows:

In the case of a 6-parameter affine motion model using three CPMVs, which are $v_{0}$, $v_{1}$, and $v_{0}$ and $v_{2}$, an MV ($mv_{x}$, $mv_{y}$) at sample location (x, y) in a CU whose size of $W\times H$ is derived as follows:

As expressed in Eqs. (3.3.4) and (3.3.5), an MV at every sample location in a CU can be derived with an affine motion model using CPMVs. On the other hand, to reduce complexity, an MV is derived for each 4${\times}$4 subblock in a CU, and motion compensation is performed with the derived 4${\times}$4 block level MV, which is an MV at the center sample position of the 4${\times}$4 block.

The affine merge mode and affine inter mode are based on the affine motion model in H.266/VVC depending on the ways of obtaining CPMVs. In the affine merge mode, which is a part of the subblock-based merge mode, the affine merge candidates, i.e., CPMV candidates, are added to the subblock-based merge candidate list so that the motion compensated prediction can be performed based on the affine motion model with the derived CPMVs. There are two types of CPMV candidates in affine merge mode: inherited affine merge candidates and constructed affine merge candidates. The inherited affine merge candidates are the CPMVs of the current CU derived from the CPMVs of the above or left neighboring CU, which is motion compensated based on the affine motion model. The constructed affine merge candidates are constructed by combining the translational motion information of neighboring CUs corresponding to the control points.

The affine inter mode, which is a type of AMVP mode, requires information such as a flag to indicate the motion model type, i.e., the 4-parameter model or 6-parameter model, and two or three motion vector differences (MVDs) between the CPMVs and the predictors from an affine MVP list. For up to two affine AMVP candidates for an affine MVP list, there are the inherited affine AMVP candidates and the constructed affine AMVP candidates similar to the affine merge mode. The inherited affine AMVP candidates are derived from the CPMVs of the above or left CU having the same reference picture to the current CU, and the constructed affine AMVP candidates are derived by combining the translation motion information of the neighboring CUs having the same reference picture to the current CU. In addition, if there are fewer than two affine AMVP candidates, the same translational MV of a neighboring CU is assigned to two or three CPMVs in the affine AMVP list, depending on the motion model type.

Fig. 12. CPMV based affine motion model.

3.3.4 Decoder-side MV Refinement Tools

Refinements of motion and prediction at the decoder side are introduced to improve the prediction quality without increasing the bit overhead of the signaling motion parameters. DMVR is used to improve the accuracy of the MVs of the regular merge mode with a low-complexity motion refinement. Unlike block-based motion compensation (MC), optical flow is expected to achieve the effect of sample-wise inter prediction. It is implemented in H.266/VVC as BDOF to improve the bi-prediction efficiency and as PROF to refine the subblock prediction of the affine MC (AMC).

Decoder-side motion vector refinement (DMVR): DMVR refines the bi-prediction motion of the regular merge mode using a bilateral search. To ensure the bilateral search with equal distance, DMVR is allowed only if the merge MV pair point to two reference pictures that have an equal and opposite temporal distance to the current picture. As shown in Fig. 13, DMVR applies bilateral matching to refine the accuracy of the input MV pair {$MV_{0},MV_{1}$}. That is, it searches the candidate MVs around the initial MVs in lists 0 and 1 with a mirrored MV offset $MV_{diff}$. The refined pair obtained by Eq. (3.3.6), {$MV'_{0},MV'_{1}$}, are used for the motion-compensated prediction of both luma and chroma CBs of a CU.

The searching process consists of an integer sample MV offset search and a fractional sample MV offset refinement. The integer sample MV search calculates the sums of the absolute differences (SADs) between each pair of candidate reference blocks in lists 0 and 1 within the search range of ${\pm}$2 integer luma samples from the initial MVs. The fractional sample refinement is derived using a parametric error surface approximation instead of additional SAD comparisons.

Bi-directional optical flow (BDOF): BDOF is another coding tool for improving the bi-prediction signal using a motion refinement performed by the decoder. In particular, BDOF aims at compensating the sample-wise fine motion that is limited in the block-based MC based on the optical flow concept at the 4${\times}$4 subblock level. It is applied to CUs coded either in merge mode or AMVP mode and assumes constant motion trajectory. As the same constraint applied to DMVR, BDOF is applied only if the two different reference pictures have an equal distance in picture order count (POC) to the current picture.

For each 4${\times}$4 subblock, a motion difference relative to CU MVs is calculated by solving an optical flow equation that minimizes the difference between the prediction subblocks of lists 0 and 1. The derived motion differences and the prediction sample gradients are then used to adjust the bi-predicted sample values.

Let $I\left(i,j,t\right)$ be the luminance value of a sample at time $t$ in position $\left(i,j\right)$. Assuming the luminance of a sample does not change during the object motion, the optical flow equation can be expressed as follows:

where the motion $(v_{x},v_{y})$ describing the remaining motion applied on top of the original MV at each sample position.

As shown in Fig. 14, the motion $(v_{x},v_{y})$ from $I_{c}$ to $I_{0}~ $is symmetrical to its motion from $I_{c}$ to $I_{1}$, where $I_{c}$, $I_{0}$ and $I_{1}$ are arrays of luminance values in the current block and the two prediction blocks from the lists 0 and 1 reference pictures, respectively. According to the aforementioned constraint that BDOF is applied only to a true bi-directional prediction with the same prediction distance, the remaining motions relative to both reference pictures are assumed to be in a mirroring relation.

In such a symmetric motion model illustrated in Fig. 14, each sample in $I_{c}~ $can be approximated from two directions, one from its correspondence $A$ in $I_{0}~ $and the other from its correspondence $B$ in $I_{1}$ using Eq. (3.3.7). By minimizing the difference between two predictions with refined motion, the value of $(v_{x},v_{y})$ is calculated as

The vector $(v_{x},v_{y})$ of each 4${\times}$4 subblock is calculated from the extended 6${\times}$6 window (denoted as $\Omega $) containing a subblock in the center, assuming that it is constant in each subblock. This way, a more stable motion field is derived with reduced computational complexity. The optimization problem in Eq. (3.3.8) can be solved using the auto- and cross-correlation of the horizontal and vertical gradients for each prediction sample ^[22].

Based on the derived motion refinement $(v_{x},v_{y})$ and the prediction sample gradients, the following adjustment is calculated for each sample in the subblock:

Finally, the bi-prediction signal of BDOF $I'_{c}\left(i,j\right)$ is calculated by adjusting the bi-prediction samples as follows:

When DMVR and BDOF are applied to a CU, DMVR is performed first and followed by BDOF. If BCW, WP, CIIP, or GPM, which include the blending process, is enabled for a CU, then the BDOF is disabled. BDOF is also disabled when a CU is coded with SMVD mode.

Prediction refinement with optical flow (PROF): PROF is used to compensate for the prediction error of a subblock-based AMC with the optical flow-based sample-wise refinement. In this way, a finer granularity of AMC, which is conducted in a block-wise manner for the trade-off between prediction accuracy and complexity, can be achieved.

After the subblock-based AMC is performed, each luma prediction sample is refined by adding a difference derived based on the optical flow equation. PROF is not applied to chroma samples.

The prediction at position $\left(i,j\right)$ in the current block $P\left(i,j\right)$ is predicted from the sample at position $\left(x,y\right)$ in the reference picture $I\left(x,y\right)$ with the subblock MV. Let $\Delta v\left(i,j\right)$ be the difference between the sample MV computed by an affine model and the MV of the subblock to which the sample $\left(i,j\right)$ belongs, as shown in Fig. 15. The prediction with the sample MV $I'\left(x,y\right)$ would be:

where $g_{x}\left(i,j\right)$ and $g_{y}\left(i,j\right)$ are the horizontal and vertical gradients of the subblock prediction, respectively, which are calculated at each sample location similar to BDOF.

In Eq. (3.3.12), the prediction refinement $\Delta I\left(i,j\right)$ is derived using the spatial gradients of each prediction sample and sample based MV offset relative to the centered subblock MV $\Delta v\left(i,j\right)$ as follows:

The prediction refinement is added to the affine subblock prediction to form the final affine prediction as

3.6.1 Adaptive Loop Filter

ALF is based upon adaptive linear filters to restore the reconducted picture to the original picture by deriving the filter coefficients from a Wiener-Hopf equation. The output samples of SAO are used as input samples, as shown in Fig. 21. When the filter coefficients are transmitted to a decoder, the derivation process of filter coefficients can be conducted using RDO to improve the coding performance. Despite the improved coding performance, the derivation process requires heavy computational loads for a practical real-time decoder of a consumer device. The filtering mechanisms are simplified during the H.266/VVC standardizations while maintaining the coding performance.

The ALF uses two 2D finite impulse response (FIR) filters with a 7${\times}$7 diamond shape and a 5${\times}$5 diamond shape applied to the luma and chroma samples, respectively ^[35]. Fig. 22 exhibits the two filter shapes to correct the center samples. The filter shapes and sizes are determined from extensive experiments with various resolutions of test videos to consider the trade-off between the coding efficiency and computational complexity ^[36,40]. The ALF uses symmetric FIR filters, in which the numbers of coefficients are 13 and 7 for the 7${\times}$7 and 5${\times}$5 filter shapes, respectively, which reduces the computational complexity ^[38,39]. A line buffer reduction is also applied to reduce the storage requirements for ALF ^[40].

The spatially neighboring samples to the center points are used for deriving the corresponding filter coefficients $c_{i}$ in Fig. 22. A filtered sample $\overset{˜}{I}\left(x,y\right)$ in the current position is corrected from the reconstructed sample $I\left(x,y\right)$ using the weighted linear combinations of $c_{i}$ with a 7-bit fractional precision and the spatially neighboring samples. Once $c_{i}$ is derived from the Wiener-Hopf equation, the filtering is defined as

where $N$ is 13 and 7 for the luma and chroma samples, respectively. $r_{i~ }$ refers to a difference between the current pixel and a neighboring sample specified in Fig. 22. Specifically, $r_{i~ }$ is computed as follows:

where $b_{i}$ refers to a clipping parameter determined by a clipping index $d_{i}$ and a sample bit depth $BD$. The clipping parameter $b_{i}$ is set to $2^{BD}$ when $d_{i}=0.$ Otherwise ($d_{i}$= 1, 2, and 3), it is $2^{BD-1-2{d_{i}}}$. The clipping index is transmitted to a decoder.

The ALF maintains a set of up to 25 filter coefficients for luma samples, which can be applied adaptively to each 4${\times}$4 subblock. A 4${\times}$4 subblock is categorized into one of 25 classes. The classification is conducted to obtain directions and activity using local gradients with Laplacian filters. Specifically, the classification index is derived from a combination of five directional properties, including a texture, strong and weak horizontal/vertical, strong and weak diagonal, and five activity properties of a subblock. As a result, a different filter can be assigned to each class. Furthermore, geometric transform, such as 90-degree rotation, diagonal, or vertical flip, can be applied to the filter coefficients before the filtering. Various directionality can be considered using the geometric transform, and the ALF handles more diverse block characteristics with fewer filter coefficients using this method ^[41].

In addition to a subblock adaptation, for a coding tree block (CTB)-level adaptation, a set of filter coefficients is obtained online using the current slice or the previous slices or 16 offline trained filter sets. For chroma samples, H.266/VVC ALF uses only the CTB-level filter adaptation using up to eight filters ^[42]. In H.266/VVC, the adaptation parameter set (APS) ^[19] is used to carry the ALF filter parameters, which include up to 25 and eight sets of filter coefficients for luma and chroma components, respectively, and clipping indices are signaled. When the same ALF coefficients are used for different slices, only the ID of a reference APS can be signaled instead of a redundant transmission. Furthermore, the ALF is controlled using on/off flags signaled in sequence, picture, slice, and CTB levels. Chroma ALF is enabled only when the luma ALF is enabled at the corresponding level.

H.266/VVC supports a wider variety of video applications with HDR and WCG, for which the in-loop filters attempt to improve the visual quality of both luma and chroma samples. A cross-component adaptive loop filtering (CC-ALF) ^[43] corrects the chroma samples in parallel with the ALF, using the correlation between the current chroma samples and the luma samples in the corresponding positions. CC-ALF applies a linear filtering operation to produce the correction of chroma samples from the luma samples as input. The CC-ALF uses diamond-shaped FIR filters without symmetric constraints.

Fig. 22. Filter shapes and sizes in VVC ALF: 7×7 and 5${\times}$5 symmetric diamond-shaped filters for luma and chroma samples, respectively.

3.6.2 Luma Mapping with Chroma Scaling

LMCS has been adopted to H.266/VVC to improve the coding efficiency by processing the dynamic ranges of input samples rather than improving visual quality directly ^[44]. The LMCS has been originally proposed to improve the coding efficiency of HDR and WCG PQ video contents in which most input video samples tend to be distributed within a relatively narrow range compared with the SDR video contents ^[45]. LMCS supports both HDR and SDR video content in H.266/VVC.

The LMCS consists of a luma mapping (LM) module and a chroma scaling (CS) module. The LM module maps luma code values from the original sample domain to an LMCS domain using a forward luma mapping ($FwdMap$), or vice versa using an inverse luma mapping ($InvMap$). In the CS module, a luma-dependent chroma residual scaling is applied to balance the impact of luma remapping ^[46].

Fig. 23 depicts the LM and CS process in H.266/VVC decoder. The shadowed regions, including inverse quantitation and transform, an intra prediction, and the luma reconstruction of an intra prediction, represent decoding processes under an LMCS domain ^[45]. Accordingly, an inter prediction and intra prediction signals go through different procedures with LMCS. For example, a residual signal and an inter prediction signal are obtained in an LMCS domain and the original domain, respectively. $FwdMap$ is conducted to convert a domain of a reference signal in a DPB for motion compensation, and the reconstruction signals are stored in a DPB after $InvMap$.

$FwdMap$ is determined using an adaptive piecewise linear model derived from related syntax elements of LMCS APS according to a dynamic range of input video samples. Specifically, the original code values are sampled uniformly into 16 pieces to calculate $OrgCW$. For each piece $i$, the number of mapped code values is defined as $binCW\left[i\right]$. To define $FwdMap$, the slope $scaleY\left[i\right]$ is calculated as

and $invScaleY\left[i\right]$ as the slopes of $InvMap$ is calculated with the inverse of $scaleY\left[i\right]$. In a decoder, $scaleY\left[i\right]$ and $invScaleY\left[i\right]$ can be derived because the difference between $OrgCW$ and $binCW\left[i\right]$ is signaled in the LMCS adaptive parameter set (APS).

For CS, forward and inverse scaling are applied to the chroma residue with a factor of $ScaleC$ and $invScaleC$, respectively. In H.266/VVC, $invScaleC$ is defined as

where $deltaCRS$ is a chroma scaling offset value. In this manner, as shown in Fig. 23, a chroma residue-scaled value is produced from a decoding process, and a chroma residue value is calculated by multiplying $invScaleC$ ^[47].

Fig. 23. LMCS in H.266/VVC decoder[26].

3.6.3 Towards Deep Learning In-loop Filter

Current in-loop filters are designed to perform a pixel-level restoration of a reconstructed image. Deep convolutional neural network (CNN) has attracted considerable attention from video coding experts because many data-driven approaches for image denoising have been actively studied in image processing and computer vision research areas ^[48]. During H.266/VVC standardization, the benefit of deep learning techniques in video compression has been discussed in AhG9 ^[48,49], and various CNN-based in-loop filters have been tested extensively in core experiments ^[50] to investigate the coding efficiency and computational complexity.

Although the CNN-based in-loop filter has not been adopted to the H.266/VVC specification, it envisioned a future research and development direction of in-loop filters in hybrid video coding standards. Currently, video coding experts continue verifying the effectiveness of neural network video coding (NNVC) ^[51].

3.7.1 Transform Skip Residual Coding

Transform skip residual coding (TSRC) is a CABAC entropy coding scheme designed especially for transform skip residual blocks. In the H.265/HEVC RExt extension, by producing a dedicated context model for flags representing absolute values greater than zero and 180-degree rotating intra predicted transform skip residuals, statistical difference between regular residual coding (RRC) and TSRC has already been considered, but only partially. TSRC, which is newly introduced to H.266/VVC, directly addresses the difference by employing the following three main features. First, instead of transmitting the last significant scan position, it encodes the quantization indices of all scan positions of a transform block. As shown in Fig. 24, the scanning direction is from top-left to bottom-right, which is a reversed version of an RRC.

Second, even when the global distribution is almost uniform, the non-stationarity of the symbol makes it possible to code the symbol indicators more efficiently using the context model. Last, the binarization of absolute level values is changed by coding more context-coded ``greater than $x$'' flags and modifying the Rice parameter derivation for the Golomb-Rice code suffix, resulting in a higher cutoff for the unary binarization prefix.

Fig. 24. Coefficient group scan for TSRC in H.266/VVC.

3.7.2 Block Differential Pulse Coded Modulation

Block differential pulse coded modulation (BDPCM), especially useful for screen content, is one of the intra prediction modes in H.266/VVC. It aims for better decorrelation of the intra-predicted residual of the screen content by replacing the usual transform of DCT or DST. Whether to do BDPCM or not is signaled by a flag at the CU level. In case BDPCM is used, an additional flag further signals the direction of block prediction in BDPCM. The BDPCM predictor is generated through either horizontal or vertical prediction using unfiltered reference samples. The residual values are quantized, and the difference signal $\overset{˜}{r}_{i,j}$ is calculated using Eq. (3.7.1) for vertical BDPCM and Eq. (3.7.2) for horizontal BDPCM.

Here, $r_{i,j}$ denotes the intra-predicted residual signal of a block at a position $\left(0\leq i\leq H-1,0\leq j\leq W-1\right)$ inside the block. The block has a size $H\times W$, and $Q\left(\cdot \right)$ denotes the quantization operation. The difference signal is transmitted to the decoder using TSRC. In the decoding process, the quantized residual $Q\left(r_{i,j}\right)$ is reconstructed using Eq. (3.7.3) for vertical BDPCM and Eq. (3.7.4) for horizontal BDPCM.

The dequantized residual $Q^{-1}\left(Q\left(r_{i,j}\right)\right)$ is added to the intra prediction to generate the reconstructed sample.

3.7.3 Intra Block Copy

Intra block copy (IBC) finds a reference block similar to a current block inside a designated area of the current frame. The reference block works as an intra-predictor of IBC, and its location is represented by a block vector (BV), which specifies its displacement from the current block. The difference in BV from its block vector predictor (BVP), noted as the block vector difference (BVD), is encoded using either the merge mode or the AMVP mode. The merge mode is the most efficient coding tools for the BV vector because it only sends the merge index with its BVD treated as zero (thus, not encoded). The merge index indicates one vector for BVP in the list of six merge candidates. The merge candidate list includes the following until it has six members in the order of the BVs of a spatially adjacent block (bottom-left and top-right), history-based BVs, and zero vectors. The AMVP mode transmits the AMVP flag and BVD, where the AMVP flag indicates one selected BVP in the AMVP candidate list of size two.

Because the IBC is developed as an intra prediction coding tool specialized for screen content, its encoder only searches the reference blocks at integer positions. Noting that the screen content is generated pixel-by-pixel by a computer, no consideration is given to representing the BV vector in the sub-pel resolution. The AMVR allows the BVD to be encoded at a higher resolution, such as in a 1-pel or 4-pel resolution. On the other hand, it is interesting to investigate the effectiveness of sub-pel BV resolution by further estimating the BV into the half-pel and quarter-pel resolution. In this context, the experiment results for screen content video ^[54] confirm that compared to camera-generated video, both half-pel and quarter-pel BVs are chosen less often in actual encoding, even if they can be used, but it is not absolute. Therefore, the screen content is rendered more by sophisticated rendering techniques, which are often used in recent screen content, as shown in Fig. 25. That is, those screen contents with many characters, lines, and graphics like ‘SlideEditing’ (this type of screen content can be referred to as legacy screen content) have chosen much fewer sub-pel BVs and permit additional BV resolutions in encoding, resulting in coding loss ^[53]. On the other hand, more natural-looking computer-generated video by sophisticated rendering processes is seen to choose sub-pel BVs, not trivially, as shown in Fig. 25 ^[54].

Fig. 25. Average ratio of BV resolution in camera-captured and screen video contents[54].

3.7.4 Adaptive Color Transform

Adaptive color transform (ACT) is a particularly effective technique for video sequences expressed in the RGB color space because it can effectively reduce the correlation among the three color components in 4:4:4 chroma format. The ACT technique in H.266/VVC has existed since H.265/HEVC-SCC. It selectively transforms the residual signal in the input color space (RGB) into the YCgCo-R luma-chroma color representation according to a CU-level flag. The maximum ACT size cannot exceed 32${\times}$32 samples to ease the cache requirement of temporarily storing all three transform blocks. The YCgCo-R transform is fully reversible and can be applied to lossless coding.

3.7.5 Palette Mode

Palette mode, one of the intra prediction modes, supports all chroma formats. Considering the relatively insignificant coding gain in small blocks and the complexity of the palette mode, it is not applied unless a CU has more samples than 16. Each sample in a CU coded in palette mode needs to signal its palette index with a representative color set called a palette. Otherwise, an escape symbol is signaled. If a sample is coded using escape symbols, its quantized component values are signaled. The palette is defined separately for the luma (Y component) and chroma (Cb and Cr components) for slices in a dual tree. In this case, the entries in the luma palette hold just Y values and the chroma palette entries include Cb and Cr values. For the single tree slices, a palette entry contains Y, Cb, and Cr values, and the palette is applied to Y, Cb, and Cr components jointly. For the single tree, the maximum palette predictor size is 63, and the maximum palette table size for a CU is 31. For the dual tree, the maximum predictor size and table size are 31 and 15, respectively.