1. Local layout effect

The LLE is a phenomenon observed in silicon semiconductor designs, where the performance metrics of a device such as the mobility, threshold voltage, and sub-threshold swing are influenced by the placement of surrounding devices and specific parameters, including the type, width, length, and gate pitch. Moreover, LLE varies significantly across different processes ^[4]. Consequently, each process can introduce new effects, eliminate previous ones, or alter established trends. Therefore, the optimal device placement is process-specific and depends on the node being utilized. We used a Gate-all-around (GAA) logic process node and nanosheet FET devices. As depicted in Fig. 1(a), when viewed from the top, the device shows a vertical red gate layer, an active region representing the doping layer that distinguishes the device type, and a nanosheet layer that becomes the actual channel. The common overlapping area of these three layers is the target area for calculating LLE parameters, with the edge serving as the reference point, as indicated by the yellow border.

In Fig. 1, the three different LLEs to be integrated into the rewards of Q-learning are depicted. The length of diffusion (LOD) term in Fig. 1(b) refers to the distance from the edge of the diffusion break to the channel edge of the target device ^[5]. The LOD effect, which is caused by STI and has been modeled analytically in prior research ^[6], exhibits a similar trend to historical findings, leading to the adoption of the term. As illustrated in Fig. 1, for a multi-finger gate structure of a device, estimating this effect requires two parameters as inputs: the distance on the left, labeled ``a,'' and the distance on the right, labeled ``b.'' Deep trench isolation (DTI) effectively mitigates the gain in parasitic NPN devices in CMOS processes by maintaining a distance between the edges of channels ^[7]. While DTI proves beneficial for enhancing device performance within the context of scaling trends, it introduces new LLEs. A simple example in Fig. 1(c) shows the placement of DTI when P-type devices are arranged. Two parameters, one for the north and one for the south, are considered from the target channel edge. `${DTI}_n$' denotes the parameter for the north direction. The newly developed LLE is proportional to the size of the DTI and the stress resulting from its application to the channel. This leads to variation in device performance. Previous research analytically modeled the well proximity effect (WPE) ^[8]. The effect manifests due to scattering during ion implantation, leading to a doping gradient in the gate channel. As illustrated in Fig. 1(d), four direction parameters, ${NW}_a$, ${NW}_b$, ${NW}_n$, and ${NW}_s$, are considered from the target channel edge.

2. Dataset preparation using PDK

The dataset for the three effects previously described was generated using a PDK procured from the foundry. Consequently, the PDK was presumed to represent an optimal model-hardware correlation and proceeded through the following method. The extent to which the threshold voltage ($V_t$) of each device changes with placement was obtained through SPICE simulation. The simulation results for all possible combinations were collected by adjusting the positions of individual devices, each of which had a single gate finger, the smallest unit used in the design.

The method used to measure $V_t$, which is universally adopted by foundries for measurement, specification, and monitoring, involved the constant current method in Equation (1). The effective width and length of each device, designated as $W_{eff}$ and $L_{eff}$, respectively, are calculated in accordance with the specific device structure. The critical current, $I_{crit}$, is set to $100$ nA, and the gate-source voltage, $V_{gs}$, is incrementally increased from $0$ V to $VDD$, with $V_{DS}$ representing the input voltage, $VDD$. Once the current flowing through the channel reaches $I_{crit}\frac{W_{eff}}{L_{eff}}$, the voltage at the saturation state is defined as the threshold voltage, $V_{tsat}$.

3. Results of surrogate models trained using LASSO

In this paper, we employed an ANN model to create LLE surrogate models. The model structure used in our study is briefly illustrated in Fig. 2. The primary reason for selecting an Artificial Neural Network (ANN) in this study is its robust capability to regress any arbitrary non-linear function that defines the relationship between input process parameters and device performance, particularly in the process setup stage where Local Layout Effects (LLE) are either under ongoing research or lack compact models ^[9]. Additionally, ANNs can be easily extended into a multi-input-multi-output (MIMO) model by simply adjusting the number of nodes in the output layer ^[10].

In the loss function $J\left(w\right)$ of Equation (2), the last term, $\sum{{\left|w\right|}^*}$, represents L1 regularization, specifically the least absolute shrinkage and selection operator (LASSO). This was employed to reduce the size of the input feature vector. Reducing the size of the input vector allows for the creation of a surrogate model comprising only features that influence the output.

Fig. 2. LLE surrogate model structure.

However, while this approach enables the creation of reduced surrogate models for known LLE effects, it is more practical to use ridge regression with L2 regularization to handle unknown LLEs that may arise in advanced process nodes. Nonetheless, in this paper, we omitted this approach as the primary objective was to verify changes in trends for known effects.

The training results of the ANN model are presented in Table 1. The input features listed in the table, prior to feature selection, are the sum of the parameters inherent to the device structure and the parameters derived from the geometric positioning information resulting from device placement. Nodes represent the number of nodes in the hidden layer, while layers indicate the total number of hidden layers.

The concept of $Accuracy^*$, as indicated in Table 1, employs the notion of a relative error margin to assess the accuracy of the trained ANN. The related equations are (3) and (4). There will always be slight errors when compared with the true values due to the limitations of computers. Hence, the concept of a relative error margin was employed. Consequently, the rationale behind the 100\% accuracy observed in Table 1 can be attributed to this concept.

From a physical design perspective, selectable LLE features have been identified as lateral LOD and vertical DTI based on currently available knowledge. The criteria for selection were determined by how sensitively the trained LLE surrogate models respond within the given input range. Using LASSO, effects not represented in the input feature matrix related to device placement were also eliminated. Consequently, by comparing Tables 1(a) and 1(b), we observed that feature-reduced surrogate models could be developed without any loss of accuracy compared to the models without feature reduction. The results of trained LOD, DTI, and WPE surrogate models are presented in Fig. 3 as two- and three-dimensional plots. The values of the $x$-, $y$- and $z$-axes were normalized. Device placement was not considered in WPE, as its variation is minimal in comparison to the other two effects.

Table 1. Dataset for training LLE surrogate models using artificial neural networks. Inputs, nodes, and layers are hyperparameters of the ANN model. (a) Without L1 regularization and (b) with L1 regularization.

The threshold voltage of a device is calculated with the LOD and DTI surrogate models in relation to the right-side distance (${LOD}_a$) and north-side distance (${DTI}_n$) in Fig. 3 on the right side of (a) and (b). The remaining parameters are held constant. Therefore, for the LOD effect, placement is optimized to a low-gradient saturation region (${LOD}_a>{0.3}$). Conversely, effects with linear characteristics, such as DTI, are managed to minimize the area while ensuring that the effect is consistently matched across devices.

Therefore, the introduced surrogate models will act as an essential tool for automatically identifying optimal placements that account for the LLEs. This will be further elucidated in Section III.

1. Problem formation

1) Quantizing design space for states ($s$) and actions ($a$)

To implement a practically computable version of Q-learning, actions have been quantized into a finite action space. For device placement, five possible actions--north, south, west, east, and stationary--have been defined within this space, considering each device as a state. Due to the requirement that each action must be selectable within a finite space, the environment in which the states move is also quantized into a grid-based format. The process employed was the GAA process node, utilizing the nanosheet FET device. The available channel widths are constrained to discrete values. Although it is possible to construct an analog block using devices with disparate channel widths, this approach requires specific gate length values and the inclusion of spacers. This method is inefficient from an area perspective and introduces an additional LLE, termed tapered RX, to neighboring devices ^[11]. Furthermore, the contact-to-poly pitch (CPP) is fixed for each device gate length, and a design rule exists that only allows devices with the same gate length to be merged. Consequently, in order to minimize the size of a circuit block with a specific function, the channel width and gate length are standardized. As a result, a grid-based environment, as illustrated in Fig. 4(b), can be implemented for device placement. The horizontal grid is defined as one CPP, and the vertical grid corresponds to the channel width of the device used in the design. The blue and orange colors represent different types of devices, abstracted in the manner shown in Fig. 4(a).

2) Reward functions with LLE surrogate models

In analog circuits, small variations in the threshold voltage ($V_{th}$) can result in significant output current deviation from the nominal value. This deviation increases in proportion to the number of devices, rendering these applications highly susceptible to threshold voltage variation. By extracting geometric parameters from the placed devices and utilizing the trained LLE surrogate models, we can directly obtain the estimated threshold voltage of the devices. This enables placement optimization under LLE considerations. Therefore, reward functions are integrated with the LLE surrogate models.

The reward of the ${i}$-th device or ${i}$-th cluster state at time step $t$ in the $x$-th sequence represented by $r^{{i}}_{x,t}$ encompasses a number of factors, including the area, wire length, LLE parameters, type clustering, sub-circuit clustering, threshold voltage difference and estimated threshold voltage by LLE surrogate models. The area reward and wire length reward ($r_{area,t}$ and $r_{wire,t}$) are set with the purpose of minimizing the area and length, respectively. They are applied as reversed rewards, where smaller related parameters result in larger rewards.

(6)

LLE parameter rewards $(r_{\Delta LOD,t}$, $r_{\Delta DTI,t})$ consisted of LOD parameter rewards and DTI parameter rewards. These rewards consider similarity of LLE parameters $( {\Delta }LOD$, ${\Delta }DTI )$ among devices ($i$ and $j$) at time step $t$. The differences in each of the LEE parameters serve as the criteria in Equation (6) for determining whether a penalty or an optimal reward value should be applied. ${\nu }_1$ and ${\nu }_2$ are hyperparameters that are controlled during reward tuning.

Clustering rewards ($r_{clustering,t}$) are used primarily for two purposes: clustering by type and clustering by same sub-circuit (ssc). Type refers to the categorization of devices into N- and P-types to maximize device matching. Within the same sub-circuits, such as differential pairs or current mirrors, or even multi-finger devices, split into multiple single-finger devices, as shown in Fig. 7(a), device pairs or groups are critical to achieve matching. Therefore, they must be as close as possible and placed in geometrically identical environments, such as symmetric and row-based arrangements. To accomplish this purpose, clustering rewards are used with LLE parameter rewards in the proposed algorithm. This will be explained in the next chapter. In Equation (7), ${{\Omega}}_t$ refers to an optimal reward that provides additional value when the device groups are placed close enough while considering layout constraints at time step t. ${\overline{d}}_{type,t}$, ${\overline{dx}}_t$, and ${\overline{d}y}_t$ in Equation (7) refer to the distance between devices of the same type, the coordinate difference in the x-axis direction, and the coordinate difference in the y-axis direction, respectively, at time step t. $\theta$, $\mu$, ${\sigma}$ and ${\omega }$ are hyperparameters that are controlled during reward tuning.

The purpose of the difference in threshold voltage reward ($r_{dvth}$, $t$) is to reduce differences in estimated threshold voltage within the same sub-circuit topology and for the same type of device. This reward is calculated through Equation (8). $p$ is the number of sub-circuits in the input circuit. $m$ is the number of devices in the same sub-circuit. $k$ and $l$ is the total number of P-type and N-type devices, respectively. $V_{th,LOD}$ and $V_{th,DTI}$ are calculated using the trained LLE surrogate models explained in Section II. $\alpha$, $\beta$, $\delta$, and $\zeta$ are hyperparameter controlled during reward tuning.

(9)

(10)

(11)

(12)

(13)

(14)

The objective of utilizing $r_{vth\_lle,t}$ is to optimize the estimated threshold voltage ($V_{th}$), which is calculated by the LOD surrogate models. As illustrated in Fig. 3(b), in contrast to DTI, there is a saturated region in proximity to the zero gradient at the maximum or minimum threshold voltage. Therefore, the objective is to maximize or minimize $V_{th, LOD}({LOD}_a,{LOD}_b)$ while simultaneously minimizing the parameters ${LOD}_a$ and ${LOD}_b$. In the case of P-type devices, Equation (9) is employed, whereas for N-type devices, Equation (11) is utilized. As a result, the outputs of Equations (9)-(12) are integrated into Equations (13) and (14) as optimal ($r_{optimal}$) and penalty ($r_{penalty}$) reward threshold values. In the case of a P-type device with a threshold voltage, $V^{{i}}_{th}$, the reward for said device ($r^{{i},P-type}_{vth\_lle,t}$) is calculated using Equation (13). Conversely, the reward for device $j$ $\left(r^{{j},N-type}_{vth\_lle,t}\right)$, an N-type device with threshold voltage $V^{{j}}_{th}$, would be calculated using Equation (14). Ultimately, like Equation (15), the reward value, $r_{vth\_lle,t}$, is determined by aggregating the outputs of the aforementioned equations. The two hyperparameters, $w_1$ and $w_2$, are calibrated during the reward tuning process.

2. Proposed algorithm

The Q-learning algorithm utilized in this paper employs Watkins-Dayan ^[12]-based Q-learning, as demonstrated in Equation (16). $\alpha$ and $\gamma$ represent the learning rate and discounted factor, respectively. Additionally, epsilon-greedy exploration is applied to balance the trade-off between exploitation and exploration, with probabilities of $1-\epsilon$ and $\epsilon$, respectively. This approach allows the agent to reinforce the evaluation of known good actions while also exploring new actions ^[13]. In this context, $x$ represents the sequence index, $t$ represents step number, ${i}$ represents the ${i}$-th device, and $n$ represents the total number of devices, respectively. The reward value, states set, and actions set of devices in sequence $x$ at step $t$ are ${\mathrm{R}}_{x,t}$, ${{\mathbf{S}}}_{x,t},$ and ${{\mathbf{A}}}_{x,t}$ in Equations (17), (18), and (19), respectively. Former related research ^[14] applied deep Q-learning that a state represents all devices' placement and agent selects actions of all devices. However, an alternative approach has been taken whereby individual Q tables ($q^{{i}}_{x,t}(s^{{i}}_t,a^{{i}}_t)$) have been applied for each state-action pair of ${i}$-th device, rather than applying a shared Q table for all devices. It was observed that selecting all actions simultaneously in a single step of Q-learning led to a failure in convergence to an optimal layout, instead resulting in a persistent oscillatory pattern. The root cause of this issue lies in the trade-off relationship between LLE rewards and traditional metrics such as area and wire length, combined with the dynamic nature of LLEs, which continuously vary and exert influence based on the real-time relative distances among all devices. Moreover, the Q-table structure enables the effective representation of state and action pairs for individual devices, thereby facilitating straightforward management. Consequently, as the number of devices increases, the number of Q-tables increases correspondingly. However, this approach may not fully capture the interactions between devices, especially when the optimal action for one device depends on the state or action of another device. To address this limitation, we applied a total reward (${\mathrm{R}}_{x,t}$) that considers all devices in the circuit, as shown in Equation (17). Fig. 5 illustrates the structure of the proposed algorithm. Based on that, in order to emulate knowledge-based design flow used by experts, a sequential algorithm is implemented to identify the optimal placement and ensure efficient convergence to the optimal solution. For this purpose, rather than simultaneously applying all reward functions previously described, each sequence selectively uses them according to Equation (20) to calculate the next reward value at step $t+1$. The agent selects the current actions for devices that yield the maximum Q values based on the current states array and the reward at sequence $x$. After taking actions, the next states at step $t+1$ include updated features for each device such as relative distances, device type, length, coordinates, and LLE parameters. Those features interact with sequentially selected rewards. Fig. 6 depicts the overall proposed algorithm.

Fig. 5. Structure of the proposed Q-learning's agent and environment.

Fig. 6. Sequential Q-learning algorithm.