1. Modified Face Recognition Pipeline in Mixed-mode

To solve the main power bottleneck due to high-

Fig. 2. Proposed analog-digital mixed-mode architecture

resolution ADCs, we propose an analog-digital mixed-mode architecture composed of an image sensor, an analog CNN processor, and a DSP as shown in Fig. 2. The ADCs used to convert raw image signals in the analog domain to high-resolution digital signals are now replaced by the analog CNN processor with a ternary quantizer. Unlike the conventional architecture processes all CNN pipelines in the digital domain, the proposed architecture pushes a part of CNN processing into the analog domain for low power consumption. The analog CNN processor computes the input layer of the CNN pipelines in the analog domain with very high resolution (> 32-bit floating point) and then quantizes the results into 3 digital values, i.e., 0, 1, 2. To this end, we modified a conventional face recognition pipeline that has two different CNNs for face detection and face recognition, respectively, to having a shared input layer and then diverge into separated CNN layers. To adopt the shared input layer, at first each face detection and face recognition network is trained independently and the input layer of the face detection network is replaced to the input layer of the face recognition network. After that, while the input layer is fixed, the rest of the layers are re-trained for fine-tuning. The detailed face detection and face recognition network configuration is shown in Fig. 3.

Fig. 3. Face detection & recognition network configuration

face recognition network configuration is shown in Fig. 3. As the roles of the input layer processing are the same from both CNNs, the modified recognition pipeline doesn’t affect algorithmic accuracy. More specifically, the baseline FR accuracy using 32-bit floating point (FP) input and weight is 97.52%. First, to perform analog to digital domain conversion and to replace the power consuming ADC, we adopt ternary quantization in output of the first layer. In that case, simulated FR accuracy is degraded to 96.9%. Then to reduce the total power consumption in analog-digital mixed-mode CNN, the input bit-width and weight bit-width of overall convolutional layers are reduced to 8-bit fixed point except the shared input layer which adopts 4-bit fixed point for the weight bit-precision. Overall, the modified face recognition pipeline achieves 96.18% recognition accuracy in the LFW dataset ⁽⁸⁾ while the baseline face recognition algorithm achieves 97.52% recognition accuracy with 32-bit floating-point weights. This 1.37% accuracy loss is promising considering we used much lower resolutions in weight values and used ternary quantization after analog computation. Additionally, compared to the latest binary weight face recognition processor ⁽⁵⁾ which achieves 96% recognition accuracy, the accuracy of 96.18% is quite reasonable.

The unified face recognition pipeline in the proposed mixed-mode architecture is summarized as follows. First, when the image sensor captures the image, the analog CNN processor processes the shared layer of face detection and face recognition CNN. It then quantizes the results to ternary value and the ternary value chooses 16-bit data in look-up-table (LUT) that goes into the digital domain. These 16-bit data are the pre-transposed digital value of half-VDD, one-third, and two-thirds of

Fig. 4. Power reduction of proposed architecture

maximum analog processing value which is pre-captured in maximum search unit in analog CNN processor. This maximum value can cover the whole range of analog processing outputs. Then, based on these outputs, the rest of face detection CNN layers detect the face region of interest from the entire image, and if a face is detected, the rest of face recognition CNN layers decide whether identification result is true or not by reusing these ternary value.

The proposed analog-digital mixed-mode architecture can eliminate the power consumption for domain conversion in conventional architecture. Fig. 4 illustrates the power reduction of proposed architecture compared to the previous conventional architecture ^(7,9,10) with different kinds of ADC. The power of each image sensor is normalized to the 320×240 pixel array and 1-fps frame rate. In the proposed architecture, the image sensor cell array and readout circuit consume 0.624 uW and 0.517 uW, respectively, and analog CNN processor consumes 57.7 uW. This total power of 58.84 uW can achieve at least 57.9% power reduction compared to the various type of ADC based image sensor and this power gain is due to the absence of high-resolution ADC array which is the power bottleneck of conventional architecture.

2. Overall Block Diagram

Fig. 5 illustrates the overall block diagram of the proposed analog-digital mixed-mode face recognition that has 3 main components: a CMOS image sensor (CIS), an analog CNN processor, and a digital processor. The CIS is composed of an array of 320×240 3T pixels and a CDS readout circuits that pass image pixel data to

Fig. 5. Overall block diagram

the analog CNN processor. The analog CNN processor, which performs the shared layer of face detection and face recognition CNN pipelines, has 5 building blocks: analog memory, current-based weighted-sum unit, analog-ReLU unit, maximum value searcher and ternary quantization unit with LUT. The image pixel data read from the CIS row-by row are buffered to the analog memory that can hold up to 320×3 pixels. Thereafter, 3×3 values in analog memory corresponding to the 3×3 kernel are selected, and the data are broadcasted to the 64 current-based weighted-sum units corresponding to a 3×3×64 weight kernel. When the required convolution operation with the 3×3 kernel is completed for the 320×3 input image, a new row of CIS is loaded and repeat the above convolution operations until the last row of CIS. Finally, the output of the current-based weighted-sum unit is converted to the digital value through the analog ReLU and ternary quantization. At this time, the maximum value of the analog convolution operation is pre-captured by the maximum value search unit which will be used as the digital value in LUT. The digital processor includes a DNN processor that accelerates the rest of the face detection and face recognition CNN pipelines with a cluster of convolution cores and an aggregator. It also has a controller for controlling both an analog processor and a DNN processor.

1. Reconfigurable CDS Readout Circuit for Adaptive Domain Conversion

The CIS pixel value is read through the CDS readout circuit and is passed to the analog CNN processor.

Fig. 6. Reconfigurable CDS readout circuit

Although the CMOS image sensor requires a high supply voltage of 2.5 V to cover a large dynamic range of illuminance, the analog CNN processor does not need a high supply voltage that causes high power consumption. Therefore, we lowered the supply voltage of the analog CNN processor to 1.2 V and reduced power consumption by 52% compared to 2.5 V supply voltage in overall analog computation. However, if the supply voltage of the analog processing is lowered, it is necessary to change the sensor output to fit into the corresponding lower supply voltage range. Although the output value of CIS is variable depending on the amount of illuminance, the readout circuit should be able to convert the output of CIS into the desired voltage range regardless of the amount of illuminance. Therefore, as shown in Fig. 6, a reconfigurable CDS readout circuit is proposed. The reconfigurable CDS readout circuit consists of a sample & hold with switched capacitor circuit for readout the pixel value and the adaptive domain conversion is implemented through the reconfigurable voltage reference circuit. Because of the switched capacitor operation, when the amount of illuminance is large, voltage reference should be increased to convert pixel value into the desired output voltage range (0.23 V ~ 0.78 V). Therefore, by using four reconfigurable voltage reference value (0.748 V, 0.922 V, 1.292 V, and 1.84 V), it can cover the full range of 60-900 illuminance when simulation conducts in 61,500 sensitivity (e-/lx*s) pixels with 2 ms exposure time. These four reconfigurable voltage reference values are implemented by a high output multi-voltage reference circuit which is presented in ⁽¹¹⁾. By using this, from the low reference value to the high reference value, a wide range of voltage references can be generated and the desired reference voltage value can be selected according to the amount of illuminance in

Fig. 7. Leakage tolerant analog memory w/ exposure time division

the digital controller. Finally, by using the proposed reconfigurable CDS readout circuit, we can realize the desired analog processing in various amounts of illuminance.

2. Leakage Tolerant Analog Memory w/Exposure Time Division

Analog memory is structured as 320×3 source follower based memory cells and it holds the 3 rows of pixel array for calculating convolution operation with 3×3 weight kernel. The input data stored in a 3x3 window in the analog memory are broadcasted into the 64 current-based weighted-sum units and each of the current-based weighted-sum units has 3×3 weight values. In this case, each row of memory is connected to the corresponding row of weight kernel by the SEL signal of the switching network. Then the convolution operation can be conducted by controlling REN signal in memory cell the same as the 3×3 weight kernel sweep. Therefore, by controlling the REN and SEL signal, convolution operations with the desired 3×3 weight kernel are possible.

In Fig. 7(a), the timing diagram of analog processing is represented. For single 3×3 convolution computing, analog memory should hold the value during the 0.9 us. Therefore, for computing whole data to the 3×3 weight kernel, analog memory should maintain a row data for 1.14 ms and the loss of image data should be kept to a minimum during this time. As shown in Fig. 7(b), the 55.6 fF of the storage capacitor has only 1.9% and 2.3% variation in minimum and maximum input value which is quite negligible in analog processing.

Additionally, considering the analog processing time in the above timing diagram, there is a problem that the exposure time of CIS is too long as 91.2ms when the rolling shutter type CIS is used for a small area. In this case, the pixel value itself can be saturated. Therefore, as shown in Fig. 7(c), the exposure time-division scheme is used during the analog CNN processing. To prevent the CIS pixel from being saturated, a controllable reset is inserted in the between fixed selection and reset signal. This reset signal can be controlled according to the amount of light in the digital controller.

3. Error Tolerant Current-based Weighted-sum Unit

The value of the analog memory transfers into the current-based weighted-sum unit and performs the convolution operation with the 3×3 weight kernel. The proposed error-tolerant current-based weighted-sum unit is shown in Fig. 8. For current-based multiplication, the output voltage of the analog memory is converted to the current domain through a linear V-I converter. The converted current of linear V-I converter is duplicated by using drain regulation and the full range of input value is converted linearly into the current as shown in Fig. 9(a). In the case of current based analog processing, it is important to reduce the main branch current to achieve low power operation. To that end, the main current is designed to be less than 0.5uA for the full range input value. This main current is multiplied by the weight using a switched drain regulation (SDR) current mirror. Overall SDR current mirror consists of a PMOS SDR current mirror which will be operated when the sign is 1, and an NMOS SDR current mirror which will be operated when the sign is 0. Because of the low value of the main current and small size of the mirroring transistor, it is important to reduce the mirroring error. By using a negative feedback loop for drain regulation and designing to have at least 100 mV overdrive voltage for the mirroring MOSFET, we reduced the mirroring error to under 6%, as shown in Fig. 9(a). This is very low considering the simple current mirror technique makes higher than 50% mirroring error. Because of the current domain advantages, the currents multiplied with the

Fig. 8. Error tolerant current-based weighted-sum unit

Fig. 9. Measurement results of error tolerant current-based weighted-sum unit

weight are converted to the voltage domain by simple accumulation of current and passive element without additional accumulation logic. Finally, 15.09 uW low power operation is possible.

Additionally, for accurate analog computation, the mismatch of the analog convolution unit should not affect the final quantization value. Therefore, as shown in Fig. 9(b), Monte-Carlo simulation is conducted in the output of the analog convolution unit. As a result, it is confirmed that when the output mismatch is the largest which is corresponding to the largest input voltage with maximum gain, the 3-sigma range of the output is 14 mV, which is within 1/2 LSB when compared with the LSB of the ternary quantization. In addition, when the output voltage is within the full 3-sigma range, it is also confirmed that the 3-sigma value is 13.02 mV which is within 1/2 LSB.

Fig. 10. Maximum value searcher & ternary quantizer

4. Maximum Value Searcher & Ternary Quantizer

The output of weighted-sum unit is converted to the digital domain through a simple analog ReLU unit consisting of a comparator and a multiplexer that do not require DC bias currents for low power operation, and ternary quantizer as shown in Fig. 10. The analog ReLU unit determines the output value based on half VDD which indicates digital zero. If the output of weighted-sum unit is larger than half VDD, it passes the output of weighted-sum unit. In opposite, the output is driven to the half VDD. After that, the output of analog ReLU unit is quantized by ternary quantizer. The threshold value of quantizer is determined by half of MAX and ReLU-enable signal (R_EN). The MAX is the pre-determined value which is the maximum value of previous frame. In order to use the maximum value of the previous frame, the maximum value searcher is integrated. The analog output of convolution result is stored in the Ctemp temporarily and compared with the Cstore which has the previous maximum value. When the output of comparator is high, Cstore is updated to the current analog convolution output. At last, by using these quantized value to the index of LUT, the final digital value (half VDD, 1/3MAX, and 2/3MAX) is generated and the remaining CNN operation is performed through the digital processor.

Fig. 11. Layout photograph

1. Layout Implementation Results

The proposed analog-digital mixed-mode face recognition processor is implemented in the Samsung 65 nm CMOS logic process. Fig. 11 depicts its overall layout photograph. The proposed processor occupies a 3.6 mm×4.4 mm area where 3T-based 320×240 CIS, analog CNN processor, and digital processor are integrated on a single chip. The processor operates at three different supply voltage domains for imaging, analog processing and digital processing. The operating clock frequency of the analog domain is set to 20 MHz while the operating frequency of the digital domain can range from 50 MHz to 200 MHz. The maximum frame rate of the implemented processor is 5-fps. At 1-fps frame rate, the analog domain consumes 58.8 uW for imaging and analog convolution operation and the digital domain consumes 146.2 uW at 0.77 V supply voltage with 50 MHz frequency.

2. Simulation Results

Fig. 12(a) shows the face detection and face recognition accuracy of the proposed mixed-mode face recognition processor. We used the UMD dataset ⁽¹²⁾ and the LFW dataset ⁽⁸⁾ for measuring face detection accuracy and face recognition accuracy, respectively. For the case of face detection, the final decision is made based on the binarized result whether a face exists in the image or not. With the modified deep CNN model proposed for mixed-mode processing, we achieved 97.7% of face detection accuracy and 96.18% face

Fig. 12. Simulation results of proposed processor

recognition accuracy, which are only 1.61% and 1.37% worse than the baseline 32-bit floating point implementation.

Fig. 12(b) shows the power reduction of the proposed processor. By reducing overhead of high-resolution ADCs, the proposed mixed-mode process dissipates only 64 uW in always-on operation mode (58.8 uW in the analog domain and 4.98uW in the digital domain), which is 33.3% reduced from the state-of-the-art face recognition processor ⁽²⁾. Overall, the proposed processor dissipates 0.205 mW power for entire face recognition which is 66.9% lower than ⁽²⁾. Fig. 12(c) represents the detailed power breakdown when the processor performs always-on operation and overall system operation. When the processor performs always-on operation, the analog CNN processor, CMOS image sensor, and digital processor consumes the 57.7 uW, 1.141 uW, and 4.98 uW power, respectively. As you can see here, the analog CNN processor consumes the largest power portion which is 90.1%. However, in case of overall system operation, as the face recognition network is much deeper than the face detection network, the number of computation processed by digital processor is increased and finally, digital domain consumes 0.146 mW power which is the 71.2% of the overall power consumption. Table 1 summarizes the comparison of the proposed processor with the latest face recognition processors. Reference ⁽²⁾ and ⁽³⁾ can support both face detection and face recognition but ⁽¹³⁾ can only support the face recognition. Unlike the previous works utilized

traditional feature-based face detection or recognition, the proposed processor adopts a deep CNN algorithm in both face detection and face recognition to obtain high recognition accuracy. Despite CNN’s large amount of computations, the proposed processor consumes the least power by reducing always-on power using low power analog circuits and by reducing the weight bit-precision in digital computation. Moreover, even the proposed system integrates the CMOS image sensor and face detection process for implementing the whole face recognition system, the overall power consumption is 17.4% lower than ⁽¹³⁾.