Evaluation and Validation of COLA in Complex Deep Neural Networks

Deep Neural Networks (DNNs) have achieved remarkable success in various applications; however, their performance can be significantly affected by hardware defects in specialized accelerators. The Error-Correcting Output Codes (ECOC) framework has been proposed to enhance fault tolerance by decomposing multi-class problems into multiple binary classification subproblems. Despite its potential, the ECOC framework faces limitations in maintaining clean accuracy and improving robust accuracy when subjected to hardware defects, primarily due to error correlation between networks. In this thesis, we evaluate the performance of a recently proposed error decorrelation framework, named COLA, which addresses the limitations of the ECOC framework. COLA primarily incorporates the Amplitude Adaptive Weight Orthogonalization technique to lower error correlation in shared layers and the Total Correlation based regularization technique to minimize output error correlation. We evaluate COLA on more complex networks, such as ResNet18 and ResNet34, using the Tiny-ImageNet dataset, demonstrating significant improvements in the performance of models employing the ECOC framework. Compared to the original ECOC model, the improved ECOC model with COLA achieves around 10% increase in clean accuracy and up to around 80% improvement in robust accuracy. However, due to the specific structure of ResNet networks, models with ECOC exhibit notably lower performance than the original networks, both with and without hardware defects. To investigate this issue, we explore the impact of shortcut connections on the ECOC model and found that the presence of these connections can increase the total correlation of the feature map, which may potentially have a detrimental effect on the performance of the model.