Evaluation of the CLIP Architecture for Zero-Shot Image Classification on the Intel Image Classification Dataset

Abstrak

The performance evaluation of various architectures in the Contrastive Language–Image Pre-training (CLIP) model was conducted in a zero-shot image classification scenario. Image classification was performed using the Intel Image Classification Dataset, which consists of 3000 images representing several environmental categories. This study compares several CLIP architectures based on ResNet and Vision Transformer. Model performance was evaluated using accuracy, F1-score, precision, and recall metrics. The experimental results show that the RN50x16 architecture achieved the best performance with an accuracy of 0.925, an F1-score of 0.925, a precision of 0.929, and a recall of 0.925. The RN101, RN50x64, and ViT-B/32 architectures also demonstrated relatively strong performance with accuracy values around 0.92. In contrast, the ViT-B/16, ViT-L/14, and ViT-L/14@336px architectures produced lower performance with accuracy values below 0.90. Furthermore, the Mean Cosine Similarity Matrix analysis indicates that models with ResNet-based architectures produce clearer class representation separation compared to several Vision Transformer variants. Overall, the results suggest that the choice of architecture significantly influences the performance of the CLIP model in zero-shot image classification, with RN50x16 emerging as the most optimal architecture for the dataset used.

Identifier

• DOI: https://doi.org/10.55749/ijdrr.v1i1.188
• OpenAlex: https://openalex.org/W7169795658