Team coursework · 2026
Image Recognition with CLIP
How do image quality and model choice affect recognition? In team coursework, we collected and refined image data, explored visual similarity with CLIP, and evaluated classification performance. I contributed to implementation and analysis while coordinating the work from experiments to the final report.
Dataset collection, implementation, analysis, and team coordination
Dataset and classifier studies complete
Classifier study · Team results on a 10,000-image test set
Collecting data we could rely on
We identified useful examples and photographed trees and poles in person to build a custom dataset. I contributed photographs and helped review which images reliably represented each category.
Trees in the background of pole photographs introduced distracting visual cues. We identified those examples and replaced unsuitable images before comparing their representations.
Background content could blur the distinction between trees and poles.
Understanding what the model sees
I worked on the CLIP embedding and heatmap code. We preprocessed images, extracted and normalized image embeddings, and calculated pairwise cosine similarity.
Across three iterations, we adjusted image order and replaced unsuitable examples. The heatmaps showed within-class grouping, alongside overlap between trees and poles.
The heatmaps showed within-category grouping, but trees and poles still overlapped.
Similarity is different from accuracy
Each cell compares the numeric representations, or embeddings, of two images. Stronger similarity does not necessarily mean the images share the same label. This analysis measures visual relationships; it does not measure classification accuracy.
From visual similarity to classification
The photo study showed how backgrounds can influence image representations. The next question was whether those representations could support reliable class predictions on a standardized dataset. We evaluated that separately using CIFAR-10.
This example shows the challenge: the true label is deer, but the classifier predicted horse. It motivated looking beyond a single accuracy score to see which categories the models confused.
What is CIFAR-10?
A benchmark of 60,000 small, 32 × 32 color images across ten object categories, including airplanes, cats, deer, and horses. This study used a 50,000-image pool, split into 25,000 training, 10,000 test, and 15,000 reserve images.
Dataset reference ↗Comparing two image encoders
I selected the training, test, and reserve split, then generated normalized 512-dimensional CLIP embeddings and trained linear classifiers for the ten categories. We compared encoders using 16 × 16 and 32 × 32 image patches with the same test set.
The patch-16 model reached 95.20% accuracy, compared with 94.01% for patch-32—a 1.19 percentage-point lead. Based on these results, we recommended patch-16 for this experiment.
Patch-16 led by 1.19 percentage points on the shared test set.
Embeddings, patches, and training settings
An embedding is a numeric representation of an image. These models divide the model input into 16 × 16 or 32 × 32 pixel patches before generating a 512-dimensional representation. The team trained linear classifiers using one-hot class labels, mean squared error (MSE) loss, and stochastic gradient descent (SGD). The report lists 3,000 epochs and a learning rate of 5.5. An epoch is one pass through the training set; the learning rate controls the size of parameter updates.
Looking beyond overall accuracy
I identified weaker categories by examining the confusion matrices: initial cat accuracy was 88% for patch-16, while deer accuracy was 89% for patch-32.
Adding 1,500 cat examples did not change the reported cat result. Adding 1,500 deer examples improved that class result by about one percentage point.
Within the assigned study of additional training data, I chose targeted additions based on the weaker classes and compared their effects.
+1,500 cat images: no change in cat accuracy. +1,500 deer images: about a 1-point improvement in deer accuracy.
How to read the confusion matrices
Rows represent the true class and columns represent the predicted class. Values along the diagonal represent correct classifications; off-diagonal values show which classes were confused. The displayed proportions help compare results within each true class.
What more training data changed
We expanded the training set from 25,000 to 40,000 images while keeping the 10,000-image test set unchanged. The extra 15,000 images came from the training reserve.
Despite a 60% increase in training data, patch-16 remained at 95.2%, and patch-32 changed only slightly. Adding more data alone produced limited benefit in this experiment.
That finding directed our recommendations toward other changes: preprocessing, classifier complexity, loss functions, and optimizers. These were proposed next steps, rather than completed experiments.
60% more training data; no overall accuracy gain for patch-16. Accuracy plateaued at 95.2% in this experiment, suggesting saturation under the tested setup.
Keeping the team’s work connected
Alongside implementation, I managed timelines, organized and categorized shared files, tracked progress, and coordinated integration of experiments and report sections.
I contributed to the report and presented part of the final results. We checked the final submission against the course requirements together.