Teaching a Computer to Read the Sky

Building an 11-class weather image classifier in PyTorch — Trained entirely from scratch (no pretrained models).

Kevin Lin 5 min read ·  2026

At first glance, weather classification looks like a pretty straightforward computer vision task. But once you start working with real images, things get messy quickly. Lighting changes, noise, and similar categories like frost and snow or fog and smog make the problem much harder than expected.

The goal

The goal of this project was to build a model that can recognize 11 different weather conditions from about 6,900 images.

I trained everything from scratch without using any pretrained models, so the model had to learn all the visual features directly from the data.

The 11 categories were dew, fog/smog, frost, glaze, hail, lightning, rain, rainbow, rime, sandstorm, and snow.

The final test accuracy was 77%, which is a lot better than random guessing (~9%). More importantly, it showed that the model was actually learning patterns from the images instead of just memorizing them.

What I built

I built a simple convolutional neural network that learns visual patterns directly from images.

The main components were:

• Stratified sampling to make sure each weather class was evenly represented in training, validation, and test sets.

• A 4-layer CNN to extract visual features like edges and textures and gradually combine them into higher-level patterns.

• A validation set to check performance during training and make sure the model wasn’t overfitting too much.

• A test set to evaluate the final performance on unseen data.

Results 

• Final test accuracy: 77% 

• Random baseline: ~9% 

So overall, the model learned meaningful patterns instead of just guessing.

Training behavior

I tracked training and validation accuracy and loss over time to see how the model was learning.

At the beginning, training was a bit unstable, which I expected since the model was trained from scratch. After a few epochs, the loss started to go down more consistently.

Adding Batch Normalization layers helped stabilize training a lot. It made the learning process smoother and faster. I also added a Dropout layer, which helped reduce overfitting.

By the end of training, the training and validation curves were pretty close, which was a good sign that the model was generalizing reasonably well.

What I learned

• Training a model from scratch really forces you to pay attention to architecture choices, even minor ones. 

• Batch Normalization layers made a noticeable difference in stabilizing training. 

• The dropout layer helped reduce overfitting, especially when the dataset is not very large. 

• PyTorch and scikit-learn made it much easier to experiment and iterate quickly.