Tiny CNN Walkthrough

A 6×6 image goes through a complete convolutional network: two 3×3 kernels, ReLU, 2×2 max pooling, flatten, a dense layer and softmax. Each stage has the matching NumPy or TensorFlow code, with the output it prints for the current values; open it with Show code. Draw by clicking pixels, and edit any kernel value, dense weight or bias. Section 7 trains the weights from scratch instead.

0Where the values come from

Every number in a neural network has one of four origins. Each stage below is tagged with the same labels, and the dots on the grid titles mark where each grid comes from.

Input dataComes from outside the model. During training it's the dataset; once the model is in use, it's whatever the user or an application sends in. Real images are usually resized and scaled to 0–1 first.Here: the 6×6 image you draw.
Design choiceDecided by whoever builds the model, before training starts: the architecture and hyperparameters. They stay fixed while the model trains.Here: 2 kernels of 3×3, stride 1, no padding, ReLU, 2×2 pooling, 4 classes.
LearnedThe model's parameters. They start as random numbers and are adjusted by training on labelled examples. Afterwards they're saved with the model (for example in a .keras file) and loaded each time it's used.Here: kernel values, dense weights and biases, set by hand instead so the output is predictable.
ComputedProduced while one image passes through the network, from the input and the parameters. Recalculated for every image and discarded afterwards. These are the activations, or feature maps.Here: conv, ReLU and pool grids, the flattened vector, logits, probabilities and the prediction.

Labels, the correct answer for each training image, are a fifth source. They come with the dataset, usually from people annotating examples, and are only used during training to measure how wrong a prediction is. They never enter a prediction, so they don't appear below.

1–3Convolution, ReLU and pooling

The 3×3 kernel slides over the image. Press Play, use the arrows, or hover any number to see which input pixels produced it. The kernel values are editable.

Input6 × 6
→
Vertical kernel3 × 3, editable
→
Conv6 − 3 + 1 = 4
→
ReLUmax(0, x)
→
Max pool4 ÷ 2 = 2
positivenegativezero
Where this comes from
  • Input dataInput image. You draw it here. In practice it comes from the dataset during training, or from the user when the model makes a prediction, already resized to 6×6 and scaled so pixels are 0 or 1.
  • Design choiceLayer settings. Two kernels, 3×3 each, stride 1, no padding, ReLU, and 2×2 max pooling. ReLU and pooling have no weights of their own; they are fixed operations.
  • LearnedKernel values. Learned during training, starting from random numbers. Each kernel also normally has a learned bias, left at 0 here.
  • ComputedConv, ReLU and pool grids. Calculated from this image and the kernel. Their sizes follow from the settings: 6 − 3 + 1 = 4, then 4 ÷ 2 = 2.
Python (NumPy)

4Flatten

Both kernels run on the same image. Their two pooled 2×2 maps are laid end to end as one vector of 8 numbers. Nothing is computed here; the shape just changes from 2×2×2 to 8.

Vertical mapx0–x3
Horizontal mapx4–x7
→
Flattened vector8 values
Where this comes from
  • ComputedThe 8 values. The two pooled maps from step 3, one per kernel, laid end to end.
  • Design choiceLength and order. 8 = 2 kernels × 2 × 2, which follows from the earlier settings. Putting the vertical map first is a convention; Keras interleaves the maps instead (see the TensorFlow code).
Python (NumPy)

5Dense layer

Each class has one row of 8 weights and a bias, all editable. Its logit is the dot product of that row with the flattened vector, plus the bias. Hover a row or edit a value to see its sum.

Where this comes from
  • Learned32 weights and 4 biases. Learned during training, starting from random weights and zero biases. Hand-set here.
  • Design choiceTable shape. 8 columns because flatten produces 8 values, 4 rows because there are 4 classes. Keras stores the same numbers as an (8, 4) kernel.
  • ComputedLogits. Calculated for this image: each row of weights times the flattened vector, plus the bias.
Python (NumPy)

6Softmax prediction

Where this comes from
  • Design choiceThe 4 classes and their names. Fixed by the labelled dataset the model is trained on. A digit recognizer would have 10.
  • ComputedProbabilities and prediction. Softmax of the logits, then the class with the highest probability.
  • Input dataCorrect label (training only). During training, the label that comes with each image is compared to these probabilities to compute the loss, which drives the weight updates. Not used when predicting.
Python (NumPy)

7Train mode

Start from random weights and let gradient descent find them. Each epoch runs all 54 example shapes through the network in batches of 8, and the optimizer nudges every weight against its gradient. To compare optimizers, train once, change the optimizer, and press Rerun from same start: the earlier loss curve stays on the chart. Steps 1–6 above update as the weights change.

Epoch0
Loss–
Accuracy–

Press Randomize weights to start from scratch, or train from the current hand-set weights.

Training set

Green border: the network currently classifies it correctly. Red: wrong. Click a shape to load it into the input above.

Teach the current input

Takes the image in the input grid, compares the prediction with the answer you pick, and applies a single plain SGD update, whatever optimizer is selected above. This is the worked example from the explanation: rows move by (p − y) × x.

Where this comes from
  • Input dataTraining images and their labels. In a real project these come from a collected and annotated dataset, often thousands of examples. Here the page generates 54 shapes in code, each with its correct class.
  • Design choiceTraining settings. Optimizer, learning rate, batch size (8), number of epochs, and how the weights are initialised (Glorot uniform, as Keras does). These are hyperparameters: tuned by hand, not learned.
  • ComputedLoss and gradients. Calculated for every batch: the loss says how wrong the predictions are, the gradients say which way to move each weight to reduce it.
  • LearnedThe weights. This is where they get their values. Each step moves every kernel value, dense weight and bias by −learning rate × gradient.
Python (NumPy)

Training code for the current settings. No output is shown: training starts from random weights, so your numbers will differ from the page. Accuracy normally climbs to 0.9–1.0.