6.1.1. CNNs

Convolutional Neural Networks

A convolutional neural network is a two-dimensional scanning network designed for data with spatial structure, such as images. A CNN applies the same learned local function (a convolutional kernel) to many overlapping spatial patches of the input, producing feature maps that detect patterns regardless of their position.

Given an input $x\in {\mathbb{R}}^{H\times W\times C}$, a convolutional layer applies a bank of $K$ filters, each of size $F\times F\times C$. The $k$-th filter produces an $(H-F+1)\times (W-F+1)$ feature map,
$a_{i,j,k}={\sum }_{u,v,c}{W}_{u,v,c}^{(k)}{x}_{i+u,j+v,c}+b_k,z_{i,j,k}=h(a_{i,j,k}),$
where $h(\cdot )$ is a nonlinearity such as ReLU. Note that padding is frequently employed to restore the feature map to the shape of the original input, such that in general, padding by $P$ rows and columns leads to a shape of $(H+2P-F+1)\times (W+2P-F+1)$. If convolution is done with a stride of $S>1$, then the dimension of the output is $(D+2P-F)//S+1$

A CNN therefore inherits the key properties of scanning networks:

• Sparse connectivity: each neuron sees only an $F\times F$ patch.
• Shared connectivity pattern: the same kernel weights $\{W,b\}$ are reused at every spatial location.
• Shift equivariance: shifting the input produces a corresponding shift in the activation map.

Stacking convolution layers grows receptive fields hierarchically, enabling early layers to detect edges and later layers to capture textures, shapes, and object-level structure. Optional pooling layers (max-pooling or average-pooling) downsample spatial dimensions by aggregating over local neighborhoods, providing robustness to small shifts and scale variations:
$z_{i,j,k}^{(\ell )}={\max }_{(u,v)\in P}{z}_{i+u,j+v,k}^{(\ell -1)}.$

After several such layers extract increasingly abstract features, the final spatial representation is flattened and passed to one or more fully connected layers that compute the output

Examples

Convolution Operation in CNNs

In this question, we will look at the mechanism of weight sharing in convolutions. Suppose that we have an input vector with $D=9$ dimensions:
$x={\left[\begin{array}{c}{x}_1,x_2,\dots ,x_9\end{array}\right]}^T.$
We compute a 1D convolution with the kernel filter that has $K=3$ weights (parameters):
$k={\left[\begin{array}{c}{x}_1,x_2,x_3\end{array}\right]}^T.$

What’s the output dimension if we apply filter $k$ with no padding and stride of $1$? What are the first and last elements of the output? Given a stride of $1$ and no padding, $D_{\text{out}}=9-3+1=7$. The first element is $k_1{x}_1+k_2{x}_2+k_3{x}_3$, whereas the last element is $k_1{x}_7+k_2{x}_8+k_2{x}_9$.

What’s the output dimension if we apply filter $k$ with padding of size 1 and stride of 2? What are the first and last elements of the output? Given a stride of $2$ and padding of $1$, we have that $D_{\text{out}}=(9-3+2\cdot 1)//2+1=5$. The first element will be $0+k_2{x}_1+k_3{x}_2$, whereas the last element will be $k_1{x}_8+k_2{x}_9+0$.

Recall that CNN filters have the property of weight sharing, meaning that different portions of an image can share the same weight to extract the same set of features. A convolution is actually a linear operation, which we can express as $x\prime =Kx$ for some weight matrix $K$.

Find $K$ for the convolution applied in the first part. What is its dimension? $K$ has a dimension of $7\times 9$: it needs to match the shape of the input $x$ which has a dimension of $3$, as well as map the input to an output vector with dimension $7$ (as calculated in the first part). Each row has 3 nonzero elements: $K=\left[\begin{array}{ccccccc}{k}_1& k_2& k_3& 0& \dots & 0& 0\\ 0& k_1& k_2& k_3& 0& \dots & 0\\ & & & ⋮& & & \\ 0& 0& 0& \dots & k_1& k_2& k_3\end{array}\right].$

Consider now a 2-dimensional example with the following image and kernel filter:
$X=\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\\ 7& 8& 9\end{array}\right],K=\frac{1}{4}\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right].$

Using no padding and a stride of $1$, compute the output, and describe the effect of this filter. We overlap the filter with each $2\times 2$ window of the image and record the values: $\hat{X}=\left[\begin{array}{c}1/\end{array}\right]$