6.1. Scanning Networks

Scanning Networks

A scanning network is a neural networkarchitecture that applies the same local computation repeatedly across many overlapping regions of a structured input. Instead of connecting every neuron to the entire previous layer, a scanning network restricts each neuron to a local receptive field and reuses the same weights at every valid position.

Formally, a scanning network consists of a local function $g:{\mathbb{R}}^k\to \mathbb{R}$ operating on a small patch of the input, and a scanning rule that applies $g$ at each spatial location $i$ of the input, producing outputs $y_i=g(x_{i+u_1},\dots ,x_{i+u_k})$, where $\{u_1,\dots ,u_k\}$ are fixed offsets defining a receptive field.

This leads to two defining properties:

• Sparse connectivity: The weight matrix is block-sparse; most connections do not exist.
• Shared connectivity pattern: Every output neuron uses the same local pattern of connections, just shifted spatially.

As such, scanning networks can be interpreted as a network where each neuron is connected to only a local subset of neurons from the previous layer, and this local connectivity pattern is repeated across positions:

\begin{document}
\begin{tikzpicture}[>=stealth, scale=1.0]
 
% ----- top subnetwork -----
\node[circle,draw,inner sep=3pt] (t1) at (0,1.5) {};
\node[circle,draw,inner sep=3pt] (t2) at (0.4,2.0) {};
\node[circle,draw,inner sep=3pt] (t3) at (0.4,1.0) {};
\node[circle,draw,inner sep=3pt] (t4) at (1.1,1.5) {};
\foreach \u in {t1,t2,t3} \draw[->] (\u) -- (t4);
 
\node[circle,draw,inner sep=3pt] (to) at (2.0,1.5) {};
\draw[->] (t4) -- (to);
 
% ----- middle subnetwork -----
\node[circle,draw,inner sep=3pt] (m1) at (0,0) {};
\node[circle,draw,inner sep=3pt] (m2) at (0.4,0.5) {};
\node[circle,draw,inner sep=3pt] (m3) at (0.4,-0.5) {};
\node[circle,draw,inner sep=3pt] (m4) at (1.1,0) {};
\foreach \u in {m1,m2,m3} \draw[->] (\u) -- (m4);
 
\node[circle,draw,inner sep=3pt] (mo) at (2.0,0) {};
\draw[->] (m4) -- (mo);
 
% ----- bottom subnetwork -----
\node[circle,draw,inner sep=3pt] (b1) at (0,-1.5) {};
\node[circle,draw,inner sep=3pt] (b2) at (0.4,-1.0) {};
\node[circle,draw,inner sep=3pt] (b3) at (0.4,-2.0) {};
\node[circle,draw,inner sep=3pt] (b4) at (1.1,-1.5) {};
\foreach \u in {b1,b2,b3} \draw[->] (\u) -- (b4);
 
\node[circle,draw,inner sep=3pt] (bo) at (2.0,-1.5) {};
\draw[->] (b4) -- (bo);
 
% ----- max node -----
\node[draw,rectangle,rounded corners,minimum width=1.0cm,minimum height=0.7cm]
     (max) at (4,0) {max};
 
\draw[->] (to) -- (max);
\draw[->] (mo) -- (max);
\draw[->] (bo) -- (max);
 
\end{tikzpicture}
\end{document}