3.6. Gradients

Gradient

The gradient of a scalar-valued function $f:{\mathbb{R}}^n\to \mathbb{R}$ is the vector of partial derivatives with respect to each component of its input vector $x=[x_1,x_2,\dots ,x_n{]}^T$ that points in the direction of steepest ascent at $x$:
$\nabla f(x)={\left[\begin{array}{cccc}\frac{\partial f}{\partial x_1}& \frac{\partial f}{\partial x_2}& \dots & \frac{\partial f}{\partial x_n}\end{array}\right]}^T\in {\mathbb{R}}^n.$

The magnitude of the gradient $\Vert \nabla f(x)\Vert$ gives the rate of increase in the direction of $x$. It is important to note that $f(x)=0$ indicates a stationary point, and not necessarily an extremum.

The Gradient Lives in the Domain domain, not the output space.

The gradient doesn’t lie on the surface of the function - it lives in the

For a function $E:{\mathbb{R}}^2\to \mathbb{R}$, the gradient $\nabla E:{\mathbb{R}}^2\to {\mathbb{R}}^2$ is 2-dimensional, describing direction and rate of change within the input plane, not in the 3D surface of $E(w)$.