7.2. Positional Encoding

Positional Encoding

Since transformers are order-invariant (i.e., self-attention treats all positions symmetrically), positional encodings are added to token embeddings to inject information about sequence order. Given token embeddings $X\in {\mathbb{R}}^{N\times d}$, the model forms position-aware representations
$\tilde{X}=X+R,$
where $P\in {\mathbb{R}}^{N\times d}$ assigns a distinct encoding $r_n$ to each position $n$.

An effective positional encoding should satisfy four criteria:

1. Uniqueness: each position receives a unique vector $p_n\in {\mathbb{R}}^d$, ensuring that the model can distinguish token order.
2. Boundedness: encodings remain on the same scale as the original features (e.g., $[-1,1]$), which stabilizes training.
3. Distance-awareness: the representation should make relative distances between positions easy to infer - an inductive bias important for both language and vision models.
4. Scalability: the scheme should extend gracefully to arbitrary sequence lengths, since inputs vary in size.

The original transformer uses sinusoidal encodings, varying between even and odd positions:
$p_n^{(2i)}=\sin \left(\frac{n}{10000^{2i/d}}\right),p_n^{(2i+1)}=\cos \left(\frac{n}{10000^{2i/d}}\right),$
since sinusoids make relative position linearly recoverable: if a query at position $n$ wants to attend to a key at position $n-\Delta$, then there exists a rotation matrix $R_{\Delta }$such that $p_{n-\Delta }=R_{\Delta }{p}_n$.

Another famously used positional encoding method is Rotary Positional Embeddings (RoPE), which does not add a position vector to embeddings, but instead rotates the query and key representations in attention by a position-dependent angle.

• The key idea is to treat pairs of coordinates as a 2D plane and rotate them; then attention scores depend naturally on relative offsets. For a head dimension $d_h$ (assumed even), define frequencies
  ${\theta }_i=\frac{1}{{\text{base}}^{2i/d_h}},i=0,\dots ,\frac{d_h}{2}-1,$
  and for token position $m$, rotate the $(2i,2i+1)$ coordinates by angle $m{\theta }_i$. Writing $x_{m,2i},x_{m,2i+1}$ for the two coordinates of a vector $x_m$ at position $m$, RoPE applies

x’_{m,2i} &= x_{m,2i}\cos(m\theta_i) - x_{m,2i+1}\sin(m\theta_i),\\ x’_{m,2i+1} &= x_{m,2i}\sin(m\theta_i) + x_{m,2i+1}\cos(m\theta_i). \end{aligned}$$ In practice this is implemented without explicit matrices by precomputing $\cos(m\theta_i)$ and $\sin(m\theta_i)$ and using a “rotate-half” operation that swaps the two halves of the vector with a sign flip. RoPE is especially attractive in decoder-only LMs because it builds in a strong **relative-position** inductive bias while remaining simple to compute and parameter-free.