Neural Sequence Models for Time Series

Neural Sequence Models for Time Series

The classical time-series toolkit covered in 2.5 and 2.6 uses linear models with Gaussian noise: well-understood, statistically efficient when the model is correctly specified, and the right default for short records or when interpretability matters. When the signal is genuinely non-linear, the dependencies span very long ranges, or the input is multi-dimensional (e.g., a sound spectrogram, an image stream, a multi-sensor recording), neural sequence models often outperform the linear ones.

Three families of neural architectures dominate sequential modeling, all of them already covered in the deep learning section. Each one is built around the same idea (apply the same parameterized local function to every position in the sequence), but they differ in what the local function takes as input and how far it can see.

The deep-learning treatment of these architectures lives in Computer Science/5. Deep Learning/, where the focus is on the architecture, training, and computer-vision/NLP applications. This note exists to point time-series readers at those notes and to summarize the trade-offs that matter when choosing between a classical linear model and a neural one for a time-series problem.

Architectures and where they live

• Convolutional Neural Networks: apply a learned local filter to overlapping fixed-size patches of the input. For 1-D time series, “patch” means a fixed-length window of consecutive observations, and the filter slides across the time axis (a 1D conv). Naturally captures local structure and translation-invariant patterns; receptive field grows linearly with depth, so very long-range dependencies require many layers or dilated convolutions.

• Recurrent Neural Networks: apply a recurrent transition function that updates a fixed-size hidden state at every time step. The hidden state in principle carries information from arbitrarily far in the past; in practice vanishing/exploding gradients limit the effective range. LSTMs and GRUs are gated variants that improve long-range learning. Strictly sequential, so they cannot be parallelized across time during training.

• Transformers: apply self-attention, allowing every position to look at every other position directly. The receptive field is the entire sequence in a single layer; training is fully parallelizable across time. The downside is $O(T^2)$ memory and compute in the sequence length, which has driven a research industry around efficient-attention variants.

When to reach for a neural model on time-series data

Reach for neural models when the classical linear toolkit underperforms in ways that suggest non-linearity or long-range structure. Concrete signals: residuals from a fitted ARIMA retain visible structure that no $(p,d,q)$ choice removes; the input is naturally multi-dimensional (audio spectrogram, image patches, multivariate sensor stream); or the dataset is large enough ($T\gg 10^4$) that the network has the data it needs to learn the non-linear structure.

Stay with the classical toolkit when records are short, when interpretability matters (the classical models report coefficients with confidence intervals; the neural models do not), when the underlying process is genuinely linear (which is more common than the deep-learning literature suggests), or when you need the fitted model to come with closed-form forecast distributions and confidence bands. Production forecasting in business and economics still uses ARIMA and exponential-smoothing variants for exactly these reasons.

Specific time-series uses

1-D CNNs for time series are the natural choice when local patterns matter: speech and audio recognition (filters detect phonemes or formants), seismic event detection, electrocardiogram classification. The same architecture used for image classification works after rotating the convolution from 2D to 1D and treating the time axis as the analog of one spatial axis. For time-lagged regression problems with very many lags, a 1D CNN with shared weights across time replaces the wide unstructured design matrix with a compact structured one: far fewer parameters, and the same regularization story.

RNNs and LSTMs for time series are the older standard for sequence forecasting and anomaly detection. They handle variable-length inputs naturally, and the recurrent structure produces forecasts step-by-step in the same way as ARMA models. Most “deep learning for time series” tutorials from 2015–2020 use LSTM forecasters as the baseline. They have largely been overtaken by transformers in research benchmarks but remain widely deployed in production because they are smaller and cheaper.

Transformers for time series have become the strongest forecasters on long-range, large-dataset problems since around 2020. The Informer, Autoformer, and PatchTST family of models adapt the transformer architecture to time series, typically by patching the input (treating each window of observations as a “token”) to control the $O(T^2)$ cost. For short or medium-length records, however, transformers often underperform the simpler classical or LSTM baselines. The architecture’s data appetite is real, and it has to be matched against the available data volume.

See Also

• 2.5 ARIMA Models: classical linear-model alternative to neural sequence models
• 2.6 State Space Models: Bayesian linear-model alternative
• 2.4.3 Time-Lagged Regression: the classical version of “predict $y$ from windowed $x$” that 1D CNNs generalize
• 6.1. Scanning Networks: the unifying scanning-network framework that all three architectures fit into
• 6.1.1. CNNs: convolutional neural networks
• 6.1.3. RNNs: recurrent neural networks (vanilla, LSTM, GRU)
• 7.1. Transformers: self-attention-based sequence models