Transformer vs. LSTM for Funding Rate Prediction: A Rigorous Comparison

Abstract Funding rates are the most-predictable noise in crypto: they are bounded, mean-reverting, and on major venues fit an Ornstein–Uhlenbeck / AR(1) process with a half-life measured in dozens of funding steps, not minutes. This makes them a punishing test case for the fashionable claim that the Transformer supersedes recurrent networks on every sequential task. We compare the two architectures not as black boxes but through their inductive biases, stripped to the cleanest possible proxies: a recurrent proxy — a one-parameter exponential-moving-average recurrence (the minimal LSTM/GRU), which places shared decaying weights on the past — and an attention proxy — a free-weight linear model on T lags (the minimal linear-attention / patch-flattened Transformer). On AR(1) data matching funding rates, the recurrent proxy reaches the irreducible noise floor from a few hundred samples, while the attention proxy pays a sample-complexity penalty proportional to context length: its excess test error scales as T·σ²/N versus σ²/N, a factor of T. Monte Carlo confirms the closed form within 15% across a 50× range of N. The attention proxy only overtakes the recurrent proxy when the true lag structure is non-exponential (multi-timescale), and only once its estimation variance has shrunk below the recurrent model's approximation bias — in our experiments a crossover near N ≈ 3,000–5,000, larger than the per-symbol funding sample (n = 4,105 for BTC). The honest conclusion for funding-rate prediction: the recurrent inductive bias is correct, the Transformer is overspecified, and switching architectures cannot substitute for switching to a regime where the signal lives. Hypotheses - H1 (high confidence): On AR(1)/OU data with the half-life of real funding rates, a shared-decay recurrent proxy reaches the irreducible noise floor from O(100) samples, while a free-weight attention proxy over the same context T pays an excess-test-error penalty scaling linearly in T. - H2 (high confidence): The penalty T·σ²/N is structural — it is the cost of estimating T−1 spurious near-zero coefficients that the recurrent model never has to fit — and is reproduced by Monte Carlo to within 15% of the closed form across N ∈ {100 … 5000}. - H3 (moderate confidence): The attention proxy overtakes the recurrent proxy only under non-exponential lag structure (multi-timescale DGP), at a crossover sample size scaling as N\ ≈ T·σ²/δ where δ is the recurrent model's irreducible approximation bias. Data Provenance | Quantity | Type | Source | Reliability | |----------|------|--------|-------------| | Funding OU half-life (per symbol) | Estimated (OLS AR(1)) | PerpsTrader fundingrates table, 944,150 rows | RELIABLE (n=4,105/symbol, MLE std-err < 0.5 steps) | | φ (AR(1) coefficient) | Estimated | Same; de-meaned, winsorized at ±200% annualized | RELIABLE | | Excess-test-error scaling T·σ²/N | Derived | Regression generalization for T free coefficients vs 1 shared decay | RELIABLE (form) | | Multi-timescale crossover N\ | Derived + simulated | Variance-shrinks-below-bias condition | RELIABLE (form) / UNRELIABLE (constant, collinearity-dependent) | | MC test MSEs | Simulated | Self-contained numpy, exact script below, fixed seeds | RELIABLE (reproducible to 3 decimals) | The data source for every simulated number is a fixed random seed; the funding-rate statistics were data collected via real-time API from the PerpsTrader venue set (from Binance, from Hyperliquid, and from Asterdex) and are used both as motivation and as the calibration anchor for the AR(1) parameters φ and σ. Simulated AR(1) and multi-timescale processes are SIMULATED, not observed market data. Motivation: Funding Rates Are Short-Memory We fit an AR(1) to the annualized funding rate of nine majors over 4,105 observations each: | Symbol | φ | Half-life (funding steps) | |--------|------|--------------------------| | ARB | 0.9611 | 17.5 | | LINK | 0.9677 | 21.1 | | BNB | 0.9694 | 22.3 | | AVAX | 0.9695 | 22.4 | | BTC | 0.9724 | 24.8 | | XRP | 0.9730 | 25.3 | | ETH | 0.9813 | 36.7 | | DOGE | 0.9837 | 42.1 | | SOL | 0.9890 | 62.6 | Every half-life is short: even SOL, the slowest, forgets half its deviation in 63 funding intervals. This is the OU regime — exactly the family a one-parameter exponential recurrence can represent without error. The population mean annualized rate is −0.45% across 944,150 observations, and the per-symbol dispersion is violent (MOVE ranges −2166% to +1.37% annualized), but the temporal structure within each symbol is simple. That simplicity is the whole argument. Analysis 1. The two proxies, stripped to their inductive bias Predict x[t] from its T most recent values. The architectures collapse to: - Recurrent proxy (LSTM/GRU minimal): ŷ = β·h, where h = Σ{k=1..T} α^{k−1}·x[t−k]. Two parameters (decay α, scale β). α→0 isolates lag 1 (the AR(1)-optimal feature); α→1 averages all history. - Attention proxy (linear-attention / flattened-patch Transformer minimal): ŷ = Σ