How can a bucket of water compute?

Let's start with what you probably already know. Neural networks are powerful because they learn. You feed them data, and through an iterative process called backpropagation, they adjust millions — sometimes billions — of internal parameters until they can perform a task.

This is remarkable. But it's also extraordinarily expensive.

Training GPT-3 cost an estimated $4.6 million in compute alone. Training a modest image classifier might take hours on a GPU. Even a simple recurrent neural network (RNN) processing time series data requires careful gradient computation through every timestep, through every connection, through every layer.

Here's what's happening inside a typical neural network during training:

Every single line in that diagram is a weight — a number that needs to be carefully adjusted. And these adjustments aren't independent; changing one weight affects what every downstream weight should be. It's an enormous, coupled optimization problem.

For recurrent neural networks — the kind that process sequences like speech, text, or stock prices — training is even harder. You have to compute gradients not just through layers, but through time. This is called Backpropagation Through Time (BPTT), and it comes with nasty problems: gradients that explode to infinity or vanish to zero.

LSTMs partially solved this with clever gating mechanisms. Transformers sidestepped it entirely by processing all timesteps in parallel with attention. But both approaches still require training every single weight in the network.

Consider the scale. GPT-3 has 175 billion parameters. Training it consumed an estimated 3.14 × 10²³ floating-point operations. Even a modest recurrent network with 1,000 neurons has about a million trainable connections, each requiring hundreds of gradient updates to converge. Multiply that by the length of your training sequences, and you begin to understand why RNN training is a nightmare.

But here's a question that might seem absurd at first: what if most of those weights don't actually need to be trained?

What if the network's complex internal dynamics — the very thing that makes it powerful — could emerge from random connections? And you only needed to train a thin layer at the very end?

This isn't hypothetical. It's exactly what reservoir computing does. And it works.

Here's the radical proposition at the heart of reservoir computing:

What if the complex, recurrent part of the network was completely random — and you never trained it at all?

Instead of laboriously tuning millions of weights through backpropagation, you just... generate them randomly. Fix them. Never touch them again. The only thing you train is a simple linear readout layer at the end — which is just linear regression. Something you can solve in one step with basic linear algebra.

This sounds insane. Like saying you could build a telescope by gluing random pieces of glass together and then just adjusting where you put your eye. But it turns out this is roughly what happens, and there's deep mathematical reasons why it works.

Think of it like a kaleidoscope. You don't design the intricate patterns inside a kaleidoscope. The mirrors are arranged in a fixed, somewhat arbitrary way. But when you look through it, simple inputs (a few colored beads) get transformed into rich, complex, beautiful patterns. All you need to do is learn to read those patterns.

Here's the key architectural difference:

The numbers are striking. A traditional recurrent network with 1,000 neurons has roughly a million trainable parameters. A reservoir with the same number of neurons? Just 1,000 — the output weights. That's a 1,000x reduction in training complexity.

And because the readout is linear, you don't need gradient descent at all. You can solve it in closed form with ridge regression. Training takes seconds, not hours.

This idea — that random projections can be useful — isn't as crazy as it sounds. Think about it this way: when you throw a stone into a pond, the resulting ripple pattern is incredibly complex. It encodes information about where the stone landed, how big it was, and even the shape of the pond's boundaries. Nobody designed those ripple patterns. They emerge from the physics of wave propagation. But they contain rich information that a clever observer could decode.

A reservoir works the same way. The random connections create complex dynamics that encode the input signal in a rich, high-dimensional state. A linear readout then decodes whatever aspect of that signal you're interested in.

But this only works if the random reservoir actually does something useful with the input. So... does it?

In 2003, Chrisantha Fernando and Sampsa Sojakka published a paper with one of the most delightful titles in computer science: "Pattern Recognition in a Bucket."

Their experiment was beautifully simple. They took a literal bucket of water, attached motors to its edges to create vibrations, and pointed a camera at the water's surface. The motors created different vibration patterns corresponding to different inputs. The camera recorded the resulting ripple patterns on the water surface.

Then — and here's the key — they took the pixel values from the camera image and used them as inputs to a simple linear classifier. No neural network. No backpropagation. Just the water doing the complex nonlinear transformation, and a linear readout interpreting the result.

It worked. The water bucket could perform nonlinear classification — separating input patterns that a simple linear classifier couldn't distinguish. The water's complex wave dynamics provided exactly the kind of nonlinear transformation needed.

Try it yourself:

The water surface is a reservoir. It takes simple inputs (stone drops) and transforms them into complex, high-dimensional patterns (ripple interference). Different input combinations produce different patterns, and these patterns are rich enough that a linear readout can distinguish them.

Why does this work for XOR specifically? XOR is the canonical example of a problem that's not linearly separable. If you try to separate XOR outputs (0 XOR 0 = 0, 0 XOR 1 = 1, 1 XOR 0 = 1, 1 XOR 1 = 0) with a straight line in 2D input space, you can't. That's why a single-layer perceptron fails at XOR — it was one of the key findings in Minsky & Papert's 1969 book that temporarily killed the field of neural networks.

But the water bucket transforms the 2D inputs into a high-dimensional representation — the pixel values of the water surface. In that space, the XOR problem is linearly separable. A simple linear classifier can draw a hyperplane through the pixel space that correctly separates the four cases. The water's nonlinear dynamics did the hard work; the linear classifier just reads the result.

This is the core insight of reservoir computing: you don't need to engineer the complex transformation. Nature — or randomness — can provide it for free. You just need to learn to read the output.

But of course, in practice, we don't usually use actual buckets of water. We use mathematical models that capture the same essential properties. Let's look at how these work.

A reservoir computer has three components, and understanding each one is straightforward:

1. Input weights (W_in) — A set of random, fixed weights that project the input signal into the reservoir. If your input is a single number (like a stock price at time t), these weights spread it across all the reservoir neurons. Think of it as taking a 1D signal and spraying it into a high-dimensional space.

2. The reservoir (W) — A recurrent neural network with random, fixed connections. Each neuron receives input from other neurons and from the input layer. The connections create feedback loops, so the reservoir has memory — its current state depends not just on the current input, but on past inputs too. This is key for processing time series.

3. Output weights (W_out) — The only trained part. A simple linear combination of all reservoir neuron activations. Training these weights is just linear regression — fast, simple, and solved in closed form.

The update rule for the reservoir state is deceptively simple:

Here's an interactive reservoir you can play with. Below you'll see three sliders: the number of neurons, the spectral radius (the largest absolute eigenvalue of the weight matrix W — it controls how long the reservoir remembers past inputs), and the input scaling. Adjust them and watch how the reservoir neurons respond to a simple sine wave input:

Notice something crucial: from a single input signal (the sine wave), the reservoir produces many different response patterns. Some neurons track the input closely. Others respond with delays. Some oscillate at different frequencies. This diversity is exactly what makes the reservoir useful — it's projecting the input into a rich, high-dimensional space where linear separation becomes possible.

This is the magic of recurrence. In a feedforward network, each neuron's response depends only on the current input. But in a recurrent reservoir, each neuron integrates information from the current input and from all other neurons' previous states. This means each neuron effectively computes a different nonlinear function of the input history — a different weighted mixture of recent inputs with different delays and nonlinear transformations.

If you have 100 reservoir neurons, you effectively have 100 different "feature detectors," each capturing a different temporal pattern in the input. And these features weren't designed — they emerged spontaneously from random connectivity. That's the beauty of it.

Try adjusting the sliders above. Increase the spectral radius and notice how the neuron responses become more complex and varied. Decrease it and they become simpler, more directly correlated with the input. Increase input scaling and the responses become more nonlinear (more tanh saturation). These are the key knobs you have for tuning a reservoir.

The spectral radius — the largest absolute eigenvalue of the weight matrix (max |λ|) — is the single most important hyperparameter. It controls how much influence past states have on the current state. We'll explore this more in the next section.

In 2001 — two years before the water bucket paper — Herbert Jaeger introduced the Echo State Network (ESN), which formalized the reservoir computing idea mathematically. Around the same time, Wolfgang Maass independently developed Liquid State Machines (LSMs) for spiking neural networks. The umbrella term "reservoir computing" came later to unite these approaches.

The key concept Jaeger introduced is the echo state property: the idea that the reservoir's state should be an "echo" of its recent input history. Inputs create ripples in the reservoir's dynamics, and these ripples gradually fade over time. After enough time passes without input, the reservoir should settle back to a neutral state — it should "forget."

This forgetting is essential. If the reservoir remembered everything forever, it would just be a recording device, not a computer. If it forgot instantly, it would have no memory at all. The sweet spot is somewhere in between — and the spectral radius is a key factor in controlling where on this spectrum the reservoir sits.

Watch what happens when we send a single pulse into a reservoir and observe how the echoes decay:

This brings us to a critical practical point: the washout period. When you first start feeding data into a reservoir, its state is arbitrary (usually zeros). The first few timesteps of reservoir activity are contaminated by this arbitrary initial state, not by the actual input signal. So in practice, you discard the first ~100 timesteps of reservoir states before training the readout. This is called the "washout" or "burn-in" period.

There's also the question of memory vs. nonlinearity. A reservoir can't have infinite memory and infinite nonlinear processing power simultaneously. There's a fundamental tradeoff: high spectral radius gives you more memory but less nonlinear transformation, and vice versa. This tradeoff is intrinsic to the dynamics of recurrent systems.

Jaeger formalized this as memory capacity — the ability of the reservoir to reconstruct past inputs from its current state. For a reservoir with N neurons, the total memory capacity is bounded by N. This is a hard information-theoretic limit: you can't store more than N independent pieces of past information in N neurons.

But memory capacity is distributed across different delays. A reservoir might be excellent at remembering what happened 5 timesteps ago, mediocre at 15 timesteps ago, and useless at 50 timesteps ago. The spectral radius controls this distribution: higher spectral radius shifts memory toward longer delays, but the total capacity stays bounded at N.

This is why reservoir size matters. More neurons means more total memory capacity and a richer nonlinear representation. In practice, a larger reservoir generally helps — up to a point. A reservoir with 500 neurons will typically outperform one with 50, though with diminishing returns. Beyond a certain size, you risk overfitting and ill-conditioned state matrices, which is why regularization in the readout matters.

So how do we set the spectral radius? How do we know what regime the reservoir is in? And what exactly happens at the boundary between order and chaos? That's where things get really interesting.

This is the question that trips people up. How can a random network — one you never trained — produce useful results?

The answer comes from a beautiful idea in machine learning called the kernel trick, and specifically from Cover's theorem (1965).

Cover's theorem states: a complex pattern classification problem cast into a high-dimensional space is more likely to be linearly separable than in a low-dimensional space.

In plain English: if your data is a tangled mess in low dimensions, project it into enough dimensions and it will untangle itself. A straight line (or hyperplane) can then separate what was previously inseparable.

Here's an analogy. Imagine you have a crumpled piece of paper with two different colors of ink on it. In 2D (looking at the paper from above), the colors overlap — you can't draw a straight line to separate them. But unfold the paper into 3D, and now you can probably find a flat plane that separates the red ink from the blue ink. The crumpled 3D shape has "spread out" the patterns.

The reservoir does exactly this — it's a random nonlinear projection into a high-dimensional space. But it's even better than a static projection: because the reservoir is recurrent, each neuron's activation encodes not just the current input but a nonlinear function of the input history. So the high-dimensional space includes temporal features too — delayed versions, moving averages, nonlinear combinations of recent inputs. All for free.

This is closely related to the "random kitchen sinks" idea from Rahimi and Recht (2007): you can replace expensive kernel methods with random nonlinear features and still get excellent results. The reservoir is essentially a temporal version of this — it creates random nonlinear features that also incorporate history.

This explains why the reservoir doesn't need training. Its job isn't to learn a specific representation — it's to create a rich enough representation that a linear readout can extract whatever information is needed. As long as the reservoir is sufficiently large and has the right dynamical properties, almost any random reservoir will work.

Of course, "right dynamical properties" is doing a lot of heavy lifting in that sentence. What are those properties? This brings us to one of the most fascinating concepts in complex systems theory.

Every dynamical system — from weather patterns to neural networks to economies — operates in one of three regimes:

Ordered: The system is too stable. Perturbations die out quickly. The system always converges to the same state regardless of input. It has no memory, no complexity, no computational power. Think of a pendulum with heavy damping — push it, and it immediately stops.

Chaotic: The system is too unstable. Tiny perturbations grow exponentially. Two nearly identical initial states diverge wildly. The system is unpredictable and useless for computation because outputs aren't reproducible. Think of turbulent fluid — drop two identical dye blobs side by side, and they end up in completely different places.

Edge of chaos: The Goldilocks zone. Perturbations neither die nor explode — they propagate through the system at a stable amplitude, creating complex but reproducible dynamics. This is where computation happens. This is where life happens.

For reservoir computing, the edge of chaos is where the largest Lyapunov exponent crosses zero. This exponent measures whether nearby trajectories in state space diverge (positive = chaos) or converge (negative = order). Right at zero, the system has maximum computational capacity.

See it for yourself:

The practical implication is clear: a well-tuned reservoir operates near the edge of chaos, where it has maximal sensitivity to inputs (good for computation) while still being stable enough that the readout layer can extract meaningful patterns.

This is related to a profound idea in complex systems theory: the edge of chaos is where information processing is maximized. Too ordered, and the system can't represent complex patterns. Too chaotic, and it can't reliably transmit information from input to output. Right at the boundary, you get the best of both worlds.

Chris Langton proposed this idea for cellular automata in the 1990s, and it's been observed in systems from sandpiles to gene networks to economies. Reservoir computing provides perhaps the cleanest demonstration of the principle in machine learning: you can literally slide a single parameter (the spectral radius) across the phase transition and watch computational performance peak at the boundary.

Now that we understand the theory, let's actually build one.

Here's the recipe for building an echo state network. It's surprisingly simple:

Step 1: Create the reservoir. Generate a random N×N weight matrix W. Make it sparse (most entries zero — say, 80% sparsity). Then scale W so its spectral radius (largest absolute eigenvalue) is your target value (start with ~0.9).

Step 2: Create the input weights. Generate a random N×1 vector W_in. Scale it by your desired input scaling factor (start with ~0.5). These weights are fixed — you never train them.

Step 3: Drive the reservoir. Your input signal is a time series: u(1), u(2), u(3), ... — just a sequence of numbers. At each timestep, you feed the next value from this sequence into the reservoir: x(t+1) = tanh(W · x(t) + W_in · u(t)). There's no feedback loop here — u(t) is simply the t-th value from your input data. The reservoir state x(t) accumulates memory of all past inputs automatically through its recurrent connections. Save each state x(t) as a row in a big matrix X.

Step 4: Discard the washout. Throw away the first ~100 timesteps of X (the burn-in period where the reservoir is still "warming up" and its initial random state still dominates).

Step 5: Train the readout. This is the only training step, and it's just linear regression. You want to find weights W_out such that y(t) = W_out · x(t) ≈ target(t). In other words: find the best weighted average of neuron activations that matches your target signal. With ridge regression (linear regression + a small regularization term λ to prevent overfitting), this has a direct closed-form solution — no iteration, no gradient descent, solved in one step.

Step 6: Predict. For new inputs, feed them through the same reservoir one step at a time (same W, same W_in), and apply the trained W_out to each new state to get predictions.

That's it. No backpropagation. No gradient descent. No vanishing gradients. Let's see it in action:

Play with the parameters. Notice how increasing the reservoir size generally improves predictions (more neurons = richer representation). Notice how the spectral radius matters — too low and the reservoir can't capture temporal patterns; too high and it becomes chaotic. The sweet spot depends on the task.

Reservoir computing isn't the best at everything. It won't beat a Transformer at language modeling or a deep CNN at image classification. But there are domains where it absolutely excels — sometimes outperforming far more complex models.

Time series prediction

This is the bread and butter of reservoir computing. Predicting chaotic time series (weather, financial markets, physiological signals) is where ESNs first made their name. For short-to-medium-term prediction of dynamical systems, they're often competitive with LSTMs while training in seconds instead of hours.

The key advantage here is speed of adaptation. In many real-world scenarios — adaptive control, real-time signal processing, brain-computer interfaces — you need to train and retrain models continuously as conditions change. An ESN can be retrained in milliseconds (just re-solve the linear regression), while retraining an LSTM takes orders of magnitude longer. This makes reservoir computing ideal for online learning scenarios.

Physical reservoir computing

Here's where things get wild. Since any sufficiently complex physical system can serve as a reservoir, researchers have built reservoir computers out of:

• Photonic systems — using light bouncing through optical fibers or semiconductor lasers. These can process data at the speed of light, enabling ultra-fast computation.
• Spintronic devices — exploiting the magnetic spin of electrons in nanoscale materials. These could power next-generation neuromorphic chips.
• Mechanical systems — including the original water bucket, but also mass-spring networks and even origami structures.
• Biological systems — gene regulatory networks, bacterial colonies, and even slime molds have been shown to exhibit reservoir-like computation.

Neuromorphic computing

The brain itself might work like a reservoir. Cortical microcircuits have the right properties — recurrent connections, nonlinear dynamics, and readout via downstream areas. Reservoir computing provides a theoretical framework for understanding how the brain might compute without training every synapse.

How does it compare?

Why hasn't it taken over?

If reservoir computing is so elegant and efficient, why isn't everyone using it? A few reasons:

• Scaling — ESNs struggle with very high-dimensional or very long-range dependencies. LSTMs and Transformers handle these better because they can learn specialized internal representations.
• Hyperparameter sensitivity — While you don't train the reservoir, you do need to choose its size, spectral radius, sparsity, input scaling, and regularization. Getting these right can be tricky.
• No feature learning — The reservoir's representation is fixed. It works great when the raw representation is already informative (time series), but struggles when deep feature extraction is needed (vision, language).
• The deep learning tsunami — When GPUs made deep learning practical, research attention (and funding) shifted massively. Reservoir computing became a niche field despite its merits.

Reservoir computing is part of a broader, somewhat contrarian idea in machine learning: not everything needs gradient descent.

This idea shows up in many places. Random features and random kitchen sinks (Rahimi & Recht, 2007). Extreme learning machines, which are feedforward networks with random hidden weights. The lottery ticket hypothesis, which suggests that most of a trained network's capacity is in a small subnetwork that could have been found without full training. Even the success of transfer learning — where most of the network is frozen and only a small readout layer is fine-tuned — echoes the reservoir philosophy.

There's a deeper philosophical point too. When Fernando and Sojakka showed that a bucket of water can compute, they weren't just making a clever demo. They were suggesting something profound: computation isn't a property of the substrate, but of how we interact with it.

The water wasn't "designed" to classify patterns. Its physics — wave interference, nonlinear fluid dynamics, surface tension — create a rich enough dynamical space that, with the right readout, computation emerges. The computational structure was always there; all that was needed was someone to read it.

This leads to a provocative question: if any dynamical system with nonlinear dynamics and fading memory can potentially serve as a computational reservoir, and the universe is full of complex dynamical systems... then perhaps the universe itself is a reservoir. Every physical process — from protein folding to galaxy formation — is performing a kind of computation. We just need to learn to read the output.

That might be too philosophical for a technical article. But there's a practical message here too.

In an era where machine learning is increasingly defined by scale — bigger models, more data, more compute — reservoir computing offers a reminder that cleverness can substitute for brute force. Not always. Not for everything. But for a surprisingly large class of problems, a random network with a linear readout can match the performance of models that cost a thousand times more to train.

The field is experiencing a quiet renaissance. Physical reservoir computing, in particular, is gaining traction as the energy cost of training large neural networks becomes impossible to ignore. When your reservoir is a photonic chip running at the speed of light and consuming microwatts of power, the efficiency argument becomes overwhelming.

So the next time someone tells you that machine learning requires massive GPU clusters and billions of parameters, tell them about a group of researchers who did it with a bucket of water. And then ask them: what else might be computing, right under our noses, if only we knew how to read the output?

• Jaeger, H. (2001) — "The 'echo state' approach to analysing and training recurrent neural networks." The original ESN technical report. Dense but essential.
• Maass, W., Natschläger, T., & Markram, H. (2002) — "Real-time computing without stable states." Introduces Liquid State Machines, the spiking neural network version of reservoir computing.
• Fernando, C. & Sojakka, S. (2003) — "Pattern Recognition in a Bucket." The famous water bucket paper. Short, delightful, and mind-expanding.
• Lukoševičius, M. & Jaeger, H. (2009) — "Reservoir computing approaches to recurrent neural network training." The best survey of the field. Start here if you want one comprehensive reference.
• Tanaka, G. et al. (2019) — "Recent advances in physical reservoir computing." Covers the fascinating world of physical implementations.
• ReservoirPy — A Python library for reservoir computing. Great for getting hands-on quickly.