Check that your loss can move

Hide a direction in high-dimensional noise and ask a neural network to find it.

The setup is a planted single index model. You draw inputs x from a standard Gaussian in d dimensions, pick one secret direction u, and make the label depend on nothing but how far along that direction you are: y = f(u·x). The idea was that the network has to locate u before it can fit anything at all, and at random initialization it has essentially no signal about where u points, because a random vector in high dimensions is almost exactly orthogonal to everything.

Training here has two phases, a long, nearly blind search, then a fast fit once the direction is found. The interesting quantity is how long the search takes, and specifically how that time grows with dimension. If it scales like d you are fine. If it scales like d² you are in trouble the moment the problem gets big.

There is real theory here for SGD. There is essentially none for adaptive optimizers like Adam, which is what everyone actually uses. This gap was particularly interesting to me and hence why this blog now exists

My first finding was very suprising to me. Adam could not do the search at all. Not slowly, not at all. Every learning rate across three decades, every dimension from 12 to 1024, budgets out to 40,000 steps. SGD found the direction in 50 steps. Adam never found it.

Didn't really know what to do at this point so I did the only sensible thing, I tried to break it. Over three sweeps I pinned down exactly which part was responsible.

• It survived the grokking regime, AdamW failed 0 for 72 under multipass training with weight decay, while SGD went 12 for 12.
• It was β₂. With β₂ = 0 plus momentum, Adam escaped every time. Any β₂ ≥ 0.9 and it failed every time. The boundary was sharp, right around β₂ ≈ 0.5.
• It was localized to the first layer. Adam on the weights with SGD on the readout failed; SGD on the weights with Adam on the readout escaped fine.
• Raising ε above roughly 10⁻³ rescued it, with a clean boundary in the (ε, lr) plane.

I also debunked several of my own inuitive explanations along the way. I initially thought it might be the per-coordinate signal to noise, however that idea was dismissed rather quickly when batch sizes up to 8192 changed nothing. The noise story failed when Adam failed just as hard at zero label noise. Each probe agreed with every other one. It was, I thought, a tidy result.

It was all completely consistent, and yet failed each time

My student network was a two-layer tanh net with no bias terms:

f(x) = (1/m) Σ a_j · tanh(w_j · x)

Every term is an odd function of x, so f is odd, flip the input and the output flips. My target was He₂(z) = z² − 1, which is even, in other words flip the input and the output is unchanged.

Under a symmetric input distribution, the best L₂ approximation to an even function by an odd one is not merely bad. It is exactly zero. The positive and negative halves cancel term for term. My network was not struggling to learn the task, it was structurally incapable of expressing any part of it.

I eventually realized that the evidence had been sitting in my logs the entire time. Normalized MSE was 1.00 in every single run of all three sweeps, 1.00 meaning "exactly as good as predicting the mean." I had seen it, and explained it away as the search phase not having finished.

Using some help (thank you claude), I realized that the fix is one line, that is giving the neurons bias terms, tanh(w·x + b) ; which breaks the odd symmetry and makes the target representable. I then ran the identical experiment both ways:

With biases on, Adam matches SGD. Meaning that the "separation" evaporates.

The bias-free SGD bar is particularly interesting and worth the one to look at. It "succeeds" on the alignment metric; it locks onto the planted direction, overlap climbing past 0.9, while its loss sits at 1.01. It found the needle and still could not use it. That bar is the whole observed issue and fix, i.e. my metric and my loss were measuring different things, and I had only been watching the metric.

What I had actually discovered was that SGD and Adam break a symmetry differently in a misspecified model. That is not a fact about learning. It is barely a fact about optimizers.

With the task fixed, I foudn the real question to be whether on a problem the network can represent, does Adam's search time scale differently in d than SGD's?

No. The exponents are d^0.96 and d^1.01,a gap of 0.06, which is nothing. Adam carries a roughly constant 3× penalty in absolute time, and constant factors are exactly what tuning absorbs. The scaling is what generalizes to problems bigger than the one on your desk, and the scaling is identical.

The answer to my original question is boring. The interesting part was how hard I had to work to stop it from being exciting.

Fixing the task was not the end of it. Two more things lined up to replace the one I had just dismissed

The detector disagreed with itself

"Found the direction" was defined in my code as the network's top singular vector reaching overlap 0.6 with the planted one. I computed that overlap with a randomized low-rank SVD, fast, and in this regime, it was wrong. It draws an internal random projection and takes no seed, so the same matrix gives a different answer every call.

Across the sweep, 44 runs recorded a final overlap below, the threshold they had supposedly crossed, with no weight updates in between. Replacing it with an exact computation - same cost, deterministic - took that count to zero. I had written that function myself, in the first week, and it had propagated through all four campaigns.

The learning rate grid was too narrow

On a harder version of the task I got an exponent gap of +0.52, comfortably past the significance threshold I had set in advance. A live finding, finally.

Except Adam's best learning rate sat at the very bottom edge of the grid I had searched, and only at the largest dimension. A truncated grid that bites at one end and not the other is precisely the shape that manufactures a slope. Twelve extra runs extending the grid downward:

That one would have been the write up and something I thought was genuinely worth exploring. It died in about four minutes of compute :) because I checked whether my tuning grid was binding before I believed my own number.

Before interpreting how a model fails, verify that it could have succeeded.

I realized that a bit too late, and thats a lesson for the future I guess. I've read more papers and realized thatt most actually do not check that their o optimizers ensure their loss can actually move before you interpret anything else. Alignment metrics, overlap measures, subspace distances, etc. All of them can look spectacular while the loss sits pinned at its trivial value. If you only report the metric, a representability failure is indistinguishable from a learning result, and it will be more internally consistent than a real effect, not less, because an artifact has no competing mechanisms to muddy it.

The specific trap that got me was an odd activation with no biases, pointed at an even target. Worth knowing that quadratic activations sidestep it entirely, which is what the theory papers in this area tend to use. The hazard arrived when someone swaps in tanh for realism and inherits a parity mismatch nobody noticed

Both surviving gaps sit under the 0.25 threshold I set before running. The bottom row is the artifact, kept visible on purpose ; it is what the result looks like one check short of the truth.

Every figure on this page is generated directly from the run files using the same tuning rule as the analysis, so a plot cannot drift from the table it illustrates. Four seeds per point on the well-posed task, three on the harder one; censored runs are recorded, never dropped.

← Back to blog · GitHub