What Is Grokking? A Short Intro & Open Questions

What is Grokking?

Grokking is a phenomenon in deep learning where a neural network achieves near-zero training loss but initially fails to generalize (i.e., the validation loss remains high), continuing to perform poorly on test data. After extended training, the model suddenly undergoes a phase transition, rapidly learning to generalize effectively to unseen data. The delay between memorizing the training set and generalizing to the test set is the whole phenomenon, and it can be enormous: in the original paper's own experiments, generalization sometimes took over a thousand times longer to arrive than memorization did.

The term was introduced by Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra, in a January 2022 paper titled “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets.” They borrowed the word from Robert Heinlein's 1961 novel Stranger in a Strange Land, where a Martian word, grok, means to understand something so completely that you merge with it, literally “to drink,” figuratively to absorb something in its entirety rather than merely know facts about it.

Literature Findings

A comprehensive 2026 survey by Bertolotti and Cazzola reviewed the field down to 45 papers judged directly relevant, which helps to understand where things stand: not one settled explanation, but eleven distinct, partially overlapping mechanisms, each with real evidence behind it.

An important point mentioned in the survey is that these mechanisms are not fully reconciled. Several papers report grokking under conditions a given mechanism predicts should prevent it, negative regularization that should push a model away from the Goldilocks zone, models that grok without ever showing the slingshot dynamics that one proposed explanation calls essential, tasks with no weight decay at all.

Can any model grok?

Grokking has now been documented well beyond the small transformers it was first observed in. The survey's own accounting includes multi-layer perceptrons, transformers, ResNets, convolutional networks, and even non-neural models like Gaussian processes and Bayesian neural networks. The data side is also broad: while the classic demonstrations use algorithmic tasks, modular arithmetic, sparse parity, permutation composition, grokking has also been reported on MNIST, CIFAR, and even masked language modeling with BERT-style models, a result usually credited to Liu, Michaud, and Tegmark's “Omnigrok” paper, which argued grokking is not restricted to algorithmic data at all.

Grokking is not a property of one specific architecture; it shows up across multiple model families. But it is also not automatic or guaranteed for any of them. It depends heavily on hyperparameters that are easy to get wrong by accident, weight decay strength, initialization scale, and how much of the available data you train on.

Toy Example

The clearest way to see grokking is to reproduce it in miniature, and the architecture below follows Liu, Kitouni, Nolte, Michaud, Tegmark, and Williams, “Towards Understanding Grokking: An Effective Theory of Representation Learning” (NeurIPS 2022). Their paper studies a deliberately simple toy model rather than a full transformer: two numbers are mapped to learned embedding vectors, the embeddings are summed, not concatenated, and the sum is passed through a small decoder MLP that predicts the result.

The task itself is one of the simplest found in the literature: given two numbers $a,b$ from $0$ to $p-1$, predict $(a+b) \bmod p$. Here $p=97$. The model trains on half of the $p^2$ possible pairs and is evaluated on the other half it never sees.

import torch
import torch.nn as nn

P = 97              
TRAIN_FRACTION = 0.5

class GrokkingMLP(nn.Module):
    def __init__(self, vocab_size=P, embed_dim=128, hidden_dim=512):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.decoder = nn.Sequential(
            nn.Linear(embed_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, vocab_size),
        )

    def forward(self, a, b):
        summed = self.embedding(a) + self.embedding(b)
        return self.decoder(summed)

The dataset is every pair, split into a training half and a held-out half:

import numpy as np
import torch

def make_dataset(p, train_fraction, seed=0):
    pairs = [(a, b, (a + b) % p) for a in range(p) for b in range(p)]
    rng = np.random.default_rng(seed)
    rng.shuffle(pairs)
    cut = int(len(pairs) * train_fraction)
    return pairs[:cut], pairs[cut:]

def to_tensors(pairs):
    a = torch.tensor([t[0] for t in pairs])
    b = torch.tensor([t[1] for t in pairs])
    c = torch.tensor([t[2] for t in pairs])
    return a, b, c

train_pairs, test_pairs = make_dataset(P, TRAIN_FRACTION)
train_a, train_b, train_c = to_tensors(train_pairs)
test_a, test_b, test_c = to_tensors(test_pairs)
print(f"P={P}: {len(train_pairs)} train pairs, {len(test_pairs)} test pairs")

Grokking depends heavily on the training regime, not just the architecture. What actually gets it to happen reliably is full-batch gradient descent and a weight decay strong enough to matter, a recipe established in Power et al.'s original paper and used throughout the follow-up literature, including Nanda et al.'s mechanistic analysis of this same task. Weight decay of 1.0 is the value most commonly reported; dropping it by two orders of magnitude has been reported to make grokking take around sixty times longer to arrive, without stopping it from happening, which is a demonstration of just how sensitive this phenomenon is.

from torch.optim import AdamW
import torch.nn.functional as F

def accuracy(model, a, b, c):
    with torch.no_grad():
        return (model(a, b).argmax(dim=1) == c).float().mean().item()

def train(model, num_steps=30000, lr=1e-3, weight_decay=1.0, log_every=1000):
    optimizer = AdamW(model.parameters(), lr=lr, weight_decay=weight_decay)
    history = []
    for step in range(1, num_steps + 1):
        optimizer.zero_grad()
        logits = model(train_a, train_b)
        loss = F.cross_entropy(logits, train_c)
        loss.backward()
        optimizer.step()

        if step % log_every == 0 or step == 1:
            train_acc = accuracy(model, train_a, train_b, train_c)
            test_acc = accuracy(model, test_a, test_b, test_c)
            history.append((step, loss.item(), train_acc, test_acc))
            print(f"step {step:6d}  loss {loss.item():.4f}  train_acc {train_acc:.3f}  test_acc {test_acc:.3f}")
    return history

model = GrokkingMLP()
history = train(model)

What you should expect to see: train_acc reaching 1.0 within the first few hundred to a few thousand steps, while test_acc stays sub-optimal for a long stretch afterward before climbing toward 1.0 as well. The exact step at which that climb happens is not fixed; it depends on the random seed, the train fraction, and the weight decay. The figure below reports the accuracy curves for this simple example.

Training vs. test accuracy demonstrating the grokking delay.

Open Questions

Several open questions remain:

• Underlying Mechanics & Phase Transitions:The exact mechanistic reasons why a neural network suddenly transitions from rote memorization to structural generalization remain debated. While theories exist around representation learning and margin maximization, a unified mathematical consensus is still an open question.
• Predicting the "Grokking" Point: Currently, it is incredibly difficult to predict when (at what epoch or under what exact conditions) a model will grok. While some recent papers started to look into this, there is still no well known established framework around this.
• Scaling to Real-World Data: Much of the foundational research on grokking has been observed using small algorithmic datasets (like modular addition or simple logic operations). It is still not yet fully understood how these dynamics scale up to massive, complex, real-world datasets and whether "delayed generalization" behaves the same way in those domains.
• Role of Hyperparameters and Regularization: The exact inter-dependencies between network architecture, weight decay, optimizers, and learning rates in inducing or preventing grokking is still not fully understood. Researchers are still investigating whether specific regularizations are strictly necessary to force the network into grokking, or if it is an inherent property of gradient descent itself.

Common questions

Is grokking the same thing as generalization?

Not really. Grokking specifically refers to the delay and the abruptness, generalization arriving long after training accuracy has already saturated, and often as a sharp transition rather than a gradual climb.

Does grokking happen with real, large language models?

Grokking-like curves have been reported well beyond toy settings, in-context learning, multi-hop reasoning, and masked language modeling among them, but the detailed understanding of why it happens is overwhelmingly built on small, controlled, algorithmic tasks. Whether the same explanations transfer to large-scale training is an open question.

Why does weight decay matter so much?

Several of the leading mechanisms converge on weight decay as a central lever, because it is what drives a network out of a high-norm, low-complexity-agnostic region where memorizing solutions are abundant, toward a lower-norm region where generalizing solutions are comparatively more common. It is not the only thing that matters, grokking has been reported without any weight decay at all in some settings, but across the literature it is the single most consistently reported factor in how long the delay lasts.

References

• Power, Burda, Edwards, Babuschkin & Misra, “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets” (2022): the paper that named and first documented the phenomenon.
• Bertolotti & Cazzola, “A Survey on Grokking”, ACM Computing Surveys (2026): the eleven-mechanism taxonomy this piece draws its literature review from.
• Liu, Kitouni, Nolte, Michaud, Tegmark & Williams, “Towards Understanding Grokking: An Effective Theory of Representation Learning”, NeurIPS (2022): the source of the minimal example's architecture.
• Nanda, Chan, Lieberum, Smith & Steinhardt, “Progress Measures for Grokking via Mechanistic Interpretability”, ICLR (2023): the reverse-engineered Fourier algorithm for modular addition referenced above.
• Liu, Michaud & Tegmark, “Omnigrok: Grokking Beyond Algorithmic Data”, ICLR (2023): grokking observed on MNIST, IMDB, and QM9.

This cookbook reflects the current understanding of grokking as of today. Because deep learning research evolves rapidly, new papers may emerge that update or challenge these concepts.