A Little RoPE, a Lot of Prior Art

How a transformer knows where things are

Attention is famously permutation-invariant. A transformer scores every pair of tokens using a query vector q and a key vector k — just a dot product, q·k. Nothing in that dot product knows whether the two tokens are adjacent or fifty positions apart. Shuffle the sequence and, absent any positional signal, the model literally cannot tell.

The standard fix — inherited from the original Transformer and used by CLIP on both towers — is the learned absolute position embedding: a trainable table with one vector per position, added to each token at the input. Patch #37 of the image gets vector #37; token #12 of the caption gets vector #12. It works, but it carries two structural costs:

• It’s absolute. The model learns “what tends to be at position 37,” not “how position 37 relates to position 36.” Relationships between positions must be rediscovered, indirectly, from data.
• It’s a fixed-size table. CLIP’s text table has exactly 77 rows. Feed it token #78 and there is no row to look up — the model doesn’t degrade past its trained length, it simply cannot run. The notorious 77-token cap on CLIP-style models is baked into a parameter’s shape.

Rotation as position: what RoPE actually does

Rotary Position Embedding (RoPE), introduced in RoFormer (2021), takes a completely different route. Instead of adding a position vector to the token, it rotates the query and key vectors by an angle proportional to their position, right before the dot product.

Take a head dimension d and pair the channels into d/2 two-dimensional planes. In each plane, a token at position m is rotated by the angle mθ:

The magic is a one-line identity. Rotations in a plane compose by adding angles, and transposing one reverses it — so R(m)^⊤R(n) = R(n−m). When a query rotated by m meets a key rotated by n:

Absolute position vanishes from the computation. Two tokens three positions apart interact identically whether they sit at (0, 3) or (500, 503). And because rotations preserve lengths, the token’s content is untouched — only its phase carries position.

One more ingredient: each plane spins at its own rate, θ_j = b^−2j/d with base b = 10,000. Picture a clock with d/2 hands — that’s what the spinning figure at the top of this page is. The fastest hands tick a full radian per token and resolve fine local order; the slowest take thousands of tokens per revolution and encode coarse, long-range structure. This frequency ladder is the detail that matters later, when the story reaches length extrapolation.

If you come from computer vision, there’s a familiar analogy: this is attention’s version of a convolution’s translation equivariance. A 3×3 conv kernel responds to a pattern the same way wherever it appears, because it only ever sees relative offsets within its window. A learned absolute embedding breaks that property (patch #37 has its own private vector); RoPE restores a soft version of it inside attention.

Two towers, one inductive bias

Here’s the observation that started this project. A text token has a 1D index m. An image patch has a 2D index (p, q) — row and column. How do you rotate by a pair of coordinates?

The standard answer — used by RoPE-ViT and EVA — is axial: split each head’s channels in half and give each axis its own independent 1D RoPE. The first half rotates by the row index, the second half by the column index. As a matrix, position (p, q) applies the block-diagonal direct sum:

Because the two halves never mix in a dot product, the attention score decomposes into two independent copies of the 1D identity:

Vertical displacement lives in one set of channels, horizontal in the other. In code, the whole mechanism is ~15 lines applied to q and k inside every attention layer. Condensed from the Rotary2D module in my OpenCLIP fork:

def rotate_half(x):                    # (x1, x2) -> (-x2, x1): a 90° turn in each plane
    x1, x2 = x.chunk(2, dim=-1)
    return torch.cat((-x2, x1), dim=-1)

class Rotary2D(nn.Module):
    def __init__(self, H, W, head_dim, base=10_000.0):
        d_half   = head_dim // 2                                   # channels per axis: [row | col]
        inv_freq = base ** (-torch.arange(0, d_half, 2) / d_half)  # the frequency ladder θ_j
        rows = torch.arange(H).repeat_interleave(W).float()        # each patch's grid coordinate;
        cols = torch.arange(W).repeat(H).float()                   # CLS gets (0,0) -> identity rotation
        ang_r = torch.outer(rows, inv_freq)                        # [L, d_half/2] angles per token
        ang_c = torch.outer(cols, inv_freq)
        emb_r = torch.cat([ang_r, ang_r], dim=-1)                  # NeoX layout, matches rotate_half
        emb_c = torch.cat([ang_c, ang_c], dim=-1)
        self.cos_r, self.sin_r = emb_r.cos(), emb_r.sin()          # cached: the patch grid never changes
        self.cos_c, self.sin_c = emb_c.cos(), emb_c.sin()

    def forward(self, x):              # x: [batch, heads, L, head_dim] — applied to q AND k
        x_r, x_c = x.chunk(2, dim=-1)
        x_r = x_r * self.cos_r + rotate_half(x_r) * self.sin_r     # rotate by ROW index
        x_c = x_c * self.cos_c + rotate_half(x_c) * self.sin_c     # rotate by COL index
        return torch.cat([x_r, x_c], dim=-1)

Two implementation notes. First, rotate_half is how every fast RoPE implementation avoids materializing rotation matrices: for a plane (x₁, x₂), the rotated vector (x₁cos θ − x₂sin θ, x₂cos θ + x₁sin θ) is exactly x*cos + rotate_half(x)*sin, elementwise. Second, RoPE adds zero parameters — the cos/sin caches are pure functions of the grid and the base. Every accuracy delta in the tables below comes at strictly equal parameter count.

Now delete rows/cols, feed torch.arange(seq_len) as the only coordinate, and skip the channel split — this same module is 1D text RoPE. Spatial RoPE (images) and sequential RoPE (text) are literally one operation applied to different index sets. One inductive bias — attention should see relative position, through rotation — with two instantiations. That framing suggested two falsifiable predictions:

• P1 — 2D RoPE should improve a sigmoid-loss (SigLIP) image tower at fixed budget, with a larger gain on retrieval (which needs fine-grained spatial alignment) than on classification (which needs one global label).
• P2 — a 1D RoPE text tower trained short should extrapolate to longer captions at inference. There is no 77-row table to run out of: position is computed, not looked up.

RoPE was thoroughly studied for LLMs, established for supervised ViTs (RoPE-ViT, ECCV 2024), and present-but-never-ablated inside EVA-CLIP’s InfoNCE tower. Under SigLIP’s sigmoid loss, I believed it was untested. Hold that thought.

June 17–20The setup: a sigmoid loss and an ablation ladder

A word on the loss, since it was the hinge of my novelty claim. Both objectives operate on the same raw material: normalized image embeddings x_i and text embeddings y_j, scored by similarity x_i·y_j with a learned temperature t. CLIP trains with InfoNCE — for each image, a softmax over every text in the batch, “pick your caption out of this lineup”:

The denominator couples every pair to every other pair in the batch, which is why CLIP’s quality is famously sensitive to batch size. SigLIP (ICCV 2023) replaces it with an independent binary decision per image–text pair — a sigmoid on the similarity, matched (z_ij = +1 on the diagonal) or not (−1 off it), with a learned bias b and no batch-wide normalization:

Same shared embedding space, different geometry of the training signal: InfoNCE only cares about each positive relative to the in-batch negatives, while the sigmoid pins every pair’s similarity against an absolute threshold. There was no published evidence, I thought, on whether a positional inductive bias tuned under one loss transfers to the other.

I forked OpenCLIP and made one methodological bet early: every run is a row in an additive ablation ladder, declared in a single registry file, differing from its neighbor by exactly one ingredient. No ad-hoc command lines — a launcher turns registry rows into runs, and the diff between any two arms is auditable at a glance. When a stock config would have changed two things at once (the off-the-shelf SigLIP config also swaps the pooling head), I split it into separate rungs. It’s slower than vibes-driven experimentation. It’s also the only reason the paper’s causal claims survived what came later.

Training: ViT-B/16, CC12M (~10M pairs), ~3 epochs, fixed global batch, H100s. The ladder: InfoNCE → SigLIP → +qk-norm → +2D RoPE → +attention pooling → +focal sigmoid. Three seeds on anything I planned to make a claim about. The RoPE rung jumped out immediately:

Because the ladder had stacked RoPE on top of qk-norm, I ran isolation arms — RoPE alone on plain SigLIP, three seeds — to be sure the win wasn’t an interaction. It wasn’t: adding qk-norm on top of RoPE moves ImageNet by 0.11 with per-seed noise up to 0.26. RoPE is the driver. A focal-weighted sigmoid variant hurt (kept as a reported negative). The retrieval gains (+4–5 R@5) exceeded the classification gains, matching P1’s fine-grained-alignment prediction. And scaling to ViT-L/14 widened the gap: 25.42 → 28.45 (+3.0), RoPE sweeping every metric on every seed.

June 22 – July 4The text side: extrapolation, a frequency trick, and a padding trap

P2 asks a sharper question than “does RoPE work in the text tower?” It asks: can a model trained only on 77-token captions read longer ones at inference?

The two position codes fail in structurally different ways past the training length, and the difference is visible in one line of code each:

# absolute: position is a LOOKUP — the table has exactly 77 rows
pos_emb = self.positional_embedding[:seq_len]        # seq_len > 77 -> those rows don't exist

# RoPE: position is COMPUTED — any seq_len works
angles = torch.arange(seq_len)[:, None] * inv_freq   # [L, d/2], built on the fly

For the absolute baseline, the honest answer is that it can’t even try — there is no row 78. The standard workaround (this is what Long-CLIP builds on) is to interpolate the table: stretch 77 learned vectors across, say, 164 slots via F.interpolate, so each position gets a blend of its trained neighbors. The positions now exist, but every token sits at coordinates the model never saw in training.

RoPE runs at any length, but naive extrapolation has its own failure mode, and it’s worth understanding because the fix is the crux of the experiment. Each plane’s phase at position m is mθ_j. Train at L = 77 and the model has only ever seen phases up to 77 θ_j in each plane; evaluate at position 150 and the slow planes present phase angles that never occurred in training — out-of-distribution inputs, just hiding in the rotation space rather than the token space. The remedy, known in the LLM world as NTK-aware scaling, is a single arithmetic change at inference — replace the base b so every wavelength stretches by just enough:

# NTK-aware rescaling: the entire "method". At inference, no retraining.
s        = L_test / L_train                           # e.g. 164 / 77 ≈ 2.13
eff_base = base * s ** (head_dim / (head_dim - 2))    # 10_000 -> ~21_800 at s ≈ 2.13
inv_freq = eff_base ** (-torch.arange(0, head_dim, 2) / head_dim)

The exponent d/(d−2) is chosen so the maximum phase reached at the longer test length matches the maximum phase seen in training: for every plane, L_test θ′_j stays within L_train θ_j at the slow end of the ladder, with the distortion spread geometrically across planes. Every rotation the model encounters at length 164 is one it has met at length 77 — the fast planes lose a little per-token resolution, and nothing else changes. No parameters, no gradient steps, three lines of code. This trick (community-discovered for LLaMA in mid-2023, formalized alongside YaRN) had, as far as I knew, never been tested on a contrastive text encoder.

My first experimental design failed in an instructive way. I trained at 248 tokens and “extrapolated” to 320 — then noticed the eval captions max out around 222 tokens. Past 248, the test set is 100% padding; a flat curve proves nothing. I threw the experiment away and redesigned: train both arms at 77 tokens (CLIP’s real-world limit, where every long caption overflows), evaluate at 77/128/164/196/248 — with 164 ≈ the p99 of real caption content on these benchmarks.

Both arms are the standard CLIP text tower, identical to the parameter (~149.7M) except the position code: learned-absolute with linear interpolation at eval time, versus 1D RoPE with the NTK rescaling applied at inference. Training on DenseFusion-1M (a million long, dense captions); evaluation on Urban-1k and DOCCI long-caption retrieval. The metric: Δ Recall@1 versus each arm’s own 77-token score — does feeding the model more of the caption help or hurt?

Two controls made me trust this table. First, the NTK-off ablation: raw RoPE extrapolation degrades about as badly as interpolation — so the win is not “RoPE extrapolates,” it’s “RoPE plus the frequency rescaling extrapolates.” The decisive ingredient is the one-line base change. Second, a corruption control: replace every token past position 77 with padding, and RoPE+NTK craters from 17.9 to 7.7 R@1 — proof the flat curve reflects genuine use of the extra caption content, not a model quietly ignoring its tail.

June 25The paper, and one proposition too many

I wrote it up: A Little RoPE Goes a Long Way. Three claimed contributions — the two experiments above, plus a theory section I was rather pleased with. It went like this: every term in the sigmoid loss is a function of inner products x_i·y_j of normalized embeddings, and inner products don’t change if you rotate the entire embedding space by any orthogonal transform Q:

So the loss constrains only the relative geometry of the space, never its absolute frame. RoPE, meanwhile, injects position through orthogonal rotations that preserve norms and encode only relative offsets. Loss and positional code agree about what is real (relative structure) and what is gauge (the absolute frame) — that, I argued, is the geometric reason rotation is the natural positional bias for contrastive training.

It’s an elegant paragraph. Keep it in mind while the next section takes it apart.

July 9The kill step

Before submitting anywhere, I ran an adversarial novelty review — briefed explicitly to kill the claims, not confirm them: three parallel literature sweeps (RoPE-in-vision, long-context CLIP text, and the theory), each required to fetch primary sources and report per-claim verdicts naming the closest prior work.

The theory died in minutes. The orthogonal invariance of dot-product contrastive losses isn’t just known — it’s a theorem: Zimmermann et al. (ICML 2021) proved that InfoNCE-family minimizers recover the true latents exactly up to an orthogonal transform; the gauge freedom I’d “observed” is the load-bearing object of a four-year-old spotlight paper. The 1D/2D unification exists at least four times over (RoPE-ViT itself presents 2D RoPE as 1D-per-axis; LieRE, STRING, and ComRoPE each build general N-dimensional frameworks). And the review surfaced a logical gap I had papered over with prose: my proposition is about one global rotation applied to the final embeddings, while RoPE applies position-dependent rotations to intermediate queries and keys inside attention. The former implies nothing about the latter. My “geometric reason” was an analogy wearing a proposition’s clothes.

P1’s framing was factually false — by six weeks. Abstract-level searches for “RoPE SigLIP” return nothing, which is exactly why I believed the abstract I wrote. The pairing lives where abstract search can’t see it: inside VLM tech reports. ByteDance’s Seed1.5-VL (May 2025) trains a 2D-RoPE ViT with the SigLIP loss. Kwai’s Keye-VL (July 2025) adds 2D RoPE to a SigLIP encoder, continues pretraining with the SigLIP loss, and even publishes a comparison table — in which RoPE doesn’t beat the non-RoPE baseline at standard resolution. Meta’s Perception Encoder had individually ablated 2D RoPE under softmax CLIP that April.

Then TULIP. The second sweep returned a paper I had never encountered: TULIP: Token-length Upgraded CLIP — Najdenkoska et al., arXiv October 2024, accepted at ICLR 2025. Last year. Reading its method section was an out-of-body experience. It replaces CLIP’s absolute text position embeddings with relative ones — “we implement P_g(i) as Rotary Positional encodings (RoPE).” It uses NTK-aware scaled RoPE, for the same reason I did. It evaluates at 248 tokens against Long-CLIP, and wins. My P2, essentially, published before I ran my first experiment — with one real difference: TULIP’s length expansion is a trained pipeline (a distillation stage, then a fine-tuning epoch on long captions), where mine is inference-time-only. And even the “training-free NTK on a contrastive text encoder” half exists: LongEmbed (EMNLP 2024) demonstrated it on text-embedding retrieval models and explicitly advocated RoPE for future embedding models.

My related-work section cited seven papers. The review surfaced five directly-adjacent works I hadn’t cited at all — one peer-reviewed and published before the project’s first commit.

The scoreboard

What survived, and what I’d do differently

The experiments survived; the claims didn’t. Every number above is real, replicated, and — as far as I can tell — the only controlled measurement of its kind. What died was the story I wrapped around them: “first to combine RoPE with X.” The durable framing is smaller and better: careful science on known ingredients — the first clean isolation of RoPE under the sigmoid loss (which resolves a contradiction sitting in the tech-report record), and evidence that in a CLIP text tower a one-line inference-time rescaling does what TULIP needed a training pipeline for.

1. Run the kill review before writing the abstract, not after. It cost one day and rewrote three contribution claims. The same review three weeks earlier would have shaped the experiments themselves — I’d have run a TULIP-style fine-tuned arm as a baseline.
2. Search where the field actually publishes. My hinge claim was falsified by two tech reports invisible to abstract-level search. If your claim is “X has never been paired with Y,” you have to grep the 80-page tech reports, not just Semantic Scholar.
3. “First to combine” is fragile; “first controlled measurement” is durable. Combinations get published constantly, in passing, deep inside VLM reports. Nobody scoops a clean ablation by accident.
4. The trash-can experiment was the most valuable one. Killing my own padding-contaminated v1 eval is why the v2 numbers held up under adversarial review. The habit that caught my flawed eval is the same habit that caught the novelty problem — it just needed to be pointed outward sooner.

The paper is being reframed accordingly — honest venue target included. The full adversarial review, with per-claim verdicts and every citation, lives in NOVELTY_REVIEW.md in the repo.