Home  ›  Insights  ›  LLM context windows
Field Note

A 128k context window can mean 64k trained. Read the config

A model card prints one context number. How the model encodes position tells you how much of that number it was trained on and how the rest was reached. Sebastian Raschka’s survey of positional embeddings is the clearest map of the mechanics. Here is what it means for a team sizing long-document work on an open-weight model.

Carriage: a token's positionDials: rotation speeds, fast to slowBlue: trained. Violet: stretchedSmolLM3 was trained to 64k tokens and uses YaRN to reach 128k at inference (vendor, Hugging Face). YaRN leaves fast rotations alone and slows the slow ones (independent, Peng et al.). Dial speeds and the split between geared and ungeared dials are illustrative, this figure's model.

The claim

The context window on a model card reads like a tested capacity: put that many tokens in and get the model’s normal quality out. Buyers size document pipelines, contract review, and retrieval budgets on it.

For many open-weight models, part of that window was trained and part was stretched at inference. The position scheme tells you which.

How models encode position

On 15 November Sebastian Raschka published a walkthrough of positional embeddings, from GPT-2 to the 2026 releases in his architecture gallery (Raschka, independent). RoPE is still the most popular method. Among RoPE models in his gallery, rotating only part of each attention head went from a minority in 2025 to a majority in 2026. The schemes differ in what they mean for length:

SchemeHow position entersWhat it means for lengthExample
Learned absolute tableA learned vector per position, added to each tokenA hard cap at the table sizeGPT-2: 1,024 positions (Raschka, independent)
RoPEQueries and keys rotate by an angle that grows with position, each feature pair at its own speedExtended by more training with a larger base (rope_theta), or scaled at inferenceMost open-weight LLMs (independent)
Partial RoPEOnly part of each head rotatesThe same extension questions, on fewer featuresMiniMax-M3: 64 of 128 features, rope_theta 5,000,000, 1,048,576 positions accepted (config, vendor)
RoPE with NoPE layersSome layers carry no position signalReported by its authors to help long contextSmolLM3: RoPE removed from every fourth layer (Hugging Face, vendor)
Relative positional embeddingA relative-position term inside attentionVendor reports better extrapolation than RoPE; your serving engine must support itInkling, up to 1M tokens (Thinking Machines, vendor)

The worked case: SmolLM3

Hugging Face documents the whole path for SmolLM3, which makes it a useful reference (Hugging Face, vendor):

  • Trained: long-context training ran from 4k to 32k tokens with rope_theta raised to 1.5M, then from 32k to 64k with it raised to 5M, over about 100B tokens.
  • Stretched: YaRN extrapolates to 128k at inference, twice the 64k training length.
  • Repaired: after alignment, long-context scores on RULER degraded. A 0.9 to 0.1 merge with a mid-training checkpoint recovered the base model’s RULER score up to 128k.

The result: the 128k on the card is 64k trained plus 64k stretched, and the long-context score depended on a repair after alignment. Hugging Face says all of this. Many model cards say only the last number.

How the stretch works

YaRN treats each rotating pair by its speed. Fast pairs complete many turns inside the training length and carry near-word order; YaRN leaves them alone. Slow pairs would rotate past any angle seen in training at the new length; YaRN slows them so they never do (Peng et al., independent). The authors report needing about 10 times fewer tokens and 2.5 times fewer training steps than earlier methods.

So the model meets no angle it has not seen. It meets slow angles packed more tightly than it was trained on. That is why stretching is cheap, and why the stretched half is untested until you test it. Fig 1 shows the mechanism.

Partial RoPE is not a memory decision

Raschka computes that MiniMax-M3, by rotating half of each head, saves 0.134 GB of rotary tables in BF16 at its 1M-token context. Our check: 1,048,576 positions × 32 pairs not rotated × 2 tables (cosine and sine) × 2 bytes = 134,217,728 bytes, about 0.134 GB. The whole table at full RoPE would be about 0.268 GB (our maths).

Khan et al. report that rotating about 10% of features matches full RoPE’s training convergence, and that removing position entirely can cause loss spikes that a little RoPE or QK-Norm prevents (Khan et al., independent). For a buyer, partial RoPE and NoPE layers are about training stability and length behaviour. They are not a line on your GPU bill.

Where this breaks

  • Stretched is not broken. SmolLM3’s authors report recovering the base model’s RULER score to 128k. A stretched window can work. It is unproven for your documents until you test it there.
  • The config is not the whole deployment. A serving engine can apply RoPE scaling from its own launch settings. Read those too.
  • We read configs and papers. Prestanda ran no long-context test for this note, and Raschka’s 2025 and 2026 shares are in charts we did not quote as numbers.

Read five things before you size the work

  1. Step 1. max_position_embeddings. The longest input the code accepts, not proof of training length. Owner: AI engineering lead.
  2. Step 2. rope_theta and rope_scaling. A scaling block with a type such as YaRN and a factor means part of the window is stretched. Original length times factor gives the card number. Owner: AI engineering lead.
  3. Step 3. partial_rotary_factor, rotary_dim, and any layers without RoPE. Field names vary by model. If you plan to extend context yourself, ask the vendor which recipe they used on this scheme. Owner: ML engineer.
  4. Step 4. The trained length in the technical report. Plan production on it. Owner: solution architect.
  5. Step 5. Test past it on your own documents. Plant known facts at depths beyond the trained length and score exact match, as in our recall test. Owner: AI engineering lead with the business owner.

A short script prints the position fields from a downloaded config.json:

import json, sys
c = json.load(open(sys.argv[1]))
c = c.get("text_config", c)
for k, v in sorted(c.items()):
    if "rope" in k or "rotary" in k or k in ("max_position_embeddings", "head_dim"):
        print(f"{k:28} {v}")

Missing

  • The trained context length on most model cards, stated beside the advertised one.
  • SmolLM3’s RULER scores at 64k and 128k in the blog text.
  • Long-context results for partial RoPE against full RoPE at the same model size and length.
  • Inkling’s long-context results against a RoPE baseline on the page we read.

Frequently asked questions

Is the context window on a model card the length the model was trained on?

Not always. Hugging Face trained SmolLM3 to 64k tokens and uses YaRN to reach 128k at inference. Check rope_scaling in config.json and the model's technical report for the trained length, and plan production on that number until you have tested further.

What is RoPE in an LLM?

Rotary position embedding rotates each query and key, in pairs of features, by an angle that grows with the token's position, and each pair turns at its own speed. Attention scores then depend on the distance between tokens. It remains the most common position method in open-weight LLMs.

Does partial RoPE save meaningful memory?

Not at the scale a buyer budgets for. By our maths, the full BF16 rotary table for 1,048,576 positions at head size 128 is about 0.268 GB, and rotating half of each head saves about 0.134 GB. Choose a model for how it behaves at your length.

Living page. Last updated 16 Nov 2026. We update this note when we run a long-context test, or when model cards start stating trained length. Sizing long-document work on an open model? Talk to us.