What changed
On 10 November Sebastian Raschka published a classification of open-weight LLM architectures by how much full attention they keep (Raschka, independent). His chart sorts releases into three groups:
- Full attention throughout: global MHA, GQA, dense MLA, or CCA in every layer.
- Some full attention: full global attention mixed with sliding-window or chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, or sparse attention.
- No full attention: zero dense global layers.
| Period (Raschka’s chart) | Models | Full throughout | Some full | No full |
|---|---|---|---|---|
| 2025 | 32 | 59% | 34% | 6% |
| 1 Jan to 25 Aug 2026 | 60 | 27% | 65% | 8% |
| 26 Aug to 5 Nov 2026 | 21 | 0% | 52% | 48% |
Our maths, rounding his percentages back to model counts: about 2 releases with no full attention in all of 2025, about 5 in the first eight months of 2026, and about 10 in the ten weeks since. Raschka does not publish his inclusion rules, so read this as one careful observer’s sample, not a census.
Three current examples
| Model | Attention layout | Size | Context | Source |
|---|---|---|---|---|
| MiniMax-M3.1-Flash | Sparse attention in every layer | About 428B total, 23B active | 1M tokens | vLLM recipe (independent) |
| Kimi K3 | Three Kimi Delta Attention layers, then one gated MLA layer, repeated | 2.8T total, about 104B active | Up to 1M tokens | Kimi K3 paper (vendor) |
| Mistral Small 4 | Latent attention in every layer | 119B total, about 6.6B active | 256K tokens | Raschka’s architecture figure (independent) |
Raschka names the two gains. Sparse attention runs the expensive attention step over a selected subset of tokens or blocks, which cuts compute and can still reach distant positions. Recurrent layers such as Gated DeltaNet keep a fixed-size state, which cuts memory. Then the catch, in his words: “compressing the history creates challenges for exact retrieval,” which is why most designs keep some attention, sparse or full, in the mix.
Where the cost moved
For a buyer the change is not cheaper tokens on its own. Two costs move.
Recall becomes a property you test. A fixed-size state cannot hold every earlier token as a separate entry (Raschka’s gallery). A sparse layer reads what its selector picks. Both are fine for summarising a long thread. Neither is guaranteed to return the exact indemnity clause, the one log line, or a function signature from the far side of a large codebase. Kimi K3’s own paper gives a hint that a bigger window is not automatically better: on BrowseComp it scored 91.2% with context compaction triggered at 300K tokens and 90.4% using the full 1M window with no context management (vendor). One benchmark, a small gap, and a vendor run. Still, the 1M window did not win.
Serving gets stricter. The vLLM recipe for MiniMax-M3.1-Flash requires --block-size 128 to match the sparse attention block, stores the KV cache and the sparse indexer cache in NVFP4 on Blackwell and MI355X and FP8 on Hopper, needs four GPUs at minimum on Blackwell or MI355X and a full eight-GPU node on H100 or H200, and leaves speculative decoding off on Hopper (independent). New attention types arrive with new kernels. Your inference stack version is now part of the model choice.
Where this argument breaks
- Hybrids keep attention for a reason. Half of the recent releases still mix in full attention to protect retrieval. “No full attention” does not mean “no exact recall”; it means you have to check.
- No matched comparison exists. Raschka’s gallery says that without matched ablations, a throughput or quality difference cannot be assigned to the attention schedule alone. Training data, MoE design, and kernels all move the result.
- Closed APIs hide all of this. If you buy tokens from a hosted frontier model, you see architecture only through price, latency, and quality. The test below still applies.
- 21 models is a small sample from one observer, over ten weeks.
Decision: what to test, by workload
| Workload | Risk with compressed or sparse history | Test before you commit |
|---|---|---|
| Contract, policy, or claims review that quotes the source | Exact clause missed or paraphrased | Plant known clauses at several depths in your own documents; score exact match, not similarity |
| Coding agents over a large codebase | Wrong signature or stale definition recalled late in a long run | Replay real trajectories at full length and with compaction; compare task success |
| Tool-heavy agents with long transcripts | Earlier tool output forgotten after many calls | Count how many calls back the agent still uses a result correctly |
| Summaries of long threads or meetings | Low: gist matters more than exact tokens | Spot-check; take the cheaper model |
Five steps before your next model choice
- Step 1. List the long-context workloads you run or plan. Mark which need exact quotation and which need gist. Owner: AI platform lead.
- Step 2. Build a recall set from your own documents. Known facts planted at 10%, 50%, and 90% of your real document lengths. Owner: the business owner of each workload.
- Step 3. Run it on one hybrid, one no-full-attention model, and your current model. Score exact match and task success, at your real lengths, not 1M by default. Owner: ML engineer.
- Step 4. Price the serving stack, not just the model. Confirm your vLLM or SGLang version supports the attention kernels, the required block size, and your GPU generation. Owner: infrastructure lead.
- Step 5. Keep a retrieval path for exact answers. Where Step 3 shows misses, route exact-quotation tasks through retrieval or a model that passed, and let the cheaper model handle the rest. Owner: AI platform lead.
Missing
- Raschka’s inclusion rules and the list behind each bar.
- Any published long-context exact-recall result for MiniMax-M3.1-Flash or Kimi K3 at depth.
- A matched test that separates attention design from everything else.
- A Prestanda measurement. We have not run Step 3 ourselves; it is a test design, not a result.
Frequently asked questions
What is a hybrid attention model?
A model that keeps full global attention in only some layers and uses cheaper mechanisms in the rest, such as sliding-window attention, sparse attention, Gated DeltaNet, Kimi Delta Attention, or Mamba-2. Kimi K3, for example, repeats three Kimi Delta Attention layers followed by one gated MLA layer.
Does a 1M-token context window mean the model recalls everything in it?
No. Recurrent layers keep a fixed-size state, so they compress history, and Sebastian Raschka notes this makes exact retrieval harder. Sparse attention reads a selected subset of tokens or blocks. Test exact recall at the depths your documents reach before relying on it.
Does this matter if we only use closed model APIs?
Less directly. Closed providers do not publish their attention layouts, so you see the effect only through price, latency, and output quality. The recall test in this note still applies to any model you send long documents to.
Living page. Last updated 11 Nov 2026. We update this note when Raschka publishes his full roundup, a vendor publishes exact-recall results at depth, or we run the Step 3 test.