The point
On 16 November Sebastian Raschka wrote that “the open-source LLM ecosystem could really benefit from an LLM building-block API” (Raschka on X, independent). Swapping one component to run an experiment, GQA for MLA, a dense feed-forward for mixture of experts, attention for Mamba2 or Gated DeltaNet, carries “a major switching cost”, and it is “very messy even with LLM-assisted coding”. His sketch is a hypothetical PyTorch-style API in which each block is one argument: mixer, ffn, norm, residual.
He names projects heading that way, torchtitan, torchtune, Megatron Core, and OLMo-core, and says they are “not quite that level of abstraction”. He also names the catch: it has to be popular to be adopted, and adopted to be popular.
We agree with him. And the switching cost is not only a researcher’s problem. A team that will never pretrain a model pays it too, when it picks an open model and has to run, quantise, and fine-tune whatever parts that model was built from.
Every open model is a parts list
Raschka’s architecture comparison is the clearest catalogue of what changed (Raschka, independent). For a buyer, each part has a running cost attached:
| Part | Choice | Example | What it changes for you |
|---|---|---|---|
| Attention | Grouped-query (GQA) | Qwen3, gpt-oss | Query heads share key and value projections, so the KV cache is smaller than one set per head. |
| Attention | Multi-head latent (MLA) | DeepSeek V3, Kimi K2 | Keys and values are compressed “into a lower-dimensional space before storing them in the KV cache”, at the cost of an extra matrix multiplication. Your engine needs the kernel. |
| Attention | Linear plus full, 3:1 | Qwen3-Next (Gated DeltaNet), Kimi Linear (Kimi Delta Attention with MLA) | Linear layers keep a fixed-size state instead of a cache that grows with length. Full layers stay because linear retrieval is less precise. |
| Feed-forward | Mixture of experts | DeepSeek V3: 256 experts per MoE module, 671B parameters in total, 37B used per step | GPU memory is sized on the total. Compute per token follows the active share. |
| Residual | mHC, Attention Residuals | DeepSeek-AI, Kimi Team | Training-side changes. Ask whether your fine-tuning stack implements them before you plan to adapt such a model. |
| Position | RoPE, partial RoPE, NoPE layers | See our context-window note | How much of the advertised window was trained. |
The residual changes are the newest. DeepSeek-AI reports that mHC adds 6.7% training time at an expansion rate of 4 (DeepSeek-AI, vendor). The Kimi Team reports a loss of 1.692 against 1.714 for its baseline and calls that a 1.25× compute advantage (Kimi Team, vendor). Both are the authors’ own results.
The config says it, if you know the field
DeepSeek V3’s config.json lists num_key_value_heads: 128, the same as its 128 attention heads (DeepSeek V3 config, vendor). Read alone, that looks like full multi-head attention with the largest possible cache. The field that matters is kv_lora_rank: 512, the latent MLA stores instead. Its experts sit in n_routed_experts: 256 and num_experts_per_tok: 8.
Qwen3-Coder-Next spells out its hybrid: 48 layer_types, three linear_attention then one full_attention, repeated, with full_attention_interval: 4, 512 experts, and 10 per token (Qwen3-Coder-Next config, vendor, read from Unsloth’s mirror). Two models, two vocabularies for the same questions. That is Raschka’s point, seen from the buyer’s side.
What exists today
| Project | What it says it is | What that means for you |
|---|---|---|
| torchtitan | “A PyTorch-native platform for pretraining and post-training generative AI models at scale” | A training platform with model families, not a parts catalogue. |
| Megatron Core | “A composable library with GPU-optimized building blocks for custom training frameworks” | The closest to building blocks, aimed at teams writing their own training framework. |
| OLMo-core | “PyTorch building blocks for the OLMo ecosystem” | Built around Ai2’s OLMo models. |
| torchtune | “No longer actively maintained”; development “wound down in 2025” | A fine-tuning library teams built on is now frozen. |
| Transformers, modular | A model file inherits from existing ones, and “a converter generates standalone files from the modular file” | Reuse when a model is added, one file per model when it runs. A swap is still a code change. |
None of these is the one-line swap Raschka sketches, and he says so. torchtune is the reminder that the switching cost also arrives from the other side: the library under your pipeline can stop.
Why a buyer should care
- Serving. Every part needs a kernel in your inference engine and a path through your quantisation tool. A new mixer or residual can reach a model before it reaches the engine version you run.
- Memory. Mixture of experts is sized on total parameters, and the attention type decides whether the cache grows with your context length.
- Tooling life. Your fine-tuning and evaluation code outlives any one model and, as torchtune shows, can outlive its library.
Hosting is your cloud’s job. Reading the parts list against your workload is yours.
Where this breaks
- Bought through an API, the parts are the provider’s problem. This note is for teams that self-host or fine-tune open weights.
- The API in Raschka’s post does not exist. His code is labelled hypothetical.
- We read configs, papers, and project pages. Prestanda ran no ablation for this note. The mHC and Attention Residuals figures are the authors’ own.
Read the parts list in five steps
- Step 1. Pull
config.jsonand list the parts. Attention (num_key_value_heads,kv_lora_rank,layer_types), experts (num_expertsorn_routed_experts,num_experts_per_tok), and position. Names vary by family. Owner: AI engineering lead. - Step 2. Check each part against your serving engine and quantisation tool, at the versions you run. Owner: platform engineer.
- Step 3. Size GPU memory on total parameters, and the cache on the attention type at your context length. Owner: solution architect.
- Step 4. Put a thin interface you own between your pipelines and any training library, so a model or library change is one adapter, not a rewrite. Owner: ML engineering lead.
- Step 5. Test the shortlist on your own tasks before you choose on architecture. Owner: AI engineering lead with the business owner.
A short script prints the parts from a downloaded config.json:
import json, sys
c = json.load(open(sys.argv[1]))
c = c.get("text_config", c)
for k in ("num_attention_heads", "num_key_value_heads", "kv_lora_rank",
"full_attention_interval", "num_experts", "n_routed_experts",
"num_experts_per_tok", "max_position_embeddings"):
if k in c:
print(f"{k:26} {c[k]}")
types = c.get("layer_types")
if types:
print("layer_types", {t: types.count(t) for t in sorted(set(types))})Where we help
We help enterprise teams choose, fit, and run open-weight models: read the parts list against the workload, size the hardware, check the serving and fine-tuning stack part by part, and test the shortlist on the client’s own documents before anyone commits. Where models must run on infrastructure you own, that is the work behind Auralis. The wider practice sits under AI solutions.
Missing
- A component API with production-grade kernels, at the level Raschka describes.
- A public, per-component support table for the major serving engines.
- Independent replications of the mHC and Attention Residuals results at production scale.
- Model cards that list their parts in one place, not across config fields with different names.
Frequently asked questions
What is an LLM building-block API?
A library where each part of a model, such as attention, the feed-forward layer, normalisation, and the residual path, is a swappable argument, so a team can compare GQA with MLA, or dense layers with mixture of experts, by changing one line. Sebastian Raschka proposed one in November 2026 and says existing projects such as torchtitan and Megatron Core are not quite at that level of abstraction.
Why does the attention type matter if we only run inference?
It decides the KV cache. GQA shares keys and values across query heads, MLA caches a compressed latent instead of full keys and values, and linear-attention layers such as Gated DeltaNet keep a fixed-size state that does not grow with context. Each needs support in the serving engine you run.
How do I see which components an open model uses?
Read its config.json. DeepSeek V3 lists kv_lora_rank 512 for MLA and 256 routed experts with 8 per token. Qwen3-Coder-Next lists its layer types: three linear-attention layers to each full-attention layer, with 512 experts and 10 per token. Field names vary by model family.
Living page. Last updated 17 Nov 2026. We update this note when a building-block library ships with production kernels, or when we run a component swap ourselves. Choosing an open model for your own infrastructure? Talk to us.