Introducing KV1-1.2B.

Today, we're releasing KV1-1.2B, the first model in the KV1 family: language models trained to run with a 2-bit KV cache. At a full 32K context, KV1-1.2B keeps its entire cache in 60 MiB, 6.4× less than the 384 MiB the same model needs in fp16. It still retains 75.8% of the full-precision model's quality.¹

KV1-1.2B is built on LFM2.5-1.2B-Instruct and is available now on Hugging Face. You can run it with the real 2.5-bit cache in KV1.cpp, our llama.cpp fork, and try it in the demo. A 2.6B model is in training, and 4B, 8B-A1B and 24B-A2B versions are planned.

A smaller cache for longer contexts

Every token a model keeps in context stores a key and a value vector in every attention layer. That KV cache grows linearly with context length, and with batch size when several conversations run at once. On small models it is often what fills memory first, long before the weights do.

LFM2.5 is already efficient here: only 6 of its 16 blocks use attention, and the other 10 are short convolutions with a small, fixed state. KV1 shrinks what's left. Each key and value is stored in 2.5 bits instead of 16, so the cache takes 1.9 KiB per token instead of 12 KiB: 15 MiB at 8K context and 60 MiB at 32K. The bf16 weights are unchanged. For comparison, Gemma 3 1B needs 139 MiB at 32K, Llama 3.2 1B needs 1,024 MiB and Qwen3-1.7B needs 3,584 MiB.

Trained to work with a 2-bit cache

Squeezing an untrained model's cache to 2 bits breaks it: LFM2.5-1.2B drops to 1.6% on GSM8K and 0.6% on HumanEval. KV1 is trained to expect the 2-bit cache from the start, using four techniques:

• Self-distillation: The teacher is the frozen original model with an fp16 cache. The student is the same model running with the 2-bit cache. The student learns to match the teacher's full next-token distribution and, with weight 0.5, its hidden state after every layer. The training text is only input, and the target is always the teacher, so KV1 keeps the original model's behavior.
• Hadamard rotation: A few channels in attention keys are much larger than the rest, and with only four levels per vector they take up all the precision. Rotating each head's vectors with a Hadamard matrix spreads those outliers evenly over all 64 dimensions. The rotation is exact, so attention scores don't change and it needs no extra storage.
• Attention sinks: The first 4 tokens of every sequence keep their keys and values in fp16. Attention relies heavily on these positions, and they cost almost nothing to keep.
• Straight-through training: The forward pass uses the rounded 2-bit values, and the backward pass treats rounding as if it weren't there, so gradients still flow.

Training took 750M tokens at 32K context on a single RTX Pro 6000, about 17 GPU hours. The mix was 60% chat and instruction following, 16% agent and tool-use trajectories, 12% code, 7% web text and 5% math.

Performance across benchmarks

With the same 2-bit cache that breaks the untrained model, KV1-1.2B recovers most of the original quality. It scores 77.6% on IFEval (vs. 43.1% untrained), 44.8% on GSM8K (vs. 1.6%) and 40.2% on HumanEval (vs. 0.6%). That keeps 94.8%, 66.3% and 72.5% of the fp16 scores, respectively.

Instruction following holds up best. KV1's IFEval score is higher than Qwen3-1.7B (68.2%), Gemma 3 1B (55.5%) and Llama 3.2 1B (48.8%), all running with full-precision caches. On GSM8K and HumanEval, KV1 is in the same range as Gemma 3 1B and Llama 3.2 1B while using 2–17× less KV memory.

On our KV1 test suite of 25 automatically checked tasks, KV1 passes 22, compared with 24 for fp16 and 7 for the untrained 2-bit cache. It passes every tool-calling task and every long-context retrieval task, at up to 16K tokens.²

Measured as benchmark score per MiB of KV cache, KV1 delivers 4.8× more than fp16, and more per MiB than an untrained 3-bit cache (3.9×). The ratio doesn't depend on context length, because every cache grows linearly with it.

Where KV1 still falls short

The 2-bit cache still costs quality in some areas:

• Knowledge-heavy tasks: MMLU-Pro recovers the least, at 54.6% of fp16 (21.1% vs. 38.7%). Multiple-choice knowledge recovers less than generation, coding and tool use. Training on short, document-separated sequences is the planned fix for the next version.
• 3-bit is still ahead on quality: An untrained 3-bit cache scores higher on every benchmark, at 84 MiB instead of 60 MiB. KV1 comes close on IFEval and HumanEval.
• GPQA-Diamond: Both 2-bit configurations score 24.2%, which is chance level for four-way multiple choice.

KV1-1.2B, available today

KV1-1.2B is released under the LFM Open License v1.0, following its base model:

• For developers: Load klein-ml/KV1-1.2B in Transformers with trust_remote_code=True. You get the exact 2-bit outputs, but the cache is still stored in bf16, so there's no memory saving. Without remote code, it loads as a plain LFM2 model.
• For on-device and server use: Download a GGUF from klein-ml/KV1-1.2B-GGUF (BF16, Q8_0 or Q4_K_M) and run it in KV1.cpp with -fa on -ctk bq2_0 -ctv bq2_0 for the real 2.5-bit cache. BQ2_0 is not in mainline llama.cpp yet.
• For everyone: Try it in the KV1 demo.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "klein-ml/KV1-1.2B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16", device_map="cuda")

msgs = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

git clone https://github.com/Klein-ML/kv1.cpp && cd kv1.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
hf download klein-ml/KV1-1.2B-GGUF KV1-1.2B-Q8_0.gguf --local-dir .
./build/bin/llama-cli -m KV1-1.2B-Q8_0.gguf -fa on -ctk bq2_0 -ctv bq2_0 -ngl 99 -c 32768