Research

Measured, with the data to check it.

Reasoning ahead is not exact, so it was measured on answers that can be checked. Every table below has its script and its JSON in the repository, and the research log keeps the working notes, newest first.

Method

Word problems with a known numeric answer, typed in Italian in real time: 5 to 8 characters per second, a few natural pauses, Enter right after the last character, no rereading. The script is research/thinking_ahead.py; it types each problem with and without reasoning ahead and records the wait for the first word of the answer, the reasoning tokens in total and the ones left after Enter, and whether the answer is right.

Models are Qwen3 Q4_K_M in thinking mode on llama.cpp (b11747). The reference machine is an RTX 2060 with 6 GB of VRAM (i5-10400, 16 GB DDR4, Windows 11): Qwen3-4B fits in VRAM at about 83 tok/s, Qwen3-14B does not (38% on the GPU, about 5 tok/s). The second machine is a MacBook Pro M2 Pro, 16 GB, with Metal. Two runs of the same 24 problems differ by up to a second in the mean: compare rows measured one after the other.

Results

Reasoning ahead, Qwen3-4B on the RTX 2060, 24 problems (research/thinking_ahead.py, data thinking_ahead_4b.json):

Same accuracy, less reasoning in total (the material gathered while typing saves the model from starting over), and almost two thirds of it done before Enter. The frame of the notes matters: a generic frame gave 22/24 and a median of 4.4 s; no frame gave 22/24, 6.9 s and 70% more reasoning. With the list frame, the example line and a logit bias against words such as user, incomplete, typing while reasoning ahead, the share of sentences in the notes about the message itself instead of its content went from 86% to 17-26%.

Exact answer ahead

Qwen3-4B, no reasoning, pause before Enter with the text unchanged: the answer itself is written ahead, and that part is exact (with greedy decoding, identical token by token to the one started after Enter; checked).

Larger models

What the model can do before Enter is bounded by typing time × its writing speed:

Even so, on the 14B (6 problems, 6/6 correct both ways) reasoning ahead halves the mean wait after Enter: 108 s → 47 s, with 164 reasoning tokens written before Enter on average. Single runs vary a lot (the same problem took 106 s once and 154 s another time), so 6 problems give the order of magnitude. Reading ahead gains more the larger the model: the 14B reads a 3,000-token pasted document in about 10 s, all of it hidden while you type.

Guessing the ending (--branches, experimental). The reasoning left after Enter is mostly about the end of the message, which arrives last. With --branches 1 the model also guesses how you will finish and reasons on the guess on a slot of its own. Same 24 problems, two runs each: 48/48 correct, mean wait 4.17 s → 3.51 s, slowest tenth 7.9 s → 5.3 s, 14 of 48 messages guessed exactly. A modest gain, mostly on the slowest answers, on 24 problems not yet a significant one (sign test p = 0.15). On the 14B no branch ever started reasoning before Enter: in the app a branch is given only to models that fit in VRAM.

A small model gathering for a large one (--gatherer, experimental). A fast Qwen3-0.6B, also fine-tuned for it, writing material while you type for the 14B to read on Enter. On the RTX 2060 it does not beat the 14B reasoning ahead by itself: 47 s against 60 s untrained and 79-86 s fine-tuned; even ideal material written by hand gives 62 s, or 49 s when presented as the model's own reasoning. The 14B reads it, then redoes the calculations. Open questions for anyone who wants to take this further are in the README.

On a Mac

MacBook Pro M2 Pro, 16 GB, macOS 26.2, llama.cpp with Metal; the same scripts and problems.

The 4B writes at 54 tok/s on the M2 Pro (83-87 on the RTX 2060), so both waits are longer than on the PC, and reasoning ahead takes about the same share of them away. The 14B is where the Mac differs: Metal may use 12 GB of the unified memory, so the whole model is on the GPU and writes at 18 tok/s. During the runs the Mac reached thermal pressure Heavy for a few minutes. Data: thinking_ahead_4b_m2pro.json, thinking_ahead_4b_m2pro_fixed.json, thinking_ahead_14b_m2pro.json.

Tried, and why not

• --recycle (answers to earlier versions of the message as n-gram drafts): +4% decode speed with natural typing, +20% on code, slightly later first token.
• --draft (Qwen3-0.6B drafting for the 14B): on the GPU it takes VRAM from the 14B, which then runs more of its layers on the CPU (4.8 → 3.5 tok/s); on the CPU it gains 8% with 4-token drafts and halves the reading speed.
• The reasoning ahead as the draft for speculative decoding after Enter: no faster on the 4B (73 tok/s either way) and slower on the 14B (5.0 → 3.1 tok/s), because the layers on the CPU compute every token of the draft and only 11% of the drafted tokens were right. Script: research/notes_as_draft.py.
• IQ4_XS instead of Q4_K_M on the 4B (RTX 4070 SUPER): 4% faster, one answer worse out of 28. Not worth it on the 4B; on the 14B, where it is 31% faster with the same answers, it is being decided.

Key fact learned: on a model split between GPU and CPU, the CPU's layers compute every token they are given. Anything that processes several tokens at once costs the 14B almost proportionally. To speed up the 14B, take work away from the CPU.

Help wanted: Radeon

MAFDET runs on Radeon cards through llama.cpp's Vulkan build, but nobody has measured it on one yet. The Vulkan path itself is checked, on the RTX 2060 (Qwen3-4B reads 2,103 tokens/s and writes 83, about as with CUDA). If you have a dedicated card, RX 5000 or later, 8 GB or more is ideal, about half an hour would fill this gap:

1. Install MAFDET and open it once: it should find your card, install llama.cpp Vulkan and download Qwen3-4B. mafdet doctor shows what it found.
2. In the app, Models, Measure Qwen3-4B.
3. From the MAFDET folder, with its environment active: python research/thinking_ahead.py radeon_4b.json. About 20 minutes; leave the machine otherwise idle.
4. Open an issue with the output of mafdet doctor, the numbers of Measure and radeon_4b.json.

Related research

• LiveMind: Low-latency Large Language Models with Simultaneous Inference
• StreamingThinker: Large Language Models Can Think While Reading
• SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
• Asynchronous Reasoning: Training-Free Interactive Thinking LLMs