People building AI memory and RAG often assume reasoning is the hard part: when multi-step questions fail, they rework the reasoning step or switch to a bigger model.
I split the answer path of my personal memory system into stages and measured how much each stage loses. The result points the other way: given correct inputs, the reasoning stage holds up; most of the loss comes earlier, when the system doesn't work out what the question is asking.
This is a research result with small samples. It does not describe how any product behaves.
The stages behind one answer
Take "Which city is my advisor's university in?" The system has to:
- Find the start: which person in its knowledge "my advisor" refers to.
- Pick the relations: which steps to take. Here, two: "teaches at", then "is located in".
- Walk the facts: follow those steps through the fact store; the user never names the university in the middle.
- Answer or abstain: give the answer with its support, or say there is no support.
A mistake at any stage ends in a wrong answer. Final accuracy alone can't tell you which stage failed.
The test: same questions, answered twice
To separate reading from reasoning, I built questions from the system's own fact store, so every fact a question needs is in the store. A wrong answer can't be blamed on missing knowledge.
How questions were made: take a chain of facts from the store, such as "person → teaches at → university → located in → city", and have a large model write the question in English and Chinese without naming the middle step or the answer. A second model answers blind, without seeing the chain; a question is kept only if its answer matches the end of the chain.
Each question is answered twice:
- Route given: the system is told the correct starting concept and relation chain. This tests reasoning only.
- Question only: the system starts from the question text. Reading the question is currently done by a fixed version of a large model.
A large model judges both runs against the same standard.
Building the test set was the first finding
Random chains of true facts mostly aren't questions anyone would ask.
Of 330 random chains, 34 survived, about 10%. Among rejected multi-step chains, the most common reason was "every fact is true, but the combination means nothing": 16 of 40 rejected two-step chains and 12 of 20 three-step chains. Choosing relation pairs that people often ask about together raised the yield to 112 kept out of 767, about 15%. Of 150 three-step chains, 1 survived.
The facts in the store are true. Which combinations are worth asking is something the store itself doesn't know. That comes back later.
The 112 kept questions each have an English and a Chinese version: 224 answers in total.
Result: 0.79 with the route, 0.57 from the question
| Route given | Question only | |
|---|---|---|
| All 224 answers | 0.79 | 0.57 |
| One-step (English / Chinese) | 0.93 / 0.85 | |
| Two-step (English / Chinese) | 0.73 / 0.71 |
Numbers are the share judged correct.
With the route given, one-step questions are nearly all right and two-step questions over 70%. When the system is set to answer only if it finds a complete path, it answers 91% of questions with precision 0.84. Given correct inputs, the reasoning stage works.
From the question alone, the same set drops 0.22. Where it goes:
- Start found correctly: 57% to 65% of questions (varies by language and question type).
- Start not found: 12% to 24%.
- The rest: bound to a different concept.
- Relation chain exactly right: 73% to 78%.
Both the start and the relation chain must be right before reasoning has a chance. The 0.22 is lost mainly in those two stages.
On questions without a guaranteed answer, the loss moves even earlier
The set above guarantees the facts are in the store. A second set of 344 multi-step questions makes no such guarantee. Stage by stage:
The share judged correct at the end is 0.196.
Among the questions that broke along the path, 98 broke at a relation type the system doesn't cover yet (46 of them "cause" relations), and 39 at a concept outside the core concept set.
These numbers also show why a single fix barely moves the total: losses multiply across stages. Improving one stage by a few points usually moves final accuracy by 0 to 2 points.
Why reading the question is hard
The core problem is the starting point. One name often maps to several closely related concepts in the knowledge, and if the wrong one is picked, correct reasoning answers a different question.
The previous post set a rule: if the system can't tell who or what is meant, it says so; no fuzzy binding. Not finding the start lets the system answer "no support". Picking the wrong start gives an answer with support that belongs to something else.
That rule makes the fix hard. I tried two ways to get better at finding the start: training a dedicated model to map different phrasings to concepts, and training a small local question reader on a large set of generated questions, so no large model is needed. Neither beat the current approach:
- When finding the start had to be at least 90% precise, one approach never got there, and the other could handle fewer than a fifth of the questions.
- Without that requirement, wrong starts went up sharply. In one run, wrong bindings per 100 starts went from 4.5 to over 40.
Answering more questions is easy. The hard part is keeping the answers right.
What builders of AI memory and RAG can use
- Measure by stage. At least separate finding the start, picking relations, and reasoning. Final accuracy alone makes reading problems look like reasoning problems.
- Give reasoning the right inputs before changing it. It's a reasoning problem only if it still fails with the route given.
- Not found beats found wrong. Count "not found" and "found the wrong one" separately; their costs are very different.
- Test with questions people would ask. Questions stitched together from a fact store are mostly not ones anyone asks, and scores on them don't say much about real use.
Limits of this result
- The route-given vs question-only comparison is 112 questions, 224 answers; three-step and rule-derived questions are too few to conclude anything here.
- Correctness was judged by a large model, not checked by hand item by item.
- Reading the question still uses a large model. My previous post said answering makes no large-language-model call; that was an earlier design, and this stage doesn't meet it yet. I'm working on replacing it with a local model.
- This is one research measurement, not a description of a product feature.
Earlier in this series: AI Long-Term Memory: Similar Is Not the Same as Known, and One Transfer Experiment: Bigger Fits Better, Transfers Worse.