How the questions were chosen, how each system was run, how answers were graded, and what the numbers do not show.
Taken from a held-out set of questions written by a lawyer, each with an expected answer derived from the acts. They are phrased the way clients ask, in Uzbek, Russian and English, and cover banking, anti-money-laundering, civil law, liability, land, tax and labour. The set is not published, so that no system can be tuned to it.
Written by us to probe the cases where general search fails. Expected answers were taken from the acts themselves, not from a lawyer. The set is published below.
| What it tests | Question |
|---|---|
| A rule behind "other cases established by law" | My loan at one bank is overdue. Can that bank debit my card at another bank without my consent? |
| Several acts at once | Which sanitary rules apply to tobacco products, what monthly fee does a retailer pay, and what liability applies for illegal circulation? |
| Figures in decrees | What State Budget revenue and expenditure did legislation set for 2010, 2011 and 2012? |
| An act three days old | Must a privately owned bank's cybersecurity staff take the training required by a resolution adopted three days before the test? |
| The rule at a past date | What minimum charter capital did a commercial bank need on 1 June 2024, and what is it now? |
| Appendix tables | How long does the review of a tourism licence application take, and what is the fee? |
| Licence conditions | Can a tour operator send citizens on Umrah? What do the licence requirements say? |
| The rule at a past date | What share of its funds could a charity spend on its own upkeep in September 2025, and now? |
| The rule at a past date | Up to how many years did childcare leave count towards the pension record in 2021, and now? |
| Regulator sanctions | Is there a fine for payment organisations that breach anti-money-laundering rules? How much? |
| A rule in a regulation's appendix | Is an unanswered electronic invoice deemed accepted after 10 days? |
| Restrictions by investor type | Can a company with foreign investment buy land? |
| System | Setup |
|---|---|
| ChatGPT, Gemini, Perplexity | Their own apps, web search on. A new conversation; the questions pasted as one numbered list; the answers copied as given. |
| ChatGPT with Recansio | The same app and prompt, with Recansio added as an MCP connector, in a new conversation. Both sets. |
| Claude | The same prompt in four setups: no tools; web search; with Recansio; with Recansio in research mode (browse acts, follow references). Each question in a fresh context. |
| Qwen 3.8 27B, local | 4-bit weights in LM Studio on a laptop (Apple M1 Max, 64 GB). Reasoning mode off, temperature 0.6, one question at a time in a fresh chat. With Recansio: the same server as the other systems, tool results cut at 12,000 characters, a final answer forced after six tool steps. |
| Claude with Recansio, one chat | An ordinary chat whose only legal source was Recansio; the 30 questions of set A pasted as one list, as with the other apps. |
The prompt asked for the rule with its numbers, dates and conditions, the act, the article or point, and an official link, and asked the system to say so when it was not sure.
Each answer was graded by three graders working separately. A grader saw the question, the expected answer and one answer at a time, with the system's name removed and the order shuffled. The verdict is the majority of the three; a three-way split counts as partly right.
| Verdict | Rule |
|---|---|
| Correct | States the expected key points (rule, numbers, deadlines, who) without a material error. |
| Partly right | The main direction is right, but a key element is missing or wrong. |
| Wrong | Contradicts the expected answer, gives a wrong number or rule, or answers confidently with invented content. |
| No answer | Says it cannot answer, without the key point. |
A confident wrong number is graded wrong. Style, length and language are not graded.
The graders are language models (separate Claude instances), from the same family as one of the tested systems. Blinding removes the system's name but not its writing style. On the local model's 60 answers the three graders agreed on every verdict.
The count of each verdict per system.
An answer is precise when it contains a link to the official registry that points to a paragraph, not just to the act. This was counted automatically from the answer text. It does not check that the paragraph is the right one; correctness covers that.
Wall-clock time from sending the question to the final answer. For ChatGPT, and for Claude in an ordinary chat, the time for the whole pasted list (12 or 30 questions), read from the chat interface. For the local model, the sum of the per-question times measured by the test script, including tool calls; questions ran one at a time. Claude on its own was timed the same way as the apps: all 30 questions in one message, with no tools, none of the operator's instructions or saved notes loaded, and high reasoning effort. Times compare a system with itself, with and without the structure; they do not compare systems, which ran on different hardware.
The risk assessment figures come from use in a bank's real product approval work, not from a controlled experiment. Before: each specialist took 5 to 10 working days to assess a product against the law, write the findings and explain the mitigating measures. With Recansio: 10 to 20 minutes to assess a product for all departments, each finding tied to its clause. The products and documents are confidential and are not shown.
The first corpus is the legislation of Uzbekistan: 57,893 acts from the official registry, in Uzbek and Russian, with 66,734 past editions and 523,943 dated links between acts. It is built by deterministic rules, with no language model inside; two builds from the same inputs give the same result. Language models read it through MCP tools: search, read a paragraph, read it as of a date, and follow a reference.