Short answer
In the latest run on 28 September 2026, 12 of 31 answers to statute questions (39%, 95% CI 24–56%) contained at least one wrong or mis-cited legal claim, 91% of cited claims were supported by the law they cited, and 0 of 164 citations pointed to a source or section that does not exist. The questions cover Norway, Sweden, Denmark and Germany; the grading is done by models and fixed rules, not by lawyers.
Key findings
| Measure | Result | Basis |
|---|---|---|
| Answers with a wrong or mis-cited legal claim | 39% (95% CI 24–56%) | 12 of 31 answers |
| Cited claims supported by the cited law | 91% | 150 of 164 claims |
| Fabricated or non-existent citations | 0% | 0 of 164 citations |
| Required points covered | 92% | 73 of 79 points |
| Accurate answers (nothing wrong, nothing missing) | 0% | 0 of 32 graded questions |
Results by jurisdiction
| Jurisdiction | Graded | Accurate | Hallucinated | Incomplete | No answer |
|---|---|---|---|---|---|
| Norway | 8 | 0 | 1 | 6 | 1 |
| Sweden | 8 | 0 | 6 | 2 | 0 |
| Denmark | 8 | 0 | 3 | 5 | 0 |
| Germany | 8 | 0 | 2 | 6 | 0 |
Sample and timeframe
| Item | Value |
|---|---|
| Questions in the latest run | 32 |
| Graded in the latest run | 32 |
| Skipped in the latest run | 0 |
| First published run | 28 September 2026 |
| Latest run | 28 September 2026 |
| Runs published | 1 |
| Build measured | 37f924ff6 |
| Models that served the run | Not recorded |
The question set now holds 100 questions, 25 per jurisdiction. A run grades the set as it stood on that day, so an older run can hold fewer. A question is skipped, not failed, when the corpus or the citation check was unavailable, the answer call failed or it timed out; skipped questions are listed in the download.
Definitions
| Outcome | Meaning |
|---|---|
| Hallucinated | A cited claim is not supported by its source, a citation points to a source or section that does not exist, the answer relies on a clearly off-topic statute, it contradicts a required point, or an uncited legal figure is contradicted by the governing statute. |
| Incomplete | Nothing is wrong, but a required point is missing, the governing section is not cited, or a legal rule is stated without a citation. |
| Accurate | Neither hallucinated nor incomplete. |
| No answer | The assistant found no authority and said so. Excluded from the hallucination rate. |
The outcome classes follow Magesh et al. (Stanford RegLab/HAI, 2024), a study of hallucination in commercial legal research tools, so that the term means the same thing. The scores are not directly comparable: that study graded 202 US questions by hand; this benchmark grades Nordic and German statute questions with a model-based verifier and fixed rules.
How answers are graded
Ask through the product pipeline
The assistant answers each statute question through the same retrieval, answer model and citation check as the product. Each question is written in the jurisdiction’s language and never names the governing statute.
Split the answer into claims
Every sentence that cites a source, or states law without one, is a claim. Deterministic checks catch citations to no retrieved source, to an act outside the corpus or to a section that does not exist.
Check each cited claim
The product’s own citation check decides whether the cited passage supports the claim. Verdicts that alone would make an answer hallucinated, and accepted claims whose legal figure is not in the passage, are re-checked on a stronger model, whose verdict is final.
Match required points
Each question has required points, such as periods, amounts and conditions from the statute. A fixed pattern matcher, not a model, looks for them in the answer.
Limitations
- The judge is a model: the citation check and the stronger re-check are language models, not lawyers. Their verdicts are in the download so you can disagree with them.
- An incorrect uncited statement that contradicts no required point and gives no legal figure is not caught as a hallucination; at most it makes the answer incomplete.
- Statute questions only: no case law, no false-premise questions and no fact recall.
- Four jurisdictions and at most 25 questions per jurisdiction, so one question moves a country’s rate by several points.
- In-house: Attorly wrote the questions and the grading rules.
The benchmark runs automatically at night after a new release has been deployed, at most once a day, so the latest run can be older than the latest release. Every run is published unedited, including failures.
Data and downloads
Who wrote this and who checked it
Published by Attorly, a product of Getia AS in Oslo, Norway. The tasks, graders and this report are written and maintained by the Attorly team, and the benchmark runs on Attorly’s own production release.
No external reviewer has checked the method or the results. Read them as a vendor’s measurement of its own product. If you find an error, email hello@attorly.ai and we correct this page and the benchmark page.
Frequently asked questions
- Does Attorly hallucinate?
- Sometimes, and this page measures how often on statute questions. Every legal claim in an answer links to its source, and the product marks citations its check could not verify, so a reader can check before relying on it.
- Why publish an error rate at all?
- Because a buyer should know the error rate before relying on any legal AI. The benchmark shows where answers still go wrong and whether that improves from run to run.
- Is the content legal advice?
- No. Attorly gives legal information. A qualified lawyer should check anything you rely on.