Short answer
In the latest run on 21 October 2026, Attorly’s legal research returned every statute that governs the matter in 36 of 37 graded tasks (97%, 95% CI 86–100%) across Norway, Sweden, Denmark and Germany. The benchmark has run 116 times since 4 August 2026. It is an in-house test of statute retrieval only, not of case law or drafting.
Results by jurisdiction
| Jurisdiction | Tasks graded | Passed | Pass rate | 95% CI |
|---|---|---|---|---|
| Norway | 11 | 11 | 100% | 74–100% |
| Sweden | 9 | 8 | 89% | 57–98% |
| Denmark | 9 | 9 | 100% | 70–100% |
| Germany | 8 | 8 | 100% | 68–100% |
| All four | 37 | 36 | 97% | 86–100% |
With 8 to 11 graded tasks per jurisdiction, one task moves a country’s rate by 9 to 13 points. Two rates whose intervals overlap are not measurably different.
Sample and timeframe
| Item | Value |
|---|---|
| Tasks in the set | 37 |
| Graded in the latest run | 37 |
| Skipped in the latest run | 0 |
| First published run | 4 August 2026 |
| Latest run | 21 October 2026 |
| Runs published | 116 |
| Build measured | f6cb895e2 |
| Models that served the run | claude-haiku-4-5-20251001 |
The benchmark runs automatically every night on the release deployed at the time, at most once a day. Every run is published unedited, including failures.
What counts as a pass
- Governing law surfaces: every statute that governs the matter appears in the retrieved sources.
- Noise stays out: known plausible-looking but irrelevant statutes do not appear.
- Rank discipline: for selected tasks, the governing statute appears at the top of the results.
How the benchmark works
Each task describes a legal matter the way a client or lawyer would, in the jurisdiction’s own language, and sends it through the same source-suggestion call the product uses. The query never names the governing statute, because naming it would make retrieval trivial.
Every statute a task expects was checked to be present in the official corpus before the task was added, so a failure measures retrieval, not a missing law. The corpora are Lovdata (Norway), Riksdagen (Sweden), Retsinformation (Denmark) and Gesetze im Internet (Germany).
A task is skipped, never failed, when the vector store is unreachable or the language-model step fell back during the task, because the run then measured a different pipeline. Every rate is shown with a 95% Wilson confidence interval.
Limitations
- In-house: Attorly wrote the tasks and the graders. It is not an independent evaluation.
- Small sample: a few dozen tasks in total, so compare intervals rather than single numbers.
- Statutes only: it says nothing about case law, contract review, drafting or questions outside the four corpora.
- Four jurisdictions: Finland, the EU and the other countries Attorly covers are not part of this set.
- Not comparable: no other legal AI tool has run these tasks, so the rate is not a ranking against competitors.
Data and downloads
Who wrote this and who checked it
Published by Attorly, a product of Getia AS in Oslo, Norway. The tasks, graders and this report are written and maintained by the Attorly team, and the benchmark runs on Attorly’s own production release.
No external reviewer has checked the method or the results. Read them as a vendor’s measurement of its own product. If you find an error, email hello@attorly.ai and we correct this page and the benchmark page.
Frequently asked questions
- Is this an independent evaluation?
- No. Attorly wrote the tasks and graders and publishes every run, including the failures, so you can check the work.
- Can I compare this rate with other legal AI tools?
- Not directly. Other tools have not run these tasks. Use it to see how Attorly performs over time and per country.
- How often is it updated?
- Every night on the release in production, at most once a day. The page always shows the latest run.