Short answer
In the latest runs (25 October 2026), on GerLeRB, the governing German law was in Attorly’s first 5 results for 111 of 248 graded questions (45%, 95% CI 39–51%); on GerDaLIR, a cited court decision was in the first 10 results for 3 of 150 graded tasks (2%, 95% CI 1–6%). Both datasets were built by third parties, so Attorly chose neither the questions nor the answers. None of the German court decisions carried vectors in the latest case-law run, so that retrieval used keyword search plus a language-model re-rank.
GerLeRB: German statute retrieval
| Measure | Result |
|---|---|
| Gold law in the first 5 results | 45% (95% CI 39–51%) |
| Questions graded | 248 of 367 |
| Mean reciprocal rank | 0.41 |
| Gold paragraph named | Not available: no result carried a section marker |
| Law (at least 5 questions) | Graded | Gold law in first 5 |
|---|---|---|
| BGB | 38 | 63% |
| SG | 21 | 43% |
| ZPO | 21 | 43% |
| FGO | 17 | 35% |
| BVerfGG | 15 | 27% |
| MarkenG | 15 | 67% |
| FamFG | 11 | 64% |
| StPO | 10 | 90% |
| AO 1977 | 9 | 100% |
| InvStG 2018 | 8 | 13% |
| EStG | 7 | 71% |
| UStG 1980 | 7 | 71% |
| Item | Value |
|---|---|
| First published run | 30 September 2026 |
| Latest run | 25 October 2026 |
| Runs published | 22 |
| Build measured | 326bbe092 |
| Models that served the run | claude-haiku-4-5-20251001 |
GerDaLIR: German case-law retrieval
| Measure | Result |
|---|---|
| Cited decision in the first 10 results | 2% (95% CI 1–6%) |
| Tasks graded | 150 of 150 |
| Mean reciprocal rank | 0.01 |
| Cited decisions with vectors in the store | 0 of 153 |
| Item | Value |
|---|---|
| First published run | 30 September 2026 |
| Latest run | 23 October 2026 |
| Runs published | 20 |
| Build measured | df53fd80e |
| Models that served the run | claude-haiku-4-5-20251001 |
The datasets
| Dataset | What it contains | Source and licence |
|---|---|---|
| GerLeRB | 367 legal questions, each paired with the one statutory paragraph that answers it, out of 90,798 paragraphs on gesetze-im-internet.de. Questions were generated with GPT-4.1 from court-decision contexts and verified by legal experts. | Weber et al., RVI 2025; Zenodo record 15745124; CC BY 4.0 |
| GerDaLIR | Passages of German court decisions that cite another decision, with the reference masked. The cited decision is the gold answer. | Wrzalik & Krechel, NLLP @ EMNLP 2021; built from Open Legal Data; MIT licence |
How the benchmark works
Each task makes the same legal-source call Attorly makes for a German matter, with the dataset’s question or masked passage as the query, and grades the product’s own ranking, source-type quotas included. GerLeRB passes when the gold law is in the first 5 results; every one of the 367 questions is a task. GerDaLIR passes when a cited decision is in the first 10 results.
GerLeRB task set: every question is a task, with no sampling. When the set was built on 29 September 2026, every gold law was confirmed to exist in Attorly’s German statute corpus with the gold paragraph in its text. A gold that later disappears from the corpus is skipped, never failed, and the task set is versioned, so a rebuild is a new version rather than an edit.
GerDaLIR sample: 150 of the 309 test-split queries whose every cited decision Attorly’s federal-court corpus (BGH, BAG, BVerfG) holds, out of 12,298 test queries, drawn with the fixed seed 20260929.
The benchmark runs automatically every night on the release deployed at the time, at most once a day. Every run is published unedited, including failures.
Limitations
- German law only: these results say nothing about Norway, Sweden, Denmark, Finland or the EU.
- GerDaLIR is a sample of 150 tasks limited to decisions Attorly holds, so it measures ranking, not corpus coverage.
- Case-law retrieval depends on whether the German decisions carry vectors; the share that did in the latest run is in the GerDaLIR table above.
- Retrieval only: the datasets grade which sources come back, not the answer written from them.
Data and downloads
Who wrote this and who checked it
Published by Attorly, a product of Getia AS in Oslo, Norway. The tasks, graders and this report are written and maintained by the Attorly team, and the benchmark runs on Attorly’s own production release.
No external reviewer has checked the method or the results. Read them as a vendor’s measurement of its own product. If you find an error, email hello@attorly.ai and we correct this page and the benchmark page.
Frequently asked questions
- Why use third-party datasets?
- Because Attorly wrote every other benchmark it publishes. On these two, someone else chose the questions and the gold answers.
- Is Attorly a good choice for German case law?
- Judge by the GerDaLIR figure above. It covers decisions of three federal courts only, and it is published whatever the result is.