We measure our legal AI in public
Every release of Attorly is benchmarked against corpus-verified legal-retrieval tasks across four jurisdictions. The latest results — passes, failures, and method — are published here.
Latest results
| Jurisdiction | Tasks | Passed | Pass rate |
|---|---|---|---|
| Norway | 8 | 8 | 100% |
| Sweden | 1 | 1 | 100% |
| Denmark | 0 | 0 | — |
| Germany | 0 | 0 | — |
How the benchmark works
Each task describes a legal matter the way a real user would — a case summary or a research question — and asks our retrieval pipeline for the most relevant legal sources. The task passes only if the statutes that actually govern the matter surface in the results.
Queries deliberately never name the governing statute — naming it would make retrieval trivial and measure nothing. Every task is verified against the official legal corpus of its jurisdiction before it is added.
Governing law surfaces
Every statute that governs the matter must appear in the retrieved sources. Missing the controlling law fails the task.
Noise stays out
Known tangential sources — plausible-looking but irrelevant statutes — must not appear. Surfacing noise fails the task.
Rank discipline
For selected tasks, the governing statute must appear at the top of the results, not merely somewhere in them.
Per-task results
Task descriptions are shown in English — they are the benchmark's source data, identical across all languages.
| Task | Jurisdiction | Result |
|---|---|---|
| no-tenancy-depositTenancy: landlord withholding a deposit after move-out | Norway | Pass |
| no-criminal-embezzlementCriminal defence: gross embezzlement (the Fredrik regression guard) | Norway | Pass |
| no-employment-dismissalEmployment: contested dismissal / unfair termination | Norway | Pass |
| no-family-custodyFamily: parental responsibility and where the child lives | Norway | Pass |
| no-sale-defectContract: defect in a commercial sale of goods | Norway | Pass |
| no-company-sharesCorporate: shareholder dispute in a limited company | Norway | Pass |
| no-personal-dataPrivacy: unlawful processing of personal data | Norway | Pass |
| no-inheritanceInheritance: distribution of an estate and forced share | Norway | Pass |
| se-employment-dismissalEmployment: contested dismissal (Lagen om anställningsskydd) | Sweden | Pass |
| se-consumer-saleConsumer: defective goods purchased by a consumer (Konsumentköplagen) | Sweden | Skipped |
| se-company-sharesCorporate: minority shareholder dispute (Aktiebolagslagen) | Sweden | Skipped |
| se-discriminationEmployment: workplace discrimination (Diskrimineringslagen) | Sweden | Skipped |
| se-family-custodyFamily: custody and residence after separation (Föräldrabalken) | Sweden | Skipped |
| se-criminal-embezzlementCriminal defence: embezzlement (Brottsbalken) | Sweden | Skipped |
| dk-employment-dismissalEmployment: salaried-employee dismissal (Funktionærloven) | Denmark | Skipped |
| dk-sale-defectSale: defective goods and right to reject (Købeloven) | Denmark | Skipped |
| dk-company-sharesCorporate: shareholder dispute in a capital company (Selskabsloven) | Denmark | Skipped |
| dk-family-custodyFamily: parental responsibility and residence (Forældreansvarsloven) | Denmark | Skipped |
| dk-inheritanceInheritance: estate distribution and forced share (Arveloven) | Denmark | Skipped |
| dk-criminal-embezzlementCriminal defence: embezzlement (Straffeloven) | Denmark | Skipped |
| de-employment-dismissalEmployment: unfair dismissal protection (Kündigungsschutzgesetz) | Germany | Skipped |
| de-criminal-embezzlementCriminal defence: embezzlement (Strafgesetzbuch) | Germany | Skipped |
| de-sale-defectSale: defective goods and buyer remedies (Bürgerliches Gesetzbuch) | Germany | Skipped |
What this does and doesn't show
- This is an in-house benchmark, authored and graded by us. It is not an independent evaluation — treat it as our own quality bar, published so you can hold us to it.
- It measures legal-source retrieval — whether the right law surfaces for a matter. It does not measure the quality of drafted text or end-to-end answers.
- Results are produced automatically for every deployed release and published unedited, including failures.