Benchmarks
We measure our legal AI in public
Every release of Attorly is benchmarked against corpus-verified legal-retrieval tasks across four jurisdictions. The latest results — passes, failures, and method — are published here.
Latest results
100%
Pass rate
37
Tasks graded
4
Jurisdictions
Sep 21, 2026
Last run
Build 9b4511f58
| Jurisdiction | Tasks | Passed | Pass rate |
|---|---|---|---|
| Norway | 11 | 11 | 100% |
| Sweden | 9 | 9 | 100% |
| Denmark | 9 | 9 | 100% |
| Germany | 8 | 8 | 100% |
How the benchmark works
Each task describes a legal matter the way a real user would — a case summary or a research question — and asks our retrieval pipeline for the most relevant legal sources. The task passes only if the statutes that actually govern the matter surface in the results.
Queries deliberately never name the governing statute — naming it would make retrieval trivial and measure nothing. Every task is verified against the official legal corpus of its jurisdiction before it is added.
Governing law surfaces
Every statute that governs the matter must appear in the retrieved sources. Missing the controlling law fails the task.
Noise stays out
Known tangential sources — plausible-looking but irrelevant statutes — must not appear. Surfacing noise fails the task.
Rank discipline
For selected tasks, the governing statute must appear at the top of the results, not merely somewhere in them.
Per-task results
Task descriptions are shown in English — they are the benchmark's source data, identical across all languages.
| Task | Jurisdiction | Result |
|---|---|---|
| no-tenancy-depositTenancy: landlord withholding a deposit after move-out | Norway | Pass |
| no-criminal-embezzlementCriminal defence: gross embezzlement (the Fredrik regression guard) | Norway | Pass |
| no-employment-dismissalEmployment: contested dismissal / unfair termination | Norway | Pass |
| no-family-custodyFamily: parental responsibility and where the child lives | Norway | Pass |
| no-sale-defectContract: defect in a commercial sale of goods | Norway | Pass |
| no-company-sharesCorporate: shareholder dispute in a limited company | Norway | Pass |
| no-personal-dataPrivacy: unlawful processing of personal data | Norway | Pass |
| no-inheritanceInheritance: distribution of an estate and forced share | Norway | Pass |
| no-bankruptcy-supplierInsolvency: supplier stopped paying, creditor weighs bankruptcy petition | Norway | Pass |
| no-house-hidden-defectsReal property: hidden defects after buying a house (avhendingslova, NOT tenancy) | Norway | Pass |
| no-stale-invoice-claimLimitation: collecting on a years-old unpaid invoice (foreldelsesloven) | Norway | Pass |
| se-employment-dismissalEmployment: contested dismissal (Lagen om anställningsskydd) | Sweden | Pass |
| se-consumer-saleConsumer: defective goods purchased by a consumer (Konsumentköplagen) | Sweden | Pass |
| se-company-sharesCorporate: minority shareholder dispute (Aktiebolagslagen) | Sweden | Pass |
| se-discriminationEmployment: workplace discrimination (Diskrimineringslagen) | Sweden | Pass |
| se-family-custodyFamily: custody and residence after separation (Föräldrabalken) | Sweden | Pass |
| se-criminal-embezzlementCriminal defence: embezzlement (Brottsbalken) | Sweden | Pass |
| se-inheritance-forced-shareInheritance: estate division and the children's reserved share (Ärvdabalken) | Sweden | Pass |
| se-tenancy-evictionTenancy: landlord terminating a lease — hyresrätt lives in Jordabalken, a hard mapping | Sweden | Pass |
| se-stale-debt-claimLimitation: enforcing an old consumer debt (Preskriptionslag) | Sweden | Pass |
| dk-employment-dismissalEmployment: salaried-employee dismissal (Funktionærloven) | Denmark | Pass |
| dk-sale-defectSale: defective goods and right to reject (Købeloven) | Denmark | Pass |
| dk-company-sharesCorporate: shareholder dispute in a capital company (Selskabsloven) | Denmark | Pass |
| dk-family-custodyFamily: parental responsibility and residence (Forældreansvarsloven) | Denmark | Pass |
| dk-inheritanceInheritance: estate distribution and forced share (Arveloven) | Denmark | Pass |
| dk-criminal-embezzlementCriminal defence: embezzlement (Straffeloven) | Denmark | Pass |
| dk-holiday-pay-exitEmployment: holiday pay owed when an employee resigns (Ferieloven) | Denmark | Pass |
| dk-employee-dataData protection: employer monitoring employee e-mail (Databeskyttelsesloven) | Denmark | Pass |
| dk-divorce-assetsFamily: division of assets at divorce (Ægtefælleloven) | Denmark | Pass |
| de-employment-dismissalEmployment: unfair dismissal protection (Kündigungsschutzgesetz) | Germany | Pass |
| de-criminal-embezzlementCriminal defence: embezzlement (Strafgesetzbuch) | Germany | Pass |
| de-sale-defectSale: defective goods and buyer remedies (Bürgerliches Gesetzbuch) | Germany | Pass |
| de-gmbh-formationCorporate: founding a GmbH, share capital and managing-director duties (GmbHG) | Germany | Pass |
| de-insolvency-filing-dutyInsolvency: over-indebted company and the duty to file (Insolvenzordnung) | Germany | Pass |
| de-working-time-overtimeEmployment: overtime limits and rest periods (Arbeitszeitgesetz) | Germany | Pass |
| de-works-council-codeterminationEmployment: works council consultation before restructuring (BetrVG) | Germany | Pass |
| de-misleading-advertisingCompetition: competitor runs misleading price advertising (UWG) | Germany | Pass |
Assistant source citations
Measures whether the assistant’s source citations stay on relevant law: for each question, the top sources are graded against a list of clearly off-domain statutes. Citing e.g. maritime-labour law on an ordinary dismissal question counts as noise.
100%
Pass rate
0%
Noise rate
0 of 36 citations
12
Tasks graded
Sep 21, 2026
Last run
Build 9b4511f58
| Task | Jurisdiction | Noise | Result |
|---|---|---|---|
| no-chip-employment-dismissalEmployment chips: office dismissal must not surface maritime-labour law | Norway | 0/4 | Pass |
| no-chip-employment-overtimeEmployment chips: unpaid retail overtime | Norway | 0/2 | Pass |
| no-chip-employment-temp-contractsEmployment chips: serial temporary contracts, claim to permanent employment | Norway | 0/3 | Pass |
| no-chip-employment-harassmentEmployment chips: whistleblowing about harassment at the office | Norway | 0/3 | Pass |
| no-chip-employment-holiday-payEmployment chips: holiday pay not paid out after quitting | Norway | 0/3 | Pass |
| no-chip-family-custodyFamily chips: residence and contact after a breakup | Norway | 0/5 | Pass |
| no-chip-family-divorce-assetsFamily chips: division of marital property and skjevdeling | Norway | 0/2 | Pass |
| no-chip-family-child-supportFamily chips: disagreement about child support | Norway | 0/3 | Pass |
| no-chip-consumer-used-carConsumer chips: used car from a dealer with hidden defects | Norway | 0/3 | Pass |
| no-chip-consumer-craftsmanConsumer chips: botched bathroom renovation by a craftsman | Norway | 0/3 | Pass |
| no-chip-consumer-online-cancellationConsumer chips: right of withdrawal on an online purchase | Norway | 0/3 | Pass |
| no-chip-housing-depositHousing chips: landlord withholding the deposit after move-out | Norway | 0/2 | Pass |
Citation verification accuracy
After every grounded answer, the assistant checks each citation against the text of the source it points at and marks it verified, partly supported or not supported. This benchmark measures that verifier itself on labeled pairs of a claim and a verbatim statute excerpt with a known answer.
Half the pairs are near-misses: the right statute but the wrong period, amount or party — the case a lenient verifier waves through. A false accept, an unsupported claim marked verified, is the number that matters; a false reject only costs the reader a double-check.
97%
Pass rate
0
False accepts
0 of 16 unsupported claims
34
Tasks graded
Sep 21, 2026
Last run
Build 9b4511f58
| Task | Jurisdiction | Ground truth | Verifier said | Result |
|---|---|---|---|---|
| no-aml-15-3-notice-one-monthaml § 15-3 (NL/lov/2005-06-17-62): default one-month mutual notice period restated | Norway | Supported | Verified | Pass |
| no-aml-15-3-notice-three-months-near-missaml § 15-3: claim states a three-month default notice period (statute says one month) | Norway | Unsupported | Not supported | Pass |
| no-aml-15-3-five-years-two-monthsaml § 15-3 (2): at least two months after five years of continuous employment | Norway | Supported | Verified | Pass |
| no-aml-15-1-consultation-before-dismissalaml § 15-1: employer must consult the employee before deciding on dismissal | Norway | Supported | Not supported | Fail |
| no-aml-15-1-consultation-optional-near-missaml § 15-1: claim says the consultation is optional for the employer (statute makes it a duty) | Norway | Unsupported | Not supported | Pass |
| no-aml-15-4-written-formaml § 15-4: dismissal must be in writing | Norway | Supported | Verified | Pass |
| no-husleie-3-5-deposit-six-monthshusleieloven § 3-5 (NL/lov/1999-03-26-17): deposit capped at six months of rent | Norway | Supported | Verified | Pass |
| no-husleie-3-5-deposit-three-months-near-misshusleieloven § 3-5: claim caps the deposit at three months (statute says six) | Norway | Unsupported | Not supported | Pass |
| no-husleie-3-5-off-topic-noticehusleieloven § 3-5 (deposit) cited for a claim about notice periods in employment | Norway | Unsupported | Not supported | Pass |
| no-fkjl-27-two-year-limitforbrukerkjøpsloven § 27 (NL/lov/2002-06-21-34): absolute two-year complaint deadline, five for durable goods | Norway | Supported | Verified | Pass |
| no-fkjl-27-three-year-near-missforbrukerkjøpsloven § 27: claim states a three-year absolute deadline (statute says two / five) | Norway | Unsupported | Not supported | Pass |
| no-ferie-5-off-topic-depositferieloven § 5 (NL/lov/1988-04-29-21, holiday length) cited for a claim about rent deposits | Norway | Unsupported | Not supported | Pass |
| se-las-11-minimum-one-monthLAS 11 § (sfs-1982-80): minimum one-month notice for both parties | Sweden | Supported | Verified | Pass |
| se-las-11-four-years-three-monthsLAS 11 §: three months after at least four but under six years of employment | Sweden | Supported | Verified | Pass |
| se-las-11-ten-years-twelve-months-near-missLAS 11 §: claim gives twelve months after ten years (statute says six) | Sweden | Unsupported | Not supported | Pass |
| se-semester-4-twenty-five-dayssemesterlagen 4 § (sfs-1977-480): twenty-five days of holiday per year | Sweden | Supported | Verified | Pass |
| se-semester-4-thirty-days-near-misssemesterlagen 4 §: claim gives thirty days (statute says twenty-five) | Sweden | Unsupported | Not supported | Pass |
| se-kkl-5-2-two-months-always-timelykonsumentköplagen 5 kap. 2 § (sfs-2022-260): a complaint within two months of noticing is always timely | Sweden | Supported | Verified | Pass |
| se-kkl-5-2-off-topic-noticekonsumentköplagen 5 kap. 2 § (complaints) cited for a claim about employment notice | Sweden | Unsupported | Not supported | Pass |
| dk-ful-2-three-months-after-sixfunktionærloven § 2 (lta/2017/1002): three months notice from the employer after six months of employment | Denmark | Supported | Verified | Pass |
| dk-ful-2-cap-six-monthsfunktionærloven § 2, stk. 3: the notice period grows by one month per three years, capped at six months | Denmark | Supported | Verified | Pass |
| dk-ful-2-cap-nine-months-near-missfunktionærloven § 2: claim caps the notice period at nine months (statute says six) | Denmark | Unsupported | Not supported | Pass |
| dk-kbl-83-two-year-limitkøbeloven § 83 (lta/2021/1853): defects must be raised within two years of delivery | Denmark | Supported | Verified | Pass |
| dk-kbl-83-five-year-near-misskøbeloven § 83: claim states a five-year absolute deadline (statute says two) | Denmark | Unsupported | Not supported | Pass |
| dk-ferie-5-off-topic-noticeferieloven § 5 (lta/2024/152, holiday accrual) cited for a claim about notice periods | Denmark | Unsupported | Not supported | Pass |
| de-bgb-622-four-weeks-basicBGB § 622 (1) (bgb): basic four-week notice to the 15th or month end | Germany | Supported | Verified | Pass |
| de-bgb-622-five-years-two-monthsBGB § 622 (2) Nr. 2: two months after five years for an employer termination | Germany | Supported | Verified | Pass |
| de-bgb-622-probation-four-weeks-near-missBGB § 622 (3): claim gives four weeks during probation (statute says two weeks) | Germany | Unsupported | Not supported | Pass |
| de-bgb-551-deposit-three-monthsBGB § 551 (1): rent deposit capped at three months of net rent | Germany | Supported | Verified | Pass |
| de-bgb-551-deposit-six-months-near-missBGB § 551: claim allows six months of rent as deposit (statute says three) | Germany | Unsupported | Not supported | Pass |
| de-kschg-1-six-months-thresholdKSchG § 1 (1) (kschg): protection applies after more than six months of uninterrupted employment | Germany | Supported | Verified | Pass |
| de-kschg-4-three-week-deadlineKSchG § 4: claim must be filed within three weeks of receiving the written termination | Germany | Supported | Verified | Pass |
| de-kschg-4-four-week-near-missKSchG § 4: claim gives a four-week filing deadline (statute says three weeks) | Germany | Unsupported | Not supported | Pass |
| de-kschg-1-off-topic-depositKSchG § 1 (dismissal protection) cited for a claim about rent deposits | Germany | Unsupported | Not supported | Pass |
What this does and doesn't show
- This is an in-house benchmark, authored and graded by us. It is not an independent evaluation — treat it as our own quality bar, published so you can hold us to it.
- It measures legal-source retrieval — whether the right law surfaces for a matter. It does not measure the quality of drafted text or end-to-end answers.
- Results are produced automatically for every deployed release and published unedited, including failures.
See also:Security & data handling