PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice

As large language models (LLMs) are increasingly applied to legal domain-specific tasks, evaluating their ability to perform legal work in real-world settings has become essential. However, existing legal benchmarks rely on simplified and highly standardized tasks, failing to capture the ambiguity, complexity, and reasoning demands of real legal practice. Moreover, prior evaluations often adopt coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. To address these limitations, we introduce PLawBench, a Practical Law Benchmark designed to evaluate LLMs in realistic legal practice scenarios. Grounded in real-world legal workflows, PLawBench models the core processes of legal practitioners through three task categories: public legal consultation, practical case analysis, and legal document generation. These tasks assess a model’s ability to identify legal issues and key facts, perform structured legal reasoning, and generate legally coherent documents. PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics, resulting in approximately 12,500 rubric items for fine-grained assessment. Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs. Experimental results show that none achieves strong performance on PLawBench, revealing substantial limitations in the fine-grained legal reasoning capabilities of current LLMs and highlighting important directions for future evaluation and development of legal LLMs. Data is available at: https://github.com/skylenage/PLawbench.

💡 Research Summary

The paper introduces PLawBench, a practical, rubric‑based benchmark designed to evaluate large language models (LLMs) on tasks that closely resemble real‑world legal practice. Existing legal benchmarks largely rely on simplified, standardized questions drawn from bar exams or academic exercises, which fail to capture the ambiguity, incomplete fact patterns, emotional narratives, and strategic considerations that lawyers routinely face. Moreover, prior evaluations typically use coarse, single‑dimensional metrics (e.g., final answer correctness or citation presence) and do not explicitly assess the multi‑step reasoning that legal work demands.

PLawBench addresses these gaps through three core design principles. First, it models the hierarchical workflow of legal professionals by defining three task categories: (1) public legal consultation, (2) practical case analysis, and (3) legal document generation. These tasks are instantiated across 13 authentic scenarios (e.g., divorce cohabitation, fraud vs. illegal fundraising) and comprise a total of 850 questions. The data are collected from multiple real sources—court records, public consultation logs, and law‑firm documents—intentionally embedding realistic “noise” such as vague queries, omitted key facts, and emotionally charged language.

Second, the benchmark embeds fine‑grained reasoning steps. Each task is broken down into six unified evaluation dimensions that reflect core competencies identified in the EU AI Act risk framework and legal risk management practice: (1) Issue & Fact Identification, (2) Legal Reasoning, (3) Legal Knowledge Application, (4) Procedural & Strategic Awareness, (5) Claim & Outcome Construction, and (6) Professional Norms & Compliance. Expert annotators first define a reasoning‑oriented framework for each task type and then tailor specific rubric criteria to individual cases, resulting in roughly 12,500 rubric items across the benchmark. This structure enables assessment not only of final conclusions but also of intermediate reasoning stages such as fact clarification, rule extraction, logical linkage, and procedural guidance.

Third, PLawBench employs a rubric‑based automatic evaluator. A “judge model,” itself an LLM fine‑tuned on human‑annotated rubric scores, predicts fine‑grained scores for each dimension. The judge model is aligned with expert judgments and demonstrates high correlation with human evaluators, allowing large‑scale, cost‑effective scoring while preserving the nuanced feedback that a human rubric would provide.

The authors evaluate ten state‑of‑the‑art LLMs (including both Chinese‑language and English‑language models) using the benchmark. Overall performance is modest: the average score is 49 out of 80 (≈61 %). The most pronounced weaknesses appear in the Legal Knowledge Application and Procedural & Strategic Awareness dimensions, where models often fail to cite the correct statutes, misinterpret procedural requirements, or overlook strategic risk points embedded in the scenarios. While models can reproduce statutory language and generate syntactically correct legal text, they struggle with the deeper, context‑dependent reasoning required to reconstruct missing facts, ask clarifying questions, and assess procedural risks.

All data, task definitions, and rubric specifications are released publicly on GitHub, ensuring reproducibility and encouraging community extensions. The authors outline future work such as adapting the rubric to multiple jurisdictions and languages, integrating the benchmark into human‑LLM collaborative workflows (e.g., lawyer‑assistant tools), and developing dynamic rubric updates to keep pace with evolving law.

In summary, PLawBench represents a significant shift in legal AI evaluation: from coarse answer‑checking toward a comprehensive, risk‑aware assessment of the reasoning processes that underpin professional legal work. The findings highlight that current LLMs, despite impressive language capabilities, are not yet ready to serve as autonomous legal practitioners and that fine‑grained, practice‑oriented benchmarks like PLawBench are essential for guiding the next generation of legally competent AI systems.