Better Call GPT, Comparing Large Language Models Against Lawyers

Are large language models about to replace junior attorneys and advocates?

Author

Ross Shepstone
February 03, 2024

In the evolving legal landscape, the advent of Large Language Models (LLMs) presents both challenges and opportunities. A recent study, "Better Call GPT: Comparing Large Language Models Against Lawyers," demonstrates LLMs' potential to match or exceed the performance of junior lawyers and legal process outsourcers in contract review tasks. This development underscores the importance of integrating technology into legal practice, not only for efficiency but also for enhancing the accuracy of legal processes. Legal practitioners and judges are encouraged to explore the benefits and implications of these technologies, balancing innovation with the ethical and practical aspects of legal work.

For a more detailed exploration of this study and its implications, legal professionals can delve into the full paper, which offers insights into the methodology, results, and potential future directions for research in this area.

Better Call GPT, Comparing Large Language Models Against Lawyers

This paper presents a groundbreaking comparison between Large Language Models and traditional legal contract reviewers, Junior Lawyers and Legal Process Outsourcers. We dissect whether LLMs can outperform humans in accuracy, speed, and cost efficiency during contract review. Our empirical analysis benchmarks LLMs against a ground truth set by Senior Lawyers, uncovering that advanced models match or exceed human accuracy in determining legal issues. In speed, LLMs complete reviews in mere seconds, eclipsing the hours required by their human counterparts. Cost wise, LLMs operate at a fraction of the price, offering a staggering 99.97 percent reduction in cost over traditional methods. These results are not just statistics, they signal a seismic shift in legal practice. LLMs stand poised to disrupt the legal industry, enhancing accessibility and efficiency of legal services. Our research asserts that the era of LLM dominance in legal contract review is upon us, challenging the status quo and calling for a reimagined future of legal workflows.

arxiv.org/abs/2401.16212

The report "Better Call GPT: Comparing Large Language Models Against Lawyers" explores the effectiveness of Large Language Models (LLMs) in legal contract review, contrasting their performance with Junior Lawyers and Legal Process Outsourcers (LPOs). It investigates accuracy, speed, and cost-efficiency, finding that LLMs match or exceed human accuracy in identifying legal issues, complete reviews significantly faster, and operate at a fraction of the cost. The study indicates a potential shift in legal practices, suggesting LLMs could significantly enhance the accessibility and efficiency of legal services.

Key Takeaways and Examples:

1. Accuracy in Identifying Legal Issues:

Explanation: LLMs demonstrate comparable or superior precision in determining legal issues within contracts, though their ability to locate specific issues is model-dependent.

Examples: GPT-4 variants show high precision and recall, mirroring or surpassing Junior Lawyers and LPOs. For instance, a leading law firm using AI for document review could serve as a practical application, streamlining contract analysis processes.

2. Speed of Contract Review:

Explanation: LLMs significantly outpace human reviewers in contract analysis speed, showcasing their capability to process and analyze documents much faster.

Examples: The fastest LLM reviewed contracts in a matter of seconds, a stark contrast to hours required by human reviewers. A comparison could be made to the adoption of AI by legal tech companies like LegalZoom, which has enhanced client service speed and efficiency.

3. Cost Efficiency:

Explanation: LLMs offer a dramatically lower cost per document for contract review, presenting a cost-effective alternative to traditional human-based review methods.

Examples: The cost per contract review by LLMs is substantially lower, indicating over 99% cost reduction compared to human reviewers. An example is the use of AI in cost-sensitive legal services provided by startups, significantly reducing operational costs.

These findings advocate for the integration of LLMs into legal practices, emphasizing their potential to redefine traditional legal workflows through improved accuracy, efficiency, and affordability.