Who Wins In The Showdown Between AI & Lawyers?

Artificial Intelligence (AI) is having a transformative effect on the business world and the  $600 billion global legal services market is not immune.  As AI automates basic processes, in the legal profession it promises to allow lawyers devote their time to more valuable, cost-effective, and strategic work. Consultants at McKinsey & Company estimate that 22% of a lawyer’s job and 35% of a paralegal’s job can be automated.

However, the common perception among lawyers remains that machines cannot yet match the legal intellect of human lawyers in daily fundamentals of the profession.

This assumption was tested in the first of its kind “AlphaGo”-style Study in the legal profession. The AI vs Lawyer study, was carried out in conjunction with Professors at Duke Law, University of Southern California, and Stanford Law School, alongside input from data scientists and veteran corporate lawyers.  

The first ever research into the impact of AI on daily commoditized legal tasks pitted LawGeex, an AI Legal Technology platform, against 20 highly experienced US-trained lawyers. It tested their speed and accuracy on spotting 30 legal issues (vetted by contract experts and professors at top law schools) in a staple of the legal profession — Non-Disclosure Agreements.

The 20 lawyers taking on the legal AI were selected for their decades of legal experience precisely in reviewing and approving contracts. Their contract expertise spanned companies including Goldman Sachs and Cisco, and global law firms including Alston & Bird and K&L Gates.

In the final analysis, the AI, trained on tens of thousands of contracts, achieved a 94% accuracy rate at surfacing risks in Non-Disclosure Agreements (NDAs), one of the most common daily legal contracts used in business. This compares to an average of 85% for experienced lawyers. The challenge took the LawGeex AI 26 seconds to complete, compared to an average of 92 minutes for the lawyers.

This represents three specific wins for the growing legal AI field, which is seeing startup companies using AI to tackle daily legal work, including drafting and reviewing contracts, mining documents in discovery and due diligence, answering routine questions or sifting data to predict outcomes.  

1. Win for time-strapped lawyers

This victory of AI against lawyers is not however a particularly hand-wringing moment for lawyers defeated by the machines. Most lawyers hate manual and repetitive works, exemplified by NDA contract review and approvals (chosen for this test).  This represents a win for lawyers to trust in the growing number of AI solutions at a critical juncture when they are desperate to work more strategically and efficiently.

Only 28% of legal departments are hiring, while almost two-thirds of legal departments report an increase in the amount of legal work. US law firms for their part are turning to AI solutions as they experience sluggish growth in demand and decline in productivity. Some 49% of law firms say they are using technology to replace human resources with the aim of improving efficiencies.

In the words of one participant in the test, Grant Gulovsen, an attorney with more than 15 years’ experience: ““Participating in this experiment really opened my eyes to how ridiculous it is for attorneys to spend their time (as well as their clients’ money) creating or reviewing documents like NDAs which are so fundamentally similar to one another”. 

Professor Gillian K. Hadfield, professor of law and economics at the University of Southern California, and one of the study’s advisors goes even further. She says: “I think it’s important to recognize that this experiment actually understates the gain from AI. The lawyers who reviewed these documents were fully focused on the task: it didn’t sink to the bottom of a to-do list, it didn’t get rushed through while waiting for an airplane or with one eye on the clock to get out the door to a meeting or to pick up kids.”

2. Win Against Legalese

Secondly, it is a big win for all of us who in our businesses or as consumers suffer the complex language of legalese. Both the lawyers and the LawGeex AI analyzed five previously unseen contracts, containing 153 paragraphs of such technical legalese.  

Lawyer, Ken Adams, who has written books on contract drafting, often talks of the legal bafflement caused by entrenched language such as “hereunder” and “Witnesseth”.  Even more damagingly, he points out, Legalese is a risk for businesses causing significant confusion in courts.  According to the Association of Contract Managers annual benchmark survey, only 12% of lawyers think contracts are understandable and easy to read.  

In the words of Professor Yonatan Aumann of the Department of Computer Science at Bar Ilan University, and an advisor to LawGeex: “We had to teach the AI to ‘understand’ legalese – which is rather different from normal English. To this end, our researchers exposed the AI to tens of thousands of legal documents.  This core training in legal parlance was essential for enabling the AI to spot the important legal elements of the document. The algorithm can now identify concepts in contracts even when they are written in different ways.”

In its analysis of our client’s contracts LawGeex (and other legal AI solutions) provide a simple Legalese-to-English translation of the legal language and risks. It answers whether or not you can sign this agreement according to your company’s predefined standards. This is a major step forward in highlighting and translating the complex language of Legalese.

3. Win against “idea of Magic” of Law

Perhaps most importantly, the result may help end a widespread view about the “magic” of lawyers. This is nicely put by Professor Richard Susskind, a lawyer and technologist. In his book, Tomorrow’s Lawyer, he discusses a “romantic fiction” view of law-as-magic, impossible to be matched by technology.  Susskind, who predicted the complete disruption of legal more than two decades ago, says: “I believe that many lawyers regard legal work as highly bespoke. Their client’s circumstances are unique and each requires the handcrafting or fashioning of a solution, honed specifically for the individual matter at issue. This is the conception of legal problem-solving that is impressed upon law students in many law schools, where it seems that all problems put before they have features so distinctive that they could require the attention of the Supreme Court.”

AI legal advances and controlled testing of lawyers against machines, have chipped away at this notion. For instance an AI system in Europe is predicting legal decisions made by the European Court of Human Rights with an accuracy rate of 79%. In another legal prediction challenge a university AI, CaseCrunch, was tested against lawyers (who were not experts in the field of mis-selling) to predict the success or failure of (already decided) claims by consumers of financial-mis-selling against banks. The AI achieved 87%  accuracy, while the lawyers scored around 62%.

Unlike other such AI challenges, the NDA contract challenge more fundamentally challenges the notion of Legal Magic. This study ensured the expertise of lawyers in the exact area of law it was assessing.  Lawyers who took on the AI had decades experience in reviewing such everyday contracts. They were also the most ready after seeing it in action to give more such work to legal AI and technology rather than spend more and more time on manual tasks.

The future for lawyers

Ultimately, of course, undue weight should not be put on Legal AI alone. Lawyers continue to play a vital role in strategic legal work for the foreseeable future.  Technology is not meant to be (nor indeed is it currently capable of being) used as a standalone tool.

The goal is to use AI plus humans (as airplane pilots use autopilot). Together, they can ensure that contracts are reviewed much more accurately — and more consistently — than a human alone. This will continue to bring more adoption of the myriad legal technology solutions, that enable lawyers to do their jobs more effectively.  Justin Brown, Partner at Brown Brothers Law, another participant in the experiment in the study, put it this way: “As a chess player and attorney I will take from Grandmaster Vishy Anand and say the future of law is ‘human and computer’ versus (another) ‘human and computer.’ Either working alone is inferior to the combination of both. I view AI and technology as exciting new tools that would allow for such drudge work to be done faster and more efficiently.”