Template:M intro technology Better call ChatGPT

“The era of LLM dominance in legal contract review is upon us”.

Oh, just listen to the tiny, AI violins.

Some legal technologists have presented a “groundbreaking comparison between large language models and “traditional legal contract reviewers” — being junior lawyers and legal process outsourcers — benchmarking then against a “ground truth” set by senior lawyers.

It did not go well for the meatware.

The researchers collated and anonymised ten “real-world” procurement contracts — NDAs were seen to be a bit easy — and fed them to a selection of junior bugs, LPOs and large language models.

An interesting finding, noted but not deeply explored by the paper, was a variance measurement across the categories of human reviewers: the least qualified, the LPOs had an “alpha” of 1, implying complete agreement about the issues (a function, we suppose, of slavish and obedient adherence to the rules). This dropped to 0.77 for junior lawyers and further still to 0.71 for senior lawyers.

You read that right: senior lawyers were less consistent about what was important in a basic contract than anyone else.

This says one of two things: either you get worse at reading contracts as you get more experienced, or there is something else, not measured in these key performance indicators, that sets apart an experienced lawyer from, well, a machine. That reading basic contracts isn’t all there is to it. Hold that thought.

In any case, perhaps to spare their blushes, the report does not tell us how senior lawyers did compared with the machines, but the good news — for paralegals— is that they came out better than any of the chatbots, both when spotting issues and when locating them in the contract (how you can spot an issue but not know where it is, we are not told) whereas junior lawyers were about the same as the machine.

It won’t be a surprise to hear that the machines were a lot quicker. Clear implication: we can expect LLMs to get better over time, so the meatware’s days are numbered.

There is something to admire in the method: deferring to the already-credentialised acknowledges the power structure in which we dance the legal tarantella. This is the paradigm: it throws out not only what is a good answer, but what counts as a good question.

Senior lawyers make the rules.

Now, if you ask a senior lawyer to craft a set of abstract guidelines that she must hand off to low-cost, rule-following units bear in mind where she will draw her boundaries: conservatively. Wherever there be scope for nuance or subtle judgment, she will not trust a dolt — whether naturally or generatively intelligent — to get it right.

Well inside the practical point of danger she will instruct her amanuenses to return home for instruction, preferring to handle exceptions that commit anything like an edge-case to a sorcerer’s apprentice.

Now, a standard-form contract without at least one howling error is unknown to legal science, so you should expect an assiduous machine reader, so instructed, to be tireless, and quickly tiresome, in rooting out minor discrepancies and exceptions. This, for the most part, is a bad thing. These are all things an experienced lawyer would roll her eyes at, or sanctimoniously tut about, but then let go, without recording for posterity that the issue was even there.

These judgment calls, we submit, account for the increasing variance among experienced lawyers. Contract review, end of the day, is an art, not a science.

A busy-body LLM that gets everything right, which cannot take a view, gives her masters a problem: they have an officious pedant on their hands. This kind of pedantry wears out of junior lawyers. LLMs have an insatiable thirst for it.

For what we are fighting against here is neither bad lawyering, nor bad machines nor bad intentions but bad design and bad process. Insatiable busybodies do not make things better. A bad process that you make faster, more accurate and more consistent remains a bad process. This is the lesson of the sorcerer’s apprentice.

We are going to make some assumptions here. The contacts in question are suitable for outsourcing to an LPO which means they are relatively high in volume, relatively low in risk and value, relatively (but not entirely) simple in structure and their rules for processing them are relatively easy to state. This is stuff you can hard off with clear instructions to a paralegal and not much can do wrong.

There is one archetypal contract that meets all these criteria: an NDA. Some complexity, some regulatory sensitivity (privacy and data protection), but the rules of the road are widely understood. It is hard to get into an intractable conceptual argument in an NDA, but passably diverting to try.

There is one peculiarity that a technologist’s formalistic approach cannot address, but we should mention: sometimes a contract’s true significance is tangential to its contents. Sometimes the finely thrashed-out detail doesn’t really matter. Sometimes the act of finely thrashing out unimportant details frustrates the true purpose of the contract, which is to fulfil a sociological function. As a commitment or competence signal.

As a mating ritual, of sorts: a performative ululation of customary cultural verities meant signal that yes, we care about the same things you do, are of the right stuff, the same mind and our ad idems are capable of consensus.

If it is that — most NDAs are that — then descending into the subterranean world of pedantry and exactitude that an LLM offers, in the service of “picking up things that even a trained paralegal might not” can be even counterproductive. The point is to carry out the ritual; accord these pleasantries the required respect but to not labour them.

Now. That aside: with high-volume, low-risk legal processes — especially where they do not play a part in the courting rites — the name of the game is not fast, efficient and precise negotiation, but no negotiation. Negotiation is the problem. If you find customers regularly negotiate your standard terms of business, or you get regular snarl-ups on procurement processes and end-user sale contracts you have bad contracts. Fix them.

This might be a matter of formal redesign, or persuading legal to come to Jesus on the preposterous width of the exclusion of liability and indemnity — but the answer is not to excellently negotiate individual contracts. This leaves you with two enduring problems: first, your portfolio of homogenous customer sale contracts are not homogenous; secondly, you have now overlaid administrative machinery upon a bad process — that generates non-standard contracts — that will be hard to remove. By appointing unskilled bureaucrats and technocrats to oversee and manage that process, and likely other unskilled bureaucrats to oversee and monitor them , you have institutionalised bad process.

John Gall’s Systemantics captures this well. Temporary fixes have a habit of becoming permanent. Bureaucrats are butterfly collectors: they do not give up responsibilities without a fight. Before long, this process will have itself sedimented into the administrative sludge that weighs your organisation down.

LLMs can’t function by themselves (yet: we are not quite at the point of skynet, however much techno-utopians might hanker for it). They imply not saved legal cost, but “waste” transferred: it will be diffused among software-as-a-service providers, the firm’s procurement complex, internal audit, operations and, yes, legal who will still have to handle exceptions, manage and troubleshoot the system, vouch for it, periodically certify its legal adequacy and present it to the opco

LLMs as finite. They necessarily mimic what has gone before. While yes alpha go might engineer a novel strategy in a zero sum game, it is not so easy on in the non linear infinitude of life. An LLM that purports to improve on its training material will be distrusted it doesn’t understand, so what good had it got of reimagining?

The perfect LLM serves up an archetypal sample of what you already have.