82,966
edits
Amwelladmin (talk | contribs) No edit summary |
Amwelladmin (talk | contribs) No edit summary Tags: Mobile edit Mobile web edit |
||
Line 6: | Line 6: | ||
The researchers collated and anonymised ten “real-world” procurement contracts — [[NDA]]<nowiki/>s were [[deemed]] a bit easy — and fed them to a selection of junior bugs, [[Legal process outsourcer|LPO]]s and [[Large language model|large language models]].<ref>It looks to have been those of OpenAI, Google, Anthropic, Amazon and Meta. Poor old Bing didn’t get a look in.</ref> | The researchers collated and anonymised ten “real-world” procurement contracts — [[NDA]]<nowiki/>s were [[deemed]] a bit easy — and fed them to a selection of junior bugs, [[Legal process outsourcer|LPO]]s and [[Large language model|large language models]].<ref>It looks to have been those of OpenAI, Google, Anthropic, Amazon and Meta. Poor old Bing didn’t get a look in.</ref> | ||
===== | ===== The buried lead: variance ''increases'' with experience ===== | ||
{{drop|A|n interesting finding,}} noted but not explored by the paper, was a variance measurement<ref>“Cronbach’s alpha” is a statistic that measures internal consistency and reliability, of a different items such as, in this case, the legal agreement reviews. A high “alpha” indicates consistency and general agreement; a low alpha indicates variance or disagreement.</ref> across the categories of human reviewers: the ''least'' qualified, the | {{drop|A|n interesting finding,}} noted but not explored by the paper, was a variance measurement<ref>“Cronbach’s alpha” is a statistic that measures internal consistency and reliability, of a different items such as, in this case, the legal agreement reviews. A high “alpha” indicates consistency and general agreement; a low alpha indicates variance or disagreement.</ref> across the categories of human reviewers: the ''least'' qualified, the [[LPO]]s had an “alpha” variance of 1.0, implying complete agreement among them about the issues (a function, we suppose, of slavish and obedient adherence that is beaten into LPO businesses). This ''dropped'' to 0.77 for junior lawyers and further still to 0.71 for senior lawyers. | ||
You read that right: experienced lawyers were ''least'' likely to agree what was important in a basic contract. | You read that right: experienced lawyers were ''least'' likely to agree what was important in a basic contract. | ||
This says one of two things: either | This says one of two things: either lawyers get worse at reading contracts as they get more experienced — by no means out of the question, and would explain a few things — or there is something not measured in these [[key performance indicator]]s that sets the veterans apart. That, maybe, linear contract analytics is the proverbial a [[machine for judging poetry]], and isn’t all there is to it. | ||
Hold that thought. | Hold that thought. | ||
====Results: all hail the paralegals?==== | ====Results: all hail the paralegals?==== | ||
{{drop|I|n any case}} | {{drop|I|n any case}}, for accuracy the LPO [[paralegal]]s did best, both in spotting issues and in locating them in the contract. (How you can ''spot'' an issue but not know where it is we are not told). Junior lawyers ranked about the ''same'' as the chatbots. Perhaps to spare their blushes the report does not say how the vets got on. | ||
But it shouldn’t surprise anyone that all the machines were | But it shouldn’t surprise anyone that all the machines were quicker that the humans of whom. LPOs were by far the slowest. There is a cost to obliging humans to behave like robots. | ||
Clear implication: we can expect | Clear implication: as we can expect [[LLM]]s to get better over time,<ref>Maybe {{Plainlink|https://www.theregister.com/2023/07/20/gpt4_chatgpt_performance/|not, actually}}, but okay.</ref> the [[meatware]]’s days are numbered. | ||
Now, if you ask | Now, if you ask an experienced lawyer to craft a set of abstract guidelines that she must hand off to low-cost, rule-following units, but for whose operation she remains accountable, expect her to draw her boundaries ''conservatively''. | ||
There being no “[[bright line test|bright lines]]” wherever there is scope for nuance or call for subtlety, she will stay well inside it, not trusting a dolt — whether naturally or generatively intelligent — to get it right. | |||
This is common sense and little more than prudent [[triage]]: well before any practical danger, her amanuenses must report to matron for further instruction. She can then send the machine back into the fray with contextualised instructions, or just handle anything properly tricky herself. This is all good best-practice outsourcing, straight from the McKinsey [[playbook]]. | |||
Now, a standard-form contract without at least one howling error is unknown to legal science, so she should expect an assiduous machine reader, so instructed, to be tireless, and quickly tiresome, in rooting out formal discrepancies and bringing them to her attention. | |||
Contrary to modernist wisdom, this, for the most part, is a bad thing. These will be things an experienced lawyer would roll her eyes at, or sanctimoniously tut about, ''but then let go'', in most cases without even recording that the “issue” was even there. This is formalistic fluff, a long way past a seasoned professional’s [[ditch tolerance]]. | |||
This, perhaps, accounts for that mysterious variance among experienced lawyers. Contract review, end of the day, is an art, not a science. Sometimes you take a point, sometimes you don’t. Some lawyers like the comfort of redundant boilerplate, others cannot abide it. Harbouring different scars, different institutions are fearful about different things. Does it matter that your contract has a [[counterparts]] clause? Does it matter that it ''doesn’t''? | |||
A busy-body LLM that sees everything and cannot take a view gives its master a problem: she has an officious pedant on her hands. This kind of pedantry usually rubs off as junior lawyers acquire experience. LLMs have an insatiable thirst for it. | |||
For what we are fighting here is not bad lawyering, nor bad machines nor bad intentions but ''bad process design''. Supporting it with machinery will make things worse. This is the lesson of the sorcerer’s apprentice. | |||
====The oblique purposes of formal contracts==== | |||
There is one peculiarity that | There is one peculiarity that this kind of formalistic approach cannot address, but we should mention: sometimes a contract’s true significance is tangential to its contents. Sometimes the finely thrashed-out detail is not the point.<ref>This is, broadly, true of all contracts from execution until formal enforcement. The overwhelming majority of contracts are never formally enforced.</ref> | ||
Sometimes the very ''act'' of finely thrashing out unimportant details frustrates the true purpose of the contract, which is to fulfil a sociological function. As a commitment signal or competence signal. | |||
As a mating ritual, of sorts: a performative ululation of customary cultural verities meant signal that yes, we care about the same things you do, are of the right stuff, the same mind and our ad idems are capable of consensus. | As a mating ritual, of sorts: a performative ululation of customary cultural verities meant signal that yes, we care about the same things you do, are of the right stuff, the same mind and our ad idems are capable of consensus. |