Template:M intro technology Better call ChatGPT: Difference between revisions

{{drop|O|h, just listen}} to the tiny, AI violins. Some legal technologists<ref>At the Onit Inc. “AI Center of Excellence” in Auckland, New Zealand.</ref>  have presented a “{{plainlink|https://arxiv.org/html/2401.16212v1|groundbreaking comparison between large language models and “traditional legal contract reviewers}}” — being junior lawyers and legal process outsourcers — benchmarking then against a “ground truth” set by senior lawyers.  

It did not go well for the meatware.

The researchers collated and anonymised ten “real-world” procurement contracts — [[NDA]]<nowiki/>s were [[deemed]] a bit easy — and fed them to a selection of junior bugs, [[Legal process outsourcer|LPO]]s and [[Large language model|large language models]].<ref>It looks to have been those of OpenAI, Google, Anthropic, Amazon and Meta. Poor old Bing didn’t get a look in.</ref>

{{drop|A|n interesting finding,}} noted but not explored by the paper, was a variance measurement<ref>“Cronbach’s alpha” is a statistic that measures internal consistency and reliability, of a different items such as, in this case, the legal agreement reviews. A high “alpha” indicates consistency and general agreement; a low alpha indicates variance or disagreement.</ref> across the categories of human reviewers: the ''least'' qualified, the LPOs had an “alpha” variance of 1.0, implying complete agreement among them about the issues (a function, we suppose, of slavish and obedient adherence that is beaten into LPO businesses). This ''dropped'' to 0.77 for junior lawyers and further still to 0.71 for senior lawyers.

You read that right: experienced lawyers were ''least'' likely to agree what was important in a basic contract.  

This says one of two things: either you get worse at reading contracts as you get more experienced, or there is something else, not measured in these [[key performance indicator]]<nowiki/>s, that sets the veterans apart. That, maybe, linear contract analytics isn’t all there is to it.  

Hold that thought.

{{drop|I|n any case}} — perhaps to spare their blushes — the report does not tell us how the vets did compared with the chatbots, but the LPO [[paralegal]]s came out best, both in spotting issues and locating them in the contract (how you can ''spot'' an issue but not know where it is, we are not told). Junior lawyers ranked about the ''same'' as the machines.  

But it shouldn’t surprise anyone that all the machines were a lot quicker that any of the humans. LPOs were the slowest.  

Clear implication: we can expect LLMs to get better over time,<ref>Maybe {{Plainlink|https://www.theregister.com/2023/07/20/gpt4_chatgpt_performance/|not, actually}}, but okay.</ref> so the [[meatware]]’s days are numbered.  

Now, if you ask a senior lawyer to craft a set of abstract guidelines that she must hand off to low-cost, rule-following units she will draw her boundaries ''conservatively''. Wherever there is scope for nuance or subtle judgment, she veer inside them, not trusting a dolt — whether naturally or generatively intelligent — to get it right.  

This is basic triage: well before any practical danger her amanuenses must report to matron for further instruction, whereupon our responsible lawyer can send the machine back into the fray with new instructions, or handle anything properly tricky herself. This is all good best-practice outsourcing, right from the playbook.

Now, a standard-form contract without at least one howling error is unknown to legal science, so she should expect an assiduous machine reader, so instructed, to be tireless, and quickly tiresome, in rooting out minor discrepancies.  

This, for the most part, is a bad thing. These are all things an experienced lawyer would roll her eyes at, or sanctimoniously tut about, ''but then let go'', in most cases without even recording that the “issue” was even there. This is formalistic fluff, a long way past a seasoned professional’s [[ditch tolerance]].

This, perhaps, accounts for that mysterious variance among experienced lawyers. Contract review, end of the day, is an art, not a science. Sometimes you take a point, sometimes you don’t.