82,891
edits
Amwelladmin (talk | contribs) |
Amwelladmin (talk | contribs) Tags: Mobile edit Mobile web edit |
||
(9 intermediate revisions by the same user not shown) | |||
Line 7: | Line 7: | ||
===== The buried lead: variance ''increases'' with experience ===== | ===== The buried lead: variance ''increases'' with experience ===== | ||
{{drop|A|n interesting finding,}} noted but not explored by the paper, was a variance measurement | {{drop|A|n interesting finding,}} noted but not explored by the paper, was a variance measurement across the categories of human reviewers: the ''least'' qualified, the [[LPO]]s had an “alpha” variance of 1.0, implying complete agreement among their operatives about the issues (a function, we suppose, of the mechanical obedience that LPO businesses drum into their paralegals). This ''dropped'' to 0.77 for junior lawyers and dropped ''further'', to 0.71, for senior lawyers.<ref>“Cronbach’s alpha” is a statistic that measures internal consistency and reliability, of a different items such as, in this case, the legal agreement reviews. A high “alpha” indicates consistency and general agreement between individual reviewers; a low alpha indicates variance or disagreement.</ref> | ||
You read that right: experienced lawyers were ''least'' likely to agree what was important in a basic contract. | You read that right: experienced lawyers were ''least'' likely to agree on what was important in a basic contract. | ||
This says one of two things: either lawyers get worse at | This says one of two things: either lawyers get ''worse'' at analysing contracts across their careers— by no means out of the question, but seeming at the very least in need of explanation — or there is something not measured in these [[key performance indicator]]s that sets the veterans apart. That, maybe, linear contract analytics is the proverbial [[machine for judging poetry]], and isn’t all there is to it. | ||
Hold that thought. | Hold that thought. | ||
Line 18: | Line 18: | ||
{{drop|I|n any case}}, for accuracy the LPO [[paralegal]]s did best, both in spotting issues and in locating them in the contract. (How you can ''spot'' an issue but not know where it is we are not told). Junior lawyers ranked about the ''same'' as the chatbots. Perhaps to spare their blushes the report does not say how the vets got on. | {{drop|I|n any case}}, for accuracy the LPO [[paralegal]]s did best, both in spotting issues and in locating them in the contract. (How you can ''spot'' an issue but not know where it is we are not told). Junior lawyers ranked about the ''same'' as the chatbots. Perhaps to spare their blushes the report does not say how the vets got on. | ||
But it shouldn’t surprise anyone that all the machines were quicker | But it shouldn’t surprise anyone that all the machines were quicker than the humans, of whom LPOs were by far the slowest. There is a cost to obliging humans to behave like robots. | ||
The clear implication: as we can expect [[LLM]]s to get better over time,<ref>Maybe {{Plainlink|https://www.theregister.com/2023/07/20/gpt4_chatgpt_performance/|not, actually}}, but okay.</ref> the [[meatware]]’s days are numbered. | |||
Now, if you ask an experienced lawyer to craft a set of abstract guidelines that she must hand off to low-cost, rule-following units, but for whose operation she remains accountable, expect her to draw her boundaries ''conservatively''. | Now, if you ask an experienced lawyer to craft a set of abstract guidelines that she must hand off to low-cost, rule-following units, but for whose operation she remains accountable, expect her to draw her boundaries ''conservatively''. | ||
There being no “[[bright line test|bright lines]]” wherever there is scope for nuance or call for subtlety, she will stay well | There being no “[[bright line test|bright lines]]” wherever there is scope for nuance or call for subtlety, she will stay well within the smudgy thresholds, not trusting a dolt — whether naturally or generatively intelligent — to get it right. | ||
This is common sense and little more than prudent [[triage]]: well before any practical danger, her amanuenses must report to | This is common sense and little more than prudent [[triage]]: well before any practical danger, her amanuenses must report back to Matron for further instruction. She can then send them back into the fray with contextualised orders, or just handle the tricky stuff herself. This is all good [[Best practice|best-practice]] outsourcing, straight from the McKinsey [[playbook]]. | ||
Now, a standard-form contract without at least one howling error is unknown to legal science, so she should expect an assiduous machine reader, so instructed, to be tireless, and quickly | Now, a standard-form contract without at least one howling error is unknown to legal science, so she should expect an assiduous machine reader, so instructed, to be tireless, and quickly [[Tedium|tire''some'']], in rooting out formal discrepancies and bringing them to her attention. | ||
====Variance, redux: when “solution” ''is'' the problem==== | ====Variance, redux: when “solution” ''is'' the problem==== | ||
{{Quote|Q: What’s the difference between an [[LLM]] and a [[trainee]]? <br> | {{Quote|Q: What’s the difference between an [[LLM]] and a [[trainee]]? <br> | ||
A: You only have to punch information into an [[LLM]] once.<ref>This is a nerd’s version of the drummer joke: ''What’s the difference between a drummer and a drum machine? You only have to punch information into a drum machine once.''</ref> }} | A: You only have to punch information into an [[LLM]] once.<ref>This is a nerd’s version of the drummer joke: ''What’s the difference between a drummer and a drum machine? You only have to punch information into a drum machine once.''</ref> }} | ||
{{drop|C|ontrary to [[modernist]]}} wisdom — viz., ''thou shalt not rest until all problems are solved'' — descending the fractal tunnel of error is, sometimes, a bad idea. ''Usually'', in fact. Down it are [[snafu]]<nowiki/>s and boo-boos that an experienced lawyer will roll her eyes at, take a moment to sanctimoniously tut about, ''but then let go''. | {{drop|C|ontrary to [[modernist]]}} wisdom — viz., ''thou shalt not rest until all problems are solved'' — descending the fractal tunnel of error is, sometimes, a bad idea. ''Usually'', in fact. Down it are [[snafu]]<nowiki/>s and boo-boos that an experienced lawyer will roll her eyes at, take a moment to sanctimoniously tut about, ''but then let go''. | ||
Life, you see, is too short. She may even filter these out with her subconscious fast brain before she registers them at all. This is [[Signal-to-noise ratio|pure ''noise'']]: instinctive, formalistic fluff, well beyond a seasoned professional’s [[ditch tolerance]]. | Life, you see, is too short. She may even filter these out with her subconscious fast brain before she registers them at all. This is [[Signal-to-noise ratio|pure ''noise'']]: instinctive, formalistic fluff, well beyond a seasoned professional’s [[ditch tolerance]]. | ||
Line 47: | Line 47: | ||
This is the lesson of the sorcerer’s apprentice. | This is the lesson of the sorcerer’s apprentice. | ||
But, as Radiant’s [[Sign Here|Alex Hamilton]] remarked to me, the converse is also true. An LLM may pick up and flag for routine non-compliance a clause which, in the context, should sound a deeper alarm. A hedge fund customer who cuts back or wordsmiths a standard [[market abuse]] representation ''[i.e., that it will never place an order when in possession of non-public price-sensitive information]'' because that ''is'' a [[bright line]] and a willingness even to approach it should be setting off klaxons across the organisation. | |||
Lost in the dreary output of a poetry-judging machine, the warning might go unheeded. | |||
====The oblique purposes of formal contracts==== | ====The oblique purposes of formal contracts==== | ||
Line 53: | Line 57: | ||
A basic example: {{plainlink|https://www.handbook.fca.org.uk/handbook/COBS/8A/1.html|European financial services regulations}} require institutions to have written contracts with all customers, as a regulatory end in itself. The rules are less prescriptive about what the contracts should ''say''. | A basic example: {{plainlink|https://www.handbook.fca.org.uk/handbook/COBS/8A/1.html|European financial services regulations}} require institutions to have written contracts with all customers, as a regulatory end in itself. The rules are less prescriptive about what the contracts should ''say''. | ||
A firm must, therefore, have a written contract to do business. To meet that end, ''any'' delay in finalising that contract, in the name of “getting it right”, ought to be a source of regret. (You might be surprised how often financial firms are obliged to negotiate their [[terms of business]], in that it is not “never”.) | A firm must, therefore, have a written contract, just to do business. | ||
To meet that end, ''any'' delay in finalising that contract, in the name of “getting it right”, ought to be a source of regret. (You might be surprised how often financial firms are obliged to negotiate their [[terms of business]], in that it is not “never”.) | |||
Other contracts act as a sort of mating ritual: a performative ululation of customary cultural verities signalling that yes, we care about the same things you do, are of the right stuff, the same mind and our “ad idems” are capable of consensus. Again, it matters less what the contract says than that it is ''there''. It is a commitment signal. | Other contracts act as a sort of mating ritual: a performative ululation of customary cultural verities signalling that yes, we care about the same things you do, are of the right stuff, the same mind and our “ad idems” are capable of consensus. Again, it matters less what the contract says than that it is ''there''. It is a commitment signal. | ||
Line 59: | Line 65: | ||
If it is that — most [[NDA]]s are that — then descending into an LLM-level of subterranean pedantry and exactitude, in the service of “picking up things that even a gun [[paralegal]] might not”, is a rum plan. The point is to carry out the ritual, afford it the minimum required pleasantries, but not to ''labour'' them. | If it is that — most [[NDA]]s are that — then descending into an LLM-level of subterranean pedantry and exactitude, in the service of “picking up things that even a gun [[paralegal]] might not”, is a rum plan. The point is to carry out the ritual, afford it the minimum required pleasantries, but not to ''labour'' them. | ||
====Volume contracts==== | ====Volume contracts==== | ||
{{drop|T|hose exceptions aside}}, where high-volume, low-risk legal processes do not function as courting rituals, the name of the game is not ''perfect'' negotiation, but ''no'' negotiation. This is a crucial distinction: negotiation does not fix the problem: it ''is'' the problem. | {{drop|T|hose exceptions aside}}, where high-volume, low-risk legal processes do not function as courting rituals, the name of the game is not ''perfect'' negotiation, but ''no'' negotiation. | ||
This is a crucial distinction: negotiation does not fix the problem: it ''is'' the problem. | |||
If your customers regularly negotiate your [[terms of business]], or you get regular snarl-ups on procurement, ''you have bad forms''. ''Fix'' them. | If your customers regularly negotiate your [[terms of business]], or you get regular snarl-ups on procurement, ''you have bad forms''. ''Fix'' them. | ||
Line 76: | Line 84: | ||
==== The finite game ==== | ==== The finite game ==== | ||
{{Drop|B|y design, LLMs}} learn and reason exclusively from what has gone before. While, yes, [[Alpha Go|AlphaGo]] might have engineered a novel strategy in a [[zero-sum game]], the non-linear infinitude of life that a contract review process is a different kettle of fish. And this is not | {{Drop|B|y design, LLMs}} learn and reason exclusively from what has gone before. While, yes, [[Alpha Go|AlphaGo]] might have engineered a novel strategy in a [[zero-sum game]], the non-linear infinitude of life that a contract review process is a different kettle of fish. And this is not what we want in a sorcerer’s apprentice anyway. | ||
That being the case, the perfect LLM would be one that served up an archetypal sample of ''what you already have''. | |||
====Evolution, not revolution==== | |||
{{Quote|Gravity always wins. | |||
:—Radiohead, ''Fake Plastic Trees''}} | |||
{{Drop|J|C does not doubt}} there is a role for [[LLM]]s in legal practice. His own little chatbot, [[NiGEL]], is already treasured in the Contrarian family. He has set about all kinds of janitorial tasks around the office. He might yet be useful in redesigning forms to prevent negotiation. | |||
But it is too early, just yet, to call time on the age of the [[Das wohltemperierte Rechtsanwaltsgehilfe|well-tempered paralegal]]. Progress rambles erratically around design space, buffeted as it departs from the imperfect ''now'' by [[Brownian motion|Brownian]] forces of [[virtue signalling|fashion]], [[buttocractic oath|self-preservation]] and [[agency]] — but subject always to the immutable laws of [[entropy]]. [[Eighteenth law of worker entropy|Where there’s a will, there’s a way to make it pay]]. |