The Jolly Contrarian

Template:M intro technology Better call ChatGPT: Difference between revisions

Template Discussion

Template:M intro technology Better call ChatGPT (view source)

Revision as of 18:15, 9 February 2024

42 bytes added ,  9 February
no edit summary
Tags: Mobile edit Mobile web edit
No edit summary
Line 7: Line 7:


===== The buried lead: variance ''increases'' with experience =====
===== The buried lead: variance ''increases'' with experience =====
{{drop|A|n interesting finding,}} noted but not explored by the paper, was a variance measurement<ref>“Cronbach’s alpha” is a statistic that measures internal consistency and reliability, of a different items such as, in this case, the legal agreement reviews. A high “alpha” indicates consistency and general agreement between individual reviewers; a low alpha indicates variance or disagreement.</ref> across the categories of human reviewers: the ''least'' qualified, the [[LPO]]s had an “alpha” variance of 1.0, implying complete agreement among their operatives about the issues (a function, we suppose, of the mechanical obedience that LPO businesses drum into their paralegals). This ''dropped'' to 0.77 for junior lawyers and dropped ''further'', to 0.71, for senior lawyers.
{{drop|A|n interesting finding,}} noted but not explored by the paper, was a variance measurement across the categories of human reviewers: the ''least'' qualified, the [[LPO]]s had an “alpha” variance of 1.0, implying complete agreement among their operatives about the issues (a function, we suppose, of the mechanical obedience that LPO businesses drum into their paralegals). This ''dropped'' to 0.77 for junior lawyers and dropped ''further'', to 0.71, for senior lawyers.<ref>“Cronbach’s alpha” is a statistic that measures internal consistency and reliability, of a different items such as, in this case, the legal agreement reviews. A high “alpha” indicates consistency and general agreement between individual reviewers; a low alpha indicates variance or disagreement.</ref>


You read that right: experienced lawyers were ''least'' likely to agree what was important in a basic contract.  
You read that right: experienced lawyers were ''least'' likely to agree on what was important in a basic contract.  


This says one of two things: either lawyers get ''worse'' at analysing contracts across their careers— by no means out of the question, but seeming at the very least in need of explanation — or there is something not measured in these [[key performance indicator]]s that sets the veterans apart. That, maybe, linear contract analytics is the proverbial a [[machine for judging poetry]], and  isn’t all there is to it.  
This says one of two things: either lawyers get ''worse'' at analysing contracts across their careers— by no means out of the question, but seeming at the very least in need of explanation — or there is something not measured in these [[key performance indicator]]s that sets the veterans apart. That, maybe, linear contract analytics is the proverbial a [[machine for judging poetry]], and  isn’t all there is to it.  
Line 18: Line 18:
{{drop|I|n any case}}, for accuracy the LPO [[paralegal]]s did best, both in spotting issues and in locating them in the contract. (How you can ''spot'' an issue but not know where it is we are not told). Junior lawyers ranked about the ''same'' as the chatbots. Perhaps to spare their blushes the report does not say how the vets got on.
{{drop|I|n any case}}, for accuracy the LPO [[paralegal]]s did best, both in spotting issues and in locating them in the contract. (How you can ''spot'' an issue but not know where it is we are not told). Junior lawyers ranked about the ''same'' as the chatbots. Perhaps to spare their blushes the report does not say how the vets got on.


But it shouldn’t surprise anyone that all the machines were quicker that the humans of whom. LPOs were by far the slowest. There is a cost to obliging humans to behave like robots.
But it shouldn’t surprise anyone that all the machines were quicker than the humans, of whom LPOs were by far the slowest. There is a cost to obliging humans to behave like robots.


Clear implication: as we can expect [[LLM]]s to get better over time,<ref>Maybe {{Plainlink|https://www.theregister.com/2023/07/20/gpt4_chatgpt_performance/|not, actually}}, but okay.</ref> the [[meatware]]’s days are numbered.  
The clear implication: as we can expect [[LLM]]s to get better over time,<ref>Maybe {{Plainlink|https://www.theregister.com/2023/07/20/gpt4_chatgpt_performance/|not, actually}}, but okay.</ref> the [[meatware]]’s days are numbered.  


Now, if you ask an experienced lawyer to craft a set of abstract guidelines that she must hand off to low-cost, rule-following units, but for whose operation she remains accountable, expect her to draw her boundaries ''conservatively''.  
Now, if you ask an experienced lawyer to craft a set of abstract guidelines that she must hand off to low-cost, rule-following units, but for whose operation she remains accountable, expect her to draw her boundaries ''conservatively''.  


There being no “[[bright line test|bright lines]]” wherever there is scope for nuance or call for subtlety, she will stay well inside it, not trusting a dolt — whether naturally or generatively intelligent — to get it right.  
There being no “[[bright line test|bright lines]]” wherever there is scope for nuance or call for subtlety, she will stay well within the smudgy thresholds, not trusting a dolt — whether naturally or generatively intelligent — to get it right.  


This is common sense and little more than prudent  [[triage]]: well before any practical danger, her amanuenses must report to matron for further instruction. She can then send the machine back into the fray with contextualised instructions, or just handle anything properly tricky herself. This is all good best-practice outsourcing, straight from the McKinsey [[playbook]].
This is common sense and little more than prudent  [[triage]]: well before any practical danger, her amanuenses must report back to Matron for further instruction. She can then send them back into the fray with contextualised orders, or just handle the tricky stuff herself. This is all good [[Best practice|best-practice]] outsourcing, straight from the McKinsey [[playbook]].


Now, a standard-form contract without at least one howling error is unknown to legal science, so she should expect an assiduous machine reader, so instructed, to be tireless, and quickly tiresome, in rooting out formal discrepancies and bringing them to her attention.  
Now, a standard-form contract without at least one howling error is unknown to legal science, so she should expect an assiduous machine reader, so instructed, to be tireless, and quickly [[Tedium|tire''some'']], in rooting out formal discrepancies and bringing them to her attention.  
====Variance, redux: when “solution” ''is'' the problem====
====Variance, redux: when “solution” ''is'' the problem====
{{Quote|Q: What’s the difference between an [[LLM]] and a [[trainee]]? <br>
{{Quote|Q: What’s the difference between an [[LLM]] and a [[trainee]]? <br>
A: You only have to punch information into an [[LLM]] once.<ref>This is a nerd’s version of the drummer joke: ''What’s the difference between a drummer and a drum machine? You only have to punch information into a drum machine once.''</ref> }}  
A: You only have to punch information into an [[LLM]] once.<ref>This is a nerd’s version of the drummer joke: ''What’s the difference between a drummer and a drum machine? You only have to punch information into a drum machine once.''</ref> }}  
{{drop|C|ontrary to [[modernist]]}} wisdom — viz., ''thou shalt not rest until all problems are solved'' — descending the fractal tunnel of error is, sometimes, a bad idea. ''Usually'', in fact. Down it are [[snafu]]<nowiki/>s and boo-boos that an experienced lawyer will roll her eyes at, take a moment to sanctimoniously tut about, ''but then let go''.  
{{drop|C|ontrary to [[modernist]]}} wisdom — viz., ''thou shalt not rest until all problems are solved'' — descending the fractal tunnel of error is, sometimes, a bad idea. ''Usually'', in fact. Down it are [[snafu]]<nowiki/>s and boo-boos that an experienced lawyer will roll her eyes at, take a moment to sanctimoniously tut about, ''but then let go''.


Life, you see, is too short. She may even filter these out with her subconscious fast brain before she registers them at all. This is [[Signal-to-noise ratio|pure ''noise'']]: instinctive, formalistic fluff, well beyond a seasoned professional’s [[ditch tolerance]].
Life, you see, is too short. She may even filter these out with her subconscious fast brain before she registers them at all. This is [[Signal-to-noise ratio|pure ''noise'']]: instinctive, formalistic fluff, well beyond a seasoned professional’s [[ditch tolerance]].
Retrieved from "https://jollycontrarian.com/index.php?title=Special:MobileDiff/83742"