Legal Prompting - Chain-of-thought e few-shot prompting nel legal

Welcome to Legal Prompting, the podcast dedicated to legal methods in the age of artificial intelligence.

I'm Nicola Fabiano.

In the previous episode, we talked about RUG and its risks in the legal domain.

Today, we take a step forward.

We discuss two techniques that change how we interact with models,

chain of thought and few-shot prompting.

Let's start with the premise.

A model does not reason like a lawyer.

It produces plausible answers, not reasoned conclusions.

But we can steer its process with precise techniques.

Let's look at them one at a time.

Chain of thought is the explicit request to lay out the logical steps before the conclusion.

Instead of simply asking, is this clause valid, we ask,

examine the clause step by step,

identify the applicable provision,

verify the conditions,

assess the exceptions,

only then formulate the conclusion.

Let's take a concrete example.

We have a clause on data transfers outside the EU.

Instead of a blunt question, we structure the reasoning as follows.

First, identify the legal basis for the transfer under Chapter 5 of the GDPR.

Second, verify the safeguards adopted.

Third, consider the SHREMS 2 case law.

Fourth, conclude on compatibility.

The result changes right away.

The model makes the path explicit.

And this allows us to verify where the reasoning holds and where it slips.

It's the difference between a statement and an argument.

A warning, though.

Chain of thought does not guarantee that the reasoning is correct.

It only guarantees that it is made explicit.

The model can build coherent logical steps on flawed premises.

Our verification remains indispensable.

Let's move to few-shot prompting.

Here, we give the model examples of how we want it to answer

before posing the actual question.

Two, three, at most five well-chosen examples.

In the legal context, it works like this.

Do we want the model to analyze a decision of the supervisory authority

in accordance with a precise structure?

We show two or three analyses already done with that structure.

Then we submit the new decision.

The model tends to replicate the format.

The quality of the examples is everything.

Generic examples produce generic results.

Precise examples with the correct technical language

and the argumentative structure we need

produce much better aligned outputs.

It's instruction by demonstration, not by explanation.

The two techniques can be combined.

We can provide examples of chain of thought reasoning

already structured in a few-shot mode.

The model learns both the format and the method.

For analyzing complex decisions, it's often the most effective combination.

Let's move to the limits.

The first is the length of the context.

Every example takes space.

In long documents, we have to balance the number of examples

and the complexity of the text we analyze.

The second limit concerns bias.

If all our examples follow a certain interpretation,

the model will apply it even where it's not appropriate.

Examples shape reasoning, not just form.

Let's choose representative examples, not convenient ones.

The third limit is the most insidious.

Unexplicit reasoning looks more reliable,

but plausibility is not legal correctness.

A well-built argument on a non-existent rule

remains a hallucination, only more convincing.

A practical caution.

When we use these techniques for real legal work,

we document everything, the prompt, the examples,

the output, and our verification.

It is part of the governance of AI use,

and it will be increasingly relevant with the AI act.

One last thought.

Chain of thought and few-shot are not tricks.

They are the way we translate our legal method

into instructions understandable to the model.

The clearer our method, the better the techniques work.

In the next episode, we will apply these techniques

to the analysis of contracts and clauses.

We will see how to build effective checklists,

how to compare versions,

and which limits remain beyond the model's reach.

Thanks for listening.

If you find this podcast useful,

share it with colleagues interested in legal methods

in the age of AI.

We'll meet again in the next episode of Legal Prompting.