Welcome back to the NicFab Podcast dedicated to legal prompting.
Download transcript (.vtt)
0:09
0:15
0:18
0:25
0:28
0:35
0:40
0:44
0:49
0:50
0:54
0:59
1:03
1:07
1:12
1:17
1:24
1:29
1:32
1:36
1:38
1:44
1:49
1:53
1:57
2:00
2:03
2:08
2:10
2:18
2:21
2:25
2:28
2:31
2:34
2:38
2:46
2:52
2:55
2:58
3:03
3:07
3:12
3:16
3:22
3:24
3:26
3:28
3:32
3:35
3:39
3:43
3:46
3:49
3:52
3:56
4:00
4:03
4:07
4:11
4:15
4:18
4:22
4:27
4:32
4:33
4:36
4:41
4:44
4:51
4:53
4:57
5:01
5:04
5:06
5:09
5:12
5:16
5:17
5:19
5:22
5:26
5:30
5:34
5:36
5:39
5:41
5:42
5:45
5:49
5:54
5:56
5:57
6:03
6:08
6:13
6:15
6:19
6:21
6:24
6:29
6:33
6:38
6:44
6:50
6:54
6:59
7:04
7:05
I am Nicola Fabiano and this is the second episode.
Last time we talked about privacy notices, how to check them,
simplify them and adapt them to different contexts.
At the end of that episode, I introduced a topic that shifts our perspective.
We are no longer talking about how to write a prompt,
but about how the model finds the information it works with.
We are talking about RAC, Retrieval Augmented Generation.
What is RAC?
RAC is an architecture. It works like this.
Instead of relying only on what the model learned during training,
the system retrieves documents from an external knowledge base
and inserts them into the context of your request.
The model then generates its response based on those documents.
In practice, you load a collection of materials, regulatory decisions,
contracts, legal opinions, legislation, and the system indexes them.
When you ask a question, the retrieval engine selects the most relevant fragments
and passes them to the model.
The model responds based on those fragments.
The idea is powerful.
You no longer depend on the model's memory, which can be inaccurate or outdated.
You work with your own documents, your own sources.
For a legal professional, this seems like the ideal solution.
It seems like it, but there are risks.
The risks are the following.
The first risk is retrieval quality.
The system does not search the way a lawyer would.
It uses semantic similarity.
It selects fragments that linguistically resemble the question, not legally.
If you ask about legitimate interest,
the system might return a fragment that mentions it
but in a completely different context.
The result is an answer that looks well-founded
but rests on an irrelevant fragment.
The second risk is fragmentation.
Documents are split into blocks, chunks, in technical terms, for indexing.
A supervisory authority decision is not a sequence of independent blocks.
It has an argumentative structure.
The premise shapes the conclusion.
If the system extracts only the conclusion without the premise,
the model works with an incomplete piece.
And it generates answers that lose the reasoning.
The third risk is outdated sources.
If your document base contains a repealed regulation or a superseded clause,
the system does not know.
It does not check validity.
It does not compare dates.
It retrieves the fragment most similar to the question,
even if it is no longer in force.
In the regulatory field, where amendments are frequent
and the differences between versions can be decisive,
this is particularly dangerous.
The fourth risk is opacity.
When the model responds using RUG,
it is not always clear which fragments it used.
Some systems show the sources, others do not.
But even when sources are shown,
you do not know how the model combined them,
which ones it prioritized, which ones it ignored.
This lack of transparency is a real problem.
A lawyer must be able to reconstruct the reasoning
with RUG that is often not possible.
The fifth risk concerns professional secrecy.
If you load confidential materials into a RUG platform,
where does that data go?
Who processes it? Where is it stored?
The infrastructure behind RUG is not just a technical issue.
It is a professional ethics issue.
And in many cases, it is a matter of GDPR and AI Act compliance.
The safeguards.
This does not mean RUG is useless.
Far from it, it means it must be used with awareness.
First safeguard.
Always verify the sources.
Do not trust the answer.
Read the fragments the system retrieved.
Check that they are relevant, complete and current.
Second safeguard.
Check the segmentation.
How are documents being split?
Do the chunks respect the logical structure of the text,
or do they cut it arbitrarily?
Poorer segmentation produces poorer retrieval.
Third safeguard.
Choose the infrastructure carefully.
Where does the data reside?
Who has access?
Are there contractual guarantees?
RUG, more than any other use of AI in law,
requires a serious infrastructure assessment.
Fourth safeguard.
Document everything.
If you use a RUG system to prepare an opinion or analyze a contract,
record the sources retrieved, the question asked and the answer received.
This is a matter of professional responsibility.
My closing remarks.
RUG promises to solve the root cause of the problem.
In part, it does.
But it introduces new risks.
Imprecise retrieval, fragmentation, outdated sources,
opacity and exposure of confidential data.
Legal prompting applied to RUG is not just about writing the right prompt.
It is about understanding what happens before the model generates its response
and taking professional responsibility for the result.
Next time, we will talk about advanced prompting techniques,
chain of thought and few-shot applied to legal reasoning.
Subscribe to the newsletter at nickfab.eu.
Thank you for listening.
Until the next episode.