LLMs have transformed the way we work, and given us new and creative ways to get things wrong. Barely a day goes by without a fresh hallucination story making headlines.

The worst involve lawsuits, hefty fines and red faces. Teams who use LLMs in a professional context know the risks and decide accordingly where to automate, but when deadlines are tight, it’s tempting to rely on their convenient and apparently convincing answers. Is it safe to do so for compliance teams?

Apparently convincing is the key. To simplify the impressively complex, LLMs are designed to generate text by predicting the most likely next word based on patterns in their training data and when you ask for an answer, they really want to find one for you. Prompts can push models beyond their trained knowledge and a eagerness to “do a good job” can result in a confident answer that sounds plausible but is inaccurate or invented. The worst offence is when the model invents sources (last year the High Court urged lawyers to stop misusing AI because models were inventing case law citations, but the problem persists).

Whilst hallucinations and unreliable sources might be making the news, they’re not the only reasons why you should be cautious when using general-purpose LLMs for due diligence.

Explainability and the law

Guidance for regulated industries is clear. The US Department of Justice requires compliance teams to understand how their AI tools work and to evidence how they assess and control the risks those tools present, and its ECCP (Evaluation of Corporate Compliance Programs, updated September 2024) stresses controls to monitor AI’s trustworthiness and reliability with clear human oversight. This is a problem for organisations experimenting with LLMs: most research agents operate as black boxes, with opaque logic and outputs that often lack citations or verifiable sources (or even show fabricated sources) making it difficult to evidence how a conclusion was reached. Whilst a purpose-built AI platform can meet this standard through clear and transparent processes and sentence-level sourcing, a general-purpose LLM-generated report is unlikely to do so with the same level of precision.

Hidden errors

The section above focused on regulated industries where the danger is being unable to evidence your process when a regulator asks. But plenty of organisations doing due diligence aren’t part of that world, including foundations, universities, charities, smaller companies and investment teams checking donors, grantees, partners and suppliers. Many are experimenting with general-purpose AI, and for them the risk is quieter – mistakes that never surface in a formal review at all.

The most dangerous errors an LLM makes in due diligence are the ones that look like good answers. Individuals or companies with similar names can be confused or folded into a single profile, creating false-positive red flags or missing genuine risks, especially across languages and differing cultural naming conventions. Most AI chatbots aren’t built for the sophisticated level of disambiguation this requires. Technology that can effectively disambiguate with the rigour of a compliance professional is remarkably complex to engineer – as Xapien has learnt after years of ironing out our own creases.

What about sensitive data?

General-purpose tools give you little visibility over where data is stored, how long it’s kept, who can access it, or whether it feeds the training of future models. Data-protection duties apply whatever your sector and the people you’re researching would reasonably expect their information to be handled appropriately. That means having a lawful basis for processing it, knowing which jurisdiction it’s held in and whether it’s crossing borders. With a general-purpose LLM, you often can’t answer any of those questions with confidence.

Building and maintaining a system in-house

General-purpose LLMs don’t integrate natively into a compliance workflow, though Claude and others are being used in increasingly inventive and exciting ways, so it’s highly possible to create your own workflow. Depending on the nature of your business it might not even take much longer than an afternoon.

Getting set up is one thing, but maintaining a process is quite another. Most companies don’t want to have to set up and run a miniature corporate investigations department in-house – not whilst there are clients to onboard and a business to run. Building a workflow that integrates an LLM – one that stays accurate, sourced and defensible as the models, data and rules around it change – is a much more complex task and one that never really finishes. LLM providers are not obliged to inform you about updates or fundamental changes to how their models work which would impact the reliability of your workflow (though you might find out about gremlins in their systems in the news).

Any better than a Google search?

Underneath all of this is a question of process, and of how much confidence you can place in it. For all their fluency, when it comes to compliance, LLMs don’t always find as much as a careful Google search would.

Searching properly was always a process: a set of well-refined queries, run methodically, with every promising result opened, read and weighed. An LLM uses some of the same techniques, but you still have to build the process around it, decide what a good report looks like, and test it thoroughly against a test set of old cases before you trust it with a live one. The catch is that you cannot control an LLM the way you would control a process of your own. It updates silently, and the model underneath can be retrained or replaced without word ever reaching you. If you have wired up integrations so it can reach restricted data, such as registries, screening sources and watchlists, you now need to maintain all of that too.

Doesn’t Xapien use LLMs?

We absolutely do, not only for our own team’s productivity but within our product. The difference is that we have experienced engineers whose whole job is to evaluate and harness the best technology available and to rigorously test it for every task at hand.

There are approximately 40–50 algorithms behind Xapien and many of them use LLM technology. Each one performs a single, specific task with restricted inputs and outputs, which allows us to test, evaluate and iterate. They are not used to generate answers in the same way as a general-purpose chat model and are therefore not vulnerable to hallucination in the same way.

In other words, Xapien is not an LLM in a due diligence “wrapper” that produces reports, but a sophisticated platform built for compliance-grade, fully-sourced, and consistent outputs. It augments the human research process by finding, distilling and presenting information accurately, repeatably and quickly.

Within Xapien, each model is only one of the tools in the toolbox for our engineers. Each individual assertion within a Xapien report is sourced, with links to the original article, so you can interrogate the information further. We’d love to tell you more about how this works and how it could transform your due diligence process on a live demo.

I think I’ll still use Claude

Sometimes risks pay off. We took a big one creating an automated due diligence platform before the world was talking about AI. We’re just not sure your compliance program is the place to take them.

For more information, check out A buyer’s guide to AI in due diligence.