New AI Tools Are Making More Mistakes and Law Firms Face the Consequences

Newer Isn’t Better When It Comes To AI

Law firms using AI in their marketing are facing an unexpected problem: the latest tools are less reliable than the ones that came before. Despite promises of smarter systems and faster output, newer models from leading AI companies are getting more facts wrong, not fewer.

These aren’t occasional slips. Testing shows a noticeable drop in accuracy compared to earlier versions of the same tools. That means law firms producing content or handling intake through AI are at greater risk of publishing false or misleading information.

The promise of AI in legal marketing has always been about saving time without sacrificing quality. But if the technology is moving in the wrong direction, law firms can’t afford to use it without safeguards.

The Accuracy Problem in Plain Numbers

Recent tests from OpenAI have confirmed that newer ChatGPT models are more likely to get things wrong.

• OpenAI’s o3 model answered general knowledge questions incorrectly 51% of the time.
• On questions about individuals (i.e. names, backgrounds, public roles), it made mistakes 33% of the time, which is double the error rate of the previous version.
• The smaller o4-mini model fared even worse, producing wrong answers in nearly 8 out of 10 responses.

These issues aren’t isolated to OpenAI. Other companies pushing advanced tools, including Google and DeepSeek, are also seeing accuracy fall off as their systems become more complex.

For law firms, this isn’t a technical footnote. The drop in reliability directly affects anything a client might see, from blog content to chatbot conversations. If the information is wrong, the consequences aren’t theoretical. They’re public.

When AI Errors Reach Your Clients

The growing inaccuracy of new AI systems is already reaching customers and creating public problems.

Last month, Cursor, a tool used by software developers, saw a wave of complaints after its AI support assistant gave out false information. It told users they couldn’t install the product on multiple machines. That wasn’t true. The confusion triggered frustration across their user base, leading to several cancellations of the service, and the company’s CEO had to personally step in to correct it.

Mistakes like this are easy to dismiss as bugs until they start damaging trust. If a law firm’s intake chatbot gives out the wrong answer about case eligibility or misstates something about fees or process, it won’t be the AI that loses credibility. It will be the firm.

No tool is perfect. But when the errors show up in front of a potential client, they stop being technical issues and start becoming brand problems.

How AI Training Is Breaking Accuracy

The drop in AI accuracy is a result of how the latest models are being trained.

Early AI systems were built using massive amounts of internet text. But that pool has already been mostly used. To keep improving performance, companies like OpenAI have moved toward reinforcement learning—a technique that teaches AI through trial and error. It helps with tasks like solving math problems or writing code, but it doesn’t support consistent factual recall.

Another factor is how modern models think through answers. Many are designed to respond step-by-step, which might sound more thoughtful, but each step is another chance to go off track. That’s why even simple questions can return confident but wrong answers.

Researcher Laura Perez-Beltrachini stated:

“The way these systems are trained, they will start focusing on one task—and start forgetting about others.”

This proves a clear tradeoff: as AI becomes more specialized in certain tasks, it starts losing accuracy in others. And for law firms, where accuracy isn’t optional, that tradeoff matters.

How to Protect Your Firm From AI Mistakes

Factual errors in AI-generated content are not only embarrassing—they can cost trust, search visibility, and even potential clients. Firms using AI need to treat these tools the same way they would treat a junior team member: useful, but never unsupervised.

Here’s how law firms can reduce the risk:

1. Review everything before it goes public
   Any AI-generated blog post, email, chatbot reply, or intake response should go through human review, especially if it’s going to be seen by a client or prospect.

2. Build a simple fact-checking checklist
   Create a short list of items to verify in every AI-generated output. This could include names, citations, stats, or references to legal processes.

3. Use AI for structure, not factual research
   AI tools can help organize content, brainstorm headlines, or suggest formats. But they shouldn’t be relied on for stating facts, legal interpretations, or policy details.

4. Consider tools that cite their sources
   Some newer systems use a method called retrieval-augmented generation, which ties AI responses to real documents or verified databases. Even ChatGPT can cite sources that are hallucinated.

5. Create a response plan for fixing errors fast
   If something inaccurate goes live, make it easy to correct. Whether it’s a blog post, intake script, or automated email, your team should have a clear process for damage control.

These steps don’t require a full tech overhaul. They’re simple habits that help law firms keep control of their message and avoid letting flawed AI output do the talking.

What Law Firms Can Do While AI Companies Catch Up

AI developers know their tools are making more mistakes, and they’re working to improve accuracy. But the legal industry can’t afford to wait for a fix. When AI generated content misleads or delivers the wrong message, the damage is immediate and possibly permanent.

The firms that succeed in using AI are the ones with the right systems in place to catch errors before clients ever see them.

That’s where TSEG comes in. We help law firms use AI in ways that support growth without compromising trust. Our approach keeps accuracy front and center—so your firm gets the benefits of modern tools without taking on unnecessary risk.

AI isn’t going away. But bad output doesn’t have to be part of your marketing. With the right strategy, you can stay efficient and credible. Let us help you get there.