How AI hallucinations could land for AFSL holders
With 90% of industry leaders believing AI will improve service and efficiency in their firms,attention is now turning to how this technology is integrated into everyday workflows, and how its behaviour shapes outcomes.
A specific pattern emerging is AI hallucinations, where AI recognises objects or patterns that don’t exist or that humans are unable to see.
The problem with this? Inaccurate, invented responses, with no supporting evidence or insight, and errors hard to detect due to the confidence AI responds with.
AI is going to be part of every Australian financial services organisation, the right approach is choosing the version that protects your client data.
AI hallucinations in the real world
In 2023, AI hallucinations appeared in the legal case of Mata v. Avianca. An attorney was representing an injury claim for a client, using ChatGPT this produced references that did not exist in the legal research.
It also falsely claimed the references came from a reputable legal database, revealing how far AI can go in producing inaccurate outputs.
The knowledge base AI utilises continued to be undermined. From 2023-2025, cases like these kept emerging, with judges worldwide issuing hundreds of filings, 90% of them in 2025 alone.
For AFSL holders, in a Statement of Advice, this can land as AI-generated errors appearing as verified content, which risks consumers being misled by incorrect facts and insights.
Why this matters to AFSL holders
AFSL holders need to ensure that consumers aren’t unethically treated through vulnerability exploitation and potential AI bias.
The accuracy of insights and facts produced to consumers must also oblige ensuring AI information isn’t misleading, unexplained, or inaccurate.
Under directors’ duties, responsibilities must be undertaken carefully and diligently when considering reliance on information from AI and the risk that it holds.
Without upholding these obligations, an uncaught hallucination in a Statement of Advice, client email or compliance file note becomes an AFCA complaint, a breach report, and a remediation bill the licensee carries.
A confident approach to AI
For AFSL holders to ensure they can meet these obligations, Microsoft 365, specifically Microsoft Copilot, is a safe way to implement AI. Designed specifically for enterprise, Copilot operates within an organisation’s tenant.
The benefits of Microsoft 365 involve:
Data protection and strict permissions: Adheres to an organisation’s existing data permissions and information, no exposure of organisational data to train the public AI model.
Automated compliance:Finds regulatory changes and prepares for audits, allowing adherence to compliance standards.
Microsoft purview:Prevents unauthorised data sharing, monitors AI prompts, and classifies sensitive data.
Many organisations already have these capabilities within their environments, they just haven’t been optimised yet. For financial services organisations, accuracy can be enhanced through role-specific AI agents, allowing for automated data reconciliation.
Reliable and permitted outputs can also be achieved through a strong SharePoint architecture and governed data access, allowing Copilot to search through categories.
The next 3 steps
There are a series of steps AFSL holders can take to ensure continued confidence within their environments.
- Embed Copilot through existing applications and actively update databases
- Ensure data governance through implementing restrictions and Microsoft Purview
- Updating policies around AI that mandate human review
These measures ensure Copilot can function optimally in your environment while maintaining data and access governance supported by updated AI policies.
Why Danet
With AI being part of how financial services will operate in 2026 and beyond, hallucinations will progressively surface.
Those who move first with the right version of AI will benefit from the productivity without the privacy risk.
Danet works with organisations to align Microsoft 365 and Copilot to their environments to ensure confidence and compliance with the use of AI, through designing and continuing to update the structures of databases and workflows.
These themes are further explored in Danet’s AI Guide for AFSL Leaders, determining how AI capabilities can be implemented into Microsoft 365 environments and how governance oversight shapes this.