What to do when something goes wrong with AI: a step-by-step response guide

AI incidents rarely look like what you expect

When most people imagine an "AI incident," they picture a dramatic failure: a chatbot giving dangerous medical advice, a deepfake causing reputational damage, an AI system making a catastrophically wrong decision. These things happen — but they're not the incidents most Australian SMEs will actually face.

The incidents that are far more common are quieter, and often more ambiguous. A staff member who used ChatGPT to summarise a client contract and didn't realise the free tier retains inputs. A vendor who suffers a data breach and notifies you 45 days later. An AI-generated report with a plausible-sounding but factually wrong legal citation that made it into a client deliverable. An approved tool that was used for a purpose it wasn't approved for.

Responding well to these situations requires a plan — and the time to make that plan is not in the middle of an incident.

Types of AI incidents to prepare for

Before covering the response process, it helps to know what you're responding to. Common AI incident types include:

• Data entry incidents: Personal or confidential information entered into an AI tool that shouldn't have been — the most common type, often caused by staff using consumer AI tools without understanding the data handling implications.
• Output reliance incidents: Acting on AI-generated information that was incorrect, misleading, or fabricated — particularly serious in legal, financial, or health contexts.
• Vendor-side breaches: An AI vendor you use suffers a data breach that affects your data or your clients' data.
• Shadow AI incidents: Discovery that staff have been using unapproved AI tools, creating risks you weren't aware of.
• Automated decision failures: An AI system making or influencing decisions in ways that are biased, incorrect, or discriminatory.

Phase 1: Detect and contain

The first priority in any incident is to stop the damage from spreading. In AI incidents, this usually means:

• Identifying what data was affected and where it went
• Suspending access to the AI tool or system involved, if appropriate
• Preventing additional data from being entered into the system until the situation is understood
• Preserving evidence — logs, screenshots, communications — that will be needed for assessment and notification

Don't delete anything at this stage. Even if it's embarrassing, you'll need it.

Phase 2: Assess

Once you've contained the immediate issue, you need to understand what actually happened and what the legal implications are. The critical question at this stage is: does this meet the threshold for a notifiable data breach under the NDB scheme?

Under the Privacy Act 1988, a data breach is notifiable if it is likely to result in serious harm to any individual whose information was affected. This requires a three-part analysis:

1. Was personal information involved?
2. Has that information been accessed by or disclosed to an unauthorised party?
3. Is there a real risk of serious harm to affected individuals?

Not every AI incident will meet this threshold. A staff member accidentally pasting their own name and email into a prompt is unlikely to. A staff member pasting a client's full medical history into a consumer AI tool is much more likely to.

Phase 3: Notify

If your assessment concludes that you have an eligible data breach, you have two notification obligations:

• OAIC notification: Submit a statement to the OAIC as soon as practicable after forming a belief that an eligible data breach has occurred. The OAIC provides a standard notification form on their website.
• Individual notification: Notify affected individuals directly, unless doing so would be impossible or would require disproportionate effort (in which case a public notice may substitute). The notification must include what happened, what information was involved, and what steps you're taking in response.

Where a vendor-side breach is involved, coordinate with the vendor on notification timing and content. You should not assume the vendor will handle notification on your behalf — check your contract and confirm explicitly.

Phase 4: Remediate

Remediation means addressing the root cause so the same incident doesn't happen again. Depending on the nature of the incident, this might include:

• Removing the AI tool from your approved list, or restricting how it can be used
• Updating your AI Acceptable Use Policy to close the gap that led to the incident
• Providing targeted training to the staff involved — and to the broader team if the behaviour was widespread
• Reviewing vendor contracts to ensure appropriate data processing terms are in place
• Implementing technical controls to prevent the same type of data from being entered into AI tools in future

Phase 5: Post-incident review

A post-incident review isn't a blame exercise — it's a learning exercise. The goal is to understand what happened, why your existing controls didn't prevent it, and what you'll do differently. Document the review and its findings. If the OAIC investigates, a well-documented post-incident review demonstrates that you took the matter seriously and acted in good faith.

Common mistakes in AI incident response

• Waiting to see if anything bad happens. The 30-day assessment clock starts when you become aware of the incident, not when harm materialises.
• Delegating notification to the vendor. You remain the APP entity responsible for your customers' data. Don't assume the vendor will handle it.
• Failing to document. Good documentation protects you in a regulatory investigation. Bad or absent documentation makes things worse.
• Treating it as an IT problem. AI incidents are business risk events. They need to involve senior leadership, legal counsel where appropriate, and whoever manages client relationships.

What to have ready before an incident happens

The businesses that respond well to AI incidents are the ones that prepared in advance. At minimum, you should have:

• A documented AI Incident Response Plan that covers the five phases above
• A clear escalation path — who is notified internally when an incident is discovered, and in what order
• Contact details for the OAIC and your legal adviser, readily accessible
• A register of your AI tools and vendors, so you can quickly identify who to contact if a vendor-side breach occurs
• Staff training that includes what to do if they suspect an AI-related incident

None of this is complicated. But it needs to exist before something goes wrong, not after.