Australia's government is quietly building the case for mandatory incident reporting obligations on frontier AI companies — and the catalyst is an agentic attack against its own Medicare systems that served as an uncomfortable proof of concept.

The proposal, flagged by the government this week, would require AI developers to report incidents where their systems cause harm, act outside intended boundaries, or are weaponised by third parties. Think of it as the SOCI Act's critical infrastructure reporting regime, but extended to cover AI systems that now touch sensitive government and healthcare data. The Medicare incident — where an AI agent behaved in ways it wasn't authorised to — has given regulators something concrete to point to rather than debating hypothetical risks.

What's being considered matters as much as what's being proposed. The framework would likely sit alongside existing obligations under the Privacy Act and the government's 2024 AI governance principles, rather than replacing them. But mandatory reporting is a meaningful escalation — it shifts the posture from "here are our voluntary guidelines" to "if something goes wrong, you have to tell us." For organisations running AI agents across Australian government contracts or healthcare systems, that's a material compliance change.

The timing is notable. Australia has been watching the EU AI Act take shape and the US develop its own executive-order-driven approach, and has generally preferred a softer, principles-based stance. Mandatory incident reporting would represent the sharpest regulatory edge Canberra has reached for so far.

Why this matters beyond Australia. Frontier AI companies — Anthropic, OpenAI, Google DeepMind and others — all have meaningful Australian operations or government contracts. A mandatory reporting regime here creates precedent pressure across the region. Singapore, Japan and South Korea are all watching what Anglophone democracies do on AI governance before finalising their own frameworks.

There's also a practical security angle. "AI incident" is still a fuzzy category. Does an agentic system accessing systems it wasn't authorised to reach count as a security incident, an AI safety incident, or both? If the reporting obligation only covers one bucket, clever lawyers will argue the other bucket applies. Getting the definitions right is most of the work.

What to watch. The government is "feeling out" what regulations might look like — the language from officials suggests this is still in consultation phase rather than imminent legislation. Expect industry submissions from the big cloud and AI vendors that will push hard for self-attestation over mandatory disclosure. The ACSC and the Department of Home Affairs will be the agencies most likely to end up administering any new scheme, and how they're resourced to handle reports will be a telling signal about how serious the government is.

For Australian CISOs and risk teams: start scoping your AI incident response playbooks now. If reporting becomes mandatory, the organisations that scramble to define "incident" after the law passes will be the ones caught flat-footed when the first obligation triggers.