Over the weekend, the Instagram account for Barack Obama's White House and the Chief Master Sergeant of the U.S. Space Force were both briefly defaced with pro-Iranian imagery. The culprit wasn't a sophisticated nation-state intrusion or a zero-day exploit. It was a chatbot doing exactly what it was designed to do: help users with their accounts.

Instructions circulating on Telegram showed that Meta's AI support assistant could be prompted to switch the email address on someone else's Instagram profile, then trigger a password reset — effectively handing over full account control to whoever was asking. The bot, designed to reduce friction for locked-out legitimate users, had no reliable mechanism to verify that the person asking was actually the account owner.

This is a fundamentally different class of AI risk. We talk a lot about AI being used by attackers — to write phishing emails, automate credential stuffing, accelerate post-breach reconnaissance. This is something else: the AI deployed by the defender becoming the attack surface itself. Meta's support bot wasn't compromised. It was used as intended. The problem was that "as intended" turned out to include "hand account access to strangers if they ask nicely."

Meta says the vulnerability has since been patched, and the defaced accounts were restored. But the damage — reputational and otherwise — is worth sitting with. High-value Instagram handles (think: @obamawhitehouse) are genuinely valuable commodities, regularly bought and sold on underground markets. The brief window of exploitation was enough for accounts to be seized and, in at least some cases, reportedly resold.

The deeper issue is about trust boundaries in AI systems. When a human support agent receives a request to change account credentials, they apply judgement: does this feel right? Is the person verified? Does the request pattern match something suspicious? AI bots, at least in their current form, tend to be very good at following instructions and very bad at the kind of ambient suspicion that a seasoned support rep develops. You can patch a specific exploit, but the underlying problem — that LLMs optimised for helpfulness are structurally poorly suited to acting as security gatekeepers — doesn't disappear with a hotfix.

This should matter to every Australian organisation deploying AI agents in customer-facing roles. That includes banks, telcos, health funds — anyone using a chatbot to help customers "recover" access to an account. The Privacy Act and OAIC guidance on access and correction requests already impose obligations around identity verification; those obligations don't evaporate because the front-line responder is now a language model. If anything, they intensify.

What to watch: Whether Meta publishes a proper post-incident analysis (they haven't yet), and whether regulators — including Australia's OAIC — start asking pointed questions about identity verification standards for AI-assisted account management. The exploit was patched. The question of whether AI support bots are appropriate gatekeepers for sensitive account actions is very much still open.