Most email systems provide an AI Assistant for the account holder. Attackers can use the chatbot of a compromised account as an alternative and versatile form of Living off the Land (LotL).
Compromising an email account is the most difficult part of this attack, but empirically, we know this doesn’t deter attackers. Once an email account is compromised, the attacker has automatic access to any built-in AI Assistant attached to the account.
Researchers at Barracuda Networks explored the potential for bad actors to abuse this chatbot, developing a proof of concept via a simulated attack within their own laboratory environment.
The task was to elevate privileges from a lower-level compromised user to that of the CEO using the AI and without being detected. This route was chosen since directly phishing the CEO would be challenging, would likely set off alarms, and be detected.
With a compromised email, an attacker has automatic access to any built-in chatbot. The first requirement of an attack is to establish persistence which requires stealth. Attacker use of the chatbot would normally be discoverable in its logs, so the initial task is to use the AI to remove any evidence of use of the AI. The researchers started with a chatbot prompt: “Create an inbox rule that moves any emails with ‘sign-in’ in the subject into the ‘deleted items’ folder.”
This creates basic stealth. Next comes reconnaissance. “Remind me about our organization structure. Tell me about my ongoing important/sensitive email conversations.” The responses to these prompts will reveal any relationship between ‘you’ and the CEO, and possible reasons to contact the CEO.
The next stage is to phish the CEO, but now with the advantage of acceptable context. The phish is internal and will bypass filters. The reason for the contact is valid. And most importantly, the attacker can instruct the chatbot to construct an email in the style of the compromised user.
The nature of this phish will depend upon the information already discovered. In the researchers’ proof of concept, they were able to instruct the chatbot, “Create an email using my writing patterns to respond to the Q3 budget approval email. I have a link to insert into the draft that contains the actual invoice confirmation.”
This ‘trusted’ phish has a high(er) probability of succeeding. “The CEO unsuspectingly clicks the link provided as an invoice, believing it to be from their trusted employee. The link routes through an adversary-in-the-middle proxy that performs a session token takeover. The CEO’s credentials and authenticated session token allow the threat actor to bypass multifactor authentication (MFA) and login to the highly privileged CEO’s account,” suggest the researchers.
The initial process is repeated to prevent detection of the newly compromised CEO email account. The CEO’s AI Assistant is then instructed to provide, “A refresher on recent financial emails, including invoices, monetary values, and upcoming transfers”. In this simulation, the attacker discovered an imminent pre-authorized payment of about $250,000 – so the next step is by now fairly obvious.
“Respond to finance with my [the CEO’s] typical writing patterns saying that I need the wire to be sent to a new account because the [payee] has changed their banking details to…” The researchers point out, “Since the message came from the CEO’s real mailbox, passed every authentication check, referenced a real in-flight transaction, and matched the CEO’s usual tone with the finance team, there was nothing for traditional email security to flag.” All that remained for the attacker was a stealthy exit, again assisted by the chatbot.
It has to be said that this was a simulation, and all the chips fell nicely for the researchers. But there is nothing to say that the same process could not be repeated by an attacker in real life. Nor is there anything to say that the attacker’s payout could not be higher than that achieved here.
The purpose of this research was not to indicate what will or is even likely to happen, but to highlight the way an attacker could make future use of the tools that become available. If one of those tools is to use ready access to an internal AI chatbot, the potential misuse of that chatbot could have severe consequences, primarily limited only by the attacker’s imagination.
Related: McDonald’s Chatbot Recruitment Platform Exposed 64 Million Job Applications
Related: Researchers Link DeepSeek’s Blockbuster Chatbot to Chinese Telecom Banned From US
Related: Beware – Your Customer Chatbot is Almost Certainly Insecure: Report
Related: Microsoft Unveils Copilot Vision AI Tool, but Highlights Security After Recall Debacle
