Malicious LLMs: WormGPT, FraudGPT and State Abuse of AI

The malicious-LLM landscape has two related strands. First, criminal tooling — purpose-built or jailbroken “dark LLMs” sold on dark-web forums and Telegram that strip safety guardrails to generate phishing, BEC and scam content on subscription (WormGPT, FraudGPT). Second, nation-state abuse of mainstream LLMs for reconnaissance, scripting, translation and social-engineering drafting — an efficiency gain, not autonomous attack generation. Both matter, but the capability is less dramatic than the branding suggests, and much early reporting was overstated.

Criminal Dark LLMs

WormGPT was first analysed by SlashNext in July 2023, reportedly built on the open-source GPT-J model and sold behind a paywall for BEC. FraudGPT surfaced on criminal forums around mid-2023; Netenrich reported subscription pricing and user counts, but those are advertising claims and should be treated as unverified. In 2025, Cato CTRL reported the “WormGPT” brand returning as jailbreak wrappers around mainstream models such as Grok and Mixtral. AML.T0054 AML.T0052.000

Nation-State Abuse of Mainstream LLMs

In February 2024, OpenAI and Microsoft jointly disclosed terminating accounts of five state-affiliated actors: Forest Blizzard (Russia), Emerald Sleet (North Korea), Crimson Sandstorm (Iran), and Charcoal Typhoon and Salmon Typhoon (China). The reported uses were reconnaissance, scripting help, vulnerability research, translation and social-engineering drafting — and OpenAI assessed its models provided only “limited, incremental” uplift. Later OpenAI threat reporting (2025) described continued abuse focused on linguistic polish rather than novel malware; we flag those specifics as lower confidence pending the primary reports. AML.T0051

Defensive Takeaways

• Assume cheap, fluent, scalable lure and scam generation; reinforce BEC and phishing controls rather than rely on content-quality tells.
• Monitor dark-web and Telegram threat intelligence for criminal-AI tooling targeting your brand or sector.
• Harden your own LLM deployments against prompt injection and jailbreak (input/output filtering, least privilege on tool-calling, ATLAS-aligned testing).
• Track vendor platform-abuse disclosures (OpenAI, Microsoft, Google) to see which state actors are experimenting, and keep endpoint and identity controls strong — AI does not yet confer major malware uplift.