By EUvsDisinfo
Ben Nimmo is the Principal Investigator on the Intelligence and Investigations team at OpenAI. He is a leading expert on online influence operations, disinformation, and information warfare, with more than a decade of experience investigating state-linked campaigns by Russia, China, Iran, and other actors. Before joining OpenAI, he co-founded the Digital Forensic Research Lab (DFRLab) at the Atlantic Council and worked as Director of Investigations at Graphika.
List of Abbreviations
- AI – Artificial Intelligence
- LLM – Large Language Model
- GAN – Generative Adversarial Network
- FIMI – Foreign Information Manipulation and Interference
- OSINT – Open-Source Intelligence
- SEO – Search Engine Optimization
- DFRLab – Digital Forensic Research Lab
Foreign Information Manipulation and Interference (FIMI) predates AI, yet large language models are reshaping how influence operations are drafted, refined, and countered. As LLMs become tools for both manipulators and investigators, the struggle over information integrity is moving into a faster, more adaptive phase. For this reason, we spoke with Ben Nimmo of OpenAI to understand what has truly changed, what remains constant, and what will matter most in the future.
Interviewer:
Influence operations didn’t start with AI, but large language models are clearly changing how they’re carried out and investigated. When you look at this intersection, what do you see as the biggest challenges facing LLMs today—technically, socially, and in terms of security?
Ben Nimmo:
At OpenAI, we think about technology end-to-end. No technology exists in a vacuum. When we consider large language models, we ask how they fit into the broader landscape of influence operations, which existed long before LLMs.
Whenever a new technology emerges, there is a period of adjustment and learning: how it will be used, how threat actors will try to abuse it, and how defenders can respond. With chatbots and LLMs, we are in that phase now.
A useful comparison comes from 2019, before AI chatbots, when generative adversarial networks (GANs) were used to create fake profile pictures. I led the first open-source investigation into a large network of fake social media accounts using GAN-generated faces.
Initially, this seemed worrying because investigators could no longer reverse-image-search stolen photos. But we quickly discovered that GAN images were highly distinctive. For example, the eyes always aligned in the same place across images. A colleague even developed a tool to detect GAN faces at scale.
What looked like an advantage for threat actors became an advantage for defenders. This illustrates why publishing findings matters. At OpenAI, we publish threat reports on how our models are abused for influence operations, scams, cybercrime, and authoritarian repression. We’ve done this for nearly two years to support collective learning and adaptation.
Interviewer:
It sounds like an ongoing competition between those creating deceptive content and those detecting it.
Ben Nimmo:
Exactly. What we see is evolution rather than revolution. Threat actors tend to integrate AI into existing workflows instead of building entirely new ones.
For example, you can generate a convincing article with AI, but without a distribution network, it reaches no one. So the core infrastructure of influence operations remains unchanged.
We’ve observed Russian and Chinese operations using ChatGPT to improve their English and avoid grammatical errors that once made them easy to spot. But removing one set of mistakes often introduces others.
In early 2024, we disrupted a Russian influence operation we call Bad Grammar. They used ChatGPT to generate dialect-appropriate English for fake personas. However, one Telegram post accidentally published the model’s refusal message, stating words to the effect of:
‘As an AI language model, I can’t give you a comment in the voice of Ethan Goldstein, a 57-year-old Jew.’
That single mistake revealed the use of AI, the lack of proofreading, and the attempted impersonation.
Our job as investigators is to understand these workflows, identify their weaknesses, and disrupt them. Sharing findings across platforms is essential. In late 2024, for example, we disrupted a scam network using ChatGPT. Meta investigated related Facebook accounts, identified scam centres in Cambodia, and shared findings back with us – allowing further disruption.
Interviewer:
How does AI help you investigate these operations?
Ben Nimmo:
I need to be careful with specifics, but broadly speaking, AI excels at large-scale analysis, pattern recognition, and translation.
Earlier in my career, I spent hours manually analysing social media accounts – counting post types, language use, and behaviour. Today, those same analytical processes can be done roughly 100 times faster using AI.
We always keep humans in the loop to verify results, but AI has been a genuine game changer in terms of speed and efficiency.
Interviewer:
What threats exist today that didn’t in the earlier social media era?
Ben Nimmo:
Again, we see evolution rather than entirely new threats. One unexpected development is how ordinary people now use AI defensively.
In our October threat report, we found that millions of people each month ask ChatGPT whether an SMS message is a scam. The model explains why it’s suspicious and what to do next. That kind of real-time assistance simply didn’t exist before.
AI tools are increasingly empowering users, especially on the scam-prevention side.
Interviewer:
How real is the threat of ‘LLM grooming,’ where hostile actors try to influence what models learn?
Ben Nimmo:
“LLM grooming” is a new term, but it relates to a long-standing concept known as data poisoning. There is extensive academic research on this.
What’s critical is distinguishing between training data and web search results. Models have a training cutoff date. Anything after that is not internal knowledge; it’s retrieved externally.
If a model references misleading content via web search, that does not mean the model has been trained on it. This distinction is often misunderstood.
Influence actors have long tried to manipulate search results by flooding information spaces, especially exploiting data voids – situations where only one narrative is actively being published. We saw this clearly during the Syrian civil war.
That’s why publishing credible, fact-based information is so important. If the vulnerability is a lack of reliable content, the solution is to fill that gap.
Interviewer:
Have hostile influence campaigns appeared in model outputs?
Ben Nimmo:
In my experience, ChatGPT references credible sources like EUvsDisinfo far more often than covert influence operations.
A peer-reviewed study of four different chatbots published in Harvard Misinformation Review found that the chatbots only occasionally referenced Kremlin-linked disinformation websites, and that rather than being a result of “LLM grooming”, the results where they did make those references were likely a symptom of data voids.
That doesn’t mean the risk doesn’t exist. Our job is to monitor it continuously and respond quickly if it does.
Interviewer:
How does OpenAI filter out propaganda networks before training?
Ben Nimmo:
I can’t share operational details, but this builds on years of research into how influence operations function. Domains linked to such operations are publicly documented by multiple organizations.
Internally, OpenAI has teams focused on data quality, pre-training, post-training, evaluation, and red teaming. After launch, investigators like me monitor emerging risks and share findings internally. It’s a multi-layered, collective process.
Interviewer:
Can you briefly explain ‘red teaming’?
Ben Nimmo:
Red teaming involves internal teams attempting to think like adversaries—testing how someone might try to bypass safeguards. By identifying weaknesses in advance, we can strengthen defenses before deployment.
Interviewer:
How has the influence threat landscape changed in the past decade?
Ben Nimmo:
Platforms change – Instagram, TikTok, and now generative AI – but many operations persist for years. Chinese Operation Spamouflage, exposed in 2019, is still active. Russian Doppelganger, exposed in 2022, continues.
The biggest change isn’t the threat actors; it’s the defenders. In 2014, very few people worked on this. Today, there are numerous investigative teams, growing transparency, and widespread threat reporting – including from AI companies themselves.
That collective growth is the most important shift.
Interviewer:
Is the counter-FIMI community strong enough to respond?
Ben Nimmo:
The key question isn’t whether it’s strong enough today, but how we make it stronger tomorrow.
That includes better tools, training, transparency, information sharing, and public awareness. AI can dramatically increase investigative efficiency, but skills and coordination matter just as much.
Interviewer:
What should governments, researchers, and AI companies focus on over the next five years?
Ben Nimmo:
Influence operations long predate AI. The key is applying lessons already learned: clear reporting, accurate risk assessment, impact measurement, and structured information sharing.
We must avoid starting from zero with each new technology. The goal is continuity – carrying institutional knowledge forward as the technological landscape evolves.
Interviewer:
How do you bridge the gap between technical reports and public understanding?
Ben Nimmo:
Rather than focusing on how the technology works, focus on how people use it.
Understanding influence operations means understanding human deception. Whether the platform is Facebook or a language model, the behaviours remain consistent. Patterns like posting at implausible hours or mimicking identities poorly are timeless indicators.
Human behaviour changes far less than technology – and that’s where public understanding can anchor.
By EUvsDisinfo
