Unflagged and Obeyed: Prompt Injection Through Small Models in LLM Financial Pipelines
University College London
Abstract
Financial firms let language models read documents they do not control—earnings transcripts, investor-relations pages, news—and may give that first reading to a small, low-cost model whose output feeds a larger model, an analyst or an agent. We plant one sentence urging a Strong Buy (or Strong Sell) in such documents and test six low-cost chat models on a stock-rating task, with synthetic companies whose quality is known by construction (18,630 calls, plus 2,550 in follow-up controls analysed to plans fixed before collection). Susceptibility differs sharply between models of the same low-cost tier: on earnings transcripts the same sentence led Qwen2.5-7B to rate every weak company Strong Buy and the two Gemini models none. The model’s own suspicion flag is not a reliable control by itself. On transcripts, a pipeline trusting every unflagged output would have passed all 120, 130 and 74 manipulated ratings from three of the six models. The attacker need not be the company: in exploratory news-page tests, one anonymous reader comment raised GPT-4.1-nano’s rating of every strong company, unflagged, and changing only who a quoted directive is credited to turns GPT-4o-mini’s flag on strong companies from 30 of 30 rounds to 0 while the rating still rises. When GPT-4.1-nano writes a short note on an injected transcript, it drops the instruction and adopts its conclusion, and Gemini-3.1-Flash-Lite, which resists the transcript itself, is moved by the note without flagging it; a current OpenAI flagship, GPT-6 Sol, is not: it cites the real figures the note still carries and declines its verdict. An added prompt rule telling the model never to follow instructions in the document did not stop GPT-4.1-nano or Qwen2.5-7B, nor reliably reduce the shift of the other two models tested. The practical conclusion is to red-team every model that reads untrusted text, the small ones upstream included, and not to rely on its flag or its prompt as the control. The threat model carries over to fraud, anti-money-laundering and payment screening, where the party being assessed can write the text and a flag can decide which cases reach a reviewer; we did not test those services. We outline a black-box test a governance function can require and report three measurement traps.
Key findings
- The flag misses the manipulation that works. On earnings transcripts, for three of the six models the “suspicious content” flag missed every rating the planted sentence moved.
- The attacker doesn’t have to be the company. On news pages, one anonymous reader comment raised GPT-4.1-nano’s rating of every strong company, unflagged.
- Who is credited switches the flag. Crediting the same directive to a reader instead of the company turned GPT-4o-mini’s flag from 30 of 30 rounds to 0, while its rating still rose (exploratory).
- A small model’s summary carries the verdict onward. GPT-4.1-nano’s note dropped the instruction but kept its conclusion, and moved Gemini-3.1-Flash-Lite, which had resisted the original document. GPT-6 Sol, a flagship model, was not moved.
- Takeaway: red-team every model that reads untrusted text, the small ones upstream included, and don’t rely on a model’s own flag or a prompt rule as the control.
Poster
Presented at Bocconi University, Milan, on 15 November 2026. Full-resolution PDF
Cite
@misc{brown2026unflagged,
title = {Unflagged and Obeyed: Prompt Injection Through Small Models in LLM Financial Pipelines},
author = {Brown, Euan},
year = {2026},
note = {Poster, ICAIF '26 Workshop on Financial AI Security, Privacy, and Safety, Milan},
howpublished = {\url{https://euanbrown08.github.io/icaif-paper/}}
}
Questions about the paper: euan.brown.26@ucl.ac.uk · LinkedIn