Prompt Injection and Data Poisoning for Ops Teams

Updated October 2026. First published January 2024.

In January 2024, the Australian Signals Directorate's Australian Cyber Security Centre published Engaging with Artificial Intelligence with agencies from ten other countries, among them CISA, the FBI, the NSA, and the UK's National Cyber Security Centre. The first version of this post summarized it. This version applies it to the AI tool operations teams are most likely to have by now: one that reads documents. FINRA's December 2025 oversight report says the top generative AI use case among member firms is summarization and information extraction, which means pulling key facts out of unstructured documents.

The guidance is written for organizations that use AI systems; developers get a separate set of guidelines. It describes the main threats, then poses questions a firm should be able to answer. Three of the threats matter most for a tool that reads inbound documents.

Prompt Injection

The guidance defines prompt injection as an attack that "attempts to insert malicious instructions or hidden commands into an AI system." The familiar version is a user typing a jailbreak into a chatbot. The version that matters for operations is indirect: the instruction arrives inside the data. NIST's March 2025 taxonomy of adversarial machine learning, NIST AI 100-2 E2025, explains why it works. Generative models "combine the data and instruction channels," so whoever controls a document the model reads can try to give it orders.

NIST's examples include instructions hidden in parts of a document a person can't see, a model tricked into writing a wrong summary of its source, and a model built into an email client that's told to forward mail to an attacker. Now think about what your desk receives from outside the firm: requests from counterparties, changes to settlement instructions, PDFs and faxed scans from firms you've never dealt with. Every one is untrusted input. NIST is blunt that current defenses "do not offer full protection," and suggests designing on the assumption that prompt injection is possible whenever a model reads untrusted input.

That assumption leads to a design rule. Let the model read and classify. Keep anything that commits the firm, such as approving a request, sending a response, or changing an instruction, behind a person or a control the document can't reach. NIST mentions running models with different permissions for different jobs, and letting a model touch untrusted data only through well-defined interfaces. FINRA's 2026 report raises the same concern about AI agents. Its list of risks starts with agents "acting autonomously without human validation and approval," followed by agents acting beyond the user's intended scope and authority.

Data Poisoning

Poisoning means manipulating what a model learns from, so that it "learns incorrect patterns" and misclassifies data. The guidance adds a point that's easy to miss: a model that learns from feedback can be manipulated by "poor quality or incorrect feedback." Its example is Microsoft's Tay, a 2016 Twitter chatbot that trained on its conversations. Users sent it abusive messages on purpose, and it started using abusive language with other users.

If your tool improves as staff correct it, those corrections are training data. So ask who can make corrections, and whether a second person reviews them before the model learns from them. A correction queue that anyone can edit is a poisoning path even when nobody means harm.

Hallucinations

Generative models sometimes produce confident output that's wrong. The guidance cites the 2023 case in the Southern District of New York in which a brief contained at least six cases ChatGPT had invented. In extraction work the failure is quieter: an account number or an amount misread from a scan. FINRA's 2026 report defines a hallucination as inaccurate or misleading information that "is presented as factual information." Its recommendations include accuracy testing, logs of prompts and outputs, a record of which model version ran when, and human review of what the model produces.

The Questions to Ask First

The joint guidance ends with a list of questions. For a document-reading tool, these are the ones we'd ask first:

Does access require phishing-resistant multi-factor authentication, such as FIDO2 keys, including access to any repository that holds training data?
Are privileges granted on least privilege, with privileged accounts revalidated regularly and disabled after a period of inactivity?
Do you know the tool's false positive and false negative rates?
Are inputs and outputs logged, and is there a baseline that shows you when the tool's behavior changes? The guidance says to watch for high-frequency, repetitive prompts, which can signal automated injection attempts.
If a vendor hosts the system, will your inputs be used to retrain its model, and what happens to your data when the contract ends?
Does staff training cover what data may go into the tool and how far its output can be relied on?
Does your incident response plan cover the AI system, and does the contract spell out the vendor's incident duties?

Loffa's FVD-R follows the design rule above. On the receive side of the Letter of Free Funds, it uses AI to index inbound requests, including email bodies, PDFs, and faxed scans, and to match each one to an account, desk, and request type. Items it can't match with confidence go to a review queue where staff confirm the account, and the model learns from those corrections. The response to the executing broker is a separate step: an authorized operations user verifies that the funds are available and selects the response under role-based permissions and supervisory sign-off.

Changes to settlement instructions are the inbound documents most worth protecting, AI or not. We cover that fraud pattern in our post on settlement instruction fraud under T+1.

This post is for informational purposes only and does not constitute legal advice. For guidance on specific regulatory obligations, consult your counsel or compliance advisor.