AI Chatbot Security: Stop Prompt Injection from Becoming a Data Breach

A safer AI chatbot assumes that the model can be persuaded—and still prevents it from accessing another customer’s data or taking an unauthorized action. A system prompt helps describe the task. It is not a replacement for authentication, authorization or transaction controls.

This guide is current as of September 10, 2026 and uses OWASP’s published guidance. The support chatbot below is a hypothetical design, not a penetration-test report or a claim that any named project has these protections. The checklist reduces risk; it does not certify an application as secure.

What is prompt injection?

Prompt injection occurs when input steers a model away from the application’s intended behavior. A direct attempt comes through the user’s message. An indirect attempt can arrive inside a retrieved document, webpage, email or other material the model reads.

OWASP’s Prompt Injection risk description explains that RAG and fine-tuning do not fully solve this problem. Better retrieval can improve factual answers while still retrieving hostile instructions.

The important question is therefore not just “did the bot obey the wrong sentence?” It is also “what could the application do as a consequence?” An incorrect summary is a different incident from sending a confidential document to an external recipient.

Threat-model one ordinary support conversation

Imagine a business chatbot that answers delivery questions. It can read the signed-in customer’s orders, retrieve the store’s policy and draft a response. A later release might allow a refund after approval.

Now imagine a policy attachment containing a forged instruction that says an internal review requires exporting the entire customer list. The attachment is relevant enough to retrieval that it enters the model’s context. The application must treat its contents as reference material, not as permission to change the task.

For this design, I would write down the trust boundaries before choosing a model:

Draw the boundaries even if the first version only answers questions. A read-only application can still disclose information. If it searches every tenant’s documents and asks the model not to mention the wrong ones, authorization has happened too late.

Authorize before retrieval—and before returning a result

In a multi-tenant product, resolve the user and organization on the server. Build the allowed document scope from that identity. Apply it before documents enter model context, not after an answer has been generated.

An illustrative server-side sequence is:

Authenticate the request
Resolve the user's tenant and document permissions
Search only the permitted document scope
Recheck access to the selected source records
Give the model the minimum relevant excerpts
Validate the answer's referenced source IDs
Return the answer with permitted source links

This is a design sequence, not production-ready authorization code. The exact implementation depends on your database, retrieval service and permission model.

Test the cache as carefully as the search. A cache keyed only by the question can return one organization’s answer to another. Scope retrieval and answer caches by tenant, relevant permissions and document version, or avoid caching sensitive generated answers. Invalidation must account for revoked access, not just updated text.

OWASP’s Sensitive Information Disclosure guidance highlights the risk of exposing personal, proprietary or confidential information. A practical response is data minimization: credentials, unrelated records and complete customer exports should never enter this chatbot’s context in the first place.

Give the model narrow tools, not a general-purpose account

Compare two hypothetical tool interfaces. One accepts arbitrary SQL and runs it as an administrator. The other accepts an order reference and returns a small delivery summary after checking ownership.

The narrow interface is easier to constrain, review and test. It also makes the intended capability visible to the team maintaining the system. The model should not supply a trusted role or tenant ID that overrides the session.

OWASP calls broad functionality, excessive permissions and unchecked autonomy Excessive Agency. Reduce all three. Removing a write tool entirely is stronger than leaving it available with a prompt asking the model not to use it.

For the first release, I would allow policy search and delivery lookup but omit refunds, email sending, arbitrary browsing and shell access. Add a capability only when a user need and an enforcement design justify it. This is the same architectural restraint discussed in AI agents vs workflows.

Approval must describe the exact action

A vague “yes, proceed” in chat should not authorize an action whose recipient, amount or target can change afterward.

Suppose an authorized employee approves a shipping refund. Present the order, amount, currency and reason in the review interface. Store the approved action outside the model’s conversation, with the reviewer, expiry and a version or hash of the approved arguments.

When executing, recheck the reviewer’s permission and the order’s current state. Reject changed arguments and expired approvals. Make the operation idempotent so a timeout and retry cannot create duplicate refunds. If the order changed after review, require a new decision.

These are proposed application controls for this example, not a universal payment implementation. The model may propose an action; your application owns the decision about whether that action is authorized.

Keep generated output from becoming another input channel

Treat model output like other untrusted content. Do not execute generated code in the application process or render raw generated HTML without an appropriate sanitization policy.

For a support interface, I would allow a small Markdown subset, restrict URL schemes, and verify citation targets against the permitted source records. Automatic loading of arbitrary remote images deserves special attention because a request can reveal data included in a URL. Where images are unnecessary, do not support them in chatbot responses.

A valid JSON shape proves only that the output fits a shape. It does not prove that the referenced order belongs to the user, the destination is approved, or a quoted policy is accurate.

OWASP’s Prompt Injection Prevention Cheat Sheet describes layered defenses, including separation of instructions and data, validation and monitoring. Filters can be one layer; a blocklist of suspicious phrases is not the security boundary.

Build a test matrix with observable outcomes

Test only systems and data you are authorized to assess. Use synthetic customers and documents, not copied production secrets. Keep expected results alongside the test fixture so a fluent answer cannot hide an incorrect action.

Run a benign counterpart for each refusal test. The permitted customer must still be able to read their own delivery status. A system that refuses everything has low utility, not successful security.

Repeat model-dependent cases, because one successful run is weak evidence. Keep deterministic access-control tests separate from behavioral evaluation. Run both when changing the model, prompt, tool schema, retriever or output renderer.

Plan for an incident before enabling writes

Decide how an operator disables a risky tool without taking down the whole product. Record which data sources and tool calls a response used, while redacting unnecessary personal information and secrets from logs.

If a suspected leak occurs, preserve the relevant audit trail, restrict the affected capability, identify the exposure and follow your incident process. Do not paste raw customer conversations into a public debugging tool. Retention and notification decisions need to follow the business’s actual obligations.

Assign ownership: who reviews rejected tool calls, who can revoke integration credentials, and who decides when to re-enable a capability? “The model provider handles safety” is not an operational plan for your application.

Frequently asked questions

Does RAG make a chatbot secure?

No. RAG retrieves information; authorization determines who may receive it. Retrieved material can also carry indirect prompt injection. My RAG vs fine-tuning guide explains the separate architecture decision.

Is human approval enough?

Not by itself. The reviewer needs a clear action preview, appropriate authority, and an execution path that cannot silently change the approved action. Approval complements access controls.

What is the safest useful first version?

For this support example, a tenant-scoped, read-only assistant with source-backed answers and human handoff. Start there, measure failures, and add actions individually. Other products may need different boundaries.

If you are planning an AI feature, get in touch with the data it needs and the actions it should perform. Those two lists tell us more about the engineering work than a promise that the chatbot will be autonomous.