Advanced: Agent Dev 06 Maps the Threats That Make Agents Untrustworthy

The more an agent can act on its own, the more carefully you need to think about how it could be exploited. Write a solid system prompt first, then set up a defense for each typical threat.

Remember These Points First

• System prompts need a framework: write a master template first, then use it to generate each agent's prompt, and keep improving it.
• Tampered instructions: validate inputs and limit the number of conversation turns.
• Unauthorized access: grant only the permissions needed, and use encryption and authentication for communication.
• Resource abuse: limit request counts and set budget caps.
• Knowledge base poisoning: regularly verify data, and only trusted people can edit it.
• Cascading failures: run agents in isolated environments, such as containers, and set up human fallbacks.

Elaboration

How to write a stable prompt. First write a 'meta-prompt' that tells the model how to write system prompts for other agents. Then write a basic role description and have the model help you optimize it. The first version rarely gets everything right in one go, so change a bit, compare, and revise again.

Threats and countermeasures. Look at the table below:

Threat	In one sentence	How to defend
Tampered instructions	Someone uses input to change its goal	Input filtering, limiting turns
Unauthorized access	It can reach too many systems	Least privilege, authentication, encryption
Resource abuse	It is used to flood your service	Request rate limiting, caps
Knowledge base poisoning	The material it uses has been altered	Regular verification, control who can edit
Cascading failures	One error spreads elsewhere	Isolated environments, human intervention

Human intervention. For important actions (payments, deleting data), having a person confirm is more reliable than any prompt.

Image from the course repository.

Try It Yourself

Think of an agent you want to build, write down the three systems it might touch, and write one 'least privilege' rule for each.

Let Your Agent Teach You

Give the following to your agent (Claude Code, ChatGPT, and Gemini all work), and it will guide you step by step like a mentor.

References

• Content adapted from Lesson 6 (06-building-trustworthy-agents) of microsoft/ai-agents-for-beginners, originally released under the MIT License.
• For the original and full code: 06-building-trustworthy-agents
• For all courses, see the Agent Course Roadmap