Advanced: Agent Dev 17 Runs Agents Locally So Your Data Stays Put

Running an agent locally means your code and documents never leave your computer, there is no pay-per-use billing, and it works without an internet connection. The trade-off is that you can only use small models.

Remember These Points First

• Three benefits: privacy, cost, offline.
• Small models are good at: small structured tasks, choosing tools and filling in parameters, and iterating quickly on your own data.
• Small models are not good at: complex reasoning across long contexts, and broad world knowledge.
• Strategy: let the small model act as a dispatcher, and hand the real work to tools.
• Only models that support function calling can power a real agent.

Going Deeper

How to do it. Run a service on your machine that exposes an OpenAI-compatible interface. Code that once called the cloud only needs its address changed to localhost. Then give the agent local tools, such as reading files and searching documents, so it can do real work.

Hardware. 8GB of RAM is the practical minimum, 16GB or more is more comfortable, and a GPU or NPU will make it faster but is not required.

The most important point: don't expect a small model to know everything. It doesn't need to know what your code looks like, it only needs to know when to call "read file" or "search documents". This is exactly what small models are good at.

Mixing. Keep sensitive data processed locally, and selectively send the parts that really need strong reasoning to the cloud.

Image from the course repository.

Try It Yourself

Install a tool on your machine that can run a small model, have it read a text file on your computer and answer three questions, and see where it gets things wrong.

Let an Agent Guide You Through Learning

Give the following to your agent (Claude Code, ChatGPT and Gemini all work), and it will guide you step by step like a mentor.

References

• Content adapted from Lesson 17 (17-creating-local-ai-agents) of microsoft/ai-agents-for-beginners, original project under the MIT license.
• For the original and full code: 17-creating-local-ai-agents
• For all courses, see the Agent course roadmap