How Local AI Agents Use Tools, Files and Permissions

A local AI agent is not a model that reaches into your filesystem; it is a model that proposes actions which your application then executes. This explainer walks through the loop of tool definitions, tool calls and results, then shows how skills and permissions decide what an agent can actually reach.

How Local AI Agents Use Tools, Files and Permissions
via YouTube

A local AI agent often gets imagined as a model that "looks at your files." That picture is wrong in a way that matters for both expectations and safety. The model itself does not open folders or run programs. It reads a prompt, and it can emit a structured request describing an action it would like taken. Something else on your machine — the application hosting the model — decides whether to carry that request out.

That separation is the whole architecture in miniature, and it explains nearly every practical question about local agents: why they sometimes claim to have done things they did not do, why the same model behaves differently in two apps, and why "local" is not automatically "safe."

The tool-calling loop

Tool calling is the mechanism. Microsoft's Foundry Local documentation describes it as a technique where the model is prompted with definitions of available tools alongside the user's text request, so the model can work out which tools should be called and with what input data. The application then calls those tools and feeds the results back into a subsequent model prompt to answer the question ¹.

Unpacked, that is a loop with four beats:

  1. The application hands the model a list of tools, each with a name, a description and a schema for its inputs.
  2. The model, given a user request, replies with a proposed call — a tool name and arguments — rather than a finished answer.
  3. The application executes the call, or refuses to.
  4. The result goes back into the conversation, and the model continues, possibly requesting another call.

The model's contribution is selection and argument construction. Execution belongs entirely to the host application. A model that "wants" to read a file has, in reality, produced a string that says so.

Two concrete examples

Consider a weather tool. The application defines a function that takes a city name and returns a forecast. Asked about tomorrow in a given city, the model emits a call with that city as the argument. The host runs the lookup and returns the data, and the model writes a sentence from it. The model never touched a network; it named a function.

Now consider files. The Foundry Local guide lists reading files on the local file system as exactly the kind of function a tool can perform, alongside things like fetching weather or accessing a user's address book — and it attaches a condition to that list: providing the application has permission to do so. That clause is doing enormous work. A file-reading tool is not a property of the model. It is a capability the host chooses to expose, scoped by whatever the host allows.

The same guide notes that not every model supports tool calling at all; the Foundry Local CLI exposes a model list where a tools task marker indicates support. So capability is layered: the model must be able to emit calls, the application must define tools, and the operating environment must grant access.

Skills are instructions, not access

A second concept is getting attention as agents become more reusable: the skill. GitHub's guide to agent skills describes a skill as a versioned directory centered on a SKILL.md file that describes when and how to do a repeatable task. Skills can be shared across agents, but the guide is explicit that the environment still has to supply the tools, credentials and permissions to carry them out — a skill can describe how to query a database without granting access to it ².

This is the cleanest way to think about the boundary. A skill is documentation for behaviour. A tool is a callable function. A permission is the host's decision about whether that function may run against a given resource. You can copy a skill between agents freely; you cannot copy the access with it. The GitHub guide frames skills as capturing repeatable engineering work in files teams can review and reuse, while tools, project context and permissions stay specific to the host that runs them.

That has a direct consequence for anyone assembling a local setup: sharing a skill folder is low-risk and portable, while granting a file-reading tool is a real decision about your disk.

Why "local" is not a safety label

Running a model on your own hardware removes the network round trip and keeps prompts on the machine. It does not remove the permission question, because the permission question lives in the application layer, not the model layer. A local agent with a broadly scoped file tool and no confirmation step can be asked, directly or indirectly, to read things you did not intend to share — and the model will simply propose the call.

The practical rules that follow are unglamorous. Define the narrowest tool set that accomplishes the task. Scope file tools to specific directories rather than a home folder. Keep a human confirmation in front of any write, delete or execute action. Treat retrieved content as data, not instructions, since a file an agent reads may contain text that tries to steer it. And check whether the model in use actually supports tool calling before building around it.

None of this requires distrusting local models. It requires putting the trust in the right place: the host application's tool definitions and permission checks, which are ordinary software you can inspect, rather than in the model's willingness to behave. The model proposes; the application disposes. Everything else about local agent safety follows from that one sentence.

Sources

  1. ¹ Use tool calling with Foundry Local - Foundry Local - მთავარ შიგთავსზე გადასვლა Ask Learn-ის საუბრის გამოცდილებაზე გადასვლა learn.microsoft.com
  2. ² Build once, run on any agent: A practical guide to agent skills github.blog