AI & Agentic Weekly Update — July 20, 2026

Agent Evaluations Go GA — Testing Finally Catches Up to Building

The most useful thing to land this week is also the least flashy: agent evaluations are now generally available in Copilot Studio. Makers can validate agent performance against customizable test sets, running an agent through a defined set of scenarios and scoring the results rather than eyeballing a handful of prompts and calling it good. Alongside GA, Microsoft added multi-turn conversation tests, which evaluate how an agent behaves across a realistic back-and-forth dialog instead of a single isolated exchange.

That second piece matters more than it sounds. Most agents that fail in production do not fail on turn one — they fail on turn four, when context has drifted, the user has changed their mind, or the agent has quietly lost the thread of what it was asked to do. Single-turn testing systematically misses this class of failure, which is exactly why so many pilots look great in a demo and fall apart in a department rollout.

The strategic read is that Microsoft has spent the past several quarters making agents trivially easy to create, and is now shipping the tooling to find out whether any of them actually work. Organizations that have accumulated a pile of agents built by enthusiastic departments now have a way to audit them. What to do: Pick your three highest-traffic agents and write a test set for each — ten to fifteen realistic scenarios, including multi-turn ones where the user corrects course mid-conversation. Establish a baseline score now, before you change anything, so you have something to measure future changes against.

The Agent Store Opens a Governed "Built by Your Org" Path

Agent Builder submissions can now flow into the Agent Store under a "Built by your org" section, after admin review and approval in the Microsoft 365 Admin Center. Someone in finance builds a useful agent, submits it, an admin reviews and approves it, and it becomes discoverable to everyone else in the tenant instead of dying in a Teams chat.

This closes a real gap. Until now, the practical outcome of easy agent creation was duplication — four teams independently building four mediocre versions of the same expense-policy agent, none of them reviewed, none of them shared. A governed submission flow turns internal agent-building from a scattering of private experiments into something closer to an internal app catalog, with an approval gate that gives admins a say in what gets blessed.

The catch is that a review gate is only as good as the review criteria behind it, and Microsoft does not supply those. An admin staring at a submitted agent with no rubric will either rubber-stamp everything or block everything. What to do: Write your approval rubric before you open submissions. At minimum, require a submitted agent to declare what data it touches, who owns it, what it is not meant to be used for, and — now that evaluations are GA — a passing test set. Publish the rubric so builders know the bar in advance.

Microsoft Scout and the Autopilot Model Get Concrete

Microsoft Scout, the always-on personal agent Microsoft introduced at Build as its first "Autopilot," is now reaching Frontier organizations and a select private preview group. The distinction Microsoft is drawing is worth taking seriously: a Copilot waits to be prompted, while an Autopilot runs continuously in the background, holds its own identity, and takes action without being asked each time.

Scout's scope is broad. It works across cloud, desktop, and web; connects to Teams, Outlook, OneDrive, and SharePoint along with mail, calendar, and contacts; creates, edits, and searches files; drives a browser to navigate pages and fill in forms; and can run terminal commands behind a tiered permission system. In practice that looks like coordinating meeting times across time zones, flagging meetings that need prep and producing the prep material, surfacing upcoming deliverables, blocking focus time, and spotting stalled decisions. Access requires Frontier enrollment, Intune policy configuration, and an opt-in attestation — a deliberately high bar for something with this much reach.

Pricing is still usage-based and not fully detailed, which is itself a planning problem: an agent that runs continuously has a fundamentally different cost profile than one that runs when someone types into a box. What to do: Do not wait for broad availability to think about this. Decide now what an always-on agent should and should not be allowed to touch in your environment, and get that written down as policy — because the Intune configuration and attestation step will force the question anyway, and it is a much better conversation to have deliberately than under deadline.

Model Choice Becomes a Real Design Decision in Copilot Studio

Copilot Studio's model lineup expanded meaningfully: ChatGPT-5 is now generally available globally for production agents, and Claude Sonnet 4.5, Claude Sonnet 4.6, and Claude Opus are all GA globally as well (excluding GCC environments). Separately, Claude Sonnet 4.5 remains available for computer-use agents, where Microsoft cites better nuanced decision-making and higher success rates on complex tasks.

The point of a multi-model platform is not novelty — it is that reasoning depth, output quality, latency, and cost are genuinely different levers, and the right setting differs by workload. A high-volume triage agent answering thousands of routine questions should not be running on the same model as an agent doing multi-step analysis over financial documents. Defaulting every agent to the most capable available model is the most common and most expensive mistake teams make here.

There is a governance dimension too. Model choice is now a per-agent configuration decision that someone owns, which means it needs to be documented and periodically revisited as models and prices change. What to do: Audit which model each of your production agents is running and why. Where an agent is doing simple, high-volume, low-ambiguity work, test a faster and cheaper model against your evaluation test set — if the score holds, you have found real savings with measurable evidence behind it.

Cost Governance Is Now the Bottleneck, Not Capability

Step back from the individual announcements and a pattern is hard to miss. Copilot Cowork bills through metered Copilot Credits at roughly a cent per credit, with light tasks running a dollar or three and heavier ones seven and up. Agent 365 runs about fifteen dollars per user per month standalone. Scout will be usage-based. Copilot Studio agents consume against model-dependent rates. The Cost Management Dashboard in the admin center exists precisely because this has become complicated enough to need one.

The shift is from seat licensing, where cost is predictable and flat, to consumption billing, where cost scales with how much work the agents actually do. That is arguably the honest model — you pay for value delivered — but it breaks the budgeting habits most IT organizations have built over two decades. A department that discovers an agent it loves can generate an invoice nobody forecast.

The tooling to manage this exists: credit allocation by group, spending policies and limits, budgets, alerts, and hard caps. Most organizations have not configured any of it. What to do: Set hard caps and alert thresholds now, before you need them, and allocate credits by group rather than tenant-wide so a single runaway workload cannot consume the whole pool. Then put a monthly review of the Cost Management Dashboard on someone's actual calendar — unmonitored consumption billing is how pilots become budget incidents.

What This Week Means for Your AI Strategy

The through-line across all of this is that Microsoft's agent story has moved past the build phase. Evaluations, an approval-gated internal store, licensed security controls, and consumption dashboards are not features you ship to people who are still experimenting — they are features you ship to people running agents in production and discovering that production is harder.

The organizations that will do well over the next two quarters are not the ones with the most agents. They are the ones that can answer basic operational questions about the agents they have: who owns this, what does it touch, how do we know it works, what does it cost, and what happens when it stops working. Almost none of those questions are technical, which is precisely why they get skipped in favor of building the next agent.

What to do: Take an inventory this month. List every agent running in your tenant, and for each one record an owner, a purpose, the data it accesses, the model it uses, and whether it has a test set. Whatever that exercise reveals is your real AI roadmap for the rest of the year — not the roadmap you wrote in January.

Sources

Back to all news