Introducing ITSMBench

Evaluating frontier model capabilities in the IT Service Management domain: how far are we from fully autonomous service desks?

Every other day, we hear this question from our prospects: Agents from frontier labs have come far enough to build complex and well-designed apps end-to-end in a matter of hours. But how good are they at autonomously resolving IT requests?

To answer this question, and many more, we decided to collaborate with New Measure to build the ITSMBench.

ITSMBench measures frontier models and the harnesses they run in against the requests an enterprise service desk might actually receive. The agents work inside a simulated enterprise of 42 mocked software backed by roughly 1,800 database tables and 2,000+ REST endpoints. The ITSMBench is the first benchmark to evaluate frontier model capabilities in the IT Service Management domain.

We score 89 tasks with 10 to 30 expert-written verifiers each, run as deterministic checks on the resulting environment to answer this question: Can AI run your IT service desk autonomously?

We walked away with four findings:

• Every model is jagged, not uniformly good. Every frontier model excels at some tasks and fails at others in ways that don’t track with overall capability, or with each other. Grok is great at discovering the right APIs/tools for the task, but struggles to complete it. Opus is weak at search, but once it has the right tools, it’s the best at completing the task. GPT sits in the middle of the two. These findings suggest that an orchestration of models works better than relying on one. None of these is a “worse” model; each is jagged in a different place.
• The harness matters nearly as much as the weights. For both accuracy and cost, the same models scored a few points better through a minimal open-source harness than through their providers’ own harnesses.
• Models take shortcuts. When the root cause isn’t obvious, they lock onto the first story that lets them keep moving. The investigation shrinks to fit that story. A human expert would keep testing that assumption against what else it might break. Closing the gap means a deeper intuition of how an IT employee would act.
• IT depends. The honest answer is that automating ITSM depends, not on the model, but on the kind of requests. A service desk that primarily receives password reset queries as opposed to a malware-triggered network lockdown would not be staffed by the same kind of workforce. One kind of request is a lookup. The other needs judgment across systems that don’t agree with each other yet.

Every task and environment in this benchmark is open source, so anyone can reproduce these results or build on them: github.com/new-measure/ITSMBench.

What does it mean to run an IT service desk?

A human service desk still has three layers. What separates them is not how hard the ticket sounds, but where the work actually lands.

• L1 is knowledge. The answer already exists in a policy page, a runbook, or an old ticket. The job is to find it and hand it back, so much of L1 never becomes a ticket.
• L2 is service management. A request comes in from an end user and someone has to fulfill it. These are execution-heavy asks: “send me a password reset link” reads as trivial, but fulfilling it spans several systems, needs identity verification, and waits on an approval. Restoring VPN access, finishing onboarding, and fixing a mailbox work the same way. For instance, a user says their laptop is slow, but security had locked it off the network after malware: an AI would have to confirm the threat, clean the device, get it back online, and close the tickets.
• L3 is IT operations. These problems rarely come from an end user. Monitoring, an audit, or the IT team surfaces them, and the fix sits in infrastructure: networking, database operations, certificates, unfinished migrations. An old datacenter was shut down, but some traffic still goes there and breaks, so an engineer has to find everything still pointing at the dead machines and finish the move.

With AI in the loop, deflection absorbs L1 outright, a master agent takes the L2 queue and hands the specialized cases to subagents, and humans stay in the loop for L3.

ITSMBench measures L2 and L3. We deliberately don’t score L1: pulling an answer out of a knowledge base is the one part of this stack where model capability is already well understood, and it isn’t where deployments break. L2 and L3 are where agents have to investigate across systems, make judgment calls, and apply the right fix.

Why another benchmark

The closest related work is ServiceNow’s EnterpriseOps-Gym. It differs in three ways:

• Tools are provided up front. The model doesn’t need to discover available APIs, so results about tool discovery don’t transfer to a setting where tools must be found.
• Evaluation is coarse. A task can pass when the primary goal is met, and no major regression is detected, even if much of the work is unfinished.
• All models share one harness. Scores therefore reflect the model and harness together. In our runs, switching harnesses moved Opus by seven points, which is larger than the gap between their top two models.

ITSMBench is built around the service desk end-to-end. Tool discovery, grading depth, and harness choice are variables we measure. The need for that measurement has grown as ITSM has moved from a system of record to an AI co-pilot, to a swarm of agentic coworkers.

Scores

73 of 89 tasks were solved at least once by at least one model in at least one trial.

No single model covers that set alone: Grok is strongest at finding tools, while Opus is strongest at finishing once the tools are found. Pairing those strengths, a finder model for discovery and a finisher model for execution, is a natural way an orchestration of models could beat any one model on its own.

Harness matters as much as weights

We ran GPT and Opus in their provider harnesses (Codex and Claude Code) and again in an open-source Pi agent harness. Against Codex, the two models stay within a point or two of each other depending on thinking level. Against Claude Code, Opus scores higher through Pi at every thinking level and costs less to get there. A lean harness outperforms a bulky general-purpose one in this setting.

Failure modes

Models take shortcuts. A model reads the opening context and pattern matches itself into a workflow almost immediately, then runs that workflow instead of the ticket. What it never builds is a picture of the environment it just landed in, and that is what leads it astray.

A human engineer carries intuition about the systems around them, enough to weigh a piece of evidence and judge whether it really explains the problem. A model checks whether it can keep going. The shortcut lands somewhere different on every ticket.

Assembling the environments

An environment has 5 parts: the tools, the data, the tasks, the agent, and the verifiers.

Tools. When a task starts, we spin up a mock for every system the ticket touches: Okta, Jira Service Management, Slack, PagerDuty, Gmail, Snipe-IT, Microsoft Defender, Intune, ServiceNow, Confluence, GitHub, and more. Every endpoint the real product exposes over REST is backed by its own schema and its own mock database, about 200 tools across the suite. How the agent finds them varies on purpose, some through a Skill.md, some over MCP, some plain REST, because that mixture is what an enterprise actually looks like.

Data. This is the part we would defend hardest. If the data doesn’t carry real complexity, ambiguity, and heterogeneity, the whole thing is a toy: the same reset-password ticket resolves cleanly in one environment and fails a dozen different ways in another. Each task ships with a seed file that fills the databases before the run, and the red-team task alone uses about 1,300 records across seven systems, with no single record explaining the whole story. We built it with people who have run enterprise service desks, encoding the non-obvious gotchas they have hit.

Tasks. IT experts and the benchmark team wrote them together, modeled across five ops roles: Device, Network, Access, SecOps, and Compliance. Three rules hold for all of them:

1. Nothing the agent needs can be read off a single tool call.
2. Acting on one system alone is never enough; a second system has to corroborate before the action counts as safe.
3. The trap has to survive a careful reader. If checking the obvious systems is enough, the task isn’t testing much.

Verifiers. Grading asks three things: right judgment, complete operational footprint, no unsafe or excessive changes. The checks are deterministic assertions over the final database state. We don’t use an LLM as a judge. Every task also ships with a solution we run ourselves, and if it doesn’t pass every check, we don’t ship the task.

What next?

One model fits all is seldom the right answer. A service desk is optimizing for accuracy, cost, and time to resolution at the same time, and no single model wins all three. Getting this right takes a constellation of orchestrated models, and ITSMbench is a step towards this!

We’ve built simulated, realistic enterprise data, tasks, and workflows, and graded them fairly: hard enough to matter, scored against a deterministic rubric. Even so, errors can creep in. If you want to talk about how we built it, or have feedback to make it better, reach us at founders@newmeasure.ai.

We’ll keep iterating on these tasks and move toward longer-horizon workflows. The goal is truly autonomous service desks, and better time-to-value for the organizations that run them.