File 08 · AI agents

IT helpdesk agent prototypeA research build testing how far a careful AI agent can go: tools, approvals, guardrails and live tests.

In-houseResearch build
FILE 08Research buildIn-house
The brief

Before an agent touches a client's systems, we want to know where careful design stops it from doing harm. This build stress-tests that.

The approach

A working helpdesk agent for a made-up company, with tools, a person approving anything risky, three layers of guardrails, a second AI checking every answer, and a built-in search test.

What we built
  1. A working IT helpdesk agent for a made-up company, built to stress-test agent design before it reaches a client.
  2. Finds answers with keyword and synonym search across the knowledge base, and shows the whole knowledge base as a 3D map.
  3. Uses six tools, and asks a person to approve anything risky before it runs a fix on a simulated laptop.
  4. Three layers of guardrails, tested with a planted document that tries to give the agent hidden orders.
  5. A second AI checks every answer against its sources, and a built-in test scores the search: 16 of 16 questions found the right article in the top three.

Deliverables

  • Agent with six tools
  • Human approval before risky actions
  • Three layers of guardrails
  • Answer checking by a second model
  • Search test: 16 of 16 in the top three

Built with

JavaScriptBM25 searchClaudeLLM-as-judgeEvals
Next step

Want something like this built?

Tell us about the job it should do. We'll come back with a pilot plan and a fixed price.

Next step

Let's engineer your next stage.

Thirty minutes, no pitch deck. Bring one process that eats your team's time and leave with a plan.