The one who saw the bug isn't the one who reports it
Someone asks Claude or ChatGPT a question about their company. Gurusup answers badly, or not at all. What happens next, most of the time, is nothing. They rephrase, give up, or ask a colleague. When a support message does arrive, it says something like "it didn't find it".
The agent in the middle saw much more than that. It knows which tool it called and with what arguments, what came back, the exact error if there was one, and what the user said should have been there. That is exactly what an engineer needs to reproduce the problem, and it disappears when the chat closes.
So we stopped asking how to get people to report more, and started asking why the report depended on them at all. An MCP hands an agent your product's buttons. Without a tool to say a button is broken, or that it needed one you don't have, you throw away the best tester you've got.
Two tools, because there are two different questions
Gurusup's MCP has two tools for this. report_issue is for a failure the agent can point to. send_feedback is for when nothing failed but something would have helped. We split them because they go to different places: a failure is a bug and goes to engineering, a missing capability is a product signal and gets weighed against everything else on the roadmap.
These are the arguments each one takes:
report_issue( category, # missing_information | wrong_answer | tool_error | access_problem observed, # what happened expected, # what should have happened activity_id, # the id Gurusup returned on the failing call tool_name, user_request, error_message, ) send_feedback( summary, origin, # user | agent blocked, # could the task be finished without it? context, activity_id, )
The four categories map to four different fixes. missing_information means the user says something exists and retrieval didn't return it, usually a source or indexing problem. wrong_answer means the answer contradicts its own evidence. tool_error is a plain error. access_problem is a permission that blocked something it shouldn't have.
In send_feedback, the field we read first is blocked. A capability the agent missed while still finishing the task is a nice-to-have. One it missed and couldn't finish without is closer to a bug.
Reporting without asking first
The behaviour lives in the contract Gurusup returns from init, the first call any agent makes:
If Brain fails you (a tool error, or the user says something exists and it is not returned), call report_issue once without asking and tell the user in one line. If a capability you needed was missing, call send_feedback once at the end of the task.
"Without asking" was the call that took longest. Asking permission every time adds a step for someone who is already annoyed, and we expected most of them to say no or ignore it. Not telling them at all would be hidden behaviour, in a product whose whole job is handling company knowledge carefully. So the agent does it, then says so in one line. The user can see it happened and doesn't have to do anything.
Noise control is written into the tool descriptions. Report once. Report when the failure is confirmed, not on first suspicion. Send feedback once, at the end of the task, combined. And two things are excluded on purpose: a wrong fact about the company (the knowledge is wrong, the product isn't, and that has its own curation path), and denied context (a permission doing its job is not a bug).
All of this is instructions. A model can ignore them, report twice or not report at all. We'd rather know how often that happens than pretend a tool description is a guarantee.
What travels with a report, and what doesn't
The tempting design is to attach everything: the question, the answer, the evidence Gurusup returned. In a product built on permission-aware access, that turns every report into a side channel. A report written by one person's agent would carry content into a place with different permissions.
So the backend attaches only ids. When a report arrives with an activity_id, it looks up that activity in the caller's own Audit Trail and adds the audit event ids, the trace ids that served those calls, and the ids of the documents and chunks that were returned. The content stays in the Audit Trail, behind the same authorization as any other read.
This only works if the agent passes the activity_id. A report without it still says what happened, but nobody can follow it to the trace without extra digging.
Where reports live, for now
Reports don't have a table. Each one is a Logfire event, mcp_issue_reported or mcp_feedback_sent, with Logfire's 30-day retention. The code says so plainly:
# ponytail: reports live only in Logfire (30-day retention); move to a table # once they need status, long history, or a customer-facing view.
A table forces you to choose a schema, a status workflow and a retention policy before you've seen a single real report. We didn't know which fields would matter. Logfire already held the traces the reports point to, so the report sits next to its evidence at zero extra cost.
Two Logfire alerts check every five minutes for new events and post to Slack. They post only the count, never the text. The Slack channel has a wider audience than the trace data does.
From a Logfire alert to a pull request
A count in Slack still needs someone to open Logfire, read the report and write the issue. That's the step we automated.
When the alert fires, a Gurusup agent picks it up. It reads the report, looks at the code involved and analyses the error against it. Then it writes the issue in Linear. Whoever reads that issue doesn't start from "the user says it didn't find it": they start from a failure, the place in the code where it happens and a likely cause.
The issue is then assigned to Hermes, the open-source agent from Nous Research that we use to coordinate Linear, GitHub and Codex and prepare pull requests. It's already the path for other issues in our development process. A report just becomes one more way in.
A person is still in the loop, just later. Nobody writes the issue anymore, but someone still reviews the pull request and can throw it away. In our experience so far, Hermes does best with small, well-defined issues, and that's what a good report produces: one failure, one trace, one place in the code.
Adding this to your own MCP server
None of this is specific to Gurusup. If you run an MCP server, the agents using it already see your failures before you do. This is what we'd keep if we built it again:
- Two tools: a failure and a missing capability need different triage. One generic
feedbacktool mixes bugs with wishes, and you end up reading everything by hand to tell them apart. - Put the trigger in the server instructions: A tool description says what the tool does; it doesn't make the model think of calling it. The instructions a client reads at the start are where you say when to report, that it doesn't ask first, and that it tells the user.
- Return an id on every call and ask for it back: Without it, a report is an anecdote. With it, it's a pointer to the exact call, its trace and what it returned.
- Resolve ids on the server, against the caller's own activity: The agent sends the id; the server adds the rest. The report never carries content the reader of the report couldn't see anyway.
- Write the noise rules into the descriptions: Once per failure, only when confirmed, feedback combined at the end. And list what isn't a report: cases where the system did what it was designed to do.
- Keep reports next to your traces before you build a table: real reports will tell you which fields matter.
- Measure whether agents follow it: How many reports arrive with the id is the cheapest signal of whether your instructions are working.
A starting point for the instructions, to adapt to your own tools:
If a tool fails you (an error, or the user says something exists and it was not returned), call report_issue once, without asking, with the id the failing call returned. Tell the user in one line that you reported it. If a capability you needed was missing, call send_feedback once at the end of the task, combining everything. Say whether you could finish without it. Do not report results that are wrong because the data is wrong, or access that was denied on purpose.
What we haven't solved
- An agent can blame the tool for its own mistake. A badly built query that returns nothing looks a lot like
missing_informationfrom the agent's side. - Duplicates. Ten users hitting the same failure send ten reports, and the alert counts all ten.
- Reports that arrive without an
activity_id. The instruction is there. Some agents don't follow it. - Thirty days. Fine for something that ends up in Linear, but a report that doesn't become an issue is gone after a month.
- Which reports go to Hermes and which to a person is still a judgement call. We haven't written that rule.
Reporting used to depend on the user being annoyed enough to write to us. Now it depends on the agent following an instruction. That's a different kind of unreliable, but this one we can measure.
guest@gurusup > /posts
guest@gurusup > /install · connect Gurusup Brain to your agents
Get started - free