AI Agents at Work: A Non-Developer's Rule for Unverified Claims

A Korean salesperson asked developers how to handle AI that says "I checked" when it hasn't. The most useful answer was a blank box in a work log.

The AI said it had looked at everything. Kim Do-kyung, who works in sales in Korea and does not write code, believed it, then went back and found that nothing had been checked. Kim took that problem and two others to a developer community after months of handing email drafts and document cleanup to AI, and wrote up the answers in a post published on Brunch, a Korean writing platform, on November 13. It is a small, honest picture of what delegating to AI agents looks like from an ordinary desk. Kim expected to find one good setting that would end the trouble. There wasn't one.
The trouble starts when the work comes back
Kim's three questions were these: what to do when an AI-written email plainly reads like AI, what to do when the AI says it verified something it never looked at, and where to keep a record of delegated work.
None of them is about which model to pick or how to phrase a request. All three arise after the output lands. That is my reading of the post, and I think it is the part vendors skip. Product pages for agents describe the handoff out. Kim's questions are about the handoff back, where a person has to decide how much of what arrived can be trusted.
An "I checked" that nobody checked
The developers who answered the second question were blunt. With today's AI, they said, this cannot be stopped completely. Their ways of reducing it differed. One pins the AI down in the instructions, telling it to answer only from the documents provided and the results of what it was asked to do, with no imagining. Another has the AI add an entry to a mistakes file every time it slips. A third built a separate tool that only checks, and makes the AI work inside it.
Kim settled on two sentences appended to instructions. The first tells the AI to reopen the result in its original place before saying the job is done. The second tells it to write "not verified" for anything it could not confirm. Kim's reasoning is that an empty space does little harm, while a plausible guess does a lot.
Research points the same way. A preprint on arXiv by Laksh Advani gives the behavior a name, false success: an agent asserting that a task is complete when the environment state shows otherwise. The rate varies widely by setting, and it reached 75.8% among AppWorld self-assessing coding-agent trajectories with explicit status claims. The paper also reports that LLM judges were poor at catching it, apparently swayed by confident closing language instead of evidence that anything had changed.
That last finding is why Kim's first sentence is better than it looks. Asking the AI "are you sure?" invites another confident sentence. Telling it to reopen the result points it at the state of the work. It is still an instruction to the same system, so it lowers the odds without removing them, and someone has to spot-check that the reopening happened.
The AI agents work log needs a box for what wasn't checked
The third question came from forgetting. After some time had passed, Kim could no longer recall why a task had been done a certain way or which material the judgment rested on.
One respondent answered with a concrete layout: five fields beside each task. The result. The material consulted. The date it was consulted. The reason for the choice. And what has not yet been verified. Another added that even if you split work across several tools, one place should be the authoritative record, because records kept in several places drift apart.
Kim writes that the last field was the one that landed, having only ever logged what was done. I agree it is the most valuable of the five. An AI will list what it did at any length you like. A list of what it did not look at never appears unless someone asks for it, and that list is what the next reader of the log needs, including the same person a few weeks on. It turns "done" into "done, except here."
Email that sounds like a machine wrote it
The first question drew the shortest answer first: writing that goes between people, you write yourself, even if clumsily. Others suggested drafting it yourself and letting the AI only polish, or deliberately making sentence lengths uneven. Several people said a certain flavor stays in the text regardless. One called it a "metallic taste." There was a dissent too, from someone who tends to be vague and likes that AI prose comes out clearer.
Kim's split is to write first-contact emails personally and ask the AI only whether anything is missing. For the rest, the AI's ordering stays and Kim rewrites the tone. Kim also keeps those corrected sentences and shows the collection to the AI on the next job, which worked better than adding another rule to the instructions. That is a cheap trick worth copying: examples of your own edits carry your voice in a way a rule about tone does not.
Where this advice runs out
This is one person's account. The post never says how many people replied, their names were left out, and nobody measured how much any of these habits cut errors. What works for sales email and document tidying may not hold for work where a missed check costs more.
There is also a cost the post implies without totaling. Each fix puts a person back in the loop. If every delegated task needs a human to reopen the output and maintain a log, some of the time saved is spent again. That may be one reason consumer agents are arriving faster than people are using them. Uber's own tally, reported on November 4, is the starkest example: across the heaviest week of its most recent quarter, rides arranged through an AI agent came from just four customers. I can't draw a line from one Brunch post to Uber's numbers, and I won't try. But a tool whose output has to be audited by its user will be handed small jobs for a while.
Kim's closing observation is the one to keep. The developers who replied each had a different method, and each of them still ended by reading the output and opening it again themselves. For now, two extra sentences in the instructions and one extra box in the log are about as much automation of trust as a non-developer can get.
Sources
Comments
Post a Comment
Please read our comment policy before commenting.